Hmmm, the new test is failing in browsers. I guess it's because the code extraction is not working in browsers. Should I move the new test to be a node test rather than a unit test? Since it was testing basic functionality function-by-function it seemed more like a unit test semantically, but perhaps because it's doing the extraction from comments it has to be only run in node? Thanks for your thoughts -- given the number of issues that were exposed, I think it would be very good to get this or something like it into the test suite. Sorry about the problem with this PR, I am not sure how I can run the browser tests on my machine before submitting a PR...

That is a really good idea!! I hadn't expected it would discover so many errors 😳 , so it's really useful.

If the test indeed requires node specific functionality, you'll have to move it to the node tests section (then it's impossible to run in the browser)

Alright, it works fine as a node test. This is a bit of a shame in that none of the actual tests require node, it's only the extraction from the comments that does. So it would be nice if the tests themselves were unit tests, but I don't see how to make that happen (e.g., if there were a build step that added unit tests, it would still only get exercised in build-and-test, since the build doesn't get run in the other tests). So I guess this is good enough... If you agree and like the basic approach of these tests, feel free to merge or request changes. Thanks.

Yeah I understand. Well, not a big deal to have it under the node-tests.

The PR looks good, thanks! I only wish the list with issues would be smaller 😉

And... It works, I think. It complains about the "TypeError: math.simpifyCore is not a function", I just merged the latest develop branch in this PR. Do we have to add this function to the list with issues?

Yes, because of the previously undetected typo "simpifyCore" (instead of "simplifyCore") in the examples, and because the return annotation has to be changed to Node "2 * x" (because math.parse({2 * x}) which is what the current annotation Node {2 * x} is being translated to, does not eval -- unless you want me to make another special check that would handle the brace annotation rather than just quotes which is already used in some cases and which does work fine). Anyhow, I am coming myself to several days when I will not be able to work much on mathjs, so if you want to add simplifyCore to the known problems list yourself feel free, otherwise I will get to this as soon as I can. Thanks!

Great let me give it a try 😄

I am coming myself to several days when I will not be able to work much on mathjs

😅 that gives me some time to catch up 😉 . Enjoy your time off!

Merged via PR #2471, will close this PR now

The tests are merged and active now in v10.4.0

Hi, this is some fantastic work

If we needed to test also the embedded docs, what would be best?

• extend this for the embedded docs
• make a new test exclusive for the embedded docs
• include the unit tests in each function to validate individually their extended docs

#2973

Well, I will be completely frank, what I think would be best would be to achieve something along the lines of #2745 and then there would not be separate issues of testing the docs and the embedded docs! Short of that, the third bullet option isn't any good unless there's something keeping the unit tests and the embedded docs in sync, such as one is generated from the other in one direction or the other. And then between the first two bullet options it's really six of one, half dozen of the other -- I don't think it matters much which way you go.

Thank you!

For the third bullet I was thinking for the unit test to only validate that the examples can run, so it would be in sync with the embedded docs.

It would validate something like this doesn't throw an error.

functionToTest = "sin"
math.evaluate(math.evaluate(`help(${ functionToTest })`).doc.examples)

Now it's clear to me that none of the options I asked are ideal, maybe some quick fix in the meantime something better comes along.

Oh, right, I forgot that currently the embedded docs examples, unlike the online docs examples, only have expressions, not the prescribed "correct" values of those expressions. So that means at this moment the only thing you can check is that the example doesn't throw an error. While I stand by my position that ultimately the best is to unify docs and embedded docs (so that in particular all examples are associated with correct values, in the meantime a script that does the above for each exported symbol and verifies that at least no error is thrown would be a reasonable stopgap and a positive addition to the current state of affairs (definitely doesn't seem worth changing every unit test script!). Of course, if there's dozens of current failures, you might have to initially check it in with a list of known failures, if it doesn't seem reasonable to simultaneously fix all crashes.

Thank you,

I agree it is a lot of work changing every unit test script also it would repeat the same test on each file.

Currently in 11.8.2 there are the following issues:

• Functions that throw an error when trying to run the examples of the embedded docs: diff, evaluate, invmod, lyap, numeric, polynomialRoot, schur, simplifyConstant, sylvester and typed. These are getting fixed in Issues with embedded docs examples #2976
• Functions that don't have embedded docs: addScalar, compile, divideScalar, equalScalar, multiplyScalar, parse, parser, replacer and reviver. Include tests for examples in docs #2973 (reply in thread)

The intention would be testing the embedded docs to avoid issues like these. Jos has an idea and I was starting to look around for references.

Thanks for sharing. We could work this script out in more detail and add it as a file in the folder ./tools. Then, we can add an npm script "test:embedded-docs" in package.json that runs the script. Is that something you would like to work out in a PR?

Originally posted by @josdejong in #2973 (reply in thread)

Thanks for the inputs, and thanks again David for picking this up!

To summarize:

• In the long term we indeed want to unify the embedded docs and the examples in the function comment
• For now, let's create a new unit test (a separate file) which automatically loops over all functions and tests whether it has embedded docs and execute the examples of the embedded docs.