RFC: project scope

[arnetheduck]

Regarding this project, I'm curious about the scope and use case - there are several reasons why you don't want this kind of test running:

• it has a very narrow use case - testing application I/O or similar
  • it's a smell in general if the application can only be tested this way - it means that it was written using global state / side effects and that's all you can test
• it's slow - it has to build many stand-alone executables
• it's not powerful - it can't orchestrate multi-process test runs (upstream Nim testament has several hacks for this which prevent good parallellisation - because global resources like the file system are used unsafely, you can't run tests in parallel without getting non-deterministic failures) and fixing this would require extensive redesign
• it contains / reimplements a poorly designed build system for finding and building tests
• it's unergonomic - one has to run the test runner instead of compiling and running the tested module directly (nim c -r)

[zah]

The scope of this project is pretty clear.

The supported testing methods are the only reasonable way to test libraries such as Chronicles (that feature extensive compile-time configuration able to produce vastly different run-time results and having invalid states that should produce compilation errors) or Confutils (that has shell integration features).

Fuzzing is another form of testing that requires compiling programs with custom toolchains and that will be simplified by this package.

Finally, it's a place where we can experiment with enhancements to Nim's unit testing libraries (such as unittest2, asyncTest, timedTest, ok/reject, better check macro, etc).

[disruptek]

@arnetheduck -- you make a number of statements that I don't feel are strictly accurate. Maybe you could take a closer look at the implementation and offer some specific criticism to support your claims?

I've stopped working on this project because it's not clear that my time is well-spent since a 20-line commit completely redefines the mechanisms by which testing occurs. It would be good to get some consensus on design direction.

Personally, I think this project should try a little harder to integrate or subsume testament, to the benefit of both. For example, testament's approach permits trivial nim c -r testing, which meets the ergonomic bar for at least one person in the thread.

[arnetheduck]

chronicles

the vast majority of chronicles does not depend on compiler options - the formatting, introspection, runtime configuration, etc etc - the compile time stuff merely turns on and off certain features - if you can't do this testing without running a full process, something is wrong with your code already - all that running an external process does is check that you can glue it together at compile time

clear

"a dump for various testing-related code - anything goes"? because it started out as a testament clone, but now there's the module test thing, and suddenly fuzzing and unittest should also go here?

@disruptek "not accurate" isn't that.. accurate... so it's hard to respond to, but I can generally expand the points.

narrow use case

normally if you write test cases, you're better off with a single binary with lots of tests in it - it's only in exceptional cases, integration testing or poorly written applications that you need to isolate tests at the process boundary - true, it's a powerful isolation in that the OS takes care of releasing resources for you, but it hides issues in your library as well because the way the OS releases those resources is not generally applicable for more normal usage

slow

lots of recompiling of the same code

it's not powerful

where is the code for orchestrating a multi-process test (which is one reason you'd need an external test runner - ie start this process then that process then wait for that process to fininsh and start another one - this would be useful for a tcp server + client test)? the upstream tester from the nim compiler only has ugly hacks to deal with this, which break under parallel runs (which is why the parallel testing in the nim compiler is done by category which is suboptimal - higher granularity runs into race conditions)

it contains / reimplements a poorly designed build system

nim-testutils/testrunner.nim

Line 166 in 71708e2

proc compile(test: TestSpec; backend: string): TestStatus =

nim-testutils/testrunner.nim

Line 434 in 71708e2

testFiles = scanTestPath(config.path)

it's unergonomic

normally, I can run the test with nim c -r which is convenient, because this is nim's build system as well as compiler. here, I need to run testrunner instead which needs to load .test files and compile nim files - which I don't remember the syntax for and which is not integrated in editors etc etc. one could imagine that for tests like chronicles which need multiple runs, a macro system would generate an executable that compiles and runs itself, say using the compiler api - using different compile time options instead - so that there's fewer moving pieces..

[disruptek]

normally if you write test cases,

Simply, this is not for those cases. For those cases, suitable solutions exist in the form of testament and unittest, right? This suite is for testing system software or poorly-written software that nonetheless resisted endless bikeshedding and came to be by no moral fault of the hapless but well-intentioned free-software contributor. This is all about compilation or execution characteristics.

If you have a better solution in mind, please share it. Only testament comes close and, as I said, I think it should probably be more tightly appreciated in testrunner design.

The code for orchestrating a multi-process...

...is in this repository and its use is described in the README.md.

I can run the test with nim c -r

I like this feature, too, and think it's nice that testament offers such a feature and comes bundled with Nim. I wish testrunner consumed the same test input.

[zah]

@arnetheduck, so what is the project scope of another brainchild of yours - nim-stew? Per your own advice, we need a place to experiment with testing-related functionality while gaining independence of the std lib.

Also, I feel your objections here are sabotaging the progress of important work. How do you propose that we start implementing fuzzing tests in NBC? Have you actually studied the existing implementation by Kim who thumbed up the proposed plan?

For better or worse, Chronicles currently relies on this form of testing. Do you propose that all progress on the library is halted until we spend weeks rewriting the testing framework to meet your ideals?

[arnetheduck]

@zah the scope is important so that we document and make informed decisions on what to fund and what not to fund presently, with due diligence - since the readme is out of date, it would be good to have it written down somewhere - both for ourselves and potential contributors.

[arnetheduck]

@disruptek this was not aimed at your work, of which we're very appreciative. It's more of a planning discussion to set a strategy for testing, and understand more what our needs are - we have testing needs all across the board, and chronicles/chronicles-like is but one small part of it.

of the solutions I'd work towards in this space would be a integrated build and test tool such that developers don't need to bother to learn multiple commands - with multiple developers working and being onboarded on the same project, this is a significant cost.

re multi-process, I've read both the code and the readme, the closest I can find being https://github.com/status-im/nim-testutils/#multiple-invocations - from what I can tell, it runs the same program with different options, but does not allow orchestrating. I've had similar issues with testament while developing nlvm.

nice that testament offers such a feature

it doesn't - the test is described in the comment header but you still need testament to parse the comment and compare the outputs. it seems that this should be possible to do with the compiler api or exec so that there's no separate runner involved. the way to do this would be that the TestSpec is produced by a macro or something - importing multiple test files would collect all testspecs - then one could unify the "feeling" that unittest has and run the test "externally compiled" using the same syntax that normal unittesting does - in a bright future, this would also be easy to integrate with a build tool that would do build&test using the same command line. this is however an unexplored topic, and since we have many disparate testing frameworks being written under the status umbrella, it would be good to explore what has the most impact before jumping in.

[disruptek]

Well, as I said, this repo meets a use-case for me that other test frameworks do not. I was happy to have an excuse to implement it and I'm happy to use it in my projects.

I have a branch of testutils that integrates with testament directly to consume testament and unittest (and testutils) specs, which seems similar to what you're suggesting.

I await your concrete "bright future" design document. It will be great to unify all these efforts!
😄

[arnetheduck]

I have a branch of testutils that integrates with testament directly to consume testament and unittest (and testutils) specs, which seems similar to what you're suggesting.

yeah, that sounds nice - I mean, it's effectively the same, except these tests that require compiling and running are a bit more convoluted to run, with compilation and all

[disruptek]

Are you saying your plan involves running tests without compiling them? Do you intend to run multiple NimScript test specs in a single VM instance?

The reason I put my testament integration on hold is that it seems to implement so many of the testutils requirements that I felt it best to wait until we could establish that they could not be unified before continuing. Testament source is not necessarily included in Nim packaging though the runtime is; the distinction may not be insignificant to some users...

[zah]

There are some missing features in Testament that drove the creation of Testutils (while it was part of the Chronicles repo):

A major assumption in Testament is that the program will have the same output in all compilation modes (the single expected output is embedded in the program source). This is not the case in Chronicles where the compilation options dramatically change the expected output. For this reason, we decided to go with separate .test files describing the compilation and the expected output. A single program often has multiple .test files using it.

Besides compiling the program in different ways, you can also run the program in different ways (this is quite relevant for Confutils). This additional requirement lead to the current design where a .test file can feature multiple Output sections.