Testing Method

[ryanhiebert]

What are your thoughts on testing, given the constraints of live-streaming? Is that something that comes later, or does it not fit well into the medium? Should tests be community contributions?

[DerekTurtleRoe]

@ryanhiebert What do you mean, testing? You mean like implementing tests run every commit, continuous integration stuff, etc.?

[ryanhiebert]

Yeah, unit tests and what not, that would run our CI, and make sure we don't break thing horribly.

I can see having some testing happening on the stream being good for a new Rust coder, but I don't necessarily think that we want to do all the tests on-stream. It might be too much.

[DerekTurtleRoe]

@ryanhiebert Yeah, that could be good. I know many C and C++ projects use it, but I have never used it with Rust. Does AppVeyor and other CI stuff support Rust?

[emoon]

@vgturtle127 if you look here https://github.com/yupferris/rustendo64/blob/master/README.md you can see travis and AppVeyor is setup

[emoon]

Also regarding unit-testing there is good support for that also. Here is an example from one of my projects https://github.com/emoon/dynamic_reload/blob/master/src/lib.rs#L405 and also the results can be sent directly to coveralls https://coveralls.io/builds/5043035/source?filename=lib.rs

The coverage measuring isn't perfect for Rust yet but being talked over here rust-lang/rust#31368

[DerekTurtleRoe]

@emoon Huh, not sure how I missed that! 😆

Cool, support for Rust seems pretty good for nearly all editors and stuff. People must really like it.

[yupferris]

Frankly I don't think unit tests will be very useful/fun for this project. Not to mention they may be a huge pain in the ass to write, considering I'm super anal about that, and would either want everything tested, or nothing. CPU tests etc are relatively straightforward, but anything more complicated that tests the system as a whole is just a nightmare. I'd rather not do tests at all - if people want to learn about unit tests in Rust there are plenty of other options already available :)

[yupferris]

Also I totally have plans to integrate at least one debugger, and I think we'll get plenty of visible correctness-checks from that over time. Unit testing is, in general, something that I feel is just another tool in the toolbox, and not something that will likely help us much in this project. When writing emulators you make tons of changes that affect entire system behavior as you learn more about the system and change your understanding, and the last thing I want is to have to fix a bunch of unit tests every time we do that, especially live on stream.

Even though we have appveyor and travis set up, I'd really prefer if nobody starts making PR's with unit tests, unless it's for a very self-contained feature. For instance, perhaps we do some UI stuff or end up wanting tests on the debugging protocol - in those cases I can see it making sense. But we'll play that by ear. For now, having these CI services set up is still helpful as they're actually building the project, which is half the battle :)

[emoon]

/me crawls into a corner and works harder on the potential debugger!

[yupferris]

hint hint wink wink :D

[emoon]

😛

[DerekTurtleRoe]

@yupferris You should take a look at the Nemu64 and Project64 debugger for inspiration. They are pretty decent, and they also are fairly complete.

Also, a graphics debugger like UltraHLE has would be nice, with things like a wireframe mode and a good trace log. This would be nice to implement before you try the Factor5 games (Star Wars, Indiana Jones, etc.), WinBack, Mario Tennis, and maybe that GDC 2011 demo as well. 😉

[yupferris]

@vgturtle127 I did check those out, and they look really rad! We'll definitely do some debugging stuff like that on our way to proper emulation; I think that's inevitable :)

[yupferris]

Closing this ticket because I don't plan on doing tests for some time now

[iherbig]

It occurred to me while watching the recording that you pretty much "test" your code by feeding it the PIF ROM and demo ROM, watching it run, and comparing the result with what you expect. This is static and should not change, regardless of how the structure of your emulator changes. I think it would provide you with a neat and simple showcase of the testing functionality built into cargo. Instead of typing clear && cargo run -- PIF_ROM DEMO_ROM you could just type clear && cargo test.

A simple test module which expects a PIF ROM file and demo ROM file, and runs the emulator that can be updated with assertions as the emulator is expected to be able to do more and more (at least, at this stage when the emulator is still more or less a simple state machine). Building a proper test is probably still more effort than it's worth at this point in time, though. There are pros and cons, I'm sure, but I'm not sure if this had been thought of as a stream-friendly alternative to extensive unit/integration testing so I wanted to put the option out there.

[yupferris]

yeah, my gut feeling is it won't provide much benefit as the emulator gets more complex, and for now it doesn't quite annoy me enough to change. Additionally, that would mean that we would need to upload the boot ROM to the various test servers for them to be able to run the tests, which is less than ideal, as we'd be distributing it at that point.

[iherbig]

Yeah, that's fair enough. I was thinking mainly from a streaming perspective, while everything is still in a nascent state and while you're still showing off Rust features, and less from a production perspective.

[yupferris]

yeah, I love rust/cargo's testing features, they're fantastic :) just not perhaps the right tool for this job