What SWFs do we use for testing?

[OmarShehata]

I know this has been tested on that historic dog flash animation on Newgrounds (whose name I forgot so I can't find it), but would it be a good idea to put some of these test files in the repo? They could be a collection of ones that have specific features, some that work, and some that don't yet. That would make it really easy for a new contributor to take on a specific new feature and have a real test case to be working towards.

Or would that bloat the size of the repo? Perhaps we could put a list of links then in the README or wiki? @Herschel what do you think?

I tried it on my oldest favorite animation (https://www.newgrounds.com/portal/view/454086) and got:

Fatal error:
Couldn't find any pixel format that matches the criteria.

[Herschel]

Hi Omar,

We definitely want to use actual SWFs for testing. If you look at the swf-rs repo, there are several test SWFs. These ones I crafted by hand to test the various SWF tags that Flash can output. I round-trip these through the reader and the writer as a test.

The next step is to do some testing for this repo. You're right, actual SWFs are beefy so maybe not the best to include directly into repo. I wonder if using Git LFS might be a good idea...? I don't have any experience with it, so it's worth investigating.

My thoughts for testing:

• Unit tests:
  • Test before-and-after state of the VM after execution of each SWF tag and AS bytecode. Done by hand in Rust.
• Integration tests:
  • Test loading and execution of actual SWF files for any panics. (Both hand-made and content found in the wild).
• Visual testing?
  • It would be good to test the visual output of what the SWF looks like, but I'm not totally sure how to do this, especially early on as the rendering code changes quickly.
  • FifoCI by the Dolphin team is an awesome example.
• Fuzz testing?
  • I don't have a lot of experience here, but there are many Rust fuzzing libraries. This would be good to catch panics from invalid input.

The error you posted is actually from the glutin backend failing to create the window/GL context. May I ask what OS are you running on?

[OmarShehata]

If you look at the swf-rs repo, there are several test SWFs. These ones I crafted by hand to test the various SWF tags that Flash can output.

These are great! This is exactly what I was looking for to start testing this.

The error you posted is actually from the glutin backend failing to create the window/GL context. May I ask what OS are you running on?

So I'm on a Windows 10, but I also tested it on a different Win 10 laptop that didn't have this issue (it correctly crashed with a "not implemented yet" which I think is expected for that animation?). The only relevant difference I can think of is that 2nd laptop has a dedicated Nvidia graphics card, whereas my main one just has a much older, integrated Intel one.

It was also curious because I could run the glutin example just fine. After a little bit of digging (PistonDevelopers/piston#1202 (comment)), I discovered setting srgb to false in main.rs:49 makes it work fine! I can see the red square when testing on /swf-rs/tests/swfs/define_shape.swf.

I wonder if using Git LFS might be a good idea...? I don't have any experience with it, so it's worth investigating.

I think either that or a git submodule would work well. I think the default setting if you have git lfs installed is that it'll pull down all the lfs files, so if it starts becoming several hundred megabytes it might be nice to have to explicitly pull that to run the full test suite.

I agree that sounds like an excellent testing roadmap. Unit tests sound like a good approachable first start here.

[larsiusprime]

May I humbly suggest the "Homestar runner challenge?"

Essentially -- use classic homestar cartoons as an additional test bed. There's a couple of strategic reasons for this that make them useful for testing:

• They're very well known and so being able to pull it off would get attention for the project
• IIRC a large majority of them were done with Flash 5, even well past that version's prime, so "getting homestar working" is probably an easier subproblem then "get any arbitrary swf" working
• There's some light interactivity in many of them that lets you know how you're doing on scripting support
• For bonus points, try to support the games as well

One obstacle might be getting permission, but I'm sure the Brothers Chaps would be very interested. I can reach out if you're interested.

This might be a step to save for when the project is more mature, but if we think of this as a Flash Emulator, then Homestar is to ruffle as Breath of the Wild is to CEMU.

[stuaxo]

re: testing, is there value in running test suites from other flash implementations ?

Shumway - based on Javascript
https://github.com/mozilla/shumway/tree/master/test

Lightspark uses flex to build their tests
https://github.com/lightspark/lightspark

[kmeisthax]

@larsiusprime That's been my entire test strategy so far - for example, I'm working on PR #120 to support the main menus, and I implemented #205 specifically because the games and toons menus need it.

A surprisingly high amount of Homestar Runner already works, including very recent animations. I'm really glad they still post Flash files.

[Herschel]

Closing as we have a fairly robust test suite now.
It is still an open question whether we move away from FLAs toward ASC/third-party generation tools for some/most tests, but our current methods seem to be working well enough for the near future.

[OmarShehata]

current methods seem to be working well enough for the near future.

Thanks for following up here @Herschel ! In case anyone else is looking for this - the current method you're referring to is the test SWF's here right?

https://github.com/ruffle-rs/ruffle/tree/master/swf/tests/swfs

I also found these demo SWF's but the links seem to be broken:

https://github.com/ruffle-rs/ruffle/wiki/Test-SWFs

For example (https://ruffle-rs.s3-us-west-1.amazonaws.com/builds/web-demo/heroes_of_cybertron.swf) gives me an AWS "NoSuchKey" error.

[Herschel]

The main Ruffle test suite is here:
https://github.com/ruffle-rs/ruffle/tree/master/core/tests/swfs

with some swf specific tests here:
https://github.com/ruffle-rs/ruffle/tree/master/swf/tests/swfs

The way to run these is:
cargo test --all
which will run the entire test suite. This involves simulating the SWFs in the core/tests folder in Ruffle and comparing the trace output to the output from the official Flash Player. Whenever we add a feature, we try to add a test SWF to verify the output.

We also have some visual tests here: https://github.com/ruffle-rs/visual-tests that are not automated yet.

Wiki is sadly out of date, I'll go ahead and update the wiki.