-
Notifications
You must be signed in to change notification settings - Fork 12.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add tests to rustbuild #48913
Add tests to rustbuild #48913
Conversation
☔ The latest upstream changes (presumably #48599) made this pull request unmergeable. Please resolve the merge conflicts. |
c09796a
to
9cd8b37
Compare
Rebased and fixed tidy. |
☔ The latest upstream changes (presumably #48549) made this pull request unmergeable. Please resolve the merge conflicts. |
9cd8b37
to
a28f666
Compare
Rebased again. |
Taking a look at this now, nice! I'm personally hesitant to land though due to the sprinkling of test-related Could we perhaps explore an alternate solution where something like |
Stubbing out run isn't really practical as we call ensure from it (which is pretty much required for the testing design here, and I think any viable testing design). We could explore an alternate design that separated out ensure calls from the run method but I don't personally think that would work very well with how rustbuild is designed today.
I agree with this in general but I think the overhead/pain is somewhat worth it as I feel the tests outweigh the pain caused by having the
I'm not entirely sure what you mean by stubbing out run, as that's mostly what this does anyway (the cfg! macros in Steps are in the steps that would be stubbed out, with basically the same relative diff). One solution we might potentially see is pulling the ensure calls for each step out of run (and moving ensure off of |
Oh sorry I meant stubbing out run for processes instead of steps which should hopefully be lower down and the costliest part of rustbuild mostly. I'm not sure I'd agree though that these test outweight the I think it's a good idea to test rustbuild but I'd want to be sure that it'd done so in a relatively non-invasive way. Stubbing out running processes at the very core would be an example, but I'm also not sure how feasible that would be unfortunately :( Other alternate designs (like the old compute-steps-first method) are more testable but have their own drawbacks too (hence the rewrite!). I'm not sure how easily testable the declarative design we have now is going to be :( |
This is true (and already done in this implementation). However, we also need/want to stub out filesystem access since we want to avoid creating files all over the build directory that are test-specific, and we ideally don't want to have to run tests single threaded or some other "solution" that avoids conflicts between tests. I would like to note that the cfg! macros are actually fairly minimally sprinkled, I've annotated them below with why they are there. Reading through the list I think it's perhaps a little more than the ideal but actually looks fairly reasonable and maintainable -- there's very few edge cases I think.
|
a28f666
to
4c62d62
Compare
☔ The latest upstream changes (presumably #48097) made this pull request unmergeable. Please resolve the merge conflicts. |
Ping from triage, @alexcrichton — will you have time to continue your review soon? |
@Mark-Simulacrum you have some conflicts as well. |
I probably won't rebase and hope to discuss this at the all-hands with @alexcrichton, so we might hold off until then. |
I'm down with that! |
4c62d62
to
24139ab
Compare
I've reimplemented and pushed a new version that goes with the |
Looks great! r=me with travis passing |
24d468b
to
5556def
Compare
@bors r=alexcrichton |
📌 Commit 5556def has been approved by |
⌛ Testing commit 5556def81df8062a7553ad3a9668b71d2eb48445 with merge e640148acd9eacd3095d682dd8d2370ae564420f... |
💔 Test failed - status-appveyor |
This ensures that each build will support the testing design of "dry running" builds. It's also checked that a dry run build is equivalent step-wise to a "wet" run build; the graphs we generate when running are directly compared node/node and edge/edge, both for order and contents.
This is too likely to cause spurious bounces on CI; what we run may be dependent on what ran successfully before hand (e.g. RLS features with Clippy), which makes this not tenable. There's no good way to ignore specifically these problematic steps so we'll just ignore everything for the time being. We still test that a dry run worked though so largely this is the same from a ensure-that-tests-work perspective. Eventually we'll want to undo this commit, though, to make our tests more accurate.
a9d8f01
to
184d3bc
Compare
@bors r=alexcrichton |
📌 Commit 184d3bc has been approved by |
@bors p=75 |
Add tests to rustbuild In order to run tests, we cfg out various parts of rustbuild. Generally speaking, these are filesystem and process-spawning operations. Then, rustbuild is run "as normal" and the various steps that where run are retrieved from the cache and checked against the expected results. Note that this means that the current implementation primarily tests "what" we build, but doesn't actually test that what we build *will* build. In other words, it doesn't do any form of dependency verification for any crate. This is possible to implement, but is considered future work. This implementation strives to cfg out as little code as possible; it also does not currently test anywhere near all of rustbuild. The current tests are also not checked for "correctness," rather, they simply represent what we do as of this commit, which may be wrong. Test cases are drawn from the old implementation of rustbuild, though the expected results may vary. r? @alexcrichton
💔 Test failed - status-travis |
Your PR failed on Travis. Through arcane magic we have determined that the following fragments from the build log may contain information about the problem. Click to expand the log.
I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact |
1 similar comment
Your PR failed on Travis. Through arcane magic we have determined that the following fragments from the build log may contain information about the problem. Click to expand the log.
I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact |
@bors retry 3 hour timeout in |
Add tests to rustbuild In order to run tests, we cfg out various parts of rustbuild. Generally speaking, these are filesystem and process-spawning operations. Then, rustbuild is run "as normal" and the various steps that where run are retrieved from the cache and checked against the expected results. Note that this means that the current implementation primarily tests "what" we build, but doesn't actually test that what we build *will* build. In other words, it doesn't do any form of dependency verification for any crate. This is possible to implement, but is considered future work. This implementation strives to cfg out as little code as possible; it also does not currently test anywhere near all of rustbuild. The current tests are also not checked for "correctness," rather, they simply represent what we do as of this commit, which may be wrong. Test cases are drawn from the old implementation of rustbuild, though the expected results may vary. r? @alexcrichton
☀️ Test successful - status-appveyor, status-travis |
In order to run tests, we cfg out various parts of rustbuild. Generally
speaking, these are filesystem and process-spawning operations. Then, rustbuild
is run "as normal" and the various steps that where run are retrieved from the
cache and checked against the expected results.
Note that this means that the current implementation primarily tests "what" we
build, but doesn't actually test that what we build will build. In other
words, it doesn't do any form of dependency verification for any crate. This is
possible to implement, but is considered future work.
This implementation strives to cfg out as little code as possible; it also does
not currently test anywhere near all of rustbuild. The current tests are also
not checked for "correctness," rather, they simply represent what we do as of
this commit, which may be wrong.
Test cases are drawn from the old implementation of rustbuild, though the
expected results may vary.
r? @alexcrichton