-
Notifications
You must be signed in to change notification settings - Fork 709
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PVF worker: Add seccomp restrictions (restrict networking) #2009
Merged
Commits on Oct 24, 2023
-
PVF worker: Add seccomp restrictions (block networking)
We're already working on sandboxing by [blocking all unneeded syscalls](#882). However, due to the wide scope it will take a while longer. This PR starts with a much smaller scope, only blocking network-related syscalls until the above is ready. For security we block the following with `seccomp`: - creation of new sockets - these are unneeded in PVF jobs, and we can safely block them without affecting consensus. - `io_uring` - as discussed [here](paritytech/polkadot#7334 (comment)), io_uring allows for networking and needs to be blocked. See below for a discussion on the safety of doing this. - `connect`ing to sockets - the above two points are enough for networking and is what birdcage does (or [used to do](phylum-dev/birdcage#47)) to restrict networking. However, it is possible to [connect to abstract unix sockets](https://lore.kernel.org/landlock/20231023.ahphah4Wii4v@digikod.net/T/#u) to do some kinds of sandbox escapes, so we also block the `connect` syscall. (Intentionally left out of implementer's guide because it felt like too much detail.) `io_uring` is just a way of issuing system calls in an async manner, and there is nothing stopping wasmtime from legitimately using it. Fortunately, at the moment it does not. Generally, not many applications use `io_uring` in production yet, because of the numerous kernel CVEs discovered. It's still under a lot of development. Android outright banned `io_uring` for these reasons. Considering `io_uring`'s status, and that it very likely would get detected either by our [recently-added static analysis](#1663) or by testing, I think it is fairly safe to block it. If execution hits an edge case code path unique to a given machine, it's already taken a non-deterministic branch anyway. After all, we just care that the majority of validators reach the same result and preserve consensus. So worst-case scenario, there's a dispute, and we can always admit fault and refund the wrong validator. On the other hand, if all validators take the code path that results in a seccomp violation, then they would all vote against the current candidate, which is also fine. The violation would get logged (in big scary letters) and hopefully some validator reports it to us. Actually, a worst-worse-case scenario is that 50% of validators vote against, so that there is no consensus. But so many things would have to go wrong for that to happen: 1. An update to `wasmtime` is introduced that uses io_uring (unlikely as io_uring is mainly for IO-heavy applications) 2. The new syscall is not detected by our static analysis 3. It is never triggered in any of our tests 4. It then gets triggered on some super edge case in production on 50% of validators causing a stall (bad but very unlikely) 5. Or, it triggers on only a few validators causing a dispute (more likely but not as bad?) Considering how many things would have to go wrong here, we believe it's safe to block `io_uring`. Closes #619 Original PR in Polkadot repo: paritytech/polkadot#7334
Configuration menu - View commit details
-
Copy full SHA for 5e100f6 - Browse repository at this point
Copy the full SHA 5e100f6View commit details -
Starting the tokio runtime was calling `socketpair` and triggering the new seccomp filter. Removed tokio since we wanted to do it soon anyway as part of #649.
Configuration menu - View commit details
-
Copy full SHA for 2bae7a6 - Browse repository at this point
Copy the full SHA 2bae7a6View commit details
Commits on Oct 25, 2023
-
Configuration menu - View commit details
-
Copy full SHA for a7ce198 - Browse repository at this point
Copy the full SHA a7ce198View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4754a5a - Browse repository at this point
Copy the full SHA 4754a5aView commit details -
Configuration menu - View commit details
-
Copy full SHA for cd10afa - Browse repository at this point
Copy the full SHA cd10afaView commit details -
Configuration menu - View commit details
-
Copy full SHA for f0b41bd - Browse repository at this point
Copy the full SHA f0b41bdView commit details -
Configuration menu - View commit details
-
Copy full SHA for 122eeb4 - Browse repository at this point
Copy the full SHA 122eeb4View commit details
Commits on Oct 26, 2023
-
Configuration menu - View commit details
-
Copy full SHA for f792d88 - Browse repository at this point
Copy the full SHA f792d88View commit details
Commits on Oct 27, 2023
-
Make best-effort attempt to check audit logs for seccomp violations
NOTE: Log, but don't change the outcome. Not all validators may have auditing enabled, so we don't want attackers to abuse a non-deterministic outcome. TESTING: Some manual testing where seccomp events were triggered confirmed that the logs are parsed correctly: Prepare worker: ``` Oct 27 09:15:42.725 WARN parachain::pvf: failed to recv a prepare response: Custom { kind: UnexpectedEof, error: "early eof" } worker_pid=2691819 Oct 27 09:15:42.726 DEBUG parachain::pvf: checking audit log for seccomp violations worker_pid=2691819 audit_log_path="/var/log/syslog" Oct 27 09:15:42.727 ERROR parachain::pvf: A forbidden syscall was attempted! This is a violation of our seccomp security policy. Report an issue ASAP! worker_pid=2691819 syscall=41 pvf=Pvf { code, code_hash: 0xdc4dc649bdebd4a529b6c3eb17da74b36afa9728eb7a0d834443510382490a35, executor_params: ExecutorParams([]), prep_timeout: 3s } ``` Execute worker: ``` Oct 27 09:17:28.006 WARN parachain::pvf: failed to recv an execute response worker_pid=2692712 validation_code_hash=0xdc4dc649bdebd4a529b6c3eb17da74b36afa9728eb7a0d834443510382490a35 error=Custom { kind: UnexpectedEof, error: "early eof" } Oct 27 09:17:28.006 DEBUG parachain::pvf: checking audit log for seccomp violations worker_pid=2692712 audit_log_path="/var/log/syslog" Oct 27 09:17:28.007 ERROR parachain::pvf: A forbidden syscall was attempted! This is a violation of our seccomp security policy. Report an issue ASAP! worker_pid=2692712 syscall=41 validation_code_hash=0xdc4dc649bdebd4a529b6c3eb17da74b36afa9728eb7a0d834443510382490a35 artifact_path="/tmp/.tmp4p6bN0/wasmtime_polkadot_v1.1.0_0xdc4dc649bdebd4a529b6c3eb17da74b36afa9728eb7a0d834443510382490a35_0x03170a2e7597b7b7e3d84c05391d139a62b157e78786d8c082f29dcf4c111314" ```
Configuration menu - View commit details
-
Copy full SHA for 99cb3b2 - Browse repository at this point
Copy the full SHA 99cb3b2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 38012ec - Browse repository at this point
Copy the full SHA 38012ecView commit details
Commits on Oct 30, 2023
-
Configuration menu - View commit details
-
Copy full SHA for c9b0aaa - Browse repository at this point
Copy the full SHA c9b0aaaView commit details -
Use a combination of rusty-fork (separate processes in rust tests) and new sessions to safely kill child workers in tests.
Configuration menu - View commit details
-
Copy full SHA for 5417818 - Browse repository at this point
Copy the full SHA 5417818View commit details
Commits on Oct 31, 2023
-
Configuration menu - View commit details
-
Copy full SHA for e89873c - Browse repository at this point
Copy the full SHA e89873cView commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.