Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PVF worker: Add seccomp restrictions (restrict networking) #2009

Merged
merged 13 commits into from
Oct 31, 2023

Commits on Oct 24, 2023

  1. PVF worker: Add seccomp restrictions (block networking)

    We're already working on sandboxing by [blocking all unneeded syscalls](#882). However, due to the wide scope it will take a while longer. This PR starts with a much smaller scope, only blocking network-related syscalls until the above is ready.
    
    For security we block the following with `seccomp`:
    
    - creation of new sockets - these are unneeded in PVF jobs, and we can safely block them without affecting consensus.
    
    - `io_uring` - as discussed [here](paritytech/polkadot#7334 (comment)), io_uring allows for networking and needs to be blocked. See below for a discussion on the safety of doing this.
    
    - `connect`ing to sockets - the above two points are enough for networking and is what birdcage does (or [used to do](phylum-dev/birdcage#47)) to restrict networking. However, it is possible to [connect to abstract unix sockets](https://lore.kernel.org/landlock/20231023.ahphah4Wii4v@digikod.net/T/#u) to do some kinds of sandbox escapes, so we also block the `connect` syscall.
    
    (Intentionally left out of implementer's guide because it felt like too much detail.)
    
    `io_uring` is just a way of issuing system calls in an async manner, and there is nothing stopping wasmtime from legitimately using it. Fortunately, at the moment it does not. Generally, not many applications use `io_uring` in production yet, because of the numerous kernel CVEs discovered. It's still under a lot of development. Android outright banned `io_uring` for these reasons.
    
    Considering `io_uring`'s status, and that it very likely would get detected either by our [recently-added static analysis](#1663) or by testing, I think it is fairly safe to block it.
    
    If execution hits an edge case code path unique to a given machine, it's already taken a non-deterministic branch anyway. After all, we just care that the majority of validators reach the same result and preserve consensus. So worst-case scenario, there's a dispute, and we can always admit fault and refund the wrong validator. On the other hand, if all validators take the code path that results in a seccomp violation, then they would all vote against the current candidate, which is also fine. The violation would get logged (in big scary letters) and hopefully some validator reports it to us.
    
    Actually, a worst-worse-case scenario is that 50% of validators vote against, so that there is no consensus. But so many things would have to go wrong for that to happen:
    
    1. An update to `wasmtime` is introduced that uses io_uring (unlikely as io_uring is mainly for IO-heavy applications)
    2. The new syscall is not detected by our static analysis
    3. It is never triggered in any of our tests
    4. It then gets triggered on some super edge case in production on 50% of validators causing a stall (bad but very unlikely)
    5. Or, it triggers on only a few validators causing a dispute (more likely but not as bad?)
    
    Considering how many things would have to go wrong here, we believe it's safe to block `io_uring`.
    
    Closes #619
    Original PR in Polkadot repo: paritytech/polkadot#7334
    mrcnski committed Oct 24, 2023
    Configuration menu
    Copy the full SHA
    5e100f6 View commit details
    Browse the repository at this point in the history
  2. Remove tokio from workers

    Starting the tokio runtime was calling `socketpair` and triggering the new
    seccomp filter. Removed tokio since we wanted to do it soon anyway as part of
    #649.
    mrcnski committed Oct 24, 2023
    Configuration menu
    Copy the full SHA
    2bae7a6 View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2023

  1. Address review

    mrcnski committed Oct 25, 2023
    Configuration menu
    Copy the full SHA
    a7ce198 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    4754a5a View commit details
    Browse the repository at this point in the history
  3. Update syscall lists

    mrcnski committed Oct 25, 2023
    Configuration menu
    Copy the full SHA
    cd10afa View commit details
    Browse the repository at this point in the history
  4. Fix benchmark

    mrcnski committed Oct 25, 2023
    Configuration menu
    Copy the full SHA
    f0b41bd View commit details
    Browse the repository at this point in the history
  5. Address some comments

    mrcnski committed Oct 25, 2023
    Configuration menu
    Copy the full SHA
    122eeb4 View commit details
    Browse the repository at this point in the history

Commits on Oct 26, 2023

  1. Configuration menu
    Copy the full SHA
    f792d88 View commit details
    Browse the repository at this point in the history

Commits on Oct 27, 2023

  1. Make best-effort attempt to check audit logs for seccomp violations

    NOTE: Log, but don't change the outcome. Not all validators may have auditing
    enabled, so we don't want attackers to abuse a non-deterministic outcome.
    
    TESTING: Some manual testing where seccomp events were triggered confirmed that
    the logs are parsed correctly:
    
    Prepare worker:
    ```
    Oct 27 09:15:42.725  WARN parachain::pvf: failed to recv a prepare response: Custom { kind: UnexpectedEof, error: "early eof" } worker_pid=2691819
    Oct 27 09:15:42.726 DEBUG parachain::pvf: checking audit log for seccomp violations worker_pid=2691819 audit_log_path="/var/log/syslog"
    Oct 27 09:15:42.727 ERROR parachain::pvf: A forbidden syscall was attempted! This is a violation of our seccomp security policy. Report an issue ASAP! worker_pid=2691819 syscall=41 pvf=Pvf { code, code_hash: 0xdc4dc649bdebd4a529b6c3eb17da74b36afa9728eb7a0d834443510382490a35, executor_params: ExecutorParams([]), prep_timeout: 3s }
    ```
    
    Execute worker:
    ```
    Oct 27 09:17:28.006  WARN parachain::pvf: failed to recv an execute response worker_pid=2692712 validation_code_hash=0xdc4dc649bdebd4a529b6c3eb17da74b36afa9728eb7a0d834443510382490a35 error=Custom { kind: UnexpectedEof, error: "early eof" }
    Oct 27 09:17:28.006 DEBUG parachain::pvf: checking audit log for seccomp violations worker_pid=2692712 audit_log_path="/var/log/syslog"
    Oct 27 09:17:28.007 ERROR parachain::pvf: A forbidden syscall was attempted! This is a violation of our seccomp security policy. Report an issue ASAP! worker_pid=2692712 syscall=41 validation_code_hash=0xdc4dc649bdebd4a529b6c3eb17da74b36afa9728eb7a0d834443510382490a35 artifact_path="/tmp/.tmp4p6bN0/wasmtime_polkadot_v1.1.0_0xdc4dc649bdebd4a529b6c3eb17da74b36afa9728eb7a0d834443510382490a35_0x03170a2e7597b7b7e3d84c05391d139a62b157e78786d8c082f29dcf4c111314"
    ```
    mrcnski committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    99cb3b2 View commit details
    Browse the repository at this point in the history
  2. Update doc

    mrcnski committed Oct 27, 2023
    Configuration menu
    Copy the full SHA
    38012ec View commit details
    Browse the repository at this point in the history

Commits on Oct 30, 2023

  1. Configuration menu
    Copy the full SHA
    c9b0aaa View commit details
    Browse the repository at this point in the history
  2. Un-ignore tests

    Use a combination of rusty-fork (separate processes in rust tests) and new
    sessions to safely kill child workers in tests.
    mrcnski committed Oct 30, 2023
    Configuration menu
    Copy the full SHA
    5417818 View commit details
    Browse the repository at this point in the history

Commits on Oct 31, 2023

  1. Configuration menu
    Copy the full SHA
    e89873c View commit details
    Browse the repository at this point in the history