Squash merge of the following commits from commaai/master:

2ab57c2 update release notes 7d3ad94 modeldata: use a constexpr function to build the IDXS (commaai#23245) e494801 update model replay ref commit after increased model idx precision 4061f50 boardd: new class USBDevice (commaai#23015) 285adde Revert "boardd: new class USBDevice (commaai#23015)" cf46622 Road Roll Compensation Rebased (commaai#23251) ffc2f64 modeld: update live calibration in main thread (commaai#23235) e4de82f ui: remove variable world_objects_visible, use rcv_frame (commaai#23264) fc7d152 Mazda: CX-5 and 3 FW (commaai#23267) 646fe7d mui 16dd7ce build for mac (commaai#23048) cd7db7f Mazda CX-5 2021 FW (commaai#23270) f25121b Mazda: alert when LKAS is disabled (commaai#23273) c92199a Revert "Mazda: alert when LKAS is disabled (commaai#23273)" 256c51e bump cereal f3e67e0 repeat immediate warning sound ea82af1 delete old scripts 2d85785 UI: less harsh controls unresponsive when not engaged 50cce27 UI: call map::offroadTransition after creating map (commaai#23283) 6aeb542 make that consistent with other labels db6c477 use km/h in settings.cc (commaai#23290) bf4d875 OnroadHud: use function loadPixmap (commaai#23292) a1ac773 Add missing COROLLA_TSS2 ESP & engine f/w (commaai#23294) 94e435c replay/camera: use existing YUV_BUFFER_COUNT (commaai#23297) b1dfdca Add missing HIGHLANDER_TSS2 engine f/w (commaai#23301) 6fd3d50 VW MQB: Add FW for 2018 Volkswagen Arteon (commaai#23303) 332f568 Pyupgrade 3.6: Update syntax with Python 3.6+ features (commaai#23305) 13bf30f VW MQB: Add FW for 2019 Volkswagen Jetta (commaai#23306) 699b20e test/profiler: fix setting carParams c9da140 replay: fix segfault or hanging on quit (commaai#23323) 21fc9d5 replay/util: initialize curl at startup (commaai#23314) 1fc1382 posenet_publish: use const references (commaai#23310) 4c7c2cc palanteer seems pretty cool 5fe809b ui: set the bg color of controllerUnresponsivePermanent to blue (commaai#23308) de6126b replay: add flag REPLAY_FLAG_FULL_SPEED to play at full speed (commaai#23324) 58a363e replay/segment: pass member function pointer to QtConcurrent::run (commaai#23312) 5387806 LongitudinalMpc: minor clean up (commaai#23296) dc96d4b Add More Types for mypy (commaai#23268) c95202b docs.comma.ai add sitemap and robots.txt (commaai#23259) c86ca38 add version to docs.comma.ai (commaai#23260) 8fe9c0e mac_setup: install git-lfs c26f294 plotjuggler: add mac support (commaai#23344) d85bb96 disable simulator job until LFS is fixed bd39969 Hyundai: Car Port for 2022 Santa Fe Plug-in Hybrid (commaai#23332) 7cd6a88 fix setup scripts if git isn't installed fed7cfe PC: dynamically set PYTHONPATH from real openpilot path (commaai#23322) 92b553b plotjuggler: fix layout param (commaai#23348) 1e309a5 replay: add flag to disable video (commaai#23353) 2ca0925 lateral_planner.py/publish: follow capnp best practices (commaai#23359) 9992ea1 LogReader: add arg to sort by time (commaai#23346) 066e3e3 CI: use ubuntu_setup.sh (commaai#23349) 9d3a821 first contribution ideas 92f194d MultiLogIterator: remove wraparound mode (commaai#23360) 30e8383 HKG: blacklist cars with lower steering max (commaai#23255) bbb47dc MultiLogIterator: fix after adding sort_by_time ab2111a Add firmware from 2021 prius in australia (commaai#23271) e180557 HKG: Add and Fix FW for 2022 Kia Niro EV (commaai#23253) 7c51b4d add supported FW version for Palisade 2022 (commaai#23337) 24c741e Toyota: merge Lexus RX params (commaai#23374) 162460e Toyota: 2020 Highlander Hybrid Engine fingerprint (commaai#23373) 1e41f47 CAN receive error is a comm issue (commaai#23362) 7601040 CI: cache LFS objects (commaai#23345) 2a99bc6 ui: update lead's z position if modelV2 has been updated after startup. (commaai#23355) f2520b6 lane_planner.py/parse_model: follow capnp best practices (commaai#23368) a1a9a0f pre-commit: fix exclude patterns (commaai#23375) 231f74c GM: Switch to AcceleratorPedal2 (commaai#23376) cd42be7 CI: run car unit tests in parallel (commaai#23379) c70fc7d Revert "soundd: back to onroad only (commaai#23236)" 7b612d4 radard/update: optimize capnp (commaai#23380) 21ee6d2 ui: emit offroadTransition in UIState::updateStatus (commaai#23265) 8dd8b19 replay tests: retry on network failure (commaai#23383) 3e5a273 tools: configure PYTHONPATH in pipenv (commaai#23361) 1ca5e1c Fix steering oscillations in 2019 RDX (commaai#22677) 6b20534 Subaru FPv2 support (commaai#1878) 15f88db add subaru fp2.0 to release notes 5dc631f longcontrol/update: optimize capnp (commaai#23382) 05b905d interfaces/apply: follow capnp best practices (commaai#23371) a11bb81 bump panda f561d84 ui/update_model: do not recalculate the max_idx If there is no lead (commaai#23356) 960e0cf controlsd/publish_logs: follow capnp best practices (commaai#23372) 9295066 ui/get_path_length_idx: line_x[i] shoud be less than or equal to path_height (commaai#23336) 0148b23 Add comment about vEgoStarting to interfaces (commaai#23327) 937a0df replay/logreader: handle abort in load (commaai#23321) f2c17bb logcatd: fix sd_journal_seek_tail doesn't go to end (commaai#23319) a1e201e MUI should be a status bar (commaai#23287) 9f88ba1 Longitudinal planner: make v_desired a FirstOrderFilter (commaai#23341) cd9813a UI: run set_brightness in QtConcurrent (commaai#23326) 5dbdad3 CameraView: sync pbo with glFence instead of calling glFinish (commaai#23293) 5a77157 Joystick: learn axis min/max (commaai#23377) 5264485 disable pytest concurrency for test_models 8655f6d cameraview.cc: call glFlush after creating fence 71132ed plotjuggler: support segment names (commaai#23263) 9cb0ebe LongitudinalMpc: Remove class accel_limit_arr (commaai#23295) d2948f2 replay: Add next engagement / disengagement jump capabilities (commaai#23248) a6aa094 Toyota: only send gas interceptor message when openpilot controls longitudinal (commaai#23357) aad7ebd bump CPU usage slightly after adding glFlush 47bb62b athena: methods for adding and cancelling multiple upload requests (commaai#23366) 9f70bea docs: setup docs for C code (commaai#23262) 9720dbe docs: conf.py dont shadow project variable 15be8a8 Subaru: initial FPv2 dump (commaai#23389) d3f064f Controlsd: fix bug in curv rate limit (commaai#23250) 7257f71 Move over all tss2 long tunes (commaai#23108) 0357cf2 controlsd: remove duplicate event relayMalfunction (commaai#23392) cf9792d Longcontrol: remove starting state (commaai#23328) df7126b paramsd: follow capnp best practices (commaai#23399) f30a702 fix up tools readme 34926df Revert "cameraview.cc: call glFlush after creating fence" 25f5147 Revert "CameraView: sync pbo with glFence instead of calling glFinish (commaai#23293)" 23346ae setup: update pyenv before installing python (commaai#23405) bafdb20 setup: remove duplicate pipenv install (commaai#23406) a2e762e longcontrol: check if speeds is empty (commaai#23410) 8d80c01 catch gl_buffer->map() returning nullptr (commaai#23411) 3ffebf4 athena: uploads, show which items fail on error (commaai#23412) 64caee8 replay: add missing import for std::optional (commaai#23416) 13d011c Add Prius 2022 to CARS.md (commaai#23417) e499ebc Hyundai: Add Niro EV to 384 STEER_MAX (commaai#23420) 7faad75 Subaru: remove Ascent FPv1 (commaai#23407) a126495 Subaru: remove all Pre-Global CAN fingerprints (commaai#23422) eb0cd9c bump panda 450656e GM: move abandoned cars into dashcam only (commaai#23423) c7fb8f4 selfdrive/car/gm/interface.py: fix typo in comment (commaai#23429) e0338fd controlsd/state_transition: remove redundant check in elif (commaai#23428) 86d7307 replay: handle abort in getRemoteFileSize (commaai#23427) ef225e3 locationd: disable falling device detector (commaai#23431) 1a6881e skip falling device test 4528075 log ERROR messages in qlogs too (commaai#23425) 1de0757 fix typo in athena/registration.py (commaai#23434) ce998be Camera transforms: denormalize with no bounds dbef0a1 Camera transforms: denormalize with no bounds at all 09a8359 optimize alertmanager (commaai#23433) c4b36ce Revert "ui/update_model: do not recalculate the max_idx If there is no lead (commaai#23356)" d77782e CameraView: sync pbo with glFence instead of calling glFinish (commaai#23409) 207ce41 tools: update README.md (commaai#23447) 6664421 mac: fix setup and re-enable CI (commaai#23365) 89f2f1c LFS migration (commaai#23448) a0eed0c hyundai enable radar points script cleanup (commaai#23451) 0c075a6 Toyota: use common DBC for TSSP Camry Hybrid (commaai#23453) f6c6e34 calibrationd: remove extra comma (commaai#23472) b2beb5e controls: check contents in tuple instead of list (commaai#23463) 2d08f8b controlsd: save lat_plan.dPathPoints to local variable (commaai#23462) 67af095 controlsd: check all_valid only if not initialized (commaai#23437) 842ba8e C2 detection (commaai#23439) 8eec818 events: cleanup function any (commaai#23479) 1b49ce6 Statsd (commaai#23413) a653461 check in tuple instead of list (commaai#23477) 4ff9ee8 monkeypatch os.link (commaai#23480) 4208fe0 LongControl: remove redundant check on active (commaai#23481) 779e602 add missing toyota fw engine (commaai#23272) 3ce4976 thermald: track engaged state in param and kmsg (commaai#23478) 1a54d7f Ignore statsd for running process check (commaai#23483) fa3055f smaller docker image (commaai#23469)
budney · Mar 24, 2022 · e2a46be · e2a46be
1 parent 7dd519f
commit e2a46be
Show file tree

Hide file tree

Showing 206 changed files with 3,118 additions and 2,022 deletions.
diff --git a/.github/workflows/selfdrive_tests.yaml b/.github/workflows/selfdrive_tests.yaml
@@ -69,52 +69,62 @@ jobs:
                           rm -rf /tmp/scons_cache/* && \
                           scons -j$(nproc) --cache-populate"
 
-  #build_mac:
-  #  name: build macos
-  #  runs-on: macos-10.15
-  #  timeout-minutes: 60
-  #  steps:
-  #  - uses: actions/checkout@v2
-  #    with:
-  #      submodules: true
-  #  - name: Determine pre-existing Homebrew packages
-  #    if: steps.dependency-cache.outputs.cache-hit != 'true'
-  #    run: |
-  #      echo 'EXISTING_CELLAR<<EOF' >> $GITHUB_ENV
-  #      ls -1 /usr/local/Cellar >> $GITHUB_ENV
-  #      echo 'EOF' >> $GITHUB_ENV
-  #  - name: Cache dependencies
-  #    id: dependency-cache
-  #    uses: actions/cache@v2
-  #    with:
-  #      path: |
-  #        ~/.pyenv
-  #        ~/Library/Caches/pip
-  #        ~/Library/Caches/pipenv
-  #        /usr/local/Cellar
-  #        ~/github_brew_cache_entries.txt
-  #      key: macos-cache-${{ hashFiles('tools/mac_setup.sh') }}
-  #  - name: Brew link restored dependencies
-  #    if: steps.dependency-cache.outputs.cache-hit == 'true'
-  #    run: |
-  #      while read pkg; do
-  #        brew link --force "$pkg" # `--force` for keg-only packages
-  #      done < ~/github_brew_cache_entries.txt
-  #  - name: Install dependencies
-  #    run: ./tools/mac_setup.sh
-  #  - name: Build openpilot
-  #    run: eval "$(pyenv init -)" && scons -j$(nproc)
-  #  - name: Remove pre-existing Homebrew packages for caching
-  #    if: steps.dependency-cache.outputs.cache-hit != 'true'
-  #    run: |
-  #      cd /usr/local/Cellar
-  #      new_cellar=$(ls -1)
-  #      comm -12 <(echo "$EXISTING_CELLAR") <(echo "$new_cellar") | while read pkg; do
-  #        if [[ $pkg != "zstd" ]]; then # caching step needs zstd
-  #          rm -rf "$pkg"
-  #        fi
-  #      done
-  #      comm -13 <(echo "$EXISTING_CELLAR") <(echo "$new_cellar") | tee ~/github_brew_cache_entries.txt
+  build_mac:
+    name: build macos
+    runs-on: macos-latest
+    timeout-minutes: 60
+    steps:
+    - uses: actions/checkout@v2
+      with:
+        submodules: true
+    - name: Determine pre-existing Homebrew packages
+      if: steps.dependency-cache.outputs.cache-hit != 'true'
+      run: |
+        echo 'EXISTING_CELLAR<<EOF' >> $GITHUB_ENV
+        ls -1 /usr/local/Cellar >> $GITHUB_ENV
+        echo 'EOF' >> $GITHUB_ENV
+    - name: Cache dependencies
+      id: dependency-cache
+      uses: actions/cache@v2
+      with:
+        path: |
+          ~/.pyenv
+          ~/.local/share/virtualenvs/
+          /usr/local/Cellar
+          ~/github_brew_cache_entries.txt
+          /tmp/scons_cache
+        key: macos-${{ hashFiles('tools/mac_setup.sh', 'update_requirements.sh', 'Pipfile*') }}
+        restore-keys: macos-
+    - name: Brew link restored dependencies
+      run: |
+        if [ -f ~/github_brew_cache_entries.txt ]; then
+          while read pkg; do
+            brew link --force "$pkg" # `--force` for keg-only packages
+          done < ~/github_brew_cache_entries.txt
+        else
+          echo "Cache entries not found"
+        fi
+    - name: Install dependencies
+      run: ./tools/mac_setup.sh
+    - name: Build openpilot
+      run: |
+        source tools/openpilot_env.sh
+        pipenv run selfdrive/manager/build.py
+
+        # cleanup scons cache
+        rm -rf /tmp/scons_cache/
+        pipenv run scons -j$(nproc) --cache-populate
+    - name: Remove pre-existing Homebrew packages for caching
+      if: steps.dependency-cache.outputs.cache-hit != 'true'
+      run: |
+        cd /usr/local/Cellar
+        new_cellar=$(ls -1)
+        comm -12 <(echo "$EXISTING_CELLAR") <(echo "$new_cellar") | while read pkg; do
+          if [[ $pkg != "zstd" ]]; then # caching step needs zstd
+            rm -rf "$pkg"
+          fi
+        done
+        comm -13 <(echo "$EXISTING_CELLAR") <(echo "$new_cellar") | tee ~/github_brew_cache_entries.txt
 
   build_webcam:
     name: build webcam
@@ -351,11 +361,12 @@ jobs:
         name: longitudinal
         path: selfdrive/test/longitudinal_maneuvers/out/longitudinal/
 
-  test_car_models:
-    name: car models
+  test_cars:
+    name: cars
     runs-on: ubuntu-20.04
     timeout-minutes: 50
     strategy:
+      fail-fast: false
       matrix:
         job: [0, 1, 2, 3]
     steps:
@@ -385,7 +396,7 @@ jobs:
     - name: Test car models
       run: |
         ${{ env.RUN }} "scons -j$(nproc) --test && \
-                        FILEREADER_CACHE=1 coverage run selfdrive/test/test_models.py && \
+                        FILEREADER_CACHE=1 pytest selfdrive/test/test_models.py && \
                         chmod -R 777 /tmp/comma_download_cache"
       env:
         NUM_JOBS: 4

diff --git a/.github/workflows/tools_tests.yaml b/.github/workflows/tools_tests.yaml
@@ -45,7 +45,20 @@ jobs:
     - uses: actions/checkout@v2
       with:
         submodules: true
-        lfs: true
+
+    # HACK: cache LFS objects since they count against our quota
+    # https://github.com/actions/checkout/issues/165#issuecomment-657673315
+    - name: Create LFS file list
+      run: git lfs ls-files -l | cut -d' ' -f1 | sort > .lfs-assets-id
+    - name: Restore LFS cache
+      uses: actions/cache@v2
+      id: lfs-cache
+      with:
+        path: .git/lfs
+        key: ${{ runner.os }}-lfs-${{ hashFiles('.lfs-assets-id') }}
+    - name: Git LFS Pull
+      run: git lfs pull
+
     - name: Build Docker image
       run: |
         eval "$BUILD"

diff --git a/.gitignore b/.gitignore
@@ -1,4 +1,5 @@
 venv/
+.env
 .clang-format
 .DS_Store
 .tags

diff --git a/.lfsconfig b/.lfsconfig
@@ -0,0 +1,3 @@
+[lfs]
+	url = https://gitlab.com/commaai/openpilot-lfs.git/info/lfs
+	pushurl = 
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -13,14 +13,14 @@ repos:
     rev: v0.910-1
     hooks:
     -   id: mypy
-        exclude: '^(pyextra)|(cereal)|(rednose)|(panda)|(laika)|(opendbc)|(laika_repo)|(rednose_repo)/'
+        exclude: '^(pyextra/)|(cereal/)|(rednose/)|(panda/)|(laika/)|(opendbc/)|(laika_repo/)|(rednose_repo/)/'
         additional_dependencies: ['git+https://github.com/numpy/numpy-stubs', 'types-requests', 'types-atomicwrites',
                                   'types-pycurl']
 -   repo: https://github.com/PyCQA/flake8
     rev: 4.0.1
     hooks:
     -   id: flake8
-        exclude: '^(pyextra)|(cereal)|(rednose)|(panda)|(laika)|(opendbc)|(laika_repo)|(rednose_repo)|(selfdrive/debug)/'
+        exclude: '^(pyextra/)|(cereal/)|(rednose/)|(panda/)|(laika/)|(opendbc/)|(laika_repo/)|(rednose_repo/)|(selfdrive/debug/)/'
         args:
         - --select=F,E112,E113,E304,E502,E701,E702,E703,E71,E72,E731,W191,W6
         - --statistics
@@ -31,15 +31,15 @@ repos:
         entry: pylint
         language: system
         types: [python]
-        exclude: '^(pyextra)|(cereal)|(rednose)|(panda)|(laika)|(laika_repo)|(rednose_repo)/'
+        exclude: '^(pyextra/)|(cereal/)|(rednose/)|(panda/)|(laika/)|(laika_repo/)|(rednose_repo/)/'
 -   repo: local
     hooks:
     -   id: cppcheck
         name: cppcheck
         entry: cppcheck
         language: system
         types: [c++]
-        exclude: '^(third_party)|(pyextra)|(cereal)|(opendbc)|(panda)|(tools)|(selfdrive/modeld/thneed/debug)|(selfdrive/modeld/test)|(selfdrive/camerad/test)/|(installer)'
+        exclude: '^(third_party/)|(pyextra/)|(cereal/)|(opendbc/)|(panda/)|(tools/)|(selfdrive/modeld/thneed/debug/)|(selfdrive/modeld/test/)|(selfdrive/camerad/test/)/|(installer/)'
         args:
         - --error-exitcode=1
         - --language=c++

diff --git a/Dockerfile.openpilot_base b/Dockerfile.openpilot_base
@@ -1,70 +1,32 @@
 FROM ubuntu:20.04
+
 ENV PYTHONUNBUFFERED 1
 
 ENV DEBIAN_FRONTEND=noninteractive
-RUN apt-get update && apt-get install -y --no-install-recommends \
-    autoconf \
-    build-essential \
-    bzip2 \
-    ca-certificates \
-    capnproto \
-    clang \
-    cmake \
-    cppcheck \
-    curl \
-    ffmpeg \
-    gcc-arm-none-eabi \
-    git \
-    iputils-ping \
-    libarchive-dev \
-    libbz2-dev \
-    libcapnp-dev \
-    libcurl4-openssl-dev \
-    libeigen3-dev \
-    libffi-dev \
-    libgles2-mesa-dev \
-    libglew-dev \
-    libglib2.0-0 \
-    liblzma-dev \
-    libomp-dev \
-    libopencv-dev \
-    libqt5sql5-sqlite \
-    libqt5svg5-dev \
-    libsqlite3-dev \
-    libssl-dev \
-    libsystemd-dev \
-    libusb-1.0-0-dev \
-    libzmq3-dev \
-    locales \
-    ocl-icd-libopencl1 \
-    ocl-icd-opencl-dev \
-    opencl-headers \
-    python-dev \
-    qml-module-qtquick2 \
-    qt5-default \
-    qtlocation5-dev \
-    qtmultimedia5-dev \
-    qtpositioning5-dev \
-    qtwebengine5-dev \
-    sudo \
-    valgrind \
-    wget \
-  && rm -rf /var/lib/apt/lists/*
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends sudo tzdata locales && \
+    rm -rf /var/lib/apt/lists/*
 
 RUN sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen && locale-gen
 ENV LANG en_US.UTF-8
 ENV LANGUAGE en_US:en
 ENV LC_ALL en_US.UTF-8
 
-RUN curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash
-ENV PATH="/root/.pyenv/bin:/root/.pyenv/shims:${PATH}"
+ENV PIPENV_SYSTEM=1
+ENV PYENV_VERSION=3.8.10
+ENV PYENV_ROOT="/root/.pyenv"
+ENV PATH="$PYENV_ROOT/bin:$PYENV_ROOT/shims:$PATH"
+
+COPY Pipfile Pipfile.lock .python-version update_requirements.sh /tmp/
+COPY tools/ubuntu_setup.sh /tmp/tools/
+RUN cd /tmp && \
+    tools/ubuntu_setup.sh && \
+    rm -rf /var/lib/apt/lists/* && \
+    rm -rf /tmp/* && \
+    rm -rf /root/.cache && \
+    pip uninstall -y pipenv && \
 
-COPY Pipfile Pipfile.lock /tmp/
-RUN pyenv install 3.8.10 && \
-    pyenv global 3.8.10 && \
-    pyenv rehash && \
-    pip install --no-cache-dir --upgrade pip==21.3.1 && \
-    pip install --no-cache-dir pipenv==2021.5.29 && \
-    cd /tmp && \
-    pipenv install --system --deploy --dev --clear && \
-    pip uninstall -y pipenv
+    # remove unused architectures from gcc for panda
+    cd /usr/lib/gcc/arm-none-eabi/9.2.1 && \
+    rm -rf arm/ && \
+    rm -rf thumb/nofp thumb/v6* thumb/v8* thumb/v7+fp thumb/v7-r+fp.sp
diff --git a/Pipfile b/Pipfile
@@ -25,16 +25,20 @@ pre-commit = "*"
 pycurl = "*"
 pygame = "*"
 pyprof2calltree = "*"
+pytest = "*"
+pytest-xdist = "*"
 reverse_geocoder = "*"
 scipy = "*"
 sphinx = "*"
+sphinx-sitemap = "*"
 sphinx-rtd-theme = "*"
+breathe = "*"
 subprocess32 = "*"
 tenacity = "*"
 
 [packages]
 atomicwrites = "*"
-casadi = "*"
+casadi = {version = "*", markers="platform_system != 'Darwin'"}
 cffi = "*"
 crcmod = "*"
 cryptography = "*"
@@ -50,7 +54,7 @@ libusb1 = "*"
 nose = "*"
 numpy = "*"
 onnx = "*"
-onnxruntime-gpu = "*"
+onnxruntime-gpu = {version = "*", markers="platform_system != 'Darwin'"}
 pillow = "*"
 psutil = "*"
 pycapnp = "==1.1.0"