Add real-time benchmark to CI as a nightly job (php#15696)

This PR integrates https://github.com/kocsismate/php-version-benchmarks/ into the CI as a nightly job running every day at 12:30 AM UTC. Roughly, the following happens: the benchmark suite spins up an AWS EC2 instance via Terraform, runs the tests according to the configuration, and then the results are committed to the https://github.com/kocsismate/php-version-benchmark-results repository. In order to have as stable results as possible, the CPU, kernel and other settings of the AWS instance are fine-tuned: - Hyper-threading is disabled - Turbo boost is disabled - C states of the CPU are limited: https://docs.aws.amazon.com/linux/al2/ug/processor_state_control.html#baseline-perf - The workload is dedicated to a single core by using taskset according to Intel's recommendations (https://web.archive.org/web/20210614053522/https://01.org/node/3774) - An io2 SSD volume is attached to the instance which has a provisioned IOPS (https://docs.aws.amazon.com/ebs/latest/userguide/provisioned-iops.html#io2-block-express) so that IO performance is nearly constant - The instance is dedicated so that the noisy neighbor effect is eliminated: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/dedicated-instance.html - ASLR is disabled (Disable ASLR for benchmark php#13769) Customizing the CPU is only supported by metal instances among recent instance types according to https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/processor_state_control.html, so at last, a c7i.metal-24xl instance is used in the eu-west-1 region. The benchmark suite compares the performance of the latest commit of the master branch in the time when the benchmark runs with the last commit of master from the day before yesterday. I.e. if the benchmark runs tomorrow morning at 2 AM, then the performance of the latest commit will be benchmarked against the last commit pushed yesterday. This makes it possible to spot outstanding regressions (or progressions) in time. Actually, the end goal is to send notifications in case of any significant changes for further analyzation. The reason why the benchmark is run for previous commits as well (while they may have already been measured the day before) is to make the results less sensitive for changes in the environment or the benchmark suite itself. I.e.: if AWS upgrades the OS, or if the code under test is modified, then the numbers will likely be affected, and the previous results will be invalidated).
TimWolla · Sep 25, 2024 · 2448a01 · 2448a01
1 parent 3c8c0df
commit 2448a01
Showing 1 changed file with 132 additions and 0 deletions.
diff --git a/.github/workflows/real-time-benchmark.yml b/.github/workflows/real-time-benchmark.yml
@@ -0,0 +1,132 @@
+name: Real-time Benchmark
+on:
+  schedule:
+    - cron: "30 0 * * *"
+permissions:
+  contents: read
+jobs:
+  REAL_TIME_BENCHMARK:
+    name: REAL_TIME_BENCHMARK
+    if: github.repository == 'php/php-src'
+    runs-on: ubuntu-22.04
+    steps:
+      - name: Install dependencies
+        run: |
+          set -ex
+          sudo apt-get update
+          sudo apt-get install gpg
+
+          wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
+          gpg --no-default-keyring --keyring /usr/share/keyrings/hashicorp-archive-keyring.gpg --fingerprint
+          echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
+          export DEBIAN_FRONTEND=noninteractive
+          sudo apt-get update -y
+          sudo apt-get install -y terraform=1.5.7-*
+      - name: Checkout benchmark suite
+        uses: actions/checkout@v4
+        with:
+          repository: 'kocsismate/php-version-benchmarks'
+          ref: 'main'
+          fetch-depth: 1
+          path: 'php-version-benchmarks'
+      - name: Checkout php-src
+        uses: actions/checkout@v4
+        with:
+          repository: 'php/php-src'
+          ref: '${{ github.sha }}'
+          fetch-depth: 100
+          path: 'php-version-benchmarks/tmp/php_master'
+      - name: Setup benchmark results
+        run: |
+          git config --global user.name "Benchmark"
+          git config --global user.email "benchmark@php.net"
+          
+          rm -rf ./php-version-benchmarks/docs/results
+      - name: Checkout benchmark data
+        uses: actions/checkout@v4
+        with:
+          repository: php/real-time-benchmark-data
+          ssh-key: ${{ secrets.PHP_VERSION_BENCHMARK_RESULTS_DEPLOY_KEY }}
+          path: 'php-version-benchmarks/docs/results'
+      - name: Set benchmark config
+        run: |
+          set -e
+
+          # Set infrastructure config
+          cp ./php-version-benchmarks/config/infra/aws/x86_64-metal.ini.dist ./php-version-benchmarks/config/infra/aws/x86_64-metal.ini
+          ESCAPED_DOCKER_REGISTRY=$(printf '%s\n' "${{ secrets.PHP_VERSION_BENCHMARK_DOCKER_REGISTRY }}" | sed -e 's/[\/&]/\\&/g')
+          sed -i "s/INFRA_DOCKER_REGISTRY=public.ecr.aws\/abcdefgh/INFRA_DOCKER_REGISTRY=$ESCAPED_DOCKER_REGISTRY/g" ./php-version-benchmarks/config/infra/aws/x86_64-metal.ini
+          cp ./php-version-benchmarks/build/infrastructure/config/aws.tfvars.dist ./php-version-benchmarks/build/infrastructure/config/aws.tfvars
+          sed -i 's/access_key = ""/access_key = "${{ secrets.PHP_VERSION_BENCHMARK_AWS_ACCESS_KEY }}"/g' ./php-version-benchmarks/build/infrastructure/config/aws.tfvars
+          sed -i 's/secret_key = ""/secret_key = "${{ secrets.PHP_VERSION_BENCHMARK_AWS_SECRET_KEY }}"/g' ./php-version-benchmarks/build/infrastructure/config/aws.tfvars
+
+          YEAR="$(date '+%Y')"
+          DATABASE="./php-version-benchmarks/docs/results/$YEAR/database.tsv"
+          if [ -f "$DATABASE" ]; then
+            LAST_RESULT_SHA="$(tail -n 2 "$DATABASE" | head -n 1 | cut -f 6)"
+          else
+            YESTERDAY="$(date -d "-2 day 23:59:59" '+%Y-%m-%d %H:%M:%S')"
+            LAST_RESULT_SHA="$(cd ./php-version-benchmarks/tmp/php_master/ && git --no-pager log --until="$YESTERDAY" -n 1 --pretty='%H')"
+          fi
+
+          BASELINE_SHA="d5f6e56610c729710073350af318c4ea1b292cfe"
+          BASELINE_SHORT_SHA="$(echo "$BASELINE_SHA" | cut -c1-4)"
+
+          # Set config for the baseline PHP version
+          cp ./php-version-benchmarks/config/php/master.ini.dist ./php-version-benchmarks/config/php/master_baseline.ini
+          sed -i 's/PHP_NAME="PHP - master"/PHP_NAME="PHP - baseline@'"$BASELINE_SHORT_SHA"'"/g' ./php-version-benchmarks/config/php/master_baseline.ini
+          sed -i "s/PHP_ID=php_master/PHP_ID=php_master_baseline/g" ./php-version-benchmarks/config/php/master_baseline.ini
+          sed -i "s/PHP_COMMIT=/PHP_COMMIT=$BASELINE_SHA/g" ./php-version-benchmarks/config/php/master_baseline.ini
+
+          # Set config for the previous PHP version
+          cp ./php-version-benchmarks/config/php/master.ini.dist ./php-version-benchmarks/config/php/master_last.ini
+          sed -i 's/PHP_NAME="PHP - master"/PHP_NAME="PHP - previous master"/g' ./php-version-benchmarks/config/php/master_last.ini
+          sed -i "s/PHP_ID=php_master/PHP_ID=php_master_previous/g" ./php-version-benchmarks/config/php/master_last.ini
+          sed -i "s/PHP_COMMIT=/PHP_COMMIT=$LAST_RESULT_SHA/g" ./php-version-benchmarks/config/php/master_last.ini
+
+          # Set config for the current PHP version
+          cp ./php-version-benchmarks/config/php/master.ini.dist ./php-version-benchmarks/config/php/master_now.ini
+          sed -i "s/PHP_COMMIT=/PHP_COMMIT=${{ github.sha }}/g" ./php-version-benchmarks/config/php/master_now.ini
+
+          # Set config for current PHP version with JIT
+          git clone ./php-version-benchmarks/tmp/php_master/ ./php-version-benchmarks/tmp/php_master_jit
+          cp ./php-version-benchmarks/config/php/master_jit.ini.dist ./php-version-benchmarks/config/php/master_now_jit.ini
+          sed -i "s/PHP_COMMIT=/PHP_COMMIT=${{ github.sha }}/g" ./php-version-benchmarks/config/php/master_now_jit.ini
+
+          # Set test configs
+          cp ./php-version-benchmarks/config/test/1_laravel.ini.dist ./php-version-benchmarks/config/test/1_laravel.ini
+          cp ./php-version-benchmarks/config/test/2_symfony_main.ini.dist ./php-version-benchmarks/config/test/2_symfony_main.ini
+          cp ./php-version-benchmarks/config/test/4_wordpress.ini.dist ./php-version-benchmarks/config/test/4_wordpress.ini
+          cp ./php-version-benchmarks/config/test/5_bench.php.ini.dist ./php-version-benchmarks/config/test/5_bench.php.ini
+          cp ./php-version-benchmarks/config/test/6_micro_bench.php.ini.dist ./php-version-benchmarks/config/test/6_micro_bench.php.ini
+      - name: Run benchmark
+        run: ./php-version-benchmarks/benchmark.sh run aws
+      - name: Store results
+        run: |
+          set -ex
+
+          cd ./php-version-benchmarks/docs/results
+          git pull --autostash
+          if [ -e ".git/MERGE_HEAD" ]; then
+            echo "Merging, can't proceed"
+            exit 1
+          fi
+          git add .
+          if git diff --cached --quiet; then
+            exit 1
+          fi
+          git commit -m "Add result for ${{ github.repository }}@${{ github.sha }}"
+          git push
+      - name: Cleanup
+        if: always()
+        run: |
+          set -ex
+
+          rm -rf ./php-version-benchmarks/tmp/
+          rm -f ./php-version-benchmarks/build/infrastructure/config/*.tfvars
+          rm -rf ./php-version-benchmarks/build/infrastructure/aws/.terraform/
+          rm -f ./php-version-benchmarks/build/infrastructure/aws/.terraform.lock.hcl
+          rm -f ./php-version-benchmarks/build/infrastructure/aws/aws.tfplan
+          rm -f ./php-version-benchmarks/build/infrastructure/aws/terraform.tfstate
+          rm -f ./php-version-benchmarks/build/infrastructure/aws/terraform.tfstate.backup
+          rm -f ./php-version-benchmarks/config/infra/aws/*.ini