From 90958c6b9cf26e6240993f6e8ee7f28cc285bea7 Mon Sep 17 00:00:00 2001 From: Ivo Anjo Date: Wed, 19 Jun 2024 10:05:58 +0100 Subject: [PATCH] [PROF-9963] Add integration testing for Ruby dir interruption monkey patch **What does this PR do?** This PR adds a new test scenario, the `ruby_dir_interruption_patch`. **Motivation:** This scenario tests https://github.com/DataDog/dd-trace-rb/pull/3720 in a very unique way: by loading the profiler, and then running the upstream Ruby test suite for the `Dir` class with our monkey patches applied. (For context, our "monkey patches" are a couple of Ruby modules which wrap the behavior of the standard library `Dir` class, and thus we need to be quite careful to make sure not to break any existing functionality.) **Additional Notes:** This test does not actually generate or assert on any profiles, it only checks that the profiler is running and the monkey patches were correctly applied prior to running the Ruby test suite. The `mspec_config.rb` file is also quite important, since it ensures that both the profiler as well as the monkey patches were correctly applied (e.g. to avoid the test passing trivially because e.g. the profiler actually didn't start). Because at time of writing https://github.com/DataDog/dd-trace-rb/pull/3720 is not yet merged, this test is expected to fail with: ``` $ TEST_RUN_SECS=5 TEST_SCENARIOS="ruby_dir_interruption_patch" go test -v -run TestScenarios === RUN TestScenarios correctness_test.go:210: Considering only scenarios in ruby_dir_interruption_patch correctness_test.go:225: Extract base image from: scenarios/ruby_dir_interruption_patch/Dockerfile === RUN TestScenarios/scenarios/ruby_dir_interruption_patch correctness_test.go:244: Folder: scenarios/ruby_dir_interruption_patch correctness_test.go:245: Json file: scenarios/ruby_dir_interruption_patch/expected_profile.json correctness_test.go:246: Docker file: scenarios/ruby_dir_interruption_patch/Dockerfile correctness_test.go:248: Built test app with: test-app correctness_test.go:137: Running docker command with output /home/ivo.anjo/datadog/prof-correctness/data/ruby_dir_interruption_patch-1618380497 correctness_test.go:138: '[/bin/sh -c bundle exec ddprofrb exec ruby mspec-master/bin/mspec-run --config mspec_config.rb spec-master/core/dir/]' correctness_test.go:151: Error running the test docker run -v /home/ivo.anjo/datadog/prof-correctness/data/ruby_dir_interruption_patch-1618380497:/app/data:rw -u 1000:1000 --security-opt seccomp=unconfined -e EXECUTION_TIME_SEC=5 -e DD_SERVICE=prof-correctness-scenarios/ruby_dir_interruption_patch test-app:latest - /app/mspec_config.rb:3:in `': uninitialized constant Datadog::Profiling::Ext::DirInstanceMonkeyPatches (NameError) if Dir.ancestors.first == Datadog::Profiling::Ext::DirInstanceMonkeyPatches && ^^^^^^^^^^^^^^^^^^^^^^^^^^ from /app/mspec-master/lib/mspec/utils/script.rb:91:in `load' from /app/mspec-master/lib/mspec/utils/script.rb:91:in `block (2 levels) in try_load' from /app/mspec-master/lib/mspec/utils/script.rb:86:in `each' from /app/mspec-master/lib/mspec/utils/script.rb:86:in `block in try_load' from /app/mspec-master/lib/mspec/utils/script.rb:85:in `each' from /app/mspec-master/lib/mspec/utils/script.rb:85:in `try_load' from /app/mspec-master/lib/mspec/utils/script.rb:102:in `load' from /app/mspec-master/lib/mspec/commands/mspec-run.rb:34:in `block in options' from /app/mspec-master/lib/mspec/utils/options.rb:111:in `process' from /app/mspec-master/lib/mspec/utils/options.rb:143:in `parse' from /app/mspec-master/lib/mspec/commands/mspec-run.rb:76:in `options' from /app/mspec-master/lib/mspec/utils/script.rb:286:in `main' from mspec-master/bin/mspec-run:7:in `
' I, [2024-06-19T08:58:37.927905 #7] INFO -- datadog: [datadog] DATADOG CONFIGURATION - CORE - {"date":"2024-06-19T08:58:37Z","os_name":"x86_64-pc-linux","version":"2.1.0","lang":"ruby","lang_version":"3.3.3","env":null,"service":"prof-correctness-scenarios/ruby_dir_interruption_patch","dd_version":null,"debug":false,"tags":null,"runtime_metrics_enabled":false,"vm":"ruby-3.3.3","health_metrics_enabled":false,"profiling_enabled":true} --- FAIL: TestScenarios (19.35s) ``` ...e.g. the monkey patches are missing. For local testing, you can tweak the `gems.rb` to pick up the new branch: ```ruby gem 'datadog', git: 'https://github.com/datadog/dd-trace-rb.git', branch: 'ivoanjo/prof-9342-dir-interruption-workaround' ``` and you'll see the test passing. Furthermore, I've created a branch where I introduce a bug in `Dir.home`: ```ruby gem 'datadog', git: 'https://github.com/datadog/dd-trace-rb.git', branch: 'ivoanjo/prof-9342-dir-interruption-workaround-deleteme' ``` and here's the Ruby test suite flagging this behavior issue: ``` $ TEST_RUN_SECS=5 TEST_SCENARIOS="ruby_dir_interruption_patch" go test -v -run TestScenarios === RUN TestScenarios correctness_test.go:210: Considering only scenarios in ruby_dir_interruption_patch correctness_test.go:225: Extract base image from: scenarios/ruby_dir_interruption_patch/Dockerfile === RUN TestScenarios/scenarios/ruby_dir_interruption_patch correctness_test.go:244: Folder: scenarios/ruby_dir_interruption_patch correctness_test.go:245: Json file: scenarios/ruby_dir_interruption_patch/expected_profile.json correctness_test.go:246: Docker file: scenarios/ruby_dir_interruption_patch/Dockerfile correctness_test.go:248: Built test app with: test-app correctness_test.go:137: Running docker command with output /home/ivo.anjo/datadog/prof-correctness/data/ruby_dir_interruption_patch-1872402346 correctness_test.go:138: '[/bin/sh -c bundle exec ddprofrb exec ruby mspec-master/bin/mspec-run --config mspec_config.rb spec-master/core/dir/]' correctness_test.go:151: Error running the test docker run -v /home/ivo.anjo/datadog/prof-correctness/data/ruby_dir_interruption_patch-1872402346:/app/data:rw -u 1000:1000 --security-opt seccomp=unconfined -e EXECUTION_TIME_SEC=5 -e DD_SERVICE=prof-correctness-scenarios/ruby_dir_interruption_patch test-app:latest - I, [2024-06-19T09:04:55.521052 #7] INFO -- datadog: [datadog] DATADOG CONFIGURATION - CORE - {"date":"2024-06-19T09:04:55Z","os_name":"x86_64-pc-linux","version":"2.1.0","lang":"ruby","lang_version":"3.3.3","env":null,"service":"prof-correctness-scenarios/ruby_dir_interruption_patch","dd_version":null,"debug":false,"tags":null,"runtime_metrics_enabled":false,"vm":"ruby-3.3.3","health_metrics_enabled":false,"profiling_enabled":true} Dir interruption patch is present! ruby 3.3.3 (2024-06-12 revision f1c7b6f435) [x86_64-linux] ...................................................................................................................................................................................................................................................................FFEEFFEF............................................................... 1) Dir.home when called without arguments returns the current user's home directory, reading $HOME first FAILED Expected nil == "/rubyspec_home" to be truthy but was false /app/spec-master/core/dir/home_spec.rb:16:in `block (3 levels) in ' /app/spec-master/core/dir/home_spec.rb:4:in `' 2) Dir.home when called without arguments returns a non-frozen string FAILED Expected nil.frozen? to be falsy but was true /app/spec-master/core/dir/home_spec.rb:20:in `block (3 levels) in ' /app/spec-master/core/dir/home_spec.rb:4:in `' (...etc...) 32 files, 330 examples, 440 expectations, 5 failures, 3 errors, 0 tagged --- FAIL: TestScenarios (19.42s) ``` --- .../ruby_dir_interruption_patch/Dockerfile | 24 +++++++++++++++++++ .../expected_profile.json | 5 ++++ scenarios/ruby_dir_interruption_patch/gems.rb | 3 +++ .../mspec_config.rb | 8 +++++++ 4 files changed, 40 insertions(+) create mode 100644 scenarios/ruby_dir_interruption_patch/Dockerfile create mode 100644 scenarios/ruby_dir_interruption_patch/expected_profile.json create mode 100644 scenarios/ruby_dir_interruption_patch/gems.rb create mode 100644 scenarios/ruby_dir_interruption_patch/mspec_config.rb diff --git a/scenarios/ruby_dir_interruption_patch/Dockerfile b/scenarios/ruby_dir_interruption_patch/Dockerfile new file mode 100644 index 0000000..bbceae6 --- /dev/null +++ b/scenarios/ruby_dir_interruption_patch/Dockerfile @@ -0,0 +1,24 @@ +FROM ruby:3.3 + +ENV DD_PROFILING_ENABLED true +ENV DD_PROFILING_DIR_INTERRUPTION_WORKAROUND_ENABLED true + +COPY ./scenarios/ruby_dir_interruption_patch/*.rb /app/ +RUN chmod 755 /app/* + +WORKDIR /app +RUN bundle install + +RUN wget https://github.com/ruby/spec/archive/refs/heads/master.zip -O spec-master.zip +RUN wget https://github.com/ruby/mspec/archive/refs/heads/master.zip -O mspec-master.zip +RUN unzip spec-master.zip +RUN unzip mspec-master.zip + +# Since we aggressively drop permissions when running, we need to manually create a folder for rubyspec to use +RUN mkdir /app/rubyspec_temp +RUN chmod 777 /app/rubyspec_temp + +# Needed by one of the ruby specs (which is testing access to home) +RUN useradd -u 1000 -ms /bin/bash someuser + +CMD bundle exec ddprofrb exec ruby mspec-master/bin/mspec-run --config mspec_config.rb spec-master/core/dir/ diff --git a/scenarios/ruby_dir_interruption_patch/expected_profile.json b/scenarios/ruby_dir_interruption_patch/expected_profile.json new file mode 100644 index 0000000..48ea741 --- /dev/null +++ b/scenarios/ruby_dir_interruption_patch/expected_profile.json @@ -0,0 +1,5 @@ +{ + "test_name": "ruby_dir_interruption_patch", + "stacks": [], + "note": "This test has no stacks, as it's not expected to emit profiles -- we're only testing that the mspec run is successful even when the profiler monkey patches are available." +} diff --git a/scenarios/ruby_dir_interruption_patch/gems.rb b/scenarios/ruby_dir_interruption_patch/gems.rb new file mode 100644 index 0000000..d20718e --- /dev/null +++ b/scenarios/ruby_dir_interruption_patch/gems.rb @@ -0,0 +1,3 @@ +source 'https://rubygems.org' + +gem 'datadog', git: 'https://github.com/datadog/dd-trace-rb.git', branch: 'master' diff --git a/scenarios/ruby_dir_interruption_patch/mspec_config.rb b/scenarios/ruby_dir_interruption_patch/mspec_config.rb new file mode 100644 index 0000000..a437d53 --- /dev/null +++ b/scenarios/ruby_dir_interruption_patch/mspec_config.rb @@ -0,0 +1,8 @@ +Datadog::Profiling.wait_until_running + +if Dir.ancestors.first == Datadog::Profiling::Ext::DirInstanceMonkeyPatches && + Dir.singleton_class.ancestors.first == Datadog::Profiling::Ext::DirClassMonkeyPatches + puts "Dir interruption patch is present!" +else + raise 'Dir interruption patch is not present!' +end