Support testing across a range of Python (and other language) versions #7
Comments
ivoanjo
added a commit
that referenced
this issue
Jun 12, 2024
…d relative rpath is needed **What does this PR do?** This PR adds a new test case that validates that DataDog/dd-trace-rb#3582 and DataDog/dd-trace-rb#3683 keep working fine. **Motivation:** As described in DataDog/dd-trace-rb#3683, this a somewhat annoying thing to test, but important to avoid regressing. **Additional Notes:** You can actually see the evolution of both of those fixes in this test. E.g. here's dd-trace-rb 1.21.0 (prior to DataDog/dd-trace-rb#3582 ): ``` W, [2024-06-12T09:34:08.759519 #7] WARN -- ddtrace: [ddtrace] (/app/vendor-moved/bundle/ruby/3.3.0/gems/ddtrace-1.21.1/lib/datadog/core/configuration/components.rb:115:in `startup!') Profiling was requested but is not supported, profiling disabled: There was an error loading the profiling native extension due to 'RuntimeError Failure to load datadog_profiling_native_extension.3.3.2_x86_64-linux due to /app/vendor-moved/bundle/ruby/3.3.0/gems/ddtrace-1.21.1/lib/datadog/profiling/../../datadog_profiling_native_extension.3.3.2_x86_64-linux.so: cannot open shared object file: No such file or directory' at '/app/vendor-moved/bundle/ruby/3.3.0/gems/ddtrace-1.21.1/lib/datadog/profiling/load_native_extension.rb:26:in `<top (required)>'' --- FAIL: TestScenarios/scenarios/ruby_extension_dir_and_rpath (14.86s) ``` in this version, we failed because we couldn't load the native extension. Then here's dd-trace-rb 1.23.1 (without DataDog/dd-trace-rb#3683 ) and if we don't move the `vendor` folder (but still delete the so from the lib folder): ``` --- PASS: TestScenarios/scenarios/ruby_extension_dir_and_rpath (18.96s) ``` ...but if we additionally move the vendor folder (aka what this PR does in the Dockerfile): ``` W, [2024-06-12T09:37:33.517188 #6] WARN -- ddtrace: [ddtrace] (/app/vendor-moved/bundle/ruby/3.3.0/gems/ddtrace-1.23.1/lib/datadog/core/configuration/components.rb:116:in `startup!') Profiling was requested but is not supported, profiling disabled: There was an error loading the profiling native extension due to 'RuntimeError Failure to load datadog_profiling_native_extension.3.3.2_x86_64-linux due to libdatadog_profiling.so: cannot open shared object file: No such file or directory' at '/app/vendor-moved/bundle/ruby/3.3.0/gems/ddtrace-1.23.1/lib/datadog/profiling/load_native_extension.rb:39:in `<top (required)>'' --- FAIL: TestScenarios/scenarios/ruby_extension_dir_and_rpath (3.25s) ``` Notice it fails BUT the error is now different from the one above -- the error is relating to loading `libdatadog_profiling.so`, not `datadog_profiling_native_extension.3.3.2_x86_64-linux.so`. And with the change in DataDog/dd-trace-rb#3683 (which will be in 1.23.2): ``` --- PASS: TestScenarios/scenarios/ruby_extension_dir_and_rpath (9.60s) ``` **NOTE**: For this test, unlike other Ruby tests we have, we're pulling in the latest **released** gem version (e.g. with `gem 'datadog'` on the `gems.rb` file), not the latest from git (as we do for other Ruby tests). This is because gems get installed in different paths when bundler downloads them directly from git, and we want to validate the path when a stable version is installed. This also means that this PR will show up as failed until the latest datadog release (which will be 2.2.0) gets released. (Or 1.23.2, but I left the test setup to test the latest 2.x releases, not the 1.x ones, although I used 1.x on my tests above to show the evolution of the issue).
ivoanjo
added a commit
that referenced
this issue
Jun 19, 2024
…patch **What does this PR do?** This PR adds a new test scenario, the `ruby_dir_interruption_patch`. **Motivation:** This scenario tests DataDog/dd-trace-rb#3720 in a very unique way: by loading the profiler, and then running the upstream Ruby test suite for the `Dir` class with our monkey patches applied. (For context, our "monkey patches" are a couple of Ruby modules which wrap the behavior of the standard library `Dir` class, and thus we need to be quite careful to make sure not to break any existing functionality.) **Additional Notes:** This test does not actually generate or assert on any profiles, it only checks that the profiler is running and the monkey patches were correctly applied prior to running the Ruby test suite. The `mspec_config.rb` file is also quite important, since it ensures that both the profiler as well as the monkey patches were correctly applied (e.g. to avoid the test passing trivially because e.g. the profiler actually didn't start). Because at time of writing DataDog/dd-trace-rb#3720 is not yet merged, this test is expected to fail with: ``` $ TEST_RUN_SECS=5 TEST_SCENARIOS="ruby_dir_interruption_patch" go test -v -run TestScenarios === RUN TestScenarios correctness_test.go:210: Considering only scenarios in ruby_dir_interruption_patch correctness_test.go:225: Extract base image from: scenarios/ruby_dir_interruption_patch/Dockerfile === RUN TestScenarios/scenarios/ruby_dir_interruption_patch correctness_test.go:244: Folder: scenarios/ruby_dir_interruption_patch correctness_test.go:245: Json file: scenarios/ruby_dir_interruption_patch/expected_profile.json correctness_test.go:246: Docker file: scenarios/ruby_dir_interruption_patch/Dockerfile correctness_test.go:248: Built test app with: test-app correctness_test.go:137: Running docker command with output /home/ivo.anjo/datadog/prof-correctness/data/ruby_dir_interruption_patch-1618380497 correctness_test.go:138: '[/bin/sh -c bundle exec ddprofrb exec ruby mspec-master/bin/mspec-run --config mspec_config.rb spec-master/core/dir/]' correctness_test.go:151: Error running the test docker run -v /home/ivo.anjo/datadog/prof-correctness/data/ruby_dir_interruption_patch-1618380497:/app/data:rw -u 1000:1000 --security-opt seccomp=unconfined -e EXECUTION_TIME_SEC=5 -e DD_SERVICE=prof-correctness-scenarios/ruby_dir_interruption_patch test-app:latest - /app/mspec_config.rb:3:in `<top (required)>': uninitialized constant Datadog::Profiling::Ext::DirInstanceMonkeyPatches (NameError) if Dir.ancestors.first == Datadog::Profiling::Ext::DirInstanceMonkeyPatches && ^^^^^^^^^^^^^^^^^^^^^^^^^^ from /app/mspec-master/lib/mspec/utils/script.rb:91:in `load' from /app/mspec-master/lib/mspec/utils/script.rb:91:in `block (2 levels) in try_load' from /app/mspec-master/lib/mspec/utils/script.rb:86:in `each' from /app/mspec-master/lib/mspec/utils/script.rb:86:in `block in try_load' from /app/mspec-master/lib/mspec/utils/script.rb:85:in `each' from /app/mspec-master/lib/mspec/utils/script.rb:85:in `try_load' from /app/mspec-master/lib/mspec/utils/script.rb:102:in `load' from /app/mspec-master/lib/mspec/commands/mspec-run.rb:34:in `block in options' from /app/mspec-master/lib/mspec/utils/options.rb:111:in `process' from /app/mspec-master/lib/mspec/utils/options.rb:143:in `parse' from /app/mspec-master/lib/mspec/commands/mspec-run.rb:76:in `options' from /app/mspec-master/lib/mspec/utils/script.rb:286:in `main' from mspec-master/bin/mspec-run:7:in `<main>' I, [2024-06-19T08:58:37.927905 #7] INFO -- datadog: [datadog] DATADOG CONFIGURATION - CORE - {"date":"2024-06-19T08:58:37Z","os_name":"x86_64-pc-linux","version":"2.1.0","lang":"ruby","lang_version":"3.3.3","env":null,"service":"prof-correctness-scenarios/ruby_dir_interruption_patch","dd_version":null,"debug":false,"tags":null,"runtime_metrics_enabled":false,"vm":"ruby-3.3.3","health_metrics_enabled":false,"profiling_enabled":true} --- FAIL: TestScenarios (19.35s) ``` ...e.g. the monkey patches are missing. For local testing, you can tweak the `gems.rb` to pick up the new branch: ```ruby gem 'datadog', git: 'https://github.com/datadog/dd-trace-rb.git', branch: 'ivoanjo/prof-9342-dir-interruption-workaround' ``` and you'll see the test passing. Furthermore, I've created a branch where I introduce a bug in `Dir.home`: ```ruby gem 'datadog', git: 'https://github.com/datadog/dd-trace-rb.git', branch: 'ivoanjo/prof-9342-dir-interruption-workaround-deleteme' ``` and here's the Ruby test suite flagging this behavior issue: ``` $ TEST_RUN_SECS=5 TEST_SCENARIOS="ruby_dir_interruption_patch" go test -v -run TestScenarios === RUN TestScenarios correctness_test.go:210: Considering only scenarios in ruby_dir_interruption_patch correctness_test.go:225: Extract base image from: scenarios/ruby_dir_interruption_patch/Dockerfile === RUN TestScenarios/scenarios/ruby_dir_interruption_patch correctness_test.go:244: Folder: scenarios/ruby_dir_interruption_patch correctness_test.go:245: Json file: scenarios/ruby_dir_interruption_patch/expected_profile.json correctness_test.go:246: Docker file: scenarios/ruby_dir_interruption_patch/Dockerfile correctness_test.go:248: Built test app with: test-app correctness_test.go:137: Running docker command with output /home/ivo.anjo/datadog/prof-correctness/data/ruby_dir_interruption_patch-1872402346 correctness_test.go:138: '[/bin/sh -c bundle exec ddprofrb exec ruby mspec-master/bin/mspec-run --config mspec_config.rb spec-master/core/dir/]' correctness_test.go:151: Error running the test docker run -v /home/ivo.anjo/datadog/prof-correctness/data/ruby_dir_interruption_patch-1872402346:/app/data:rw -u 1000:1000 --security-opt seccomp=unconfined -e EXECUTION_TIME_SEC=5 -e DD_SERVICE=prof-correctness-scenarios/ruby_dir_interruption_patch test-app:latest - I, [2024-06-19T09:04:55.521052 #7] INFO -- datadog: [datadog] DATADOG CONFIGURATION - CORE - {"date":"2024-06-19T09:04:55Z","os_name":"x86_64-pc-linux","version":"2.1.0","lang":"ruby","lang_version":"3.3.3","env":null,"service":"prof-correctness-scenarios/ruby_dir_interruption_patch","dd_version":null,"debug":false,"tags":null,"runtime_metrics_enabled":false,"vm":"ruby-3.3.3","health_metrics_enabled":false,"profiling_enabled":true} Dir interruption patch is present! ruby 3.3.3 (2024-06-12 revision f1c7b6f435) [x86_64-linux] ...................................................................................................................................................................................................................................................................FFEEFFEF............................................................... 1) Dir.home when called without arguments returns the current user's home directory, reading $HOME first FAILED Expected nil == "/rubyspec_home" to be truthy but was false /app/spec-master/core/dir/home_spec.rb:16:in `block (3 levels) in <top (required)>' /app/spec-master/core/dir/home_spec.rb:4:in `<top (required)>' 2) Dir.home when called without arguments returns a non-frozen string FAILED Expected nil.frozen? to be falsy but was true /app/spec-master/core/dir/home_spec.rb:20:in `block (3 levels) in <top (required)>' /app/spec-master/core/dir/home_spec.rb:4:in `<top (required)>' (...etc...) 32 files, 330 examples, 440 expectations, 5 failures, 3 errors, 0 tagged --- FAIL: TestScenarios (19.42s) ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Currently, we test on hardcoded versions. It would be better to have a test matrix that covers the set of supported versions.
Pre-requisite is to work on factorizing some of the test logics. We want to have the tests and expected as part of dd-trace-*** repository. Then we want to trigger the analysis in prof-correctness (possibly along with building the images, TBD).
Originally posted by @r1viollet in #4 (comment)
The text was updated successfully, but these errors were encountered: