Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[PROF-9963] Add integration testing for Ruby dir interruption monkey patch #41

Merged
merged 3 commits into from
Jul 2, 2024

Commits on Jun 19, 2024

  1. [PROF-9963] Add integration testing for Ruby dir interruption monkey …

    …patch
    
    **What does this PR do?**
    
    This PR adds a new test scenario, the `ruby_dir_interruption_patch`.
    
    **Motivation:**
    
    This scenario tests DataDog/dd-trace-rb#3720
    in a very unique way: by loading the profiler, and then running the
    upstream Ruby test suite for the `Dir` class with our monkey patches
    applied.
    
    (For context, our "monkey patches" are a couple of Ruby modules which
    wrap the behavior of the standard library `Dir` class, and thus we
    need to be quite careful to make sure not to break any existing
    functionality.)
    
    **Additional Notes:**
    
    This test does not actually generate or assert on any profiles, it only
    checks that the profiler is running and the monkey patches were
    correctly applied prior to running the Ruby test suite.
    
    The `mspec_config.rb` file is also quite important, since it ensures
    that both the profiler as well as the monkey patches were correctly
    applied (e.g. to avoid the test passing trivially because e.g. the
    profiler actually didn't start).
    
    Because at time of writing
    DataDog/dd-trace-rb#3720 is not yet merged,
    this test is expected to fail with:
    
    ```
    $ TEST_RUN_SECS=5 TEST_SCENARIOS="ruby_dir_interruption_patch" go test -v -run TestScenarios
    === RUN   TestScenarios
        correctness_test.go:210: Considering only scenarios in ruby_dir_interruption_patch
        correctness_test.go:225: Extract base image from: scenarios/ruby_dir_interruption_patch/Dockerfile
    === RUN   TestScenarios/scenarios/ruby_dir_interruption_patch
        correctness_test.go:244: Folder: scenarios/ruby_dir_interruption_patch
        correctness_test.go:245: Json file: scenarios/ruby_dir_interruption_patch/expected_profile.json
        correctness_test.go:246: Docker file: scenarios/ruby_dir_interruption_patch/Dockerfile
        correctness_test.go:248: Built test app with: test-app
        correctness_test.go:137: Running docker command with output /home/ivo.anjo/datadog/prof-correctness/data/ruby_dir_interruption_patch-1618380497
        correctness_test.go:138: '[/bin/sh -c bundle exec ddprofrb exec ruby mspec-master/bin/mspec-run --config mspec_config.rb spec-master/core/dir/]'
        correctness_test.go:151: Error running the test docker run -v /home/ivo.anjo/datadog/prof-correctness/data/ruby_dir_interruption_patch-1618380497:/app/data:rw -u 1000:1000 --security-opt seccomp=unconfined -e EXECUTION_TIME_SEC=5 -e DD_SERVICE=prof-correctness-scenarios/ruby_dir_interruption_patch test-app:latest - /app/mspec_config.rb:3:in `<top (required)>': uninitialized constant Datadog::Profiling::Ext::DirInstanceMonkeyPatches (NameError)
    
            if Dir.ancestors.first == Datadog::Profiling::Ext::DirInstanceMonkeyPatches &&
                                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^
            	from /app/mspec-master/lib/mspec/utils/script.rb:91:in `load'
            	from /app/mspec-master/lib/mspec/utils/script.rb:91:in `block (2 levels) in try_load'
            	from /app/mspec-master/lib/mspec/utils/script.rb:86:in `each'
            	from /app/mspec-master/lib/mspec/utils/script.rb:86:in `block in try_load'
            	from /app/mspec-master/lib/mspec/utils/script.rb:85:in `each'
            	from /app/mspec-master/lib/mspec/utils/script.rb:85:in `try_load'
            	from /app/mspec-master/lib/mspec/utils/script.rb:102:in `load'
            	from /app/mspec-master/lib/mspec/commands/mspec-run.rb:34:in `block in options'
            	from /app/mspec-master/lib/mspec/utils/options.rb:111:in `process'
            	from /app/mspec-master/lib/mspec/utils/options.rb:143:in `parse'
            	from /app/mspec-master/lib/mspec/commands/mspec-run.rb:76:in `options'
            	from /app/mspec-master/lib/mspec/utils/script.rb:286:in `main'
            	from mspec-master/bin/mspec-run:7:in `<main>'
            I, [2024-06-19T08:58:37.927905 #7]  INFO -- datadog: [datadog] DATADOG CONFIGURATION - CORE - {"date":"2024-06-19T08:58:37Z","os_name":"x86_64-pc-linux","version":"2.1.0","lang":"ruby","lang_version":"3.3.3","env":null,"service":"prof-correctness-scenarios/ruby_dir_interruption_patch","dd_version":null,"debug":false,"tags":null,"runtime_metrics_enabled":false,"vm":"ruby-3.3.3","health_metrics_enabled":false,"profiling_enabled":true}
    --- FAIL: TestScenarios (19.35s)
    ```
    
    ...e.g. the monkey patches are missing.
    
    For local testing, you can tweak the `gems.rb` to pick up the new
    branch:
    
    ```ruby
    gem 'datadog', git: 'https://github.com/datadog/dd-trace-rb.git', branch: 'ivoanjo/prof-9342-dir-interruption-workaround'
    ```
    
    and you'll see the test passing.
    
    Furthermore, I've created a branch where I introduce a bug in
    `Dir.home`:
    
    ```ruby
    gem 'datadog', git: 'https://github.com/datadog/dd-trace-rb.git',  branch: 'ivoanjo/prof-9342-dir-interruption-workaround-deleteme'
    ```
    
    and here's the Ruby test suite flagging this behavior issue:
    
    ```
    $ TEST_RUN_SECS=5 TEST_SCENARIOS="ruby_dir_interruption_patch" go test -v -run TestScenarios
    === RUN   TestScenarios
        correctness_test.go:210: Considering only scenarios in ruby_dir_interruption_patch
        correctness_test.go:225: Extract base image from: scenarios/ruby_dir_interruption_patch/Dockerfile
    === RUN   TestScenarios/scenarios/ruby_dir_interruption_patch
        correctness_test.go:244: Folder: scenarios/ruby_dir_interruption_patch
        correctness_test.go:245: Json file: scenarios/ruby_dir_interruption_patch/expected_profile.json
        correctness_test.go:246: Docker file: scenarios/ruby_dir_interruption_patch/Dockerfile
        correctness_test.go:248: Built test app with: test-app
        correctness_test.go:137: Running docker command with output /home/ivo.anjo/datadog/prof-correctness/data/ruby_dir_interruption_patch-1872402346
        correctness_test.go:138: '[/bin/sh -c bundle exec ddprofrb exec ruby mspec-master/bin/mspec-run --config mspec_config.rb spec-master/core/dir/]'
        correctness_test.go:151: Error running the test docker run -v /home/ivo.anjo/datadog/prof-correctness/data/ruby_dir_interruption_patch-1872402346:/app/data:rw -u 1000:1000 --security-opt seccomp=unconfined -e EXECUTION_TIME_SEC=5 -e DD_SERVICE=prof-correctness-scenarios/ruby_dir_interruption_patch test-app:latest - I, [2024-06-19T09:04:55.521052 #7]  INFO -- datadog: [datadog] DATADOG CONFIGURATION - CORE - {"date":"2024-06-19T09:04:55Z","os_name":"x86_64-pc-linux","version":"2.1.0","lang":"ruby","lang_version":"3.3.3","env":null,"service":"prof-correctness-scenarios/ruby_dir_interruption_patch","dd_version":null,"debug":false,"tags":null,"runtime_metrics_enabled":false,"vm":"ruby-3.3.3","health_metrics_enabled":false,"profiling_enabled":true}
            Dir interruption patch is present!
            ruby 3.3.3 (2024-06-12 revision f1c7b6f435) [x86_64-linux]
            ...................................................................................................................................................................................................................................................................FFEEFFEF...............................................................
    
            1)
            Dir.home when called without arguments returns the current user's home directory, reading $HOME first FAILED
            Expected nil == "/rubyspec_home"
            to be truthy but was false
            /app/spec-master/core/dir/home_spec.rb:16:in `block (3 levels) in <top (required)>'
            /app/spec-master/core/dir/home_spec.rb:4:in `<top (required)>'
    
            2)
            Dir.home when called without arguments returns a non-frozen string FAILED
            Expected nil.frozen?
            to be falsy but was true
            /app/spec-master/core/dir/home_spec.rb:20:in `block (3 levels) in <top (required)>'
            /app/spec-master/core/dir/home_spec.rb:4:in `<top (required)>'
    
            (...etc...)
    
            32 files, 330 examples, 440 expectations, 5 failures, 3 errors, 0 tagged
    --- FAIL: TestScenarios (19.42s)
    ```
    ivoanjo committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    90958c6 View commit details
    Browse the repository at this point in the history

Commits on Jul 2, 2024

  1. Configuration menu
    Copy the full SHA
    bb8eb69 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    af8650b View commit details
    Browse the repository at this point in the history