Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve payload cleaning performance #601

Merged
merged 13 commits into from
Jul 14, 2020

Commits on Jun 24, 2020

  1. Improve performance of cleaning objects

    This is a first iteration of improving how we clean objects when
    preparing to JSON encode them
    
    Currently we iterate over the payload multiple times; sometimes to clean
    up encoding errors/recursion and other times to filter sensitive data
    
    Ideally we should be iterating over the payload once, which is slightly
    complicated because we should only be filtering parts of the payload
    (the metadata and breadcrumb metadata)
    imjoehaines committed Jun 24, 2020
    Configuration menu
    Copy the full SHA
    9f404dc View commit details
    Browse the repository at this point in the history
  2. Move some tests to report_spec

    Helper no longer breaks recursion in 'trim_if_needed', so these tests
    no longer apply there. However we are still breaking recursion and so
    can test the same thing elsewhere
    imjoehaines committed Jun 24, 2020
    Configuration menu
    Copy the full SHA
    3888357 View commit details
    Browse the repository at this point in the history
  3. Tidy report delivery

    imjoehaines committed Jun 24, 2020
    Configuration menu
    Copy the full SHA
    6907d59 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    4d5f7b5 View commit details
    Browse the repository at this point in the history
  5. Provide Cleaner with scopes that it should filter

    For example, in this hash:
    
    { a: { b: 'c' } }
    
    'c' lives in scope 'a.b' and so should only be filtered if Cleaner
    is given 'a.b' in its 'scopes_to_filter'
    imjoehaines committed Jun 24, 2020
    Configuration menu
    Copy the full SHA
    3275fe6 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    5762b38 View commit details
    Browse the repository at this point in the history
  7. Add pending rubocop cops

    imjoehaines committed Jun 24, 2020
    Configuration menu
    Copy the full SHA
    7d9b8c6 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    0699f90 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    1203022 View commit details
    Browse the repository at this point in the history
  10. Add changelog entry for #601

    imjoehaines committed Jun 24, 2020
    Configuration menu
    Copy the full SHA
    e87761b View commit details
    Browse the repository at this point in the history

Commits on Jul 6, 2020

  1. Remove unused method

    imjoehaines committed Jul 6, 2020
    Configuration menu
    Copy the full SHA
    86d7ce1 View commit details
    Browse the repository at this point in the history

Commits on Jul 7, 2020

  1. Make Cleaner a shared instance

    This exposed a pretty big regression where filters wouldn't match when
    they should have. This was caused by us filtering the entire report
    object in one go, which means the scopes were nested deeper than they
    were before
    
    Previously we filtered the events.metaData directly, so scopes would not
    include 'events.metaData' and therefore a filter of 'foo' would match
    'events.metaData.foo'. Now that we filter the entire report, if a filter
    relied on 'deep_filters', it would apply and so things that should be
    redacted wouldn't have been
    
    To solve this, we strip each 'scope_to_filter' from the scope before
    matching it, if deep_filters are enabled
    
    The tests passed before this change because we set 'scopes_to_filter'
    in each test. Now that the instance is shared, the scopes are fetched
    from the Configuration so this isn't possible and it exposed the bug
    
    The tests now cover this case, because they can't set 'scopes_to_filter'
    directly anymore, so they are testing that the filters they're using
    match with the Configuration's 'scopes_to_filter'
    imjoehaines committed Jul 7, 2020
    Configuration menu
    Copy the full SHA
    8c669ae View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    5700938 View commit details
    Browse the repository at this point in the history