Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

39 - implement ancestry chunking memory optimization #93

Merged
merged 30 commits into from
Jan 16, 2024

Commits on Jan 2, 2024

  1. Configuration menu
    Copy the full SHA
    4431c64 View commit details
    Browse the repository at this point in the history
  2. expand AncestryMemoryOptimizedTestCase

    now covers case where a collection with no shared member refs exists
    alexdunnjpl committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    ec39198 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    a1f09c3 View commit details
    Browse the repository at this point in the history

Commits on Jan 3, 2024

  1. Configuration menu
    Copy the full SHA
    d6a8d7a View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    3007222 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    a375260 View commit details
    Browse the repository at this point in the history

Commits on Jan 5, 2024

  1. Configuration menu
    Copy the full SHA
    ea7c3be View commit details
    Browse the repository at this point in the history

Commits on Jan 11, 2024

  1. implement search-after database query

    only applied to some queries, thus far.
    initial test suggests it may take up to 20% longer, but this may not be the case where pagination is deep
    alexdunnjpl committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    0b3830f View commit details
    Browse the repository at this point in the history
  2. implement fix to search-after (track_total_hits)

    also improves logging
    alexdunnjpl committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    84fcd90 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    9f9deff View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    6bcc6ed View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    f5c6dd0 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    b6eab9a View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    c6e2066 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    03cbc53 View commit details
    Browse the repository at this point in the history
  9. replace nonaggregate core iteration with more space-efficient mutatio…

    …n approach with minimal use of objects
    alexdunnjpl committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    279ac94 View commit details
    Browse the repository at this point in the history
  10. remove noisy log

    alexdunnjpl committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    01afc35 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    19bb333 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    13ec291 View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    684d041 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    0e4d666 View commit details
    Browse the repository at this point in the history
  15. satisfy mypy, may it rot

    missing stubs failure persists despite installation of psutil library stubs
    alexdunnjpl committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    c8c80a7 View commit details
    Browse the repository at this point in the history
  16. Configuration menu
    Copy the full SHA
    f7e0eb3 View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    f1331e2 View commit details
    Browse the repository at this point in the history
  18. Configuration menu
    Copy the full SHA
    2f5fef2 View commit details
    Browse the repository at this point in the history
  19. Configuration menu
    Copy the full SHA
    4daedda View commit details
    Browse the repository at this point in the history
  20. implement memory recovery of non-dumped history chunk

    this may be causing OOM condition where it should not occur
    alexdunnjpl committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    1decf2b View commit details
    Browse the repository at this point in the history
  21. cut back default ancestry disk dump memory threshold

    this is due to OOM exit on en-prod - either the estimation is off or the task containers are subject to a <100% kill threshold
    alexdunnjpl committed Jan 11, 2024
    Configuration menu
    Copy the full SHA
    10a8905 View commit details
    Browse the repository at this point in the history

Commits on Jan 16, 2024

  1. Configuration menu
    Copy the full SHA
    1a562f7 View commit details
    Browse the repository at this point in the history
  2. excise log cruft

    alexdunnjpl committed Jan 16, 2024
    Configuration menu
    Copy the full SHA
    df2ed50 View commit details
    Browse the repository at this point in the history