Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch MKI sorting from quicksort to mergesort #19

Merged
merged 1 commit into from
May 8, 2024

Conversation

dfsnow
Copy link
Member

@dfsnow dfsnow commented May 8, 2024

While working on ccao-data/data-architecture#422, I discovered that mki() doesn't produce stable results i.e. it returns slightly different values for each run. This is due to the default pandas.sort_values() function using quicksort, which is not stable and returns different index orders for ties.

Per this SO post, This PR switches the sort method to mergesort, which is stable.

@dfsnow dfsnow added the bug Something isn't working label May 8, 2024
@dfsnow dfsnow requested review from wrridgeway and Damonamajor May 8, 2024 15:08
@dfsnow dfsnow self-assigned this May 8, 2024
Copy link

codecov bot commented May 8, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.74%. Comparing base (534d741) to head (6ed1aac).

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #19   +/-   ##
=======================================
  Coverage   98.74%   98.74%           
=======================================
  Files          12       12           
  Lines         478      478           
=======================================
  Hits          472      472           
  Misses          6        6           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Member

@wrridgeway wrridgeway left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this

Copy link

@jeancochrane jeancochrane left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@jeancochrane
Copy link

Quick follow-up question: We're not using quicksort anywhere else, are we?

@dfsnow
Copy link
Member Author

dfsnow commented May 8, 2024

A quick ripgrep of all our repos shows the following. I'm not sure if any of these would be impactful. Perhaps only the comps one is a concern @jeancochrane

projects/model-sales-val/glue/flagging_script_glue/flagging.py
485:        original_df.sort_values("meta_sale_date")
628:        dups.sort_values("meta_sale_date")
657:        original_df.sort_values("meta_sale_date")

models/model-res-avm/python/comps.py
42:    sorted_comparison_df = comparison_df.sort_values(

models/model-land-avm/notebooks/data_exploration.ipynb
286:    "price_sorted = cop.sort_values(by=['sale_price']).loc[:, 'sale_price']"

projects/scratch-res-avm/python/comps.py
42:    sorted_comparison_df = comparison_df.sort_values(

packages/assesspy/assesspy/formulas.py
179:    df = df.sort_values(by="sp", kind="mergesort")  # for stable sort results

packages/assesspy/docs/source/notebooks/example-ratio-study.ipynb
782:    "gini_data = combined[['sale_price', 'assessed']].sort_values(by='sale_price')\n",

@dfsnow dfsnow merged commit 342de02 into main May 8, 2024
9 checks passed
@dfsnow dfsnow deleted the dfsnow/use-stable-sorting branch May 8, 2024 22:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants