PERF: Use fastpath for accessing option value in internals

[jorisvandenbossche]

Related to #49729

Accessing an option value is in general quite fast, but when it is in code that is called repeatedly in a for loop, it still adds up it seems.

Using the example from #49729:

import pandas as pd

def foo(df):
    for idx in df.index:
        df.at[idx, "bar"] = 3

df = pd.DataFrame(range(10000))
df["bar"] = 0

%timeit foo(df)
# 442 ms ± 16.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  <-- main
# 316 ms ± 10.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  <-- PR

I think this should be a safe way to access the option (but will add a test that ensures you can switch back and forth with CoW within a session).

Are we OK with this pattern?

• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature. -> added in PERF: first try inplace setitem for .at indexer #49772

[jbrockmendel]

is this much faster than using pandas._config.config.options?

[jorisvandenbossche]

It's in any case faster, since _global_config is just a plain python dict (which is quite fast), while the options object is a custom class mimicking attribute access:

In [32]: %timeit _global_config["mode"]["copy_on_write"]
52.6 ns ± 1.85 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

In [36]: %timeit pd._config.config.options.mode.copy_on_write
2.78 µs ± 22 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

Strangely it seems that this is even slower than the pd.get_option("mode.copy_on_write"):

In [37]: %timeit pd.get_option("mode.copy_on_write")
1.38 µs ± 14.6 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

[jorisvandenbossche]

If we really want to avoid a private import, this is also an option:

In [39]: %timeit pd.options.d["mode"]["copy_on_write"]
85.9 ns ± 2.39 ns per loop (mean ± std. dev. of 7 runs, 10,000,000 loops each)

The options DictWrapper class seems to have a "public" d attribute exposing the underlying dict (_global_config)

[jbrockmendel]

not worth tying ourself in knots over, thanks for checking

[jbrockmendel]

hmm at one point we had a code check that checked for private imports. guess that got lost somewhere along the way

[WillAyd]

Curious if you've looked at making get_option a class and using __slots__ instead for this? Seems like it would be the idiomatic approach to speeding up attribute access

[jorisvandenbossche]

@WillAyd you mean the DictWrapper class that get_option/options uses? That is currently using a dynamic __getattr__ to look up attributes in a global dictionary that actually stores the option values.
What you suggest might be an option (but not directly an idea, didn't look into that), but that seems a bigger change in the internals of the options machinery. While what I am doing here is just using the fact that the current options internals is just a plain dictionary.

[jbrockmendel]

Best Practices-wise, the best-case would be to try to optimize get_option. Next-best would be to make _global_config public-for-inernal-usage and canonize this as how we do option checks within pandas

[jorisvandenbossche]

best-cast would be to try to optimize get_option

That would require a change of (or addition to) how get_option works, I think. Currently that takes a string (eg get_option("mode.copy_on_write")), and thus that requires parsing the string, and also has some overhead.

It's easy to rename _global_config to global_config (its module is already private) if that is preferred.

[jbrockmendel]

yah even if we did optimize get_option a dict lookup will always be way faster

[WillAyd]

lgtm