General problem: Featurization takes too long!

[ardunn]

Featurizing all of MP takes at least one day. This is way, way too long.

Edit:
Easiest way to fix this is by:

• Converting all featurizers over to inplace=False
• ability to just cache dfs

[utf]

Do you know what is taking the most time?

[ardunn]

@utf Structure featurization.

[utf]

Is it SiteStatsFeaturizer (with CrystalNN preset) or something else?

[ardunn]

To featurize 83k structures, it is taking roughly 12+ hours on beefy lawrencium Xeon compute nodes.

[ardunn]

It is many of them. SiteStatsFeaturizer is among the worst offenders

[ardunn]

Try featurizing elastic tensor dataset with the autofeaturizer "best" preset and see for yourself.

[utf]

It could be that we keep recalculating the bonding which takes a long time for big structures.

We could check by seeing if there is a big speed up using MultipleFeaturizer with the additional caching. This is not a long term solution though, as we would then lose a lot of the fidelity of timing per featurizer.

If calculating the bonding is taking the most amount of time, we could alter the featurizers to also accept a BondedStructure object, in which case the bonding would not be recalculated. And then just have a StructureToBondedStructure conversion featurizer.

[ardunn]

So that is one problem. Another problem is that it is just too slow for even one SiteStatsFingerprint for many large structures. For example running it on MP towards the larger structures I get times of ~1s/sample (running on a LR4 node on Lawrencium). As far as I can tell, it is running in fully parallel mode (n_jobs = cput_count() = 24)...

[ardunn]

After looking into this, not sure caching actually really improves the performance (or maybe I am just using wrong?)

[ardunn]

done via 7f4f99e