Fix slow import time for Parcels

[VeckoTheGecko]

Parcels version

main

Description

Currently import parcels takes several seconds to complete. This should be sped up

Links:

• https://scientific-python.org/specs/spec-0001/
• https://pytutorial.com/optimize-python-import-performance/

Code sample

No response

[VeckoTheGecko]

Nick (@VeckoTheGecko) — 13:13
Does anyone have recommendations on dealing with/debugging long import times when creating packages? I'm working on a package that has an import time of ~10s. I don't really know what is "reasonable" out in the wild or what workarounds are. numpy and scipy are both quite quick, xarray (which is our core dependency) takes ~5s.
hoxbro — 13:36
I usually check import time like this: python -X importtime -c 'import xarray' 2> tuna.log && tuna tuna.log where tuna is https://pypi.org/project/tuna/.

• Scientific Python discord

Gave it a try - this works quite well. We can add it to our Pixi tooling

[VeckoTheGecko]

xgcm/xgcm#697

[VeckoTheGecko]

Once patching xgcm, these are the import times (i.e., 2.4s)

Maybe we can get further improvements

[erikvansebille]

Do we really need to always import sklearn? Is that not also a dependency that will not always be used? That could also shave off quite some import time?

[VeckoTheGecko]

Do we really need to always import sklearn? Is that not also a dependency that will not always be used? That could also shave off quite some import time?

Yes! good call

[VeckoTheGecko]

This sklearn import is due to outdated v3 compat code for when running with MPI. I don't think MPI is on our roadmap right @erikvansebille ? I'm happy to remove this compat code, but should we clean out the other files as well?

I was actually surprised sklearn was even installed since its not in our dependency list, but turns out its a transitive dep of uxarray (go figure, already knew they had a heavy environment 😅 )

[erikvansebille]

Yep, MPI is not on our v4 roadmap, so can indeed be removed as far as I'm concerned

[VeckoTheGecko]

With the xgcm and sklearn changes, import time is down from 5s to 1.674 s for parcels

So much better. It still feels a tad slow - let's keep this open in case we can make further improvements down the line.