dask - Python for climatology, oceanograpy and atmospheric science

dask

Dask provides advanced parallelism for analytics, enabling performance at scale for the tools you love

Documentation

Embarrassingly parallel Workloads — Dask Examples documentation

https://examples.dask.org/applications/embarrassingly-parallel.html

Best Practices — Dask documentation

https://docs.dask.org/en/latest/best-practices.html

If you have a machine with 100 GB and 10 cores, then you might want to choose chunks in the 1GB range. You have space for ten chunks per core which gives Dask a healthy margin, without having tasks that are too small

Array Best Practices — Dask documentation

https://docs.dask.org/en/latest/array-best-practices.html

Select a good chunk size

While optimal sizes and shapes are highly problem specific, it is rare to see chunk sizes below 100 MB in size. If you are dealing with float64 data then this is around (4000, 4000) in size for a 2D array or (100, 400, 400) for a 3D array.

DataFrame Best Practices — Dask documentation

https://docs.dask.org/en/latest/dataframe-best-practices.html

Tutorial

Dask: Introduction - YouTube

https://www.youtube.com/watch?v=nnndxbr_Xq4

Dask Live by Coiled - YouTube

2021/10/07

https://www.youtube.com/watch?v=nHMcqEYZ5qY&t=898

https://github.com/coiled/dask-mini-tutorial/blob/main/README.md

Tips

Data Pre-Processing in Python: How I learned to love parallelized applies with Dask and Numba

https://towardsdatascience.com/how-i-learned-to-love-parallelized-applies-with-python-pandas-dask-and-numba-f06b0b367138