Fugue is a pure abstraction layer that adapts to different computing frameworks such as Spark and Dask. It is to unify the core concepts of distributed computing and to help you decouple your logic from specific computing frameworks.
pip install fugue
Fugue has these extras:
- sql: to support Fugue SQL
- spark: to support Spark as the ExecutionEngine
- dask: to support Dask as the ExecutionEngine
For example a common use case is:
pip install fugue[sql,spark]
To read the complete static docs, click here
The best way to start is to go through the tutorials. We have the tutorials in an interactive notebook environent.
But it runs slow on binder, the machine on binder isn't powerful enough for a distributed framework such as Spark. Parallel executions can become sequential, so some of the performance comparison examples will not give you the correct numbers.
Alternatively, you should get decent performance if running its docker image on your own machine:
docker run -p 8888:8888 fugueproject/tutorials:latest