zio-spark

A functional wrapper around Spark to make it work with ZIO

Documentation

You can find the documentation of zio-spark here.

Help

You can ask us (Dylan, Jonathan) for some help if you want to use the lib or have questions around it : https://calendly.com/zio-spark/help

Latest version

If you want to get the very last version of this library you can still download it using:

libraryDependencies += "io.univalence" %% "zio-spark" % "0.4.0"

Snapshots

If you want to get the latest snapshots (the version associated with the last commit on master), you can still download it using:

resolvers += Resolver.sonatypeRepo("snapshots"),
libraryDependencies += "io.univalence" %% "zio-spark" % "<SNAPSHOT-VERSION>"

You can find the latest version on nexus repository manager.

Spark Version

zio-spark is built against spark 2.4.x and scala 2.12. Spark is provided, you should your Spark version (as you would usually).

libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.4.7" % Provided

Why ?

There are many reasons why we decide to build this library, such as:

• allowing user to build Spark pipeline with ZIO easily.
• making better code, pure FP, more composable, more readable Spark code.
• stopping the propagation of implicit SparkSessions.
• improving some performances.
• taking advantage of ZIO allowing our jobs to retry and to be run in parallel.

Alternatives

• ZparkIO a framework for Spark, ZIO

Contributions

Pull requests are welcomed. We are open to organize pair-programming session to tackle improvements. If you want to add new things in zio-spark, don't hesitate to open an issue!

You can also talk to us directly using this link if you are interested to contribute https://calendly.com/zio-spark/contribution.