Skip to content

dyzsasd/sparklingpandas

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

60 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

SparklingPandas

SparklingPandas aims to make it easy to use the distributed computing power of PySpark to scale your data anlysis with Pandas.

Documentation

None (right now). You can find some slides in the slides/ directory

Requirements

The primary requirement of SparklingPandas is that you have a recent (v1.0-SNAPSHOT currently) version of Spark installed - http://spark.apache.org

Using

Make sure you have the SPARK_HOME enviroment variable set correctly, as SparklingPandas uses this for including the PySpark libraries

Other than that you can install SparklingPandas with pip and just import it. The primary unit of SparklingPandas is a PRDD (Pandas Resillent Distributed Data Set)

State

This is in early development and should not be considered usable.

Support

There isn't really a mailing list, but if you want to use please feel free to e-mail me ( [email protected] ) with any questions.

About

Pandas On PySpark(POPS)

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published