Ongoing research training transformer models at scale

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Build a Jekyll blog in minutes, without touching the command line.

Optimized primitives for collective multi-GPU communication

FastFlow is a system that automatically detects CPU bottlenecks in deep learning training pipelines and resolves the bottlenecks with data pipeline offloading to remote resources .

Tensors and Dynamic neural networks in Python with strong GPU acceleration

An Open Source Machine Learning Framework for Everyone

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch…

An open-source storage layer that brings scalable, ACID transactions to Apache Spark™ and big data workloads.

Apache Druid: a high performance real-time analytics database.

Web-based notebook that enables data-driven, interactive data analytics and collaborative documents with SQL, Scala and more.

Apache Spark - A unified analytics engine for large-scale data processing