Flow is a PHP based, strongly typed ETL (Extract Transform Load), asynchronous data processing library with constant memory consumption.

Supported PHP versions

• 
• 

Installation

This package is a monorepo. Please check below packages and select only those that you are going to use, this will reduce the number of unnecessary dependencies in your project (less maintenance).

• ETL
• Adapters
  • amphp
  • avro
  • chartjs
  • csv
  • doctrine
  • elasticsearch
  • google sheet
  • meilisearch
  • http
  • json
  • logger
  • parquet
  • reactphp
  • text
  • xml
• Libraries
  • array-dot - auto included
  • doctrine-dbal-bulk

For example if you want to work with json/csv files here are dependencies you will need to install:

composer require flow-php/etl:^0.1 flow-php/etl-adapter-csv:^0.1 flow-php/etl-adapter-json:^0.1

Usage

In order to understand how Flow works, please read documentation

Usage Examples

Features

• low and constant memory consumption
• asynchronous data processing
• reading from any data source
• writing to any data source
• rich collection of data transformation functions
• direct access to remote filesystems
• partitioning
• grouping & aggregating
• remote files processing
• joins
• sorting
• displaying datasets as ASCII table
• validation against schema
• window functions
• caching

Asynchronous Processing

• etl-adapter-amphp
• etl-adapter-reactphp

Building blocks

• DataFrame - Lazy data processing frame.
• Rows - Immutable collection of Row objects.
• Row - Immutable, strongly typed collection of Entry objects.
• Entry - Immutable, strongly typed object representing cell in a row.
• Extractor (Reader) - Memory safe, Data Source returning \Generator, yielding Rows to the Pipeline
• Transformer - Data transformer receiving and returning Rows (in most cases transformer), one instance of Rows at once.
• Loader (Writer) - Memory safe representation of Data Sink, responsibility of Loader is to write Rows into destination storage, one at time.
• Pipeline - Interface representing ETL process, each received Rows instanced is passed through all Pipes, also responsible for error handling.
• Pipe - Loader of Transformer instance existing in Pipes collection.

GitHub Stars

Sponsors

Flow PHP is sponsored by:

• Blackfire - the best PHP profiling and monitoring tool!