Factotum

A dag running tool designed for efficiently running complex jobs with non-trivial dependency trees.

The zen of Factotum

1. A Turing-complete job is not a job, it's a program
2. A job must be composable from other jobs
3. A job exists independently of any job schedule

User quickstart

Assuming you're running 64 bit Linux:

wget https://bintray.com/artifact/download/snowplow/snowplow-generic/factotum_0.1.0_linux_x86_64.zip
unzip factotum_0.1.0_linux_x86_64.zip
./factotum --help

Factotum currently accepts one argument, which is a factotum factfile that describes the job to run. For example, to run the sample sleep.factotum:

wget https://raw.githubusercontent.com/snowplow/factotum/release/0.1.0/samples/sleep.factotum
./factotum sleep.factotum

For more information on this file format and how to write your own jobs, see the Factfile format section below.

Factfile format

Factfiles are self-describing JSON which declare a series of tasks and their dependencies. For example:

{
    "schema": "iglu:com.snowplowanalytics.factotum/factfile/jsonschema/1-0-0",
    "data": {
        "name": "Factotum demo",
        "tasks": [
            {
                "name": "echo alpha",
                "executor": "shell",
                "command": "echo",
                "arguments": [ "alpha" ],
                "dependsOn": [],
                "onResult": {
                    "terminateJobWithSuccess": [],
                    "continueJob": [ 0 ]
                }
            },
            {
                "name": "echo beta",
                "executor": "shell",
                "command": "echo",
                "arguments": [ "beta" ],
                "dependsOn": [ "echo alpha" ],
                "onResult": {
                    "terminateJobWithSuccess": [],
                    "continueJob": [ 0 ]
                }
            },
            {
                "name": "echo omega",
                "executor": "shell",
                "command": "echo",
                "arguments": [ "and omega!" ],
                "dependsOn": [ "echo beta" ],
                "onResult": {
                    "terminateJobWithSuccess": [],
                    "continueJob": [ 0 ]
                }
            }
        ]
    }
}

This example defines three tasks that run shell commands - echo alpha, echo beta and echo omega. echo alpha has no dependencies - it will run immediately. echo beta depends on the completion of the echo alpha task, and so will wait for echo alpha to complete. echo omega depends on the echo beta task, and so will wait for echo beta to be complete before executing.

Given the above, the tasks will be executed in the following sequence: echo alpha, echo beta and finally, echo omega. Tasks can have multiple dependencies in factotum, and tasks that are parallelizable will be run concurrently. Check out the samples for more sample factfiles or the wiki for a more complete description of the factfile format.

Developer quickstart

Factotum is written in Rust.

Using Vagrant

• Clone this repository - git clone [email protected]:snowplow/factotum.git
• cd factotum
• Set up a Vagrant box and ssh into it - vagrant up && vagrant ssh
  • This will take a few minutes
• cd /vagrant
• Compile and run a demo - cargo run -- samples/echo.factotum

Using stable Rust without Vagrant

• Install Rust
  • on Linux/Mac - curl -sSf https://static.rust-lang.org/rustup.sh | sh
• Clone this repository - git clone [email protected]:snowplow/factotum.git
• cd factotum
• Compile and run a demo - cargo run -- samples/echo.factotum

Copyright and license

Snowplow is copyright 2016 Snowplow Analytics Ltd.

Licensed under the [Apache License, Version 2.0] license (the "License"); you may not use this software except in compliance with the License.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.