Skip to content

Latest commit

 

History

History
 
 

timer

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
examples
examples
 
 
.gitignore
.gitignore
 
 
README.md
README.md
 
 

README.md

pipedream

In this README, you'll learn how to run any Node.js code on a schedule using Pipedream.

Quickstart

Watch this short video or step through the commands below.

Install the Pipedream CLI:

curl https://cli.pipedream.com/install | sh

Create a file with a single console.log() statement:

echo 'console.log("Hello, world")' > cronjob.js

And run that code once every 15 seconds:

pd deploy --run cronjob.js --timer --frequency 15s  # cron expressions also supported with --cron

You'll be asked to sign up or sign in to Pipedream at this step. Once that's done, the CLI should deploy your code.

Each time this script runs, it prints "Hello, world". The CLI will automatically display new logs as the job runs. You can also press Ctrl-C and listen for new logs anytime by running:

pd logs cronjob-js

You can delete the job and all its logs by running:

pd delete cronjob-js

Reference

Overview

Pipedream is a platform for running hosted, backend components.

In this README, you'll learn what components are and how to use them to run code on a schedule. You can run Node code directly on Pipedream or use components to trigger external services.

Pipedream components are reusable Node.js modules that run code on specific events: timers and HTTP requests. Components are free to run and simple to learn. Here's a component that prints "Hello, world" once a minute:

module.exports = {
  name: "cronjob",
  version: "0.0.1",
  props: {
    timer: {
      type: "$.interface.timer",
      default: {
        cron: "* * * * *",
      },
    },
  },
  async run() {
    console.log("Run any Node.js code here");
  },
};

You can use components to:

Components come with a built-in key-value store, an interface for passing input via props, and more. You deploy and manage components using Pipedream's REST API, CLI, or UI.

Components can emit events, which can be retrieved programmatically via CLI, API or SSE. They can also trigger Pipedream workflows on every event. For example, you can process items from an RSS feed and access the items via REST API, or trigger code to run on every new item using the SSE interface or a workflow. Components that emit events are called event sources.

Running code on a schedule

You can run code from a few different sources:

In the examples below, you'll use the Pipedream CLI to deploy scheduled jobs. This is the simplest way to interact with Pipedream while you're learning. Install the Pipedream CLI before you begin:

curl https://cli.pipedream.com/install | sh

You can also deploy components using the Pipedream UI or REST API.

Run local Node code

Create a local file, job.js, that contains the following Node code:

console.log("Hello, world");

You can run this script on a schedule using pd deploy:

pd deploy --timer --cron "0 0 * * *" --run job.js

The argument passed to the --cron option — the "0 0 * * *" — is a cron expression. Cron expressions allow you to specify your job's frequency using a terse, but expressive, syntax. Cron expressions are tied to the UTC timezone.

You can also run a job every N seconds, minutes, or hours using the --frequency option:

pd deploy --timer --frequency 60s --run job.js  # Runs every 60 seconds
pd deploy --timer --frequency 30m --run job.js  # Runs every 30 minutes
pd deploy --timer --frequency 2h --run job.js   # Runs every 2 hours

When you run the command above, Pipedream generates a component from the code in job.js, placing it into the component's run method. Then, the CLI deploys that component to Pipedream's infrastructure and runs it at the schedule you specified.

You can view the component code in the Configuration tab tied to that component in the Pipedream UI, or by running

pd describe <component-name> --code

If you'd like to save the component code locally, just run:

pd describe <component-name> --code > component.js

Run a script hosted on Github

You can also deploy any Node.js code hosted on Github:

pd deploy --timer --cron "0 0 * * *" --run https://github.com/PipedreamHQ/pipedream/blob/master/interfaces/timer/examples/code.js

You'll need to copy the URL tied to the file in the Github repo UI, not the raw URL tied to the file's contents you get when you press the Raw button. That is, pass this:

https://github.com/PipedreamHQ/pipedream/blob/master/interfaces/timer/examples/code.js

not this:

https://raw.githubusercontent.com/PipedreamHQ/pipedream/master/interfaces/timer/examples/code.js

Create your own component

Components are Node.js modules that export an object with the following properties:

module.exports = {
  name: "cronjob", // required
  version: "0.0.1", // required
  props,
  methods,
  run() {
    console.log("Run any Node.js code here");
  },
};

The component API is deliberately simple. Your component must define a name, version, a run function, and optional props that allow the component to accept inputs on deploy.

To run code on a schedule, just include it in your run function. Here's a simple console.log() statement that runs once an hour:

module.exports = {
  name: "cronjob",
  version: "0.0.1",
  props: {
    timer: {
      type: "$.interface.timer",
      default: {
        cron: "0 0 * * *",
      },
    },
  },
  run() {
    console.log("Run any Node.js code here");
  },
};

Save this file and deploy it to Pipedream:

pd deploy job.js

Notice the difference between this and the pd deploy commands in the examples above. Before, you deployed Node code directly from a local script or Github. That code wasn't wrapped in a component, so you had to specify the --timer flag to indicate it should be run on a schedule, and pass the --run flag to tell the CLI what code to run. This directed the CLI to generate a component with the correct timer prop, a name, version, and put the code you passed to the --run flag into the component's run method.

Here, you've written the component yourself, so you can pd deploy that directly to Pipedream.

Send an HTTP request

If you already have an HTTP endpoint in front of code hosted on another platform, and just need to make an HTTP request to invoke it, use axios or another HTTP client library to make the request from your run function:

module.exports = {
  name: "invoke-http-function",
  version: "0.0.1",
  props: {
    timer: {
      type: "$.interface.timer",
      default: {
        cron: "0 0 * * *",
      },
    },
  },
  async run() {
    const axios = require("axios");

    const resp = await axios({
      method: "GET",
      url: `https://swapi.co/api/films/`,
    });

    console.log(resp.data);
  },
};

You can deploy this example component by running:

pd deploy https://github.com/PipedreamHQ/pipedream/blob/master/interface/timer/examples/send-http-request.js

See our guide on making HTTP requests with axios for more example code.

Different schedule types

You can schedule code using a cron expression or a frequency in seconds, minutes, or hours. See the examples below for how to specify these schedules using the CLI and via component props.

Using the CLI

Schedule code using a cron expression:

pd deploy --timer --cron "0 0 * * *" --run job.js

All cron schedules are tied to the UTC timezone.

You can also run a job every N seconds, minutes, or hours using the --frequency option:

pd deploy --timer --frequency 60s --run job.js  # Runs every 60 seconds
pd deploy --timer --frequency 30m --run job.js  # Runs every 30 minutes
pd deploy --timer --frequency 2h --run job.js   # Runs every 2 hours

Using component props

If you're deploying a component directly, you can specify a cron schedule directly in props using the cron property:

module.exports = {
  name: "cronjob",
  version: "0.0.1",
  props: {
    timer: {
      type: "$.interface.timer",
      default: {
        cron: "0 0 * * *",
      },
    },
  },
  run() {
    console.log("Run any Node.js code here");
  },
};

All cron schedules are tied to the UTC timezone.

You can also schedule a job to run every N seconds using the intervalSeconds property, instead:

module.exports = {
  name: "cronjob",
  version: "0.0.1",
  props: {
    timer: {
      type: "$.interface.timer",
      default: {
        intervalSeconds: 60,
      },
    },
  },
  run() {
    console.log("Run any Node.js code here");
  },
};

Notes and Limitations

Any cron schedule you specify using the --cron option is tied to the UTC timezone. If you need help converting your schedule to UTC, Google can help. If you'd like to run a job every day at 8:00pm local time, for example, just Google "8:00pm local time to UTC".

Scheduled code can run up to once every 15 seconds. If you need to run a job at this frequency, you'll need to use the --frequency scheduling option, since cron expressions cannot exceed frequencies of once per minute:

pd deploy --timer --frequency 15s --run code.js

More generally, components are subject to the limits of the Pipedream platform.

Using the REST API

You can also create a scheduled job using the REST API.

You can deploy a new component by sending a POST request to the /sources endpoint. This accepts two parameters:

The examples/create-component/api-payload.json file contains a sample payload, with a component that prints a single console.log statement once a minute. You can deploy this component using this cURL command:

curl -d @examples/create-component/api-payload.json \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer <PIPEDREAM_API_KEY>' \
  https://api.pipedream.com/v1/sources

or using the sample Node code in the examples/create-component directory:

const axios = require("axios");
require("dotenv").config();
const data = require("./api-payload.json");

axios({
  method: "POST",
  url: "https://api.pipedream.com/v1/sources",
  headers: {
    Authorization: `Bearer ${process.env.PIPEDREAM_API_KEY}`,
  },
  data,
})
  .then((res) => console.log(res))
  .catch((err) => console.log(`Error: ${err}`));

Example cron jobs

This repo contains a number of example components that run code on a schedule.

Logs

Each time your job runs, Pipedream marks its start and end times in the LOGS attached to your component in the UI.

Pipedream refers to logs as "observations" in certain contexts, since they contain a superset of standard output / error, as well as events emitted by your component, and start and end markers for each job run.

Any standard output or errors raised by your component are also logged here. You can watch these logs in realtime using the pd logs CLI command:

pd logs <component-name>

Consuming event data from your own app, outside Pipedream

Components can emit data that you can access from any application. Within your run method, pass the data you'd like to emit to the this.$emit function:

this.$emit({
  name: "Luke Skywalker",
});

Each time you run this.$emit(), you emit the data as an event. Components that emit data are called event sources.

Events can be retrieved using the REST API, CLI, or SSE stream tied to your cron job. For example, you can use the CLI to retrieve the last 3 events:

λ pd events -n 3 <job-name>
{ name: "Luke Skywalker" }
{ name: "Leia Organa" }
{ name: "Han Solo" }

This makes it easy to retrieve data processed by your cron job from another app. Typically, you'll want to use the REST API to retrieve events in batch, and connect to the SSE stream to process them in real time.

Using npm packages

To use an npm package in a code step, just require it:

const _ = require("lodash");

When you deploy a component, Pipedream downloads these packages and bundles them with your deployment. There's no need to include a package.json file with your component.

Some packages — for example, packages like Puppeteer, which includes large dependencies like Chromium — may not work on Pipedream. Please reach out if you encounter a specific issue.

Pause a component

You can stop a component from running using pd update:

pd update <component-name> --active false

You can activate a component again by running:

pd update <component-name> --active true

Change a component's schedule

You can update a component's schedule by downloading the component's code and changing its timer props.

For example, let's say you have a component that runs once an hour:

module.exports = {
  name: "cronjob",
  version: "0.0.1",
  props: {
    timer: {
      type: "$.interface.timer",
      default: {
        cron: "0 0 * * *",
      },
    },
  },
  run() {
    console.log("Run any Node.js code here");
  },
};

Download this code locally:

pd describe <component-name> --code > component.js

and modify the cron prop (or intervalSeconds, if you specified the job to run at a frequency). For example, if you want to change the job to run once an hour, change cron to:

cron: "0 * * * *"

Once you update the code, run

pd update <component-name> --code component.js

Delete a component

You can stop a component from running, deleting all of its events and logs, using pd delete:

pd delete <component-name>

Pricing

Pipedream is currently free (paid tiers are coming soon), subject to the limits noted below.

If you exceed any of these limits, please reach out.

Limits

Components are subject to the limits of the Pipedream platform. Key limits include:

If you exceed any of these limits, please reach out.

Getting Support

You can get help on our public Slack or reach out to our team directly with any questions or feedback. We'd love to hear from you!