Civilization progresses by the number of operations it can perform without conscious effort.
— Whitehead

Record your screen & mic 24/7 and connect it to LLMs. Inspired by adept.ai, rewind.ai, Apple Shortcut. Written in Rust. Free. You own your data.

screenpipe is a library that allows you to gather all your life context and connect it to LLMs easily for:

• search (e.g. go beyond your limited human memory)
• automation (such as making actions on the web while you work, syncing company's knowledge, etc.)
• etc.

Example vercel/ai-chatbot that query screenpipe autonomously

Check this example of screenpipe which is a chatbot that make requests to your data to answer your questions

070424.mp4

Status

Alpha: runs on my computer (Macbook pro m3 32 GB ram). Uses 2x less energy than Rewind.ai

• screenshots
• mp4 encoding to disk (30 GB / month)
• sqlite local db
• OCR
• audio + stt
• api
• TS SDK
• cloud storage options (s3, pqsql, etc.)
• cloud computing options
• fast, optimised
• bug-free

Usage

Keep in mind that it's still experimental.

Linux

curl -sSL https://raw.githubusercontent.com/louis030195/screen-pipe/main/install.sh | sh

Now you should be able to screenpipe. (You may need to restart your terminal, or find the CLI in $HOME/.local/bin)

MacOS

On Mac you need to build the CLI yourself.

1. Install dependencies:

# On Mac
brew install ffmpeg

Install Rust.

2. Clone the repo:

git clone https://github.com/louis030195/screen-pipe
cd screen-pipe

3. Run the API:

# This runs a local SQLite DB + an API + screenshot, ocr, mic, stt, mp4 encoding
# if you are on mac, use "cargo build --release --features metal" to leverage M series acceleration
cargo build --release
./target/release/pipe


# or only stream audio + speech to text to stdout
./target/release/pipe-audio

# or only stream screenshots + ocr to stdout
./target/release/pipe-vision

# or only record mp4 videos + json containing ocr
./target/release/pipe-video

Struggle to get it running? I'll install it with you in a 15 min call.

We are working toward making it easier to try, feel free to help!

What's next?

Examples to query the API

# 1. Basic search query
curl "http://localhost:3030/search?q=test&limit=5&offset=0"

# 2. Search with content type filter (OCR)
curl "http://localhost:3030/search?q=test&limit=5&offset=0&content_type=ocr"

# 3. Search with content type filter (Audio)
curl "http://localhost:3030/search?q=test&limit=5&offset=0&content_type=audio"

# 4. Search with pagination
curl "http://localhost:3030/search?q=test&limit=10&offset=20"

# 6. Search with no query (should return all results)
curl "http://localhost:3030/search?limit=5&offset=0"

Now pipe this into a LLM to build:

• memory extension apps
• automatic summaries
• automatic action triggers (say every time you see a dog, send a tweet)
• automatic CRM (fill salesforce while you spam ppl on linkedin)

Check example with vercel/ai-chatbot project (nextjs)

Why open source?

Recent breakthroughs in AI have shown that context is the final frontier. AI will soon be able to incorporate the context of an entire human life into its 'prompt', and the technologies that enable this kind of personalisation should be available to all developers to accelerate access to the next stage of our evolution.

Principles

This is a library intended to stick to simple use case:

• record the screen & associated metadata (generated locally or in the cloud) and pipe it somewhere (local, cloud)

Think of this as an API that let's you do this:

screenpipe | ocr | llm "turn what i see into my CRM" | api "send data to salesforce api"

Any interfaces are out of scope and should be built outside this repo, for example:

• UI to search on these files (like rewind)
• UI to spy on your employees
• etc.

Contributing

Contributions are welcome! If you'd like to contribute, please fork the repository and use a feature branch. Pull requests are warmly welcome.

Say 👋 in our public Discord channel . We discuss how to bring this lib to production, help each other with contributions, personal projects or just hang out ☕.

Bit more details on the architecture here.

Licensing

The code in this project is licensed under MIT license. See the LICENSE file for more information.

Related projects

This is a very quick & dirty example of the end goal that works in a few lines of python: https://github.com/louis030195/screen-to-crm

Very thankful for https://github.com/jasonjmcghee/xrem which was helpful. Although screenpipe is going in a different direction.

FAQ

Where is the data stored?

• 100% of the data stay local in a SQLite database and mp4 files
• if you use an LLM like OpenAI, part of your data will be sent to Microsoft servers, you can use a local LLM like Chrome AI