Skip to content

Commit

Permalink
First CLI
Browse files Browse the repository at this point in the history
  • Loading branch information
pldubouilh committed Feb 20, 2018
1 parent 182e487 commit c52e777
Show file tree
Hide file tree
Showing 8 changed files with 171 additions and 37 deletions.
47 changes: 22 additions & 25 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,49 +5,46 @@ live-torrent

Simple proof-of-concept for a live streaming solution based on [webtorrent](https://github.com/webtorrent/webtorrent). Video player courtesy of [hls.js](https://github.com/video-dev/hls.js/).



### Demo
Yes please ! Live demo at [live.computer](https://live.computer)
Yes please ! Live demo with sintel at [live.computer](https://live.computer)

### Run it yourself
```sh
# Download sintel (or use any mp4, h264 encoded file)
cd feed
wget http://peach.themazzone.com/durian/movies/sintel-1024-surround.mp4

# Generate a HLS stream - needs ffmpeg 3+
ffmpeg -i sintel-1024-surround.mp4 -codec: copy -start_number 0 -hls_time 10 -hls_list_size 0 -f hls chunks.m3u8
mv chunks.m3u8 manifest.m3u8
# Install
npm i live-torrent

# Install deps and run
cd ..
npm i
npm run test
# Start with example live-feed
live-torrent -u http://wms.shared.streamshow.it/carinatv/carinatv/ -p playlist.m3u8

# Open browser at http://127.0.0.1:8008
```

### How is it working ?
### FAQ
> I have a regular feed already
TLDR(ish); A server script parses the video manifest and generates a torrent magnet-link from the video chunks. The magnets are pushed on the manifest.
live-torrent can convert your feed into a webtorrent enabled feed. Just install the CLI tool and start converting your feed.

Now on the browser side, the videoplayer downloads the manifest, the SW hijacks the request, extract the magnet, and tries to download the chunks via webtorrent. If if fails, it falls back to server (and then seed), otherwise, well p2p - yay !
> I want to create a feed !
Basically 3 different pieces are needed :
1. server script to pre-compute the torrent magnet link of the video chunks, and add the magnet link to the manifest
2. serviceworker to proxy the manifest/chunks requests
3. client script, that's the bit utilizing webtorrent (no webrtc in SW !)
Have a look in the `server/` directory !

> How to implement on a website ?
### FAQ
Just host the script/sw yourself. Also, there are some limitations to the use of SW ; it needs to be served from HTTPS, and it should be located at the root of the domain (e.g. `https://live.computer/sw.js`). Also feel free to open an issue if something's acting weird :)

> Is it ok to use an external storage solution for my video chunks, or should I deliver them from here too ?
### How is it working ?

Just set the location of the chunks in [the server script](https://github.com/pldubouilh/live-torrent/blob/master/server/slicendice.js#L8).
TLDR(ish); A server script parses the video manifest and generates a torrent magnet-link from the video chunks. The magnets are pushed on the manifest.

> How to implement on a website ?
Now on the browser side, the videoplayer downloads the manifest, the SW hijacks the request, extracts the magnet, and tries to download the chunks via webtorrent. If if fails, it falls back the url provided (and then seed), otherwise, well p2p - yay !

Just host the script/sw yourself. Also, there are some limitations to the use of SW ; it needs to be served from HTTPS, and it should be located at the root of the domain (e.g. `https://live.computer/sw.js`). Also feel free to open an issue if something's acting weird :)
Basically 3 different pieces are needed :
1. cli.js, the server script that takes in a HLS feed and adds the magnet links to it
2. serviceworker to proxy the manifest/chunks requests
3. client script, that's the bit utilizing webtorrent (no webrtc in SW !)

### TODO:
- [x] Implement CLI tool that could live on top of existing feeds
- [ ] Optimise p2p - shave off more time for webtorrent to download the chunks
- [ ] Implement CLI tool that could live on top of existing feeds
121 changes: 121 additions & 0 deletions cli.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
#!/usr/bin/env node
const parseTorrent = require('parse-torrent')
const createTorrent = require('create-torrent')
const request = require('request-promise-native')
const argv = require('yargs').argv
const express = require('express')
const app = express()

let playlistLocation = ''
let playlistName = ''
let chunksLocation = ''
const announceList = [['wss://tracker.openwebtorrent.com']]
let manifest
let sequence = 0
const fileToMagnet = {}
const magnetsOrder = []
const entries = []

const help = `
🛰 Live-torrent 🛰
-u manifest location
-p playlist name
-c video chunk location - default same as url
-r manifest refresh rate (in seconds) - default 5
eg. live-torrent -u http://wms.shared.streamshow.it/carinatv/carinatv/ -p playlist.m3u8
`

function die (msg, code) {
console.log(help + '\n' + msg)
process.exit(code)
}

const computeMagnet = (file, fn) => {
return new Promise(async (resolve, reject) => {
file.name = fn
createTorrent(file, { announceList }, (err, t) => {
if (err) return console.log(err)
const magnet = parseTorrent.toMagnetURI(parseTorrent(t))
resolve(magnet)
})
})
}

async function makeMagnet (fn) {
// Fetch payload and compute magnet
const payload = await request(chunksLocation + fn, { encoding: null })
const magnet = await computeMagnet(payload, fn)

// Store magnet computed
fileToMagnet[fn] = magnet
magnetsOrder.push(fn)

if (magnetsOrder.length > 20) {
const oldMagnet = magnetsOrder.shift()
delete fileToMagnet[oldMagnet]
}
}

async function makeAllMagnets (files) {
return Promise.all(files.map(makeMagnet))
}

async function doManifest (path = '') {
const _manifest = await request(playlistLocation + (path || playlistName))

// Head over to the playlist, if what we got was a link to a playlist
if (_manifest.includes('.m3u8')) {
const m3u8 = _manifest.split('\n').find(l => l.includes('.m3u8'))
return doManifest(m3u8)
}

const split = _manifest.split('\n')

// Get sequenece number
const _sequence = split.filter(l => l.includes(`#EXT-X-MEDIA-SEQUENCE:`))[0].replace('#EXT-X-MEDIA-SEQUENCE:', '')
if (_sequence === sequence) {
return console.log('\nManifest unchanged\n')
}

// Split manifest
sequence = _sequence
const files = split.filter(l => l.includes('.ts'))
const times = split.filter(l => l.includes('#EXTINF'))
const head = split.filter(l => !times.includes(l) && !files.includes(l))
const lastFile = files[files.length - 1]

// Add entries with webtorrent
if (!entries.length) {
await makeAllMagnets(files)
entries.push(`${times[0]}\n###${fileToMagnet[files[0]]}\n${chunksLocation + files[0]}`)
entries.push(`${times[1]}\n###${fileToMagnet[files[1]]}\n${chunksLocation + files[1]}`)
entries.push(`${times[2]}\n###${fileToMagnet[files[2]]}\n${chunksLocation + files[2]}`)
} else {
entries.shift()
await makeMagnet(lastFile)
entries.push(`${times[2]}\n###${fileToMagnet[files[2]]}\n${chunksLocation + files[2]}`)
}

manifest = head.join('\n') + entries.join('\n')
console.log(manifest)
}

if (argv.h || argv.help) {
die('', 0)
}

if (argv.u && argv.p) {
console.log('Starting server\n')
app.get('*.m3u8', (req, res) => res.send(manifest))
app.use(express.static('client'))
app.listen(8008)

chunksLocation = argv.c || argv.u
playlistLocation = argv.u
playlistName = argv.p
doManifest()
setInterval(doManifest, (argv.r || 5) * 1000)
} else {
die('', 0)
}
2 changes: 1 addition & 1 deletion client/build.js
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ const WebTorrent = require('webtorrent')
const client = new WebTorrent()

const torrents = [] // {name, magnetURI}
const announceList = [['wss://tracker.btorrent.xyz']]
const announceList = [['wss://tracker.openwebtorrent.com']]

console.logColor = (msg, color) => console.log('%c' + msg, `color: ${color}; font-size: 11px;`)

Expand Down
2 changes: 1 addition & 1 deletion client/loader.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ const WebTorrent = require('webtorrent')
const client = new WebTorrent()

const torrents = [] // {name, magnetURI}
const announceList = [['wss://tracker.btorrent.xyz']]
const announceList = [['wss://tracker.openwebtorrent.com']]

console.logColor = (msg, color) => console.log('%c' + msg, `color: ${color}; font-size: 11px;`)

Expand Down
7 changes: 0 additions & 7 deletions feed/readme.md

This file was deleted.

10 changes: 8 additions & 2 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"description": "",
"main": "index.js",
"scripts": {
"test": "browserify client/loader.js -o client/build.js && node server/slicendice.js"
"test": "browserify client/loader.js -o client/build.js && node cli.js -u http://wms.shared.streamshow.it/carinatv/carinatv/ -p playlist.m3u8"
},
"keywords": [
"webtorrent",
Expand All @@ -22,6 +22,9 @@
"engines": {
"node": ">=8"
},
"bin": {
"live-torrent": "./cli.js"
},
"author": "pldubouilh",
"license": "MIT",
"bugs": {
Expand All @@ -35,7 +38,10 @@
"fs-extra": "^5.0.0",
"m3u8": "0.0.7",
"parse-torrent": "^5.8.3",
"webtorrent": "^0.98.21"
"request": "^2.83.0",
"request-promise-native": "^1.0.5",
"webtorrent": "^0.98.21",
"yargs": "^11.0.0"
},
"devDependencies": {
"standard": "^11.0.0"
Expand Down
17 changes: 17 additions & 0 deletions server/readme.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
Here's a quick howto on how to get started from a video file.

```sh
# Download test file, or use any mp4, h264 encoded file
mkdir feed && cd feed
wget http://peach.themazzone.com/durian/movies/sintel-1024-surround.mp4

# Convert to HLS (needs ffmpeg 3+)
ffmpeg -i sintel-1024-surround.mp4 -codec: copy -start_number 0 -hls_time 10 -hls_list_size 0 -f hls chunks.m3u8

# Rename manifest
mv chunks.m3u8 manifest.m3u8

# Start server script that will chop the HLS manifest into a live strean, and then serve your chunks and the client test page
cd ..
node server/slicendice.js
```
2 changes: 1 addition & 1 deletion server/slicendice.js
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ const extraPath = ''
const pathChunks = 'feed/'
const chunkName = 'chunks'
const manifestName = 'manifest.m3u8'
const announceList = [['wss://tracker.btorrent.xyz']]
const announceList = [['wss://tracker.openwebtorrent.com']]
let targetDuration = 10

let magnets = {}
Expand Down

0 comments on commit c52e777

Please sign in to comment.