doc: Various documentation updates.

README.md

@@ -84,14 +84,14 @@ Note that the default HTTP port or playback (8935) is different from the CLI API
 ### Using Amazon S3 for storing stream's data
 
 You can use S3 to store source and transcoded data.
-For that livepeer should be run like this `livepeer -s3bucket region/bucket -s3creds accessKey/accessKeySecret`. Stream's data will be saved into directory `MANIFESTID_NONCE`, where MANIFESTID - id of the manifest associeated with stream, NONCE - random number that uniquely identifies this stream. In this directory will be saved all the segments data, plus manifest, named `MANIFESTID_full.m3u8`.
+For that livepeer should be run like this `livepeer -s3bucket region/bucket -s3creds accessKey/accessKeySecret`. Stream's data will be saved into directory `MANIFESTID`, where MANIFESTID - id of the manifest associated with stream. In this directory will be saved all the segments data, plus manifest, named `MANIFESTID_full.m3u8`.
 Livepeer node doesn't do any storage management, it only saves data and never deletes it.
 
-### Becoming a Transcoder
+### Becoming an Orchestrator
 
 We'll walk through the steps of becoming a transcoder on the test network.  To learn more about the transcoder, refer to the [Livepeer whitepaper](https://github.com/livepeer/wiki/blob/master/WHITEPAPER.md) and the [Transcoding guide](http://livepeer.readthedocs.io/en/latest/transcoding.html).
 
-- `livepeer --rinkeby --transcoder` to start the node as a transcoder.
+- `livepeer --rinkeby --transcoder` to start the node as an orchestrator with an attached local transcoder .
 
 - `livepeer_cli` - make sure you have test ether and test Livepeer token.  Refer to the Quick Start section for getting test ether and test tokens.
 
@@ -101,7 +101,21 @@ We'll walk through the steps of becoming a transcoder on the test network.  To l
 
 - Wait for the next round to start, and your transcoder will become active.
 
-- If running on Rinkeby or mainnet, ensure your transcoder is *publicly accessible* in order to receive jobs from broadcasters. The only port that is required to be public is the one that was set during the transcoder registration step (default 8935).
+- If running on Rinkeby or mainnet, ensure your orchestrator is *publicly accessible* in order to receive jobs from broadcasters. The only port that is required to be public is the one that was set during the transcoder registration step (default 8935).
+
+### Standalone Orchestrators
+
+Orchestrators can be run in standalone mode without an attached transcoder. Standalone transcoders will need to connect to this orchestrator in order for the orchestrator to process jobs.
+
+- `livepeer --rinkeby --orchestrator -orchSecret asdf`
+
+The orchSecret is a shared secret used to authenticate remote transcoders. It can be any arbitrary string.
+
+### Standalone Transcoders
+
+A standalone transcoder can be run which connects to a remote orchestrator. The orchestrator will send transcoding tasks to this transcoder as segments come in.
+
+- `livepeer -standaloneTranscoder -orchAddr 127.0.0.1:8935 -orchSecret asdf`
 
 ## Contribution
 Thank you for your interest in contributing to the core software of Livepeer.

cmd/livepeer/livepeer.go

@@ -80,7 +80,7 @@ func main() {
 	ipfsPath := flag.String("ipfsPath", fmt.Sprintf("%v/.ipfs", usr.HomeDir), "IPFS path")
 	noIPFSLogFiles := flag.Bool("noIPFSLogFiles", false, "Set to true if log files should not be generated")
 	offchain := flag.Bool("offchain", false, "Set to true to start the node in offchain mode")
-	serviceAddr := flag.String("serviceAddr", "", "Transcoder only. Public address:port that broadcasters can use to contact this node; may be an IP or hostname. If used, should match the on-chain ServiceURI set via livepeer_cli")
+	serviceAddr := flag.String("serviceAddr", "", "Orchestrator only. Overrides the on-chain serviceURI that broadcasters can use to contact this node; may be an IP or hostname.")
 	initializeRound := flag.Bool("initializeRound", false, "Set to true if running as a transcoder and the node should automatically initialize new rounds")
 	s3bucket := flag.String("s3bucket", "", "S3 region/bucket (e.g. eu-central-1/testbucket)")
 	s3creds := flag.String("s3creds", "", "S3 credentials (in form ACCESSKEYID/ACCESSKEY)")

doc/networking.md

@@ -6,120 +6,97 @@
 [Sequence diagram source](https://sequencediagram.org/index.html#initialData=C4S2BsFMAIBkQG6QA6UgJ2gOUsA7gPboDWIAdgObQIBMAUHcgIbqgDGIzZw0ASpBRABnYOgCedAELoCTACZsmIjAFoAfP0EjxALmgB5dGwAWkbU2BE+kAI51Nw0WJXrpshUuAY9hk2dEWVvxCyAxu8orK6Oq+puaW6DoAOmQAKuhMZEJsBHIY1nbhHlEAPC6x-hkJOumZ2bn5AJJkAGYEDAzgBAShRZFe0Wq1WTl5idAAygIAtpDcBVIyEZ4YZSrD9WN6UxSz88Ghc3J0QA)
 
 
-## Terminology
-
-1. **Claimable** : A job is claimable if the current block is within 256 blocks of the job's creation, or a claim has already been submitted for the job.
-
 ## Broadcaster to Registry
 
-The broadcaster does an on-chain lookup to retrieve the orchestrator's ServiceURI in order to initiate an transcoding job.
+The broadcaster does an on-chain lookup to retrieve the orchestrator's ServiceURI in order to discover prices, capabilities and to initiate an transcoding job.
 
 ## Broadcaster to Orchestrator:
 
-### gRPC `GetTranscoder`
+### gRPC `GetOrchestrator : OrchestratorRequest -> OrchestratorInfo`
 
-This is called by the broadcaster to request information on a transcoder from an orchestrator. The request contains the following data:
+The `GetOrchestrator` method is called by the broadcaster to discover information about an orchestrator. The request contains the following data:
 
 ```protobuf
-// This request is sent by the broadcaster in `GetTranscoder` to request
-// information on which transcoder to use.
-message TranscoderRequest {
+// This request is sent by the broadcaster in `GetOrchestrator` to request
+// information on the orchestrator.
+message OrchestratorRequest {
 
-  // ID of the job that the broadcaster needs a transcoder for
-  int64 jobId = 1;
+  // Ethereum address of the broadcaster
+  bytes address = 1;
 
-  // Broadcaster's signature over the jobId
-  bytes sig   = 2;
+  // Broadcaster's signature over its hex-encoded address
+  bytes sig     = 2;
 }
 ```
 
-The `sig` field is the broadcaster's signature using its Ethereum key, over the jobId expressed as a string. The signature consists of the resulting bytes without any further encoding.
-`sig = broadcaster.Sign(jobId.String())`
-
-Verification of `TranscoderRequest` consists of the following steps:
-1. Fetch the job based on JobId.
-    * TODO distinguish if an orchestrator is lagging behind and hasn't yet received the block with the job. Might require incorporating the node's **current** block number into the request.
-2. Ensure the orchestrator is assigned to the job.
-3. Ensure the current block falls within the valid range of `CreationBlock` ... `EndBlock` for the job.
-4. Ensure the job is still claimable.
-5. Verify that the signature comes from the broadcaster.
-
-The `TranscoderInfo` response contains:
-
-```protobuf
-// The orchestrator sends this in response to `GetTranscoder`, containing the
-// transcoder URI, associated credentials authorizing the broadcaster to
-// use the transcoder, and miscellaneous data related to the job.
-message TranscoderInfo {
-
-  // URI of the transcoder to use for submitting segments
-  string transcoder  = 1;
-
-  // Signals the authentication method to expect within `credentials`. This
-  // field is opaque to the broadcaster, and should be passed to the transcoder.
-  string authType    = 2;
-
-  // Credentials to verify the request has been authorized by an orchestrator.
-  // This field is opaque to the broadcaster.
-  string credentials = 3;
+The `sig` field is the broadcaster's signature using its Ethereum key, over the broadcaster's public Ethereum address, expressed as a hex-encoded string. The signature consists of the resulting bytes without any further encoding.
+`sig = broadcaster.Sign(address.Hex())`
 
-  // Transcoded streamId list to update the master manifest on the broadcaster.
-  map<string, string> streamIds = 16;
-}
-```
+Verification of `OrchestratorRequest` consists of the following steps:
+1. Check the signature `sig` was produced by the address given by `address`.
 
-### Notes
-The authorization token will become more useful when the transcoder and orchestrator are separate entities. Additionally, the auth token can incorporate additional fields to avoid an on-chain lookup, such as the StreamID, token start time, and approximate job end time. This is stored as a base64-encoded, serialized protbuf struct.
+The `OrchestratorInfo` response contains:
 
 ```protobuf
-// AuthToken is sent by the orchestrator and encoded in the `credentials` field
-// This record is opaque to the broadcaster and is only relevant between the
-// orchestrator and the transcoder.
-message AuthToken {
+// The orchestrator sends this in response to `GetOrchestrator`, containing the
+// miscellaneous data related to the job.
+message OrchestratorInfo {
 
-  // Signature of the orchestrator over the remaining fields
-  bytes sig   = 1;
+  // URI of the orchestrator to use for submitting segments
+  string orchestrator = 1;
 
-  int64 jobId = 16;
+  // Orchestrator's preferred object storage, if any
+  repeated OSInfo storage = 32;
 }
 ```
 
 ## Broadcaster to Transcoder
 
 ### POST `/segment`
 
-Invoked each by the broadcaster for each segment that needs to be transcoded. The transcoder address is taken from the `transcoder` field in `TranscoderInfo`.
+Invoked each by the broadcaster for each segment that needs to be transcoded. The orchestrator address is taken from the `orchestrator` field in `OrchestratorInfo`.
 
 #### Required Headers:
 
-* **Authorization**
-Taken from the `authType` field in TranscoderInfo.
-* **Credentials**
-Taken from the `credentials` field in TranscoderInfo.
 * **Livepeer-Segment**
 Proves that the broadcaster generated this segment. Serialized protobuf struct, base64 encoded.
 ```protobuf
+// Data included by the broadcaster when submitting a segment for transcoding.
 message SegData {
-  int64 seq  = 1;
-  bytes hash = 2;
-  bytes sig  = 3;
+
+  // Manifest ID this segment belongs to
+  bytes manifestId = 1;
+
+  // Sequence number of the segment to be transcoded
+  int64 seq        = 2;
+
+  // Hash of the segment data to be transcoded
+  bytes hash       = 3;
+
+  // Transcoding profiles to use
+  bytes profiles   = 4;
+
+  // Broadcaster signature for the segment. Corresponds to:
+  // broadcaster.sign(manifestId | seqNo | dataHash | profiles)
+  bytes sig        = 5;
+
+  // Broadcaster's preferred storage medium(s)
+  repeated OSInfo storage = 32;
 }
 ```
 * Content-Type
-`video/MP2T`
+`video/MP2T` or `application/vnd+livepeer.uri`
 
-The composition of the body (and certain headers) varies based on the content-type. For the content-type of `video/MP2T` , the body is composed of the bytes of the segment.
+The composition of the body (and certain headers) varies based on the content-type. For the content-type of `video/MP2T` , the body is composed of the bytes of the segment. For the content-type of `application/vnd+livepeer.uri`, the body holds a URI where the data can be downloaded from.
 
 Processing a `/segment` request consists of the following steps:
 
-1. Verify the credentials generated by the orchestrator
-2. Verify that the job hasn't expired and is still claimable.
-3. Verify the segment signature from the broadcaster.
-4. Download the body
-5. Verify the keccak256 hash of the body matches `SegData.hash`
-6. Return 200 OK header. If any of the above steps fail, a non-200 response is returned.
-7. Transcoder performs additional checks and transcodes the segment.
-8. Return a `TranscodeResult` body based on the results of the transcode.
+1. Verify the segment signature from the broadcaster.
+1. Download the body
+1. Verify the keccak256 hash of the body matches `SegData.hash`
+1. Return 200 OK header. If any of the above steps fail, a non-200 response is returned.
+1. Transcoder performs additional checks and transcodes the segment.
+1. Return a `TranscodeResult` body based on the results of the transcode.
 
 #### Response:
 
@@ -174,3 +151,22 @@ Self-signed, with the DNSName field is set to the host name as specified in the
 Orchestrator/transcoder certificates are self-signed, and generated anew each time the node starts up. The current TLS implementation in the broadcaster will fail out if the DNSName field does not match, otherwise the self-signed certificate is not verified.
 
 IPs will also work in the DNS Name field (at least, the go client does not fail out). However, this may be problematic for orchestrators that are on unstable IPs or otherwise "move around". Arguably, orchestrators shouldn't move around, so perhaps this would serve to discourage that mode of operation.
+
+## Design Considerations
+
+### gRPC and HTTP
+
+The division of the protocol into gRPC and plain-HTTP parts may seem odd. There
+is a method to the madness: gRPC messages are encoded using Protocol Buffers,
+and Protocol Buffers was not designed to handle [large blobs of data](https://developers.google.com/protocol-buffers/docs/techniques#large-data).
+Hence, when we need to transmit a large blob (such as a video segment), the
+transmission is done through raw HTTP. For purposes other than sending large
+blobs, gRPC gives us a convenient framework for building network protocols.
+
+### Upgrade Path
+
+See the [official recommendations](https://developers.google.com/protocol-buffers/docs/proto#updating)
+for updating Protocol Buffers messages. The same principles of "add but don't
+modify or remove fields" carry over to gRPC service definitions: new services
+can be added, but names should not be changed, nor should services be removed
+unless the intent is to break backwards compatibility.