Skip to content

Golem Network

Jump to bottom
badb edited this page Dec 8, 2017 · 17 revisions

Warning: many parts may be deprecated

Golem Network

Golem is using TCP/IP. For making and keeping connections it uses twisted.internet.

To connect with the network seed address and port is required. Seed data are placed in app_cfg.ini configuration file and can be changed by editing this file or by editing properties in Golem GUI. New instance of Golem starts with a set of preconfigured, hardcoded seeds. Next time it tries to connect to the list of already known nodes.

Network topology should be based on Kademlia-like protocol (there is a beginning but it should be modified and optimized to be more like RLPx or Coral). This protocol should allow to find specific node in the network and find new nodes. Nodes should be identified by it's public key (see: Authentication).

Limits and settings

  • TCP connection timeout: 5 seconds
  • One data package limit - 1 Mb
  • Endianess: binary or string - no need to change order. One potential problem - the number that proceed the data with message id or data length. For this number we should use network format (big-endian)

Protocols

Basic messages (shared between P2P, Task and Resource):

  • MessageHello - introduction message sends at the beginning of each session.
  • MessagePing - ping other node (to check if it's still active).
  • MessagePong - answer to ping.
  • MessageDisconnect - ask for disconnection and give a reason.
  • MessageRandVal - send back signed random value that was added to hello message.
  • MessageChallengeSolution - send back signed solution to challenge that was added to hello message.

Starting connection when node2 doesn't have enough connections:

  1. node1 --MessageHello --> node2
  2. node2 verifies it's key by sending signed random value:
    node2 --MessageRandVal--> node1
    or by sending signed challenge solution:
    node2 --MessageChallengeSolution--> node1
  3. node2 -- MessageHello --> node1 (node1 does steps 1-2, but doesn't do step 3).

Starting connection when node2 has enough connections:

  1. node1 --MessageHello --> node2
  2. Send information about node2 closest neighboursh: node2 --MessagePeers --> node1
  3. node2 --> MessageDisconnect --> node1

Network syncing:

  1. node1 --MessagePing--> node2
  2. node2 --MessagePong--> node1 (inform P2PServer that node2 is still active).

P2P network specific messages

P2P messages:

  • MessageGetPeers - ask for known peers.
  • MessagePeers - send information about other peers.
  • MessageFindNode - ask about peers that are close to given id.

Getting new random peers:

  1. node1 --MessageGetPeers--> node2
  2. node2 --MessagePeers --> node1

Getting peers closest to the given address:

  1. node1 --MessageFindNode --> node2
  2. node2 --MessagePeers --> node1

Task messages:

  • MessageGetTasks - ask for known tasks.
  • MessageTasks - send task headers.
  • MessageRemoveTask - remove task from known task headers.

Getting new tasks:

  1. node1 -- MessageGetTasks --> node2
  2. node2 -- MessageTasks --> node1

Nat traversal messages (for task session):

  • MessageWantToStartTaskSession - ask this node to start task session with given node.
  • MessageSetTaskSession - pass information about node that want to start task session with another node.
  • MessageNatHole - inform node that there may be speciallyl prepared nat hole that may allow him to start task session connection with another node.
  • MessageNatTraverseFailure - inform node that prepared nat hole that other node wasn't able to use this.
  • MessageInformAboutNatTraverseFailure - pass information about nat traverse failure to specific node.

If node1 cannot start task connection with node2 but has P2P connection with him:

  1. node1 --MessageWantToStartTaskSession--> node2
  2. node2 starts task network connection with node1.
  3. On task network after hello messages: node2--MessageStartSessionResponse--> node1

If node1 cannot start task connection with node2 and doesn't have P2P connection with him:

  1. node1 --MessageSetTaskSession --> node3
  2. node3 --MessageSetTaskSession --> to all his neighbours
  3. nodex --MessageWantToStartTaskSession --> node2
  4. On task network after hello messages: node2--MessageStartSessionResponse--> node1

If node1 want connection with node2 but node2 cannot start connection with node1 either and his NAT service is not probably supporting task punching: start middleman procedure (see task network).

If node1 want connection with node2 but node2 cannot start connection with node1 either and his NAT service is probably supporting task punching: start nat punching procedure.

  1. node2 start task connection with node3 (or other node with public address).
  2. On task network: node2 --MessageNatPunch --> node3
  3. On task network: node3 --MessageWaitForNatTraverse --> node2
  4. node2 disconnects on task network with node3 and opens a listening socket on port used to connect with node3.
  5. node3 --MessageNatHole --> node1
  6. node1 opens task network connection with node2 using nat hole.

If node1 cannot use nat hole:

  1. node1 --MessageInformAboutNatTraverseFailure --> node3
  2. node3 --MessageInformAboutNatTraverseFailure --> all his neighbours
  3. nodex --MessageNatTraverseFailure--> node2.
  4. node2 starts middleman procedure (see task network)

Resource messages:

  • MessageGetResourcePeers - ask about known resource network nodes.
  • MessageResourcePeers - send list of known resource network nodes.

Getting resource peeers:

  1. node1 --MessageGetResourcePeers --> node2
  2. node2 --MessageResourcePeers --> node1

Reputation messages:

  • MessageDegree - send node degree in p2p network (useful for differential gossip protocol).
  • MessageGossip - send gossip vector to chosen nodes.
  • MessageStopGossip - inform other nodes that gossip vector has converged.
  • MessageLocRank - send local rank change to neighbours (after changing your local rank).

Task network specific messages

Task negotiation:

  • MessageWantToComputeTask - volunteer to compute task.
  • MessageCannotAssignTask - refuse to assign task to this node.
  • MessageTaskToCompute - send ComputeTaskDef to node when he was chosen for computation.

Refuse to assign task:

  1. node1 --MessageWantToComputeTask--> node2
  2. node2 --MessageCannotAssignTask --> node1

Assigning a task:

  1. node1 --MessageWantToComputeTask --> node2
  2. node2 --MessageTaskToCompute --> node1

Sending task resources:

  • MessageGetResource - ask requesting node for resources to assigned task.
  • MessageResourceFormat - send information about resource format (sending files with producer or zipped file in one message).
  • MessageAcceptResourceFormat - accept proposed resource format and allow requester nodes to start sending.
  • MessageResource - old method for sending resource through task network.
  • MessagetDeltaParts - new method for sending information about resources that need to be collected through resource network.

Getting resources for the assigned task through task network (old method):

  1. node1 --MessageGetResources --> node2
  2. node2 --MessageResourceFormat--> node1
  3. node1 --MessageAcceptResourceFormat--> node2
  4. node2 sends resource file using FileProducer

Getting resoures for the assigned task through resource network (new method):

  1. node1 --MessageGetResources --> node2
  2. node2 --MessageResourceFormat--> node1
  3. node1 --MessageAcceptResourceFormat--> node2
  4. node2 --MessageDeltaParts--> node1
  5. node1 add node2 to his resource peers and start pull resource operation on resource network.

Returning results:

  • MessageReportComputedTask - inform requester about finished computation.
  • MessageGetTaskResult - ask computing node to send results (with or without possible delay).
  • MessageTaskFailure - inform requester about task computation failure.
  • MessageSubtaskResultAccepted - inform computing node that task result pass verification.
  • MessageSubtaskResultRejected - inform computing node that task result doesn't pass verification.
  • MessageTaskResult - old message, not used anymore computing node is sending task results inside this message.

Send task results (without delay):

  1. node1 --MessageReportComputedTask--> node2
  2. node2 --MessageGetTaskResult--> node1
  3. node1 sends files or data using FileProducer or DataProducer

Send task results (with delay):

  1. node1 --MessageReportComputedTask--> node2
  2. node2 --MessageGetTaskResult (delay > 0) --> node1
  3. drop connection, node1 wait for delay seconds and starts send task results again.

Nat traversal messages:

  • MessageStartSessionResponse - inform node that asked for task session with this node that it has been started and he can send his request.
  • MessageNatPunch - if a node1 can't start connection with node2 he send this message to node3 and asks him for help with nat punch.
  • MessageWaitForNatTraverse - node3 send this message to node1 to inform him that he establish connection with node2 and node1 should end this connection and start listening on the same port.
  • MessageNatPunchFailure - inform node1 that node2 was not able to establish connection through nat hole.

See messages description in P2P nat traverse method.

Middleman messages:

  • MessageMiddleman - ask other node to became middleman, keep two task sessions with nodes that want to communicate with each other and pass messages between them.
  • MessageBeingMiddlemanAccepted - inform node that this node accepted being middleman.
  • MessageJoinMiddlemanConn - ask node communicate with other through middleman connection (sending node is the middleman and connection with other node is already opened).
  • MessageMiddlemanAccepted - accept join middleman proposal.
  • MessageMiddlemanReady - inform node that send MessageMiddleman that connection with other side has been established.

Middleman connection steps:

  1. node1 --> X cannot connect node2
  2. node1 -- MessageMiddleman --> node3
  3. node3 -- MessageBeingMiddlemanAccepted --> node1
  4. node3 -- MessageJoinMiddlemanConn --> node2
  5. node2 -- MessageMiddlemanAccepted --> node3
  6. node3 -- MessageMiddlemanReady --> node1
  7. node1 -- message encrypted with node2 key --> node3 -- message encrypted with node2 key --> node2

Resource network specific messages

  • MessagePushResource - inform that expected number of copies of resource should be pushed to the network.
  • MessageHasResource - answer that this node already has this resource.
  • MessageWantResource - ask node to send specific resource.
  • MessagePullResource - ask node if it has specific resource.
  • MessagePullAnswer - inform other node whether this node has or has not specific node.
  • MessageSendResource - old message, not used anymore ask node for a specific resource.

Pushing new file:

  1. node1 --MessagePushResource --> node2
  2. node2 --MessageWantResource --> node1
  3. node1 -- send file using FileProducer --> node2
  4. node2 --MessageHasResource --> node1
  5. If copies > 1: node2 --MessagePushResource --> node3 (copies-1)

Pushing old file:

  1. node1 --MessagePushResource --> node2
  2. node2 --MessageHasResource --> node1
  3. If copies > 1: node2 --MessagePushResource --> node3 (copies-1)

Pulling file (node2 has it):

  1. node1 --MessagePullResource --> node2
  2. node2 --MessageAnswerPullResource(yes) --> node1
  3. node2 --send file using FileProdure --> node1

Pulling file (node2 doesn't have it ):

  1. node1 --MessagePullResource --> node2
  2. node2 --MessageAnswerPullResource(no) --> node1
  3. node2 collects information about new resource peers that he can share with node1

Incoming pyaments

  1. MessageSubtaskPaymentRequest --> sent by worker waiting for payment for computed subtask
  2. MessageSubtaskPayment --> sent by requestor

See class docstrings for additional information.

Switching to pydevp2p

Basic Messages are gone:

  • MessageHello - gone
  • MessagePing - gone
  • MessagePong - gone
  • MessageDisconnect - gone
  • MessageRandVal - gone
  • MessageChallengeSolution - gone

This is done by devp2p internally

P2P Messages are gone

  • MessageGetPeers - gone
  • MessagePeers - gone
  • MessageFindNode - gone

This is done by devp2p internally

Task messages are slightly changed

  • MessageGetTasks - is replaced
  • MessageTasks - is replaced
  • MessageRemoveTask - is replaced
  • MessageWantToComputeTask
  • MessageCannotAssignTask
  • MessageTaskToCompute

Golem still depends on direct connections to send task descriptions

Nat traversal messages

  • MessageWantToStartTaskSession
  • MessageSetTaskSession
  • MessageStartSessionResponse

Still very important. This is very basic but reliable way, if one of the nodes is behind NAT, it will simply ask p2p network to forward request to connect back. NOTE this wont work if both nodes are behind NAT, without forwarded ports or if NAT type is symmetric, which will not let STUN servers to work. In such case it may be good to research and check if other NAT traversal messages (NAT punching) are reliable and/or needed.

  • MessageNatPunch
  • MessageWaitForNatTraverse
  • MessageNatPunchFailure
  • MessageNatHole
  • MessageNatTraverseFailure
  • MessageInformAboutNatTraverseFailure

Resource messages

  • MessagePushResource

  • MessageHasResource

  • MessageWantResource

  • MessagePullResource

  • MessagePullAnswer

  • MessageSendResource

  • MessageGetResource

  • MessageResourceFormat

  • MessageAcceptResourceFormat

  • MessageResource

  • MessagetDeltaParts

  • MessageGetResourcePeers

  • MessageResourcePeers

After switch to hyperg, they are probably not needed but someone else should confirm this.

Reputation messages

  • MessageDegree
  • MessageGossip
  • MessageStopGossip
  • MessageLocRank

Are disabled for now, need to revisit this

Returning task result messages

  • MessageReportComputedTask
  • MessageGetTaskResult
  • MessageTaskFailure
  • MessageSubtaskResultAccepted
  • MessageSubtaskResultRejected
  • MessageTaskResult

Should be revisited by someone else, but after switch they are not changed

Middleman messages

  • MessageMiddleman
  • MessageBeingMiddlemanAccepted
  • MessageJoinMiddlemanConn
  • MessageMiddlemanAccepted
  • MessageMiddlemanReady

Not changed but its worth to revisit this. It is a good concept if it could work as something like dynamically set up TURN servers for golem network (so it could scale up). Maybe with supernodes concept it could solve all NAT issues? But who will voluntary agree on being middleman without gratification?

Incoming Payments messages

  • MessageSubtaskPaymentRequest
  • MessageSubtaskPayment

Not changed.

Clone this wiki locally