forked from robotology/yarp
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathyarp_cluster.dox
231 lines (185 loc) · 9.2 KB
/
yarp_cluster.dox
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
/**
\page yarp_cluster Setting up your network for YARP
\tableofcontents
Along with TCP, YARP supports UDP and multicast communication on a
local network. These are unreliable network protocols, but can be
quite efficient. We discuss here steps that are useful for making your
network friendly to UDP and multicast. But if you find yourself
spending a lot of time on this, you may need to reconsider whether you
have made the right choice of protocol.
NOTE: if you are just getting started with YARP, you
don't need to look at any of this yet! It only matters when you
are working with a cluster of computers using unreliable protocols,
or run into bandwidth problems. But it is important to remember that
this page is here for when you need it.
\section yarp_cluster_iperf iperf is your friend
If you are doing any non-trivial networking, you need to learn how
to test your network and debug throughput issues. A key resource
in this battle is the "iperf" program.
- Download from http://software.es.net/iperf/
- Linux: apt-get install iperf (or similar command)
- macOS: brew install iperf (if you use homebrew)
Please make sure you have iperf available on all your machines
before going any further.
\section yarp_cluster_tcp Checking TCP communication
TCP is the simplest protocol to work with, so let's start with it.
Let's check the throughput between two of your machines.
Pick two machines, and choose one (arbitrarily) to be the iperf
server. Determine its IP address of domain name -- let's say it is
10.0.0.26. Run the following commands:
\verbatim
[On server, e.g. 10.0.0.26] iperf -s
[On client, e.g. 10.0.0.27] iperf -c 10.0.0.26
\endverbatim
This will run for a while and then describe your TCP throughput, for example:
\verbatim
------------------------------------------------------------
Client connecting to 10.0.0.26, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 3] local 10.0.0.25 port 34160 connected with 10.0.0.26 port 5001
[ 3] 0.0-10.0 sec 1.08 GBytes 924 Mbits/sec
...
\endverbatim
\section yarp_cluster_udp Checking UDP communication
UDP is an unreliable protocol, and there are lots of possible pitfalls
in using it. YARP will try to send messages broken into rather large
UDP packets (around 60K), and your network may or may not actually
pass those packets through.
To test UDP behavior on your network with iperf, add the "-u" flag to
the client and server. To the server, we can add "-i 1" to get reports
at regular intervals. To the client, we can specify target bandwidth,
e.g. "-b 50m" for a target of 50 MBits/sec.
\verbatim
[On server, 10.0.0.26] iperf -s -u -i 1
[On client, 10.0.0.27] iperf -c 10.0.0.26 -b 50m
\endverbatim
To test a gigabit network:
\verbatim
[On server, 10.0.0.26] iperf -s -u -i 1
[On client, 10.0.0.27] iperf -c 10.0.0.26 -b 1000m
\endverbatim
On a healthy gigabit network you should get a bandwidth of ~950 Mbits/sec.
Example result on the client side:
\verbatim
------------------------------------------------------------
Client connecting to 10.0.0.26, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 122 KByte (default)
------------------------------------------------------------
[ 3] local 10.0.0.221 port 52990 connected with 10.0.0.222 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 6.1 sec 700 MBytes 956 Mbits/sec
[ 3] Sent 499273 datagrams
[ 3] Server Report:
[ 3] 0.0- 6.1 sec 699 MBytes 955 Mbits/sec 0.014 ms 575/499272 (0.12%)
[ 3] 0.0- 6.1 sec 1 datagrams received out-of-order
\endverbatim
Result on the server side:
\verbatim
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 122 KByte (default)
------------------------------------------------------------
[ 3] local 10.0.0.26 port 5001 connected with 10.0.0.27 port 52990
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 3] 0.0- 1.0 sec 114 MBytes 954 Mbits/sec 0.012 ms 88/81245 (0.11%)
[ 3] 1.0- 2.0 sec 114 MBytes 955 Mbits/sec 0.011 ms 92/81263 (0.11%)
[ 3] 2.0- 3.0 sec 114 MBytes 955 Mbits/sec 0.014 ms 90/81287 (0.11%)
[ 3] 3.0- 4.0 sec 114 MBytes 955 Mbits/sec 0.012 ms 98/81273 (0.12%)
[ 3] 4.0- 5.0 sec 114 MBytes 954 Mbits/sec 0.012 ms 99/81263 (0.12%)
[ 3] 5.0- 6.0 sec 114 MBytes 955 Mbits/sec 0.012 ms 95/81288 (0.12%)
[ 3] 0.0- 6.1 sec 699 MBytes 955 Mbits/sec 0.014 ms 575/499272 (0.12%)
[ 3] 0.0- 6.1 sec 1 datagrams received out-of-order
\endverbatim
A key test for YARP is that you can send big datagrams okay. This is
necessary for efficient transmission of large images using UDP or
MCAST. To check 63KB datagrams, do:
\verbatim
[On server, 10.0.0.26] iperf -s -u -l 63K -w 200K -i 1
[On client, 10.0.0.27] iperf -c 10.0.0.26 -u -l 63K -w 200K -b 1000m
\endverbatim
Make sure you try this in both directions! For example, we've seen an
asymmetric problems going from a Linux machine to a Windows machine on
a gigabit network with a particular switch. If you see problems,
reduce the "-l" size until things work just to understand your system
(but if you want to send large images you'll need to tweak your system
until large packets work).
It could also be worth checking several machines since different network cards
can have different performance.
\section yarp_cluster_mcast Checking multicast communication
For testing multicast, make sure you have the most recent version
of iperf, and try this:
\verbatim
iperf -s -u -B 224.0.55.55 -i 1 -T 32 -l 60k
iperf -c 224.0.55.55 -u -T 32 -t 100 -i 1 -l 60k -b 1000m
\endverbatim
Note the IP addresses are multicast-style, they don't correspond to
individual machines.
On the client ("-c") side, you should see something like:
\verbatim
------------------------------------------------------------
Client connecting to 224.0.55.55, UDP port 5001
Sending 61440 byte datagrams
Setting multicast TTL to 32
UDP buffer size: 122 KByte (default)
------------------------------------------------------------
[ 3] local 10.0.0.221 port 60992 connected with 224.0.55.55 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 114 MBytes 958 Mbits/sec
[ 3] 1.0- 2.0 sec 114 MBytes 956 Mbits/sec
[ 3] 2.0- 3.0 sec 114 MBytes 956 Mbits/sec
[ 3] 3.0- 4.0 sec 114 MBytes 956 Mbits/sec
\endverbatim
Server side:
\verbatim
------------------------------------------------------------
Server listening on UDP port 5001
Binding to local address 224.0.55.55
Joining multicast group 224.0.55.55
Receiving 61440 byte datagrams
UDP buffer size: 122 KByte (default)
------------------------------------------------------------
[ 3] local 224.0.55.55 port 5001 connected with 10.0.0.221 port 60992
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 3] 0.0- 1.0 sec 111 MBytes 931 Mbits/sec 0.380 ms 50/ 1945 (2.6%)
[ 3] 1.0- 2.0 sec 111 MBytes 934 Mbits/sec 0.442 ms 46/ 1947 (2.4%)
[ 3] 2.0- 3.0 sec 112 MBytes 941 Mbits/sec 0.280 ms 31/ 1946 (1.6%)
[ 3] 3.0- 4.0 sec 112 MBytes 935 Mbits/sec 0.362 ms 43/ 1946 (2.2%)
[ 3] 4.0- 5.0 sec 110 MBytes 920 Mbits/sec 0.459 ms 75/ 1947 (3.9%)
[ 3] 5.0- 6.0 sec 111 MBytes 934 Mbits/sec 0.418 ms 45/ 1946 (2.3%)
[ 3] 6.0- 7.0 sec 111 MBytes 933 Mbits/sec 0.380 ms 48/ 1947 (2.5%)
[ 3] 7.0- 8.0 sec 111 MBytes 933 Mbits/sec 0.365 ms 49/ 1947 (2.5%)
[ 3] 8.0- 9.0 sec 112 MBytes 936 Mbits/sec 0.319 ms 41/ 1946 (2.1%)
\endverbatim
\section yarp_cluster_suggest YARP's suggestion
If MCAST/UDP packets drop when using YARP, YARP may suggest this:
\verbatim
The UDP/MCAST system buffer limit on your system is low.
You may get packet loss under heavy conditions.
To change the buffer limit on linux: sysctl -w net.core.rmem_max=8388608
(Might be something like: sudo /sbin/sysctl -w net.core.rmem_max=8388608)
To change the limit use: sysctl for Linux/FreeBSD, ndd for Solaris, no for AIX
\endverbatim
This controls the maximum allowed size for a RECEIVER buffer, so that
a number of packets adding up to a large amount of data can be
received and stored while your program is doing something else. YARP
will try to get a large buffer, and your operating system may not
comply unless you do the above. This configuration step is not
necessary on MS-Windows.
There are a lot more suggestions for Linux here:
- http://wwwx.cs.unc.edu/~sparkst/howto/network_tuning.php
\section yarp_cluster_tweak YARP connection problems
If you are confident of your network, but are having trouble with
making a YARP connection of some kind, read about that kind of
connection in \ref carrier_config and see if there's a tweak that
can be made. There are some parameters you can control for
the \ref carrier_config_udp and \ref carrier_config_mcast.
\section yarp_cluster_code Relevant source code
In case you need to fix something, the relevant source code in YARP is:
- `src/libYARP_os/src/yarp/os/impl/DgramTwoWayStream.h`
- `src/libYARP_os/src/yarp/os/impl/DgramTwoWayStream.cpp`
We appreciate patches to these files. We only have a few networks to test
on, and this topic can be quite network-dependent and subtle.
*/