forked from robotology/yarp
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathcarrier_h264_howto.dox
185 lines (132 loc) · 10.1 KB
/
carrier_h264_howto.dox
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
/**
\page carrier_h264_howto h264 carrier
\tableofcontents
If your robot has cameras with high resolution, you need to compress and to stream their images.
In order to achieve this, we use Gstreamer, a free framework for media applications.
This document contains a brief introduction to Gstreamer tool and explains how the high-resolution cameras stream is encoding and how a yarp application can read it.
\note They are still work in progress and should be considered experimental. Please report any problems.
\section gstreamer_introduction Gstreamer: brief introduction
Gstreamer is a free framework for media applications; it provides a set of plugins that let the user to build applications by connecting them as in a pipeline. It has been ported to a wide range of operating systems, processors and compilers. Also Nvidia developed plugins for its platforms and we employ its h264 encode in order to take advantage from its hardware codec.
A Gstreamer application is composed by a chain of elements, the base construction block of a Gstreamer application. An element takes an input stream from previous element in the chain, carries out its function, like encode, and passes the modified stream to the next element. Usually each element is a plugin.
The user can develop application in two way: the first consists in write an application in c/c++, where the elements are connected using API, while the second uses the gst-launch command-line tool. In the following an example of how to use gst-launch command:
\verbatim
gst-launch-1.0 -v videotestsrc ! ‘video/x-raw, format=(string)I420, width=(int)640, height=(int)480’ ! x264enc ! h264parse ! avdec_h264 ! autovideosink \endverbatim
This command creates a source video test with the properties specified in this string <em> “video/x-raw, format=(string)I420, width=(int)640, height=(int)480”</em>; after it is encoded in h264, then decoded and shown. Each element of this pipeline, except the property element, is plugins dynamically loaded. The videotestsrc element lets the user to see a stream without using camera.
The previous command works on Linux, but since Gstreamer is platform independent, we can launch the same command on Windows taking care to change only hardware dependent plugin. So the same command on Window is:
\verbatim
gst-launch-1.0 -v videotestsrc ! “video/x-raw, format=(string)I420, width=(int)640, height=(int)480” !
openh264enc ! h264parse ! avdec_h264 ! autovideosink
\endverbatim
It’s important to notice that the changed element is the encoder (openh264enc), while the decoder is the same. This because the decoder belongs to the plugin that wraps libav library, a cross-platform library to convert stream in a wide range of multimedia formats. [see \ref references chapter]
<em> Please see \ref notes section about commands in this tutorial. </em>
\section how_to_stream_h264 How to stream in h264
The server grabs images from cameras, so it needs to run on where cameras are connected.
The server is a Gstreamer command pipeline, while the client could be a yarp or a Gstreamer application connected to the robot’s network.
Since Gstreamer is wide spread framework for media application, wrap it into yarp it is not interesting and is worthless.
Instead is more intriguing that a yarp application can read “standard” streams using the h264 carrier.
For these reasons, the server streamer is a native Gstreamer pipeline, while the client side has been developed in yarp like a carrier.
\subsection server_side Server side:
The server application consists in the following Gstreamer command:
\verbatim
gst-launch-1.0 -v v4l2src device="/dev/video1" ! ‘video/x-raw, width=1280, height=480, format=(string)I420’ !
omxh264enc ! h264parse ! rtph264pay pt=96 config-interval=5 ! udpsink host=224.0.0.1 auto-multicast=true port=33000
\endverbatim
\li <em>v4l2src device="/dev/video1"</em>: reads the cameras using v4l (video for linux) driver
\li <em>‘video/x-raw, width=640, height=480, format=(string)I420’ </em>: this is a property-element that specifies image size and format. In order to know which sizes and format are available you can use “v4l2-ctl” utility.
\li <em>omxh264enc</em>: encodes in h264: plugin developed by Nvidia
\li <em>h264parse</em>: signals downstream the format of the stream
\li <em>rtph264pay</em>: puts h264 stream in rtp packets
\li <em>udpsink</em>: this is the last element and sends out the stream. In this case we use multicast, but it is possible to send the stream using unicast in this way: udpsink host=IP_ADDRESS_OF_CLIENT port=NOT_WELL_KNOWN_PORT_NUMBER
Currently is not available a Gstreamer application that implements the pipeline. In the near future, it could be developed in order to make easier configure the server.
\subsection client_side Client side
The client can read the stream using Gstreamer native command or yarp.
In the first case the Gstreamer command is:
\verbatim
gst-launch-1.0 -v udpsrc multicast-group=224.0.0.1 auto-multicast=true port=3000 caps="application/x-rtp, media=(string)video, encoding-name=(string)H264, payload=(int)96" !
rtph264depay ! h264parse ! avdec_h264! autovideosink
\endverbatim
If you want use a yarp application to read the stream you need to:
\verbatim
1) install Gstreamer (see \ref how_to_install_gstreamer )
2) compile yarp with “ENABLE_yarpcar_h264” option enabled.
3) register the stream server to the yarp server: “yarp name register <SERVER_NAME_PORT> h264 <SERVER_IP_ADRESS> <SERVER_IP_PORT>”
4) run your application
5) connect client and server port using h264 carrier: “yarp connect <SERVER_NAME_PORT> <CLIENT_PORT> h264”
\endverbatim
\subsection some_options Some options
\li <b>set the frame rate </b>: in server side exist the parameter framerate=30/1, that configures the framerate to which grab images. Insert in in property element: 'video/x-raw, format=(string)I420, width=(int)640, height=(int)480, framerate=30/1'
\li In some cases could be useful that server streams video at <b>constant rate</b>. You can achieve this adding this parameter to the encoder plugin: control-rate=2 bitrate=5000000
\li <b>remove the jitter</b>: in the client it is possible adding the rtpjitterbuffer plugin in this way:
If you want to remove the jitter in h264 yarp carrier, please add parameter “+removeJitter.1” in connect command. (Note that the syntax is the usual used to specify parameters to yarp carriers). Therefore the connect command to your application could be:
\verbatim
yarp connect <SERVER_NAME_PORT> <CLIENT_PORT> h264+removeJitter.1
\endverbatim
\li The yarp carrier lets you to \b crop each frame specifying the number of pixel to crop on each side of image in carrier parameter in connection command. For example if you want to crop 200 pixel on top the command appears like:
\verbatimyarp connect <SERVER_NAME_PORT> <CLIENT_PORT> h264+cropTop.200.\endverbatim
Instead, if you want to use native Gstreamer client the plugin “videocrop” performs this operation.
\verbatim
gst-launch-1.0 -v udpsrc port=33000 caps="application/x-rtp, media=(string)video, encoding-name=(string)H264, payload=(int)96" !
rtph264depay ! h264parse ! avdec_h264 ! videocrop left=10, right=30, top=50, bottom=50 ! autovideosink
\endverbatim
\section how_to_install_gstreamer How to install Gstreamer
Currently we are using 1.8.3 version
\subsection ubuntu On Ubuntu
\li Packages required to build
- libgstreamer1.0-dev
- libgstreamer-plugins-base1.0-dev
\li Packages required to run
- gstreamer1.0-plugins-base (for videoconvert and appsink)
- gstreamer1.0-plugins-good (for udpsrc and rtph264depay)
- gstreamer1.0-plugins-bad (for h264parse)
- gstreamer1.0-libav (for avdec_h264)
\li Useful packages but not required
- gstreamer1.0-tools
\subsection windows On windows
You need to download both the main package and the devel package from here:
https://gstreamer.freedesktop.org/data/pkg/windows/
Installation of package Gstreamer:
\li choose typical installation
\li redo installation adding:
- Gstreamer 1.0 devtools
- Gstreamer 1.0 Libav wrapper
Installation of grstreamer devel package:
\li choose typical installation
\li redo installation adding:
- Gstreamer 1.0 devtools
- Gstreamer 1.0 Libav wrapper
\li Add in path environment variable the path to executable (Usually is C:\\gstreamer\\1.0\\x86_64\\bin)
\subsection check_installation Verify your installation
First of all, you need to verify if Gstreamer has been installed successfully by using these commands:
\subsubsection server Server side (example on Windows)
\verbatim
gst-launch-1.0 -v videotestsrc ! "video/x-raw, format=(string)I420, width=(int)640, height=(int)480" !
openh264enc ! h264parse ! rtph264pay pt=96 config-interval=5 ! udpsink host=<YOUR_IP_ADDRESS> port=<A_PORT_NUMBER>
\endverbatim
\subsubsection client Client side
\verbatim
gst-launch-1.0 -v udpsrc port=<A_PORT_NUMBER> caps="application/x-rtp, media=(string)video, encoding-name=(string)H264, payload=(int)96" !
rtph264depay ! h264parse ! avdec_h264! autovideosink
\endverbatim
After you can substitute the client side with a yarp application, for example yarpview.
So, after yarp server had been launched, please run:
\li yarpview
\li yarp name register /gst h264 <YOUR_IP_ADDRESS> <A_PORT_NUMBER>
\li yarp connect /gst /yarpview/img:i h264
Now on yarp view you can see the following image, where in the bottom right box there is snow pattern.
\image html h264GstVideoTestSrc.png ""
\section notes Notes
\subsection tricks Tricks
\li On Ubuntu 18.04 the plugin “autovideosink” has a bug, please use “xvimagesink”
\li On Windows the property element uses ‘ instead of “
\section references References
[1] Gstreamer documentation
https://gstreamer.freedesktop.org/documentation/
[2] Gstreamer plugins
https://gstreamer.freedesktop.org/documentation/plugins.html
[3] Libav library documentation
https://www.libav.org/index.html
[4] Gstreamer Libav plugin
https://gstreamer.freedesktop.org/modules/gst-libav.html
[5] h264 yarp plugin
http://www.yarp.it/classyarp_1_1os_1_1H264Carrier.html
*/