Jetson Linux Multimedia API Reference

32.4.3 Release

 All Data Structures Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Groups Pages
multimedia_api/ll_samples/docs/l4t_mm_backend.md
Go to the documentation of this file.
1 Copyright (c) 2016-2017, NVIDIA CORPORATION. All rights reserved.
2 <!-- Sample is in the backend dir. -->
3 
4 @page nvvid_backend_group Backend
5 @{
6 
7  - [Overview](#overview)
8  - [Building and Running](#build_and_run)
9  - [Flow](#flow)
10  - [Command Line Options](#options)
11  - [Key Structure and Classes](#key)
12 
13 - - - - - - - - - - - - - - -
14 <a name="overview">
15 ## Overview ##
16 
17 This application implements a typical appliance performing intelligent video analytics.
18 Application areas include public safety, smart cities, and autonomous machines. This example demonstrates
19 four (4) concurrent video streams going through a decoding process using the on-chip decoders, video scaling using on
20 chip scalar, and GPU compute. For simplicity of demonstration, only one of the channels uses
21 NVIDIA<sup>&reg;</sup> TensorRT<sup>&trade;</sup> to perform object identification
22 and generate bounding box around the identified object. This sample also uses video converter functions
23 for various format conversions. It also uses EGLImage to demonstrate buffer sharing and image display.
24 
25 In this sample, object detection is limited to identifying cars in video streams
26 of 960 x 540 resolution, running up to 14 FPS. The network is based on
27 GoogleNet. The inference is performed on a frame-by-frame basis and no object
28 tracking is involved. Note that this network is intended to be an example
29 that shows how to use TensorRT to quickly build the compute pipeline. The sample
30 includes trained GoogleNet, which was trained with NVIDIA Deep Learning GPU
31 Training System (DIGITS). The training was done with roughly
32 3000 frames taken from 5-10 feet elevation. Varying levels of detection accuracy
33 are expected based on the video samples fed in. Given that this sample is locked
34 to perform at half-HD resolutions under 10 FPS, video feeds with higher FPS for
35 inference will show stuttering during playback.
36 
37 This sample does not require a camera.
38 
39 <a name="build_and_run">
40 ## Building and Running ##
41 
42 #### Prerequisites ####
43 * You have followed Steps 1-3 in @ref mmapi_build.
44 * You have installed:
45  * CUDA Toolkit
46  * OpenCV
47 * Optionally, you have installed NVIDIA<sup>&reg;</sup>
48  TensorRT<sup>&trade;</sup> (previously known as GPU Inference Engine (GIE))
49 
50 ### To build
51 1. If you want to run the sample without TensorRT, set the following in the Makefile:
52 
53  ENABLETRT := 0
54  By default, TensorRT is enabled.
55 
56 2. Enter:
57 
58  $ cd backend
59  $ make
60 
61 ### To run
62 * Enter:
63 
64  $ ./backend 1 ../../data/Video/sample_outdoor_car_1080p_10fps.h264 H264 \
65  --trt-deployfile ../../data/Model/GoogleNet_one_class/GoogleNet_modified_oneClass_halfHD.prototxt \
66  --trt-modelfile ../../data/Model/GoogleNet_one_class/GoogleNet_modified_oneClass_halfHD.caffemodel \
67  --trt-mode 0 --trt-proc-interval 1 -fps 10
68 
69 
70 
71 ### To quit
72 * Enter `q`.
73 
74 ### To view command-line options
75 * Enter:
76 
77  $ cd backend
78  $ ./backend -h
79 
80 
81 <a name="flow">
82 - - - - - - - - - - - - - - -
83 ## Flow
84 
85 The following image shows the movement of data through the sample when TensorRT
86 is not enabled.
87 
88 ![ ](l4t_mm_backend_one.jpg)
89 
90 The following image shows data flow details for the channel using TensorRT.
91 
92 ![ ](l4t_mm_backend_two.jpg)
93 
94 `NvEGLImageFromFd` is an NVIDIA API that returns an `EGLImage` pointer from the file
95 descriptor buffer that is allocated via the Tegra mechanism. TensorRT then uses
96 the `EGLImage` buffer to render the bounding box to the image.
97 
98 ### X11 Details
99 For X11 technical details, see:
100 
101 http://www.x.org/docs/X11/xlib.pdf
102 
103 - - - - - - - - - - - - - - -
104 <a name="key">
105 ## Key Structure and Classes ##
106 
107 The `context_t structure` (backend/v4l2_backend_test.h) manages all resources in sample applications.
108 
109 |Element|Description|
110 |-------|-----------|
111 |[NvVideoDecoder](classNvVideoDecoder.html)|Contains all video decoding-related elements and functions.|
112 |[NvVideoConverter](classNvVideoConverter.html)|Contains elements and functions for video format conversion.|
113 |NvEglRenderer|Contains all EGL display rendering-related functions.|
114 |EGLImageKHR|The EGLImage used for CUDA processing. This type is from the EGL open source graphical library.|
115 
116 ### %NvVideoDecoder ###
117 
118 The [NvVideoDecoder](classNvVideoDecoder.html) class creates a new V4L2 Video Decoder.
119 The following table describes the key %NvVideoDecoder members that this sample uses.
120 
121 |Member|Description|
122 |-------------|---|
123 |NvV4l2Element::output_plane |Holds the V4L2 output plane.|
124 |NvV4l2Element::capture_plane |Holds the V4L2 capture plane.|
125 |NvVideoDecoder::createVideoDecoder |Static function to create video decode object.|
126 |NvV4l2Element::subscribeEvent |Subscribes event.|
127 |NvVideoDecoder::setExtControls |Sets external control to V4L2 device.|
128 |NvVideoDecoder::setOutputPlaneFormat |Sets output plane format.|
129 |NvVideoDecoder::setCapturePlaneFormat |Sets capture plane format.|
130 |NvV4l2Element::getControl |Gets the value of a control setting.|
131 |NvV4l2Element::dqEvent |Dequeues the devent reported by the V4L2 device.|
132 |NvV4l2Element::isInError |Checks if under error state.|
133 
134 ### %NvVideoConverter ###
135 
136 The NvVideoConverter class packages all video
137 converting related elements and functions. It performs color space conversion,
138 scaling and conversion between hardware buffer memory and software buffer
139 memory. The following table describes the key %NvVideoConverter members that
140 this sample uses.
141 
142 |Member|Description|
143 |-------------|---|
144 |NvV4l2Element::output_plane |Holds the output plane.|
145 |NvV4l2Element::capture_plane |Holds the capture plane.|
146 |NvVideoConverter::waitForIdle |Waits until all the buffers queued on the output plane are converted and dequeued from the capture plane. This is a blocking method.|
147 |NvVideoConverter::setCapturePlaneFormat |Sets the format on the converter capture plane.|
148 |NvVideoConverter::setOutputPlaneFormat |Sets the format on the converter output plane.|
149 
150 `NvVideoDecoder` and `NvVideoConverter` contain two key elements:
151 `output_plane` and `capture_plane`. These objects are instantiated from the
152 NvV4l2ElementPlane class type.
153 
154 ### %NvV4l2ElementPlane ###
155 
156 [NvV4l2ElementPlane](group__l4t_mm__nvv4lelementplane__group.html) creates an [NVv4l2Element](classNvV4l2Element.html) plane.
157 The following table describes the key %NvV4l2ElementPlane members used in this
158 sample. `v4l2_buf` is a local variable inside the NvV4l2ElementPlane::dqThreadCallback
159 function and, thus, the scope exists only inside the callback function. If other
160 functions of the sample must access this buffer, a prior copy of the buffer
161 inside callback function is required.
162 
163 |Member |Description|
164 |-------------------|---|
165 |NvV4l2ElementPlane::setupPlane |Sets up the plane of V4l2 element.|
166 |NvV4l2ElementPlane::deinitPlane |Destroys the plane of V4l2 element.|
167 |NvV4l2ElementPlane::setStreamStatus |Starts/Stops the stream.|
168 |NvV4l2ElementPlane::setDQThreadCallback |Sets the callback function of the `dqueue` buffer thread.|
169 |NvV4l2ElementPlane::startDQThread |Starts the thread of the `dqueue` buffer.|
170 |NvV4l2ElementPlane::stopDQThread |Stops the thread of the `dqueue` buffer.|
171 |NvV4l2ElementPlane::qBuffer |Queues a V4l2 buffer from the plane.|
172 |NvV4l2ElementPlane::dqBuffer |Dequeues a V4l2 buffer from the plane.|
173 |NvV4l2ElementPlane::getNumBuffers |Gets the number of the V4l2 buffer.|
174 |NvV4l2ElementPlane::getNumQueuedBuffers |Gets the number of the V4l2 buffer in the queue.|
175 |NvV4l2ElementPlane::getNthBuffer |Gets the \c %NvBuffer queue object at index N.|
176 
177 
178 ### %TRT_Context ###
179 
180 TRT_Context provides a
181 series of interfaces to load Caffe model and perform inference. The following
182 table describes the key %TRT_Context members used in this sample.
183 
184 |%TRT_Context|Description|
185 |-----------|-----------|
186 |TRT_Context::destroyTrtContext |Destroys the TRT_context.|
187 |TRT_Context::getNumTrtInstances |Gets the number of TRT_context instances.|
188 |TRT_Context::doInference |Interface for inference after TensorRT model is loaded.|
189 
190 ### Functions to Create/Destroy EGLImage ##
191 
192 The sample uses 2 global functions to create and destroy EGLImage from `dmabuf`
193 file descriptor. These functions are defined in nvbuf_utils.h.
194 
195 |Global Function|Description|
196 |---------------|-----------|
197 |NvEGLImageFromFd() |Creates EGLImage from dmabuf fd.|
198 |NvDestroyEGLImage() |Destroys the EGLImage.|
199 
int setupPlane(enum v4l2_memory mem_type, uint32_t num_buffers, bool map, bool allocate)
Helper method that encapsulates all the method calls required to set up the plane for streaming...
static NvVideoDecoder * createVideoDecoder(const char *name, int flags=0)
Creates a new V4L2 Video Decoder object named name.
void destroyTrtContext(bool bUseCPUBuf=false)
bool setDQThreadCallback(dqThreadCallback callback)
Sets the DQ Thread callback method.
int stopDQThread()
Force stops the DQ Thread if it is running.
void doInference(queue< vector< cv::Rect > > *rectList_queue, float *input=NULL)
int qBuffer(struct v4l2_buffer &v4l2_buf, NvBuffer *shared_buffer)
Queues a buffer on the plane.
Defines a helper class for V4L2 Video Decoder.
int dqEvent(struct v4l2_event &event, uint32_t max_wait_ms)
Dequeues an event from the element.
Defines a helper class for operations performed on a V4L2 Element plane.
int setCapturePlaneFormat(uint32_t pixfmt, uint32_t width, uint32_t height, enum v4l2_nv_buffer_layout type)
Sets the format on the converter output plane.
NvV4l2ElementPlane capture_plane
Sets the capture plane.
Class representing a buffer.
Definition: NvBuffer.h:85
bool(* dqThreadCallback)(struct v4l2_buffer *v4l2_buf, NvBuffer *buffer, NvBuffer *shared_buffer, void *data)
This is a callback function type method that is called by the DQ Thread when it successfully dequeues...
NvV4l2ElementPlane output_plane
Sets the output plane.
int setOutputPlaneFormat(uint32_t pixfmt, uint32_t sizeimage)
Sets the format on the decoder output plane.
virtual int isInError()
EGLImageKHR NvEGLImageFromFd(EGLDisplay display, int dmabuf_fd)
Creates an instance of EGLImage from a DMABUF FD.
int dqBuffer(struct v4l2_buffer &v4l2_buf, NvBuffer **buffer, NvBuffer **shared_buffer, uint32_t num_retries)
Dequeues a buffer from the plane.
int getControl(uint32_t id, int32_t &value)
Gets the value of a control.
int startDQThread(void *data)
Starts DQ Thread.
Defines a helper class for V4L2 Video Converter.
int setCapturePlaneFormat(uint32_t pixfmt, uint32_t width, uint32_t height)
Sets the format on the decoder output plane.
NvBuffer * getNthBuffer(uint32_t n)
Gets the NvBuffer object at index n.
uint32_t getNumQueuedBuffers()
Gets the number of buffers currently queued on the plane.
int setStreamStatus(bool status)
Starts or stops streaming on the plane.
int setExtControls(struct v4l2_ext_controls &ctl)
Sets the value of several controls.
uint32_t getNumTrtInstances() const
int waitForIdle(uint32_t max_wait_ms)
Waits until all buffers queued on the output plane are converted and dequeued from the capture plane...
NvEglRenderer is a helper class for rendering using EGL and OpenGL ES 2.0.
Definition: NvEglRenderer.h:74
int subscribeEvent(uint32_t type, uint32_t id, uint32_t flags)
Subscribes to an V4L2 event.
void deinitPlane()
Helper method that encapsulates all the method calls required to deinitialize the plane for streaming...
int setOutputPlaneFormat(uint32_t pixfmt, uint32_t width, uint32_t height, enum v4l2_nv_buffer_layout type)
Sets the format on the converter output plane.
uint32_t getNumBuffers()
Gets the number of buffers allocated/requested on the plane.
int NvDestroyEGLImage(EGLDisplay display, EGLImageKHR eglImage)
Destroys an EGLImage object.