在RK3588上使用Gstreamer做推拉流并推理记录

2023-09-08 08:37

短信预约 -IT技能 免费直播动态提醒

在RK3588上使用Gstreamer做推拉流并推理记录

基础环境

角色	版本
rknn-toolkit2	1.2.0
python	3.7.3

1. 视频流使用结构

视频流拉取：

video/rtsp/camera ! decoder ! videoconvert ! appsink

视频流推流或显示：

appclass="lazy" data-src ! display

推理结构：

appsink——rknn.inference(model)——appclass="lazy" data-src

2. 环境安装

2.1 Gstreamer安装

apt-get updateapt-get install libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-bad1.0-dev gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav gstreamer1.0-doc gstreamer1.0-tools gstreamer1.0-x gstreamer1.0-alsa gstreamer1.0-gl gstreamer1.0-gtk3 gstreamer1.0-qt5 gstreamer1.0-pulseaudioapt-get install libunwind8-devpkg-config --cflags --libs gstreamer-1.0# 查看效果dpkg -l | grep gstreamer

测试，在终端输入以下正常运行则安装成功

gst-launch-1.0 fileclass="lazy" data-src location=./test.mp4  ! decodebin ! videoconvert ! ximagesink

2.2 opencv安装

不能使用python3-opencv，通过pip安装的opencv无法支持gstreamer，必须使用源码编译安装

git clone https://github.com/opencv/opencv.gitunzip opencv.zip # 解压zip压缩包cd opencv # 切换到源码包目录mkdir build && cd build # 创建build目录并切换进去# 其中需要配置python目录以及对gstreamer支持cmake -D WITH_GSTREAMER=ON -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D BUILD_opencv_python2=OFF -D BUILD_opencv_python3=ON -D  PYTHON3_PACKAGES_PATH=/usr/local/python3.7/lib/python3.7/site-packages/ -D PYTHON3_LIBRARY=/usr/local/python3.7/lib/libpython3.7m.so -D OPENCV_GENERATE_PKGCONFIG=YES  ..

确保opencv开启对gstreamer与python支持的api。

 Video I/O:    DC1394:                      YES (2.2.5)    FFMPEG:                      YES      avcodec:                   YES (58.54.100)      avformat:                  YES (58.29.100)      avutil:                    YES (56.31.100)      swscale:                   YES (5.5.100)      avresample:                YES (4.0.0)    GStreamer:                   YES (1.16.2)   # 识别到gstreamer    v4l/v4l2:                    YES (linux/videodev2.h)

Python 3:    Interpreter:                 /usr/local/bin/python3 (ver 3.7.3)    Libraries:                   /usr/local/python3.7/lib/libpython3.7m.so (ver 3.7.3)    numpy:                       /usr/local/python3.7/lib/python3.7/site-packages/numpy/core/include (ver 1.21.6)    install path:                /usr/local/python3.7/lib/python3.7/site-packages//cv2/python-3.7

编译安装

make -j4 # 编译make install # 安装opencv库到/usr/local下cd /etc/ld.so.conf.d/ # 切换目录touch opencv.conf # 新建opencv配置文件echo /usr/local/lib/ > opencv.conf # 填写opencv编译后库所在的路径sudo ldconfig # 使配置文件生效

最后需要把cv2模块的代码目录中的cv2.so拷贝到python的site-packages目录中,替换原本python3-opencv的cv2.so。

    注：#  如果查不到opencv的版本，需要将opencv4.pc 重命名为opencv.pc

运行以下程序正常表示安装成功

root@ztl:/# python3Python 3.7.3 (default, Jul  6 2022, 07:24:11) [GCC 9.4.0] on linuxType "help", "copyright", "credits" or "license" for more information.>>> import cv2>>> print(cv2.getBuildInformation())

3. 使用案例

拉流：

rtsp拉取：in_gst = ('rtspclass="lazy" data-src location={} latency=0 ! rtph264depay ! decodebin ! videoconvert ! video/x-raw,format=(string)BGR ! appsink  drop=true').format(url)mp4视频拉取解码：in_gst = ('fileclass="lazy" data-src location={} ! decodebin ! videoconvert ! video/x-raw,format=BGR! appsink drop=true caps=video/x-raw,format=BGR').format(url)

推流：

本地显示：out_gst = 'appclass="lazy" data-src is-live=true caps=video/x-raw,format=(string)BGR,framerate=(fraction)30/1 ! videoconvert !  ximagesink sync=false'rtmp推流：后续更新

推理：

# 视频帧数据处理cap_in = cv2.VideoCapture(in_gst, cv2.CAP_GSTREAMER)cap_out = cv2.VideoWriter(out_gst, cv2.VideoWriter_fourcc(*'H264'), fps, (640, 320), True)# 逐帧推理output = model.inference(image)# 视频流输出cap_out = cv2.write(image)