RTP/UDP vs RTSP for h.264 video?

Ask question on using Emgu CV with C#
Post Reply
Posts: 1
Joined: Wed Jan 15, 2020 6:49 am

RTP/UDP vs RTSP for h.264 video?

Post by xor31four » Wed Jan 15, 2020 6:52 am

Apologies for my inexperience in this domain..I am trying to implement an algorithm that **detects the occurrence of a particular event in real-time**. The particular event is a consecutive growth of motion across 5 consecutive frames.. almost analogous to a growing sphere or beach ball.

I am able to detect the event on pre-recorded video that is in .avi format (mjpeg frames) with EmguCV (C# wrapper for OpenCV). The method I use is based on **background subtraction**.. outlined here https://www.pyimagesearch.com/2015/05/2 ... nd-opencv/

The problem is that the live video **transport stream** is usually in the format **rtsp://XXX.XXX.X.XX/stream1.sdp**

***EmguCV on windows can't decode this h.264 stream*** for some reason that I am still trying to figure out ... I tried the same url using Python and OpenCV and received a ***non-matching transport in server reply*** message similar to this one https://stackoverflow.com/questions/496 ... vif-camera - the answer didn't work for me.

I can open the rtsp URL using VLCPlayer and its corresponding C# library - from my understanding it is using ffmpeg, although I may be wrong. ***FFmpeg on the command line can access the stream***.

**EmguCV also uses ffmpeg as a backend** which is why I am very confused as to why it can't open the rtsp URL.

Here is an image of the module tree when VLCPlayer opens the rtsp stream: [![enter image description here][1]][1]. From my understanding, EmguCV doesn't use live555 or avcodec..

I've noticed that if I change the streamer configuration to use **UDP or RTP** rather than RTSP, EmguCV **can** access the h.264 URL, although the URL is now in the format **rtp/udp://XXX.XXX.X.XX:XXXXX** - *no .sdp extension*.

I would highly appreciate if someone with more experience can give me some pointers.
I have a great deal to learn even though I have spent a lot of time researching this topic. In regards to the detections remaining successful would it be recommended to process H.264 frames with possible distortion or MJPEG frames?

*I can't afford a delay longer than 1-2 seconds*, and would ideally like to continue with the current method used to detect the event.

From my current understanding, here are the routes I can take:

1) Use **RTP/UDP and process h.264 video** using EmguCV - there is some *distortion* in the video when there is a large amount of movement.. I also receive several h264 error messages during the stream

[h264 @ 00000124f13a5080] SPS unavailable in decode_picture_timing
[h264 @ 00000124f13a5080] non-existing PPS 0 referenced
[h264 @ 00000124f13a5080] decode_slice_header error
[h264 @ 00000124f13a5080] no frame!
[h264 @ 00000124f135eac0] Missing reference picture, default is 0
[h264 @ 00000124f135eac0] decode_slice_header error
[h264 @ 00000124f13a5080] cbp too large (6929) at 11 20
[h264 @ 00000124f13a5080] error while decoding MB 11 20
[h264 @ 00000124f135eac0] top block unavailable for requested intra mode -1
[h264 @ 00000124f135eac0] error while decoding MB 3 0
[h264 @ 00000124f124e580] cbp too large (96) at 33 0
[h264 @ 00000124f124e580] error while decoding MB 33 0
[h264 @ 00000124f19940c0] top block unavailable for requested intra mode
[h264 @ 00000124f19940c0] error while decoding MB 1 1

[![H.264 video][2]][2]

2) **Keep RTSP** protocol, use **libav to decode the frames and pass to EmguCV**.. following this answer https://www.raspberrypi.org/forums/view ... hp?t=83127 .. I'm not sure if this will introduce a huge delay

3) **Keep RTSP** protocol, use **ffmpeg to convert h.264 stream to MJPEG and access this URL** instead?
Again I'm not sure if this will be a feasible solution if it will introduce a great delay.

4) Use a Linux machine rather than windows and configure a gstreamer backend - not ideal

Thank you for taking the time to read this post.

[1]: https://i.stack.imgur.com/IbbWo.png
[2]: https://i.stack.imgur.com/cmHj8.png

Post Reply