RTSP 连接过程如下图

看下实际过程中FFmpeg 的日志情况：

[tcp @ 0000014CC3256D40] No default whitelist set
[tcp @ 0000014CC3256D40] Original list of addresses:
[tcp @ 0000014CC3256D40] Address ::1 port 8554
[tcp @ 0000014CC3256D40] Address 127.0.0.1 port 8554
[tcp @ 0000014CC3256D40] Interleaved list of addresses:
[tcp @ 0000014CC3256D40] Address ::1 port 8554
[tcp @ 0000014CC3256D40] Address 127.0.0.1 port 8554
[tcp @ 0000014CC3256D40] Starting connection attempt to ::1 port 8554
[tcp @ 0000014CC3256D40] Successfully connected to ::1 port 8554
===========================================================================================
[rtsp @ 0000014CC325DA80] Sending:
OPTIONS rtsp://:8554/test RTSP/1.0
CSeq: 1
User-Agent: Lavf59.34.101
--
[rtsp @ 0000014CC325DA80] line='RTSP/1.0 200 OK'
[rtsp @ 0000014CC325DA80] line='Server: VLC/3.0.17.4'
[rtsp @ 0000014CC325DA80] line='Content-Length: 0'
[rtsp @ 0000014CC325DA80] line='Cseq: 1'
[rtsp @ 0000014CC325DA80] line='Public: DESCRIBE,SETUP,TEARDOWN,PLAY,PAUSE,GET_PARAMETER'
[rtsp @ 0000014CC325DA80] line=''
===========================================================================================
[rtsp @ 0000014CC325DA80] Sending:
DESCRIBE rtsp://:8554/test RTSP/1.0
Accept: application/sdp
CSeq: 2
User-Agent: Lavf59.34.101
--
[rtsp @ 0000014CC325DA80] line='RTSP/1.0 200 OK'
[rtsp @ 0000014CC325DA80] line='Server: VLC/3.0.17.4'
[rtsp @ 0000014CC325DA80] line='Date: Thu, 17 Nov 2022 06:45:09 GMT'
[rtsp @ 0000014CC325DA80] line='Content-Type: application/sdp'
[rtsp @ 0000014CC325DA80] line='Content-Base: rtsp://[::1]:8554/test'
[rtsp @ 0000014CC325DA80] line='Content-Length: 542'
[rtsp @ 0000014CC325DA80] line='Cache-Control: no-cache'
[rtsp @ 0000014CC325DA80] line='Cseq: 2'
[rtsp @ 0000014CC325DA80] line=''
[rtsp @ 0000014CC325DA80] SDP:
v=0
o=- 16654407034258208241 16654407034258208241 IN IP6 tayu-pc
s=Unnamed
i=N/A
c=IN IP6 ::
t=0 0
a=tool:vlc 3.0.17.4
a=recvonly
a=type:broadcast
a=charset:UTF-8
a=control:rtsp://[::1]:8554/test
m=audio 0 RTP/AVP 14
b=AS:128
b=RR:0
a=rtpmap:14 MPA/90000/2
a=control:rtsp://[::1]:8554/test/trackID=0
m=video 0 RTP/AVP 96
b=RR:0
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1;profile-level-id=640020;sprop-parameter-sets=Z2QAIKzZQFAGWwFqBAQCgAAAAwCAAAAFR4wYyw==,aOvjyyLA;
a=control:rtsp://[::1]:8554/test/trackID=1
===========================================================================================
[rtp @ 0000014CC32558C0] No default whitelist set
[udp @ 0000014CC3262100] No default whitelist set
[udp @ 0000014CC3262100] end receive buffer size reported is 393216
[udp @ 0000014CC3262000] No default whitelist set
[udp @ 0000014CC3262000] end receive buffer size reported is 393216
===========================================================================================
[rtsp @ 0000014CC325DA80] Sending:
SETUP rtsp://[::1]:8554/test/trackID=0 RTSP/1.0
Transport: RTP/AVP/UDP;unicast;client_port=20332-20333
CSeq: 3
User-Agent: Lavf59.34.101
--
[rtsp @ 0000014CC325DA80] line='RTSP/1.0 200 OK'
[rtsp @ 0000014CC325DA80] line='Server: VLC/3.0.17.4'
[rtsp @ 0000014CC325DA80] line='Date: Thu, 17 Nov 2022 06:45:21 GMT'
[rtsp @ 0000014CC325DA80] line='Transport: RTP/AVP/UDP;unicast;client_port=20332-20333;server_port=61687-61688;ssrc=416BF87E;mode=play'
[rtsp @ 0000014CC325DA80] line='Session: a242cd292f307f4b;timeout=60'
[rtsp @ 0000014CC325DA80] line='Content-Length: 0'
[rtsp @ 0000014CC325DA80] line='Cache-Control: no-cache'
[rtsp @ 0000014CC325DA80] line='Cseq: 3'
[rtsp @ 0000014CC325DA80] line=''
===========================================================================================
[rtsp @ 0000014CC325DA80] Sending:
SETUP rtsp://[::1]:8554/test/trackID=1 RTSP/1.0
Transport: RTP/AVP/UDP;unicast;client_port=20334-20335
CSeq: 4
User-Agent: Lavf59.34.101
Session: a242cd292f307f4b
--
[rtsp @ 0000014CC325DA80] line='RTSP/1.0 200 OK'
[rtsp @ 0000014CC325DA80] line='Server: VLC/3.0.17.4'
[rtsp @ 0000014CC325DA80] line='Date: Thu, 17 Nov 2022 06:45:46 GMT'
[rtsp @ 0000014CC325DA80] line='Transport: RTP/AVP/UDP;unicast;client_port=20334-20335;server_port=61688-61689;ssrc=4ACDD3CA;mode=play'
[rtsp @ 0000014CC325DA80] line='Session: a242cd292f307f4b;timeout=60'
[rtsp @ 0000014CC325DA80] line='Content-Length: 0'
[rtsp @ 0000014CC325DA80] line='Cache-Control: no-cache'
[rtsp @ 0000014CC325DA80] line='Cseq: 4'
[rtsp @ 0000014CC325DA80] line=''
===========================================================================================
[rtsp @ 0000014CC325DA80] Sending:
PLAY rtsp://[::1]:8554/test RTSP/1.0
Range: npt=0.000-
CSeq: 5
User-Agent: Lavf59.34.101
Session: a242cd292f307f4b
--
[rtsp @ 0000014CC325DA80] line='RTSP/1.0 200 OK'
[rtsp @ 0000014CC325DA80] line='Server: VLC/3.0.17.4'
[rtsp @ 0000014CC325DA80] line='Date: Thu, 17 Nov 2022 06:45:56 GMT'
[rtsp @ 0000014CC325DA80] line='RTP-Info: url=rtsp://[::1]:8554/test/trackID=0;seq=12541;rtptime=1127806156, url=rtsp://[::1]:8554/test/trackID=1;seq=942;rtptime=1127806156'
[rtsp @ 0000014CC325DA80] line='Range: npt=58.978649-'
[rtsp @ 0000014CC325DA80] line='Session: a242cd292f307f4b;timeout=60'
[rtsp @ 0000014CC325DA80] line='Content-Length: 0'
[rtsp @ 0000014CC325DA80] line='Cache-Control: no-cache'
[rtsp @ 0000014CC325DA80] line='Cseq: 5'
[rtsp @ 0000014CC325DA80] line=''
===========================================================================================

日志到Play 就没再记录时因为后面日志就比较多了，没仔细看。
总体逻辑就是客户端向服务端发送几次请求，并会收到服务端的一些响应。
Sending: OPTIONS Cseq: 1
Receive: 'RTSP/1.0 200 OK Cseq: 1
…
Sending: DESCRIBE Accept: application/sdp Cseq: 2
Receive: 'RTSP/1.0 200 OK Cseq: 2

SDP:
v=0
o=- 16654407034258208241 16654407034258208241 IN IP6 tayu-pc
s=Unnamed
i=N/A
c=IN IP6 ::
t=0 0
a=tool:vlc 3.0.17.4
a=recvonly
a=type:broadcast
a=charset:UTF-8
a=control:rtsp://[::1]:8554/test
m=audio 0 RTP/AVP 14
b=AS:128
b=RR:0
a=rtpmap:14 MPA/90000/2
a=control:rtsp://[::1]:8554/test/trackID=0
m=video 0 RTP/AVP 96
b=RR:0
a=rtpmap:96 H264/90000
a=fmtp:96 packetization-mode=1;profile-level-id=640020;sprop-parameter-sets=Z2QAIKzZQFAGWwFqBAQCgAAAAwCAAAAFR4wYyw==,aOvjyyLA;
a=control:rtsp://[::1]:8554/test/trackID=1

…
Sending: SETUP Transport: RTP/AVP/UDP;unicast;client_port=20332-20333 trackID=0 Cseq: 3 //视频轨道
Receive: 'RTSP/1.0 200 OK Cseq: 3

'Transport: RTP/AVP/UDP;unicast;client_port=20332-20333;server_port=61687-61688;ssrc=416BF87E;mode=play

…
Sending: SETUP Transport: RTP/AVP/UDP;unicast;client_port=20332-20333 trackID=1 Cseq: 4 //音频轨道
Receive: 'RTSP/1.0 200 OK Cseq: 4

'Transport: RTP/AVP/UDP;unicast;client_port=20332-20333;server_port=61687-61688;ssrc=416BF87E;mode=play

…
Sending: PLAY Session: a242cd292f307f4b Cseq: 5
Receive: 'RTSP/1.0 200 OK Session: a242cd292f307f4b;timeout=60 Cseq: 5
…

上面说了这么多消息对应早FFmpeg 中其实就是一个API 做的事情：

API avformat_open_input

当返回值为0时上面的操作就基本都完成了。
需要注意的是 SDP 协议是使用TCP 连接，RTSP 是会建立RTP 连接，FFMpeg 默认使用UDP 建立RTP连接，从上面日志中也可以看出来。
所以如果API 一直失败可以考虑是否不是UDP 的连接，可以使用AVDictionary了修改为TCP 连接。

UDP代码

  auto ret = avformat_open_input(&format_ctx, "rtsp://192.168.10.182/video/pri", NULL, nullptr);

TCP 代码

  AVDictionary* format_opts = NULL;av_dict_set(&format_opts, "stimeout", std::to_string(2 * 1000000).c_str(), 0); //设置链接超时时间（us）av_dict_set(&format_opts, "rtsp_transport", "tcp", 0); //设置推流的方式，默认udp。auto ret = avformat_open_input(&format_ctx, "rtsp://192.168.10.182/video/pri", NULL, &format_opts);

到这里基本RTSP 的连接工作就结束了。。。。

API av_read_frame

 while (av_read_frame(format_ctx, &packet) >= 0 && cnt < 1000)

可以循环使用这个API 从RTSP 中读取h264的流，packet 中包换的nal 的buffer。

到这里思路就应该是如何从h264得到yuv或rgb/bgr了。我们使用Opencv 就获取BGR24即可。

之后的代码和解码代码类似，直接放全部代码仅供参考。

#include 
#include 
#include 
#include 
#include extern "C" {
#include "libavcodec/avcodec.h"
#include "libavformat/avformat.h"
#include "libavformat/avio.h"
#include "libswscale/swscale.h"
}
#include "opencv2/imgcodecs.hpp"
#include "opencv2/highgui.hpp"
#include "opencv2/imgproc.hpp"using namespace cv;
int main(int argc, char** argv) {// Open the initial context variables that are neededSwsContext* img_convert_ctx;AVFormatContext* format_ctx = avformat_alloc_context();AVCodecContext* codec_ctx = NULL;int video_stream_index;av_log_set_level(AV_LOG_TRACE); //AV_LOG_TRACE //AV_LOG_DEBUG//open RTSPAVDictionary* format_opts = NULL;av_dict_set(&format_opts, "stimeout", std::to_string(2 * 1000000).c_str(), 0); //设置链接超时时间（us）av_dict_set(&format_opts, "rtsp_transport", "tcp", 0); //设置推流的方式，默认udp。auto ret = avformat_open_input(&format_ctx, "rtsp://:8554/test", NULL, &format_opts);ret = avformat_find_stream_info(format_ctx, NULL);//search video streamfor (int i = 0; i < format_ctx->nb_streams; i++) {if (format_ctx->streams[i]->codecpar->codec_type == AVMEDIA_TYPE_VIDEO)video_stream_index = i;}AVPacket packet;av_init_packet(&packet);AVStream* stream = NULL;int cnt = 0;//start reading packets from stream and write them to fileav_read_play(format_ctx);    //play RTSP// Get the codecconst AVCodec* codec = avcodec_find_decoder(AV_CODEC_ID_H264);if (!codec) {exit(1);}std::ofstream myfile;// Add this to allocate the context by codeccodec_ctx = avcodec_alloc_context3(codec);avcodec_parameters_from_context(format_ctx->streams[video_stream_index]->codecpar, codec_ctx);std::ofstream output_file;codec_ctx->width = 640;codec_ctx->height = 480;codec_ctx->pix_fmt = AV_PIX_FMT_YUV420P;if (avcodec_open2(codec_ctx, codec, NULL) < 0)exit(1);img_convert_ctx = sws_getContext(codec_ctx->width, codec_ctx->height, codec_ctx->pix_fmt, codec_ctx->width, codec_ctx->height, AV_PIX_FMT_BGR24,SWS_BICUBIC, NULL, NULL, NULL);AVFrame* picture = av_frame_alloc();picture = av_frame_alloc();AVFrame* picturergb = av_frame_alloc();picturergb = av_frame_alloc();if (!picture) {fprintf(stderr, "Could not allocate video frame\n");exit(1);}picture->format = codec_ctx->pix_fmt;picture->width = codec_ctx->width;picture->height = codec_ctx->height;picturergb->format = AV_PIX_FMT_BGR24;picturergb->width = codec_ctx->width;picturergb->height = codec_ctx->height;ret = av_frame_get_buffer(picture, 0);ret = av_frame_get_buffer(picturergb, 0);AVCodecParserContext* parser;parser = av_parser_init(codec->id);if (!parser) {fprintf(stderr, "parser not found\n");exit(1);}while (av_read_frame(format_ctx, &packet) >= 0 && cnt < 1000){if (stream == NULL){stream = avformat_new_stream(format_ctx, codec);avcodec_parameters_from_context(stream->codecpar, codec_ctx);}ret = avcodec_send_packet(codec_ctx, &packet);if (ret < 0) {continue;}while (ret >= 0) {ret = avcodec_receive_frame(codec_ctx, picture);if (ret == AVERROR(EAGAIN) || ret == AVERROR_EOF)break;else if (ret < 0) {fprintf(stderr, "Error during decoding\n");return -1;}ret = sws_scale(img_convert_ctx, picture->data, picture->linesize, 0, 480, picturergb->data, picturergb->linesize);Mat src(cv::Size(640,480), CV_8UC3, picturergb->buf[0]->data);imshow("test", src);waitKey(1);}}av_free(picture);av_read_pause(format_ctx);return (EXIT_SUCCESS);
}