根據RFC3984以RTP 封裝H.264 raw data來作video streaming.
1.H.264 raw data
以00 00 01 或 00 00 00 01作為開頭(Start Code),接著是8 bit NALU
NALU的format
+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
|F|NRI| Type |
+---------------+
F : forbidden zero bit, 一定為0
NRI : nal_ref_idc, 表示資料的重要性, 00為最不重要.
Type :nal_unit_type, H.264只定義1~23的範圍
一個H.264 raw data看起來像這樣
00 00 00 01 09 30 ......
2.RTP header
因為一個H.264 video frame資料的大小往往會在數k bytes到數十K bytes,
在傳送封包時就會將資料切割分別封裝,也因此需要加入一些額外的參數讓
接收端可以正確組合被分割的video frame.這也是RFC3984最主要的目的.
RTP header 中有三個參數要注意
timestamp : 以90KHz作為基準,以30 fps為例,timestamp遞增 90000 / 30.
實務上是以payload實際間隔時間作計算.同一個video frame的
分割資料timestamp是相同的
sequence : 每個RTP封包sequence number都遞增.
mark bit : RTP封包封裝的是最後一個分割的video frame時mark bit 為 1.
2.Payload format
1~23 : Single NAL unit packet.
RFC3984使用了H.264 NALU中未定義的type 24~29 (相當於增加H.264 nal_unit_type定義)
24 : STAP-A 單一時間組合
25 : STAP-B 單一時間組合
26 : MTAP16 多個時間組合
27 : MTAP32 多個時間組合
28 : FU-A 分割資料
29 : FU-B 分割資料
比較常見的是28,29.
3.Single NAL unit packet
當資料少於MTU的大小就用此方式封裝.
H.264 raw data foramt 為 [Start code][NALU][Raw Data]
封裝時去掉Start Code即可.Format 如下
[RTP Header][NALU][Raw Date]
4.FU-A (Fragmentation unit)
當資料大於MTU以此方式分割
H.264 raw data foramt 為 [Start code][NALU][Raw Data]
去掉[Start code],[NALU],以不超過MTU大小分割[Raw Data]
以NALU產生FU indicator, FU Header.
RFC 3984
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FU indicator | FU header | |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
| |
| FU payload |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 14. RTP payload format for FU-A
The FU indicator octet has the following format:
+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
|F|NRI| Type |
+---------------+
The FU header has the following format:
+---------------+
|0|1|2|3|4|5|6|7|
+-+-+-+-+-+-+-+-+
|S|E|R| Type |
+---------------+
[FU indicator] = (NALU & 0x60) | 28;
[FU Header] = (start << 7) | (end << 6) | (NALU & 0x1f);
format如下
[RTP Header][FU indicator][FU header][Raw Data Part 0]
[RTP Header][FU indicator][FU header][Raw Data Part 1]
[RTP Header][FU indicator][FU header][Raw Data Part 2]
...
5.FU-B (Fragmentation unit)
RFC 3984
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| FU indicator | FU header | DON |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-|
| |
| FU payload |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| :...OPTIONAL RTP padding |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 15. RTP payload format for FU-B
format如下
[RTP Header][FU indicator][FU header][DON][Raw Data Part 0]
[RTP Header][FU indicator][FU header][DON][Raw Data Part 1]
[RTP Header][FU indicator][FU header][DON][Raw Data Part 2]
...
6.Encoding sample code with TI DSP
/* Write the encoded frame to the network */
if (Buffer_getNumBytesUsed(hOutBuf)) {
struct iovec data[3];
rtp_hdr_t rtp;
rtp.version = 2;
rtp.p = 0;
rtp.x = 0;
rtp.cc = 0;
rtp.m = 0;
rtp.pt = 96;
rtp.seq = htons( sequence );
rtp.ts = htonl( timestamp );
rtp.ssrc = 10;
data[0].iov_base = &rtp;
data[0].iov_len = sizeof(rtp_hdr_t);
unsigned char *ptr = (unsigned char *)Buffer_getUserPtr(hOutBuf);
size_t len = Buffer_getNumBytesUsed(hOutBuf);
size_t mtu = DEFAULT_MTU;
/* Skip NAL Start Code 00 00 00 01 */
unsigned char nalType = ptr[4] & 0x1f;
//printf("Processing Buffer with NAL TYPE=%d\n", nalType);
if(len < (mtu - sizeof(rtp_hdr_t))) {
/* Remove NAL Start Code 00 00 00 01 */
data[1].iov_base = (void *)ptr + 4;
data[1].iov_len = len - 4;
//printf("NAL Unit fit in one packet size=%d\n", len);
/* only set the marker bit on packets containing access units */
if (IS_ACCESS_UNIT (nalType))
rtp.m = 1;
writev(sfd, data, 2);
sequence++;
} else {
int start = 1, end = 0;
/* We keep 2 bytes for FU indicator and FU Header */
unsigned payload_len = mtu - sizeof(rtp_hdr_t) - 2;
unsigned char nalHeader = ptr[4];
/* Remove NAL Start Code 00 00 00 01 and NAL Type*/
ptr += 5;
len -= 5;
//printf("Using FU-A fragmentation for data size=%d\n", len);
while(end == 0) {
unsigned char fu[2];
payload_len = len < payload_len ? len : payload_len;
if (payload_len == len)
end = 1;
if (IS_ACCESS_UNIT (nalType))
rtp.m = end;
/* FU indicator */
fu[0] = (nalHeader & 0x60) | 28;
/* FU Header */
fu[1] = (start << 7) | (end << 6) | (nalHeader & 0x1f);
rtp.seq = htons( sequence );
data[1].iov_base = fu;
data[1].iov_len = 2;
data[2].iov_base = (void *)ptr;
data[2].iov_len = payload_len;
writev(sfd, data, 3);
start = 0;
ptr += payload_len;
len -= payload_len;
sequence++;
}
}
if (Time_delta(hTimestamp, &time) < 0) {
printf("Failed to get timer delta\n");
goto cleanup;
}
//printf("Read time: %uus\n", (Uns)time);
timestamp += (unsigned int)(0.09 * time); /*15 fps : 90000 / 15*/
}
張貼者： 史帝夫嘰嘰叫 於 下午11:02 3 則留言: 以電子郵件傳送這篇文章 BlogThis！ 分享至 Twitter 分享至 Facebook 分享到 Pinterest frame hspace="0" marginheight="0" marginwidth="0" scrolling="no" tabindex="0" vspace="0" width="100%" id="I0_1434462927787" name="I0_1434462927787" src="https://apis.google.com/u/0/se/0/_/+1/fastbutton?usegapi=1&source=blogger%3Ablog%3Aplusone&size=medium&width=300&annotation=inline&hl=zh_TW&origin=http%3A%2F%2Fstevegigijoe.blogspot.com&url=http%3A%2F%2Fstevegigijoe.blogspot.com%2F2009%2F11%2Fh264-rtp-streaming.html&gsrc=3p&jsh=m%3B%2F_%2Fscs%2Fapps-static%2F_%2Fjs%2Fk%3Doz.gapi.zh_CN.DPWmWNjgXoM.O%2Fm%3D__features__%2Fam%3DAQ%2Frt%3Dj%2Fd%3D1%2Ft%3Dzcms%2Frs%3DAGLTcCO9yox-OxH5hAQyVIlQ7D7YxOtk_A#_methods=onPlusOne%2C_ready%2C_close%2C_open%2C_resizeMe%2C_renderstart%2Concircled%2Cdrefresh%2Cerefresh&id=I0_1434462927787&parent=http%3A%2F%2Fstevegigijoe.blogspot.com&pfname=&rpctoken=29089232" data-gapiattached="true" title="+1" style="position: static; top: 0px; width: 300px; margin: 0px; border-style: none; left: 0px; visibility: visible; height: 20px;"> 標籤： 未分類資料夾
2009年11月3日 星期二
F3F 20091018 龍蟠
張貼者： 史帝夫嘰嘰叫 於 上午2:37 沒有留言: 以電子郵件傳送這篇文章 BlogThis！ 分享至 Twitter 分享至 Facebook 分享到 Pinterest frame hspace="0" marginheight="0" marginwidth="0" scrolling="no" tabindex="0" vspace="0" width="100%" id="I1_1434462927797" name="I1_1434462927797" src="https://apis.google.com/u/0/se/0/_/+1/fastbutton?usegapi=1&source=blogger%3Ablog%3Aplusone&size=medium&width=300&annotation=inline&hl=zh_TW&origin=http%3A%2F%2Fstevegigijoe.blogspot.com&url=http%3A%2F%2Fstevegigijoe.blogspot.com%2F2009%2F11%2Ff3f-20091018.html&gsrc=3p&jsh=m%3B%2F_%2Fscs%2Fapps-static%2F_%2Fjs%2Fk%3Doz.gapi.zh_CN.DPWmWNjgXoM.O%2Fm%3D__features__%2Fam%3DAQ%2Frt%3Dj%2Fd%3D1%2Ft%3Dzcms%2Frs%3DAGLTcCO9yox-OxH5hAQyVIlQ7D7YxOtk_A#_methods=onPlusOne%2C_ready%2C_close%2C_open%2C_resizeMe%2C_renderstart%2Concircled%2Cdrefresh%2Cerefresh&id=I1_1434462927797&parent=http%3A%2F%2Fstevegigijoe.blogspot.com&pfname=&rpctoken=26222080" data-gapiattached="true" title="+1" style="position: static; top: 0px; width: 300px; margin: 0px; border-style: none; left: 0px; visibility: visible; height: 20px;"> 標籤： F3F
OMAP3530 DSP Howto ( Really works by DSPLINK)
搞了一段很長的時間終於把DSP功能弄起來了,該是回頭紀錄一下"眉角"了.
1.Before touch anything
目前有兩種方式可以控制DSP,一個是TI DVSDK提供的DSPLINK,另一個則是DSPBRIDGE.
以我從google搜尋之後的了解DSPBRIDGE目前是沒有繼續maintain.
DSPLINK是TI比較建議的方式.
所搭配的media library為
DSPLINK + gstreamer
DSPBRIDGE + openmax + gstreamer
http://felipec.wordpress.com/2009/10/13/new-project-gst-dsp-with-beagleboard-demo-image/
http://felipec.wordpress.com/2008/12/12/gst-openmax-demo-on-the-beagleboard/
這邊我選擇DSPLINK控制DSP
2.Software
dvsdk_3_00_02_44
gst-ti-plugin-full-1.00.02
gst-omapfb
Build and install to target root file system (過程就省略了)
可參考以下作法
http://ossie.wireless.vt.edu/trac/wiki/BeagleBoard_CodecEngine
安裝目錄
DVSDK : /opt/dvsdk
gstreamer : /opt
3.Setup
DSPLINK透過share memory的方式跟DSP交換資料所以有另一個driver叫cmemk.ko負責建立memory pool. 但這塊memory必須脫離kernel的掌控因此必須要空出一塊memory讓cmemk.ko使用.至於大小要看實際memory size跟DSP所需要的size來決定.
http://pixhawk.ethz.ch/wiki/tutorials/omap/dsplink/memorymap
依照以上說明修改以下檔案
dsplink_1_61_03/packages/dsplink/config/all/CFG_OMAP3530_SHMEM.c
dsplink_1_61_03/packages/dsplink/dsp/inc/DspBios/5.XX/OMAP3530/dsplink-omap3530-base.tci
dsplink_1_61_03/packages/dsplink/config/all/CFG_OMAP3530_SHMEM.c
我的target board有256MB DDR,因此記憶體配置如下
# 0x40200000 60 KB CMEM (internal memory, see comment below)
# 0x80000000 200 MB Linux
# 0x8C900000 16 MB CMEM (external memory)
# 0x8D900000 2 MB DSPLINK (MEM)
# 0x8DB00000 unused
kernel boot command 加上 mem=200M
For kernel 2.6.27
optargs=init=/init [email protected] vram=12M omapfb.vram=4M,4M,4M omapfb.debug=1 omap-dss.def_disp=lcd omap-dss.debug=1 mem=200M
For kernel 2.6.29
setenv optargs 'init=/init omapdss.def_disp=dvi omapfb.mode=dvi:[email protected] vram=12M omapfb.vram=0:4M,1:4M,2:4M mem=200M'
memory pool配置如下
insmod cmemk.ko phys_start=0x8C900000 phys_end=0x8D900000 pools=20x4096,10x131072,2x1048576,1x5250000,4x829440,2x691200
gstreamer需要設定target上的環境變數好讓系統找的到執行檔及plug-in
export PATH=$PATH:/opt/gstreamer/bin
export GST_PLUGIN_PATH=/opt/gstreamer/lib/gstreamer-0.10
export LD_LIBRARY_PATH=/opt/gstreamer/lib
3.Get DSP work
載入drivers
#cd /opt/dvsdk
#./loadmodules.sh
CMEMK module: built on Oct 30 2009 at 20:23:52
Reference Linux version 2.6.28
File /home/gigijoe/OMAP3530/dvsdk_3_00_02_44/linuxutils_2_24_02/packages/ti/sdo/linuxutils/cmem/src/module/cmemk.c
ioremap_nocache(0x8c900000, 16777216)=0xcf000000
allocated heap buffer 0xcf000000 of size 0x32c000
cmem initialized 6 pools between 0x8c900000 and 0x8d900000
DSPLINK Module (1.61.03) created on Date: Oct 30 2009 Time: 20:20:31
# lsmod
lpm_omap3530 8276 0 - Live 0xbf02c000
dsplinkk 114052 1 lpm_omap3530, Live 0xbf00b000
cmemk 24428 0 - Live 0xbf000000
測試omapfb, 應該會在畫面上看到彩色條紋.
gst-launch videotestsrc num-buffers=1000 ! omapfbsink
播放TI h.264 raw data sample
gst-launch --gst-debug-level=1 filesrc location="/media/dcim/100andro/ntsc.264" ! typefind ! TIViddec2 ! omapfbsink
Play avi file
gst-launch --gst-debug-no-color --gst-debug=TI*:2 filesrc location=/media/dcim/100andro/aris.mpeg ! typefind ! qtdemux name=demux demux.audio_00 ! queue max-size-buffers=8000 max-size-time=0 max-size-bytes=0 ! typefind ! TIAuddec1 ! audioconvert ! osssink demux.video_00 ! typefind ! TIViddec2 ! omapfbsink
Play mpeg4 file
gst-launch --gst-debug-no-color --gst-debug=TI*:2 filesrc location=/media/dcim/100andro/KenBlock-TopGear.mp4 ! typefind ! qtdemux name=demux demux.audio_00 ! queue max-size-buffers=8000 max-size-time=0 max-size-bytes=0 ! typefind ! TIAuddec1 ! audioconvert ! osssink demux.video_00 ! typefind ! TIViddec2 ! omapfbsink
Play H.264 mpeg4 file without mp3 audio
gst-launch filesrc location=/media/dcim/100andro/sany0014.mp4 ! typefind ! qtdemux name=demux demux.video_00 ! typefind ! TIViddec2 ! omapfbsink
4.Debug
http://tiexpressdsp.com/wiki/index.php?title=Debugging_DSPLink_using_SET_FAILURE_REASON_prints
http://tiexpressdsp.com/index.php/Enabling_trace_in_DSPLink
一開始有一段時間kernel-2.6.29 DSP_init總是失敗
# ./messagegpp ./message.out 1000
========== Sample Application : Failure [0x8000800b] in [0x401] at line 522
MESSAGE ==========
Entered MESSFailure [0x8000802d] in [0x401] at line 544
AGE_Create ()
Failure [0x8000802d] in [0x401] at line 544
Failure [0x8000802d] in [0x401] at line 544
ISR_Install:445
request_irq 28
request_irq failed with error: -16
Failure [0x80008008] in [0x502] at line 459
Failure [0x80008008] in [0x80a] at line 824
Failure [0x80008008] in [0x80a] at line 1061
Failure [0x80008008] in [0x801] at line 597
Failure [0x80008008] in [0x701] at line 370
DSP_init status [0x80008008]
Assertion failed ((isrObj!= NULL) && (ISR_InstalledIsrs [isrObj->dspId][isrObj->irq] == isrObj)). File : /home/gigijoe/OMAP3530/dvsdk_3_00_02_44/dsplink_1_61_03/packages/dsplink/gp
p/src/../../gpp/src/osal/Linux/2.6.18/isr.c Line : 507
Failure [0x80008000] in [0x502] at line 515
Failure [0x80008008] in [0x300] at line 476
Failure [0x80008008] in [0x300] at line 563
PROC_attach () failed. Status = [0x80008008]
PROC_setup () failed. Status = [0x80008008]
Leaving MESSAGE_Create ()
Entered MESSAGE_Delete ()
Assertion failed (IS_VALID_MSGQ (msgqQueue)). File : msgq.c Line : 484
MSGQ_release () failed. Status = [0x8000800b]
Assertion failed (IS_VALID_MSGQ (msgqQueue)). File : msgq.c Line : 335
Leaving MESSAGE_Delete ()
====================================================
Trace DSPLINK driver發現request_irq 28 居然失敗了.很明顯IRQ被佔據了.
看看到底是誰
cat /proc/interrupts
CPU0
11: 0 INTC prcm
12: 1 INTC DMA
18: 0 INTC sr1
19: 0 INTC sr2
24: 0 INTC omap-iommu.1, Omap 3 Camera ISP
25: 1 INTC OMAP DSS
28: 0 INTC omap-iommu.2
56: 347 INTC i2c_omap
61: 0 INTC i2c_omap
72: 1 INTC serial idle
73: 1 INTC serial idle
74: 94 INTC serial idle, serial
77: 0 INTC ehci_hcd:usb2
83: 0 INTC mmc0
86: 14 INTC mmc1
92: 0 INTC musb_hdrc
93: 0 INTC musb_hdrc
95: 641 INTC gp timer
160: 0 GPIO mmc1
167: 0 GPIO user
181: 8 GPIO eth0
378: 0 twl4030 twl4030_usb
379: 0 twl4030 rtc0
384: 0 twl4030 mmc0
原來是iommu啊,OK,調整一下kernel config
Disable
Device Drivers->Multimedia Devices->Video Capture Adapters->OMAP 3 Camera Support
System Type->TI OMAP Implemenation->IOMMU Support
這樣就OK囉
其實還有一些疑慮沒有解決.
IOMMU 以 Shared IRQ 方式 request_irq而DSPLINK Driver request_irq 並不是.
因此造成問題,DSPLINK Driver是否應該以Shared IRQ方式request_irq呢?
Problem :
# gst-launch --gst-debug=TI*:2 v4l2src always_copy=FALSE num-buffers=100 ! video
/x-raw-yuv,width=320,height=240 ! ffmpegcolorspace ! TIVidenc1 codecName=h264enc
engineName=codecServer contiguousInputFrame=FALSE genTimeStamps=FALSE ! filesin
k location=/tmp/enc.264
(gst-launch-0.10:636): GLib-WARNING **: getpwuid_r(): failed due to unknown user id (0)
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
0:00:01.011901868 636 0x9ebb0 WARN TIVidenc1 gsttividenc1.c:1432:gst_tividenc1_codec_start: error: failed to open codec engine "codecServer"
ERROR: from element /GstPipeline:pipeline0/GstTIVidenc1:tividenc10: failed to open codec engine "codecServer"
Additional debug info:
gsttividenc1.c(1432): gst_tividenc1_codec_start (): /GstPipeline:pipeline0/GstTIVidenc1:tividenc10
Execution ended after 382354731 ns.
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
0:00:01.016387951 636 0x9ebb0 WARN TIVidenc1 gsttividenc1.c:1601:gst_tividenc1_encode_thread: error: failed to start codec
gst-launch-0.10: BufTab.c:440: BufTab_getNumBufs: Assertion `hBufTab' failed.
Aborted
重新compile ti-gstreamer就好了...
Problem:
ERROR: from element /GstPipeline:pipeline0/GstTIViddec2:tividdec20: Failed to determine target board
這個問題是因為所使用的OMAP3530 board不是TI support的.
自己修改來support吧
Get board name first
# cat /proc/cpuinfo
Processor : ARMv7 Processor rev 3 (v7l)
BogoMIPS : 499.92
Features : swp half thumb fastmult vfp edsp thumbee neon vfpv3
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x1
CPU part : 0xc08
CPU revision : 3
Hardware : OMAP3 Thunder Board
Revision : 0020
Serial : 0000000000000000
So, the board name is OMAP3 Thunder Board
Edit dmai_2_05_00_04/packages/ti/sdo/dmai/linux/Cpu.c
...
}
else if ((strcmp(valBuf,"OMAP3EVM Board") == 0) ||
(strcmp(valBuf, "OMAP3 EVM") == 0) ||
(strcmp(valBuf, "OMAP3 Beagle Board") == 0) ||
(strcmp(valBuf, "OMAP3 Thunder Board") == 0)) { <<-- Add here
*device = Cpu_Device_OMAP3530;
}
else {
Dmai_err0("Unknown Cpu Type!\n");
return Dmai_EFAIL;
}
return Dmai_EOK;
}
...
Then rebuild again ...
![jrtplib-3.9.1+jthread-1.3.1使用cmake-3.1.0在windows下生成jrtplib的VS2010工程[图]](data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=)