天天看点
返回

gstreamer pad capabilities介绍

作者：程序员修行

Gstreamer pipeline由多个元素组成来处理多媒体数据，元素间通过pad通信。pad 间通信有什么要求？

pad caps

pad capabilities定义了可以流经此pad的数据类型。通过gst-inspect-1.0可以查看某插件支持的pad capabilities，譬如下面查看了audioconvert插件，Capabilities字段显示了src和sink pad支持的capabilities(。。。表示省略部分内容)。

# gst-inspect-1.0 audioconvert
。。。
Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      audio/x-raw
                 format: { (string)S8, (string)U8, (string)S16LE, (string)S16BE, (string)U16LE, (string)U16BE, (string)S24_32LE, (string)S24_32BE, (string)U24_32LE, (string)U24_32BE, (string)S32LE, (str
ing)S32BE, (string)U32LE, (string)U32BE, (string)S24LE, (string)S24BE, (string)U24LE, (string)U24BE, (string)S20LE, (string)S20BE, (string)U20LE, (string)U20BE, (string)S18LE, (string)S18BE, (string)U18
LE, (string)U18BE, (string)F32LE, (string)F32BE, (string)F64LE, (string)F64BE }
                   rate: [ 1, 2147483647 ]
               channels: [ 1, 2147483647 ]
                 layout: { (string)interleaved, (string)non-interleaved }
  
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      audio/x-raw
                 format: { (string)S8, (string)U8, (string)S16LE, (string)S16BE, (string)U16LE, (string)U16BE, (string)S24_32LE, (string)S24_32BE, (string)U24_32LE, (string)U24_32BE, (string)S32LE, (str
ing)S32BE, (string)U32LE, (string)U32BE, (string)S24LE, (string)S24BE, (string)U24LE, (string)U24BE, (string)S20LE, (string)S20BE, (string)U20LE, (string)U20BE, (string)S18LE, (string)S18BE, (string)U18
LE, (string)U18BE, (string)F32LE, (string)F32BE, (string)F64LE, (string)F64BE }
                   rate: [ 1, 2147483647 ]
               channels: [ 1, 2147483647 ]
                 layout: { (string)interleaved, (string)non-interleaved }
。。。

两pad 能够连接的前提条件是它们的pad capabilities有交集，两pad经协商，最终会使用相同的pad capability。我们再看下wavescope插件支持的capabilities

# gst-inspect-1.0 wavescope
。。。
Pad Templates:
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      audio/x-raw
                 format: S16LE
                 layout: interleaved
                   rate: [ 8000, 96000 ]
               channels: 2
           channel-mask: 0x0000000000000003
  
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw
                 format: BGRx
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
。。。

wavescope插件sink pad capabilities和audioconvert插件 src pad capabilities有交集，这两个插件的pad可以连接，其中交集如下：

Capabilities:
      audio/x-raw
                 format: S16LE
                 layout: interleaved
                   rate: [ 8000, 96000 ]
               channels: 2
           channel-mask: 0x0000000000000003

有些插件的pad是sometimes pad，通过gst-inspect-1.0不能直接查看其支持的pad capabilities，譬如查看uridecodebin的pad capabilities如下：

# gst-inspect-1.0 uridecodebin
。。。
Pad Templates:
  SRC template: 'src_%u'
    Availability: Sometimes
    Capabilities:
      ANY
。。。

对于此情况，我们可以在pipeline执行时，实时打印某个元素的pad capability。

代码示例

我们创建一个pipeline(代码文件demo.c省略部分代码)，观察下souce元素sometimes pad和audio_convert元素 src pad的capabilities

gstreamer pad capabilities介绍

pipeline1

。。。

/* Functions below print the Capabilities in a human-friendly format */
// 以阅读友好方式打印capabilitiy field
static gboolean print_field (GQuark field, const GValue * value, gpointer pfx) {
  gchar *str = gst_value_serialize (value);

  g_print ("%s  %15s: %s\n", (gchar *) pfx, g_quark_to_string (field), str);
  g_free (str);
  return TRUE;
}

// 打印pad capabilities
static void print_caps (const GstCaps * caps, const gchar * pfx) {
  guint i;

  g_return_if_fail (caps != NULL);

  if (gst_caps_is_any (caps)) {
    g_print ("%sANY\n", pfx);
    return;
  }
  if (gst_caps_is_empty (caps)) {
    g_print ("%sEMPTY\n", pfx);
    return;
  }

  for (i = 0; i < gst_caps_get_size (caps); i++) {
    GstStructure *structure = gst_caps_get_structure (caps, i);

    g_print ("%s%s\n", pfx, gst_structure_get_name (structure));
    gst_structure_foreach (structure, print_field, (gpointer) pfx);
  }
}

/* Shows the CURRENT capabilities of the requested pad in the given element */
// 根据pad 名称获取元素pad，然后打印pad capabilities
static void print_pad_capabilities (GstElement *element, gchar *pad_name) {
  GstPad *pad = NULL;
  GstCaps *caps = NULL;

  /* Retrieve pad */
  // 根据pad 名称获取元素pad
  pad = gst_element_get_static_pad (element, pad_name);
  if (!pad) {
    g_printerr ("Could not retrieve pad '%s'\n", pad_name);
    return;
  }

  /* Retrieve negotiated caps (or acceptable caps if negotiation is not finished yet) */
  // 若协商完成获取协商好的pad capabilities；若协商未完成获取此pad 可接受的capabilities
  caps = gst_pad_get_current_caps (pad);
  if (!caps)
    caps = gst_pad_query_caps (pad, NULL);

  g_print ("Caps for the %s pad:\n", pad_name);
  print_caps (caps, "      ");
  gst_caps_unref (caps);
  gst_object_unref (pad);
}

/* Handler for the pad-added signal */
// 定义函数处理pad-added 信号
static void pad_added_handler (GstElement *src, GstPad *pad, CustomData *data);

int main(int argc, char *argv[]) {
。。。。
  /* Create the elements */
 // 根据插件名称，创建元素
  data.source = gst_element_factory_make ("uridecodebin", "source");
  data.tee = gst_element_factory_make ("tee", "tee");
  data.audio_queue = gst_element_factory_make ("queue", "audio_queue");
  data.audio_convert = gst_element_factory_make ("audioconvert", "audio_convert");
  data.audio_resample = gst_element_factory_make ("audioresample", "audio_resample");
  data.audio_sink = gst_element_factory_make ("autoaudiosink", "audio_sink");
  data.video_queue = gst_element_factory_make ("queue", "video_queue");
  data.visual = gst_element_factory_make ("wavescope", "visual");
  data.video_convert = gst_element_factory_make ("videoconvert", "video_convert");
  data.video_sink = gst_element_factory_make ("autovideosink", "video_sink");

  。。。

  /* Set the URI to play */
  // 设置source元素的uri 配置
  g_object_set (data.source, "uri", 
    "https://www.freedesktop.org/software/gstreamer-sdk/data/media/sintel_trailer-480p.webm", NULL);
  /* Configure elements */
   // 设置visual元素
  g_object_set (data.visual, "shader", 0, "style", 1, NULL);

  /* Link all elements that can be automatically linked because they have "Always" pads */
  // 含有always pad 的元素可以自动连接
  gst_bin_add_many (GST_BIN (data.pipeline), data.source, data.tee, data.audio_queue, data.audio_convert,
      data.audio_resample, data.audio_sink, data.video_queue, data.visual, data.video_convert, data.video_sink, NULL);
  if (gst_element_link_many (data.audio_convert, data.tee, NULL) != TRUE ||
      gst_element_link_many (data.audio_queue, data.audio_resample, data.audio_sink, NULL) != TRUE ||
      gst_element_link_many (data.video_queue, data.visual, data.video_convert, data.video_sink, NULL) != TRUE) {
    g_printerr ("Elements could not be linked.\n");
    gst_object_unref (data.pipeline);
    return -1;
  }
  。。。
  /* Connect to the pad-added signal */
  // 当source元素发出pad-added信号时，调用pad_added_handler 函数
  g_signal_connect (data.source, "pad-added", G_CALLBACK (pad_added_handler), &data);

  /* Start playing the pipeline */
  // 把gst pipeline 状态设置为GST_STATE_PLAYING
  gst_element_set_state (data.pipeline, GST_STATE_PLAYING);

  /* Wait until error or EOS */
  // 直到出现错误或视频流结束时才退出pipeline
  bus = gst_element_get_bus (data.pipeline);

  do {
    msg = gst_bus_timed_pop_filtered (bus, GST_CLOCK_TIME_NONE, GST_MESSAGE_ERROR | GST_MESSAGE_EOS |
        GST_MESSAGE_STATE_CHANGED);

    /* Parse message */
    // 解析bus 消息
    if (msg != NULL) {
      GError *err;
      gchar *debug_info;

      switch (GST_MESSAGE_TYPE (msg)) {
			。。。
        case GST_MESSAGE_STATE_CHANGED:
          /* We are only interested in state-changed messages from the pipeline */
          // 这里我们只关注pipeline 的状态变化消息
          if (GST_MESSAGE_SRC (msg) == GST_OBJECT (data.pipeline)) {
            GstState old_state, new_state, pending_state;
            gst_message_parse_state_changed (msg, &old_state, &new_state, &pending_state);
            g_print ("\nPipeline state changed from %s to %s:\n",
                gst_element_state_get_name (old_state), gst_element_state_get_name (new_state));
            // 打印audio_convert元素src pad capabilities
            print_pad_capabilities(data.audio_convert, "src");
          }
          break;
        default:
          /* We should not reach here because we only asked for ERRORs, EOS and STATE_CHANGED */
          g_printerr ("Unexpected message received.\n");
          break;
      }
      gst_message_unref (msg);
    }
  } while (!terminate);
  。。。
}

/* This function will be called by the pad-added signal */
// 这个函数处理pad-added 信号，这里的new_pad 就是sometimes pad
static void pad_added_handler (GstElement *src, GstPad *new_pad, CustomData *data) {
  GstPad *sink_pad = gst_element_get_static_pad (data->audio_convert, "sink");
  。。。

  /* Check the new pad's type */
  // 检查new_pad类型是否为audio/x-raw, 若不是则退出
  new_pad_caps = gst_pad_get_current_caps (new_pad);
  new_pad_struct = gst_caps_get_structure (new_pad_caps, 0);
  new_pad_type = gst_structure_get_name (new_pad_struct);
  if (!g_str_has_prefix (new_pad_type, "audio/x-raw")) {
    g_print ("It has type '%s' which is not raw video. Ignoring.\n", new_pad_type);
    goto exit;
  }

  /* Attempt the link */
  // 连接new_pad和sink_pad
  ret = gst_pad_link (new_pad, sink_pad);
  if (GST_PAD_LINK_FAILED (ret)) {
    g_print ("Type is '%s' but link failed.\n", new_pad_type);
  } else {
    g_print ("Link succeeded (type '%s').\n", new_pad_type);
  }

  // 打印刚产生的audio sometimes pad capabilities
  GstCaps *caps = gst_pad_get_current_caps (new_pad);
  g_print("-----------------------uridecodebin audio sometimes pad-------------------------\n");
  print_caps (caps, "      ");
  gst_caps_unref (caps);
  g_print("-----------------------uridecodebin audio sometimes pad-------------------------\n");

。。。
}

代码解析

在pipeline状态变化时，打印audio_convert元素的src pad capabilities。

case GST_MESSAGE_STATE_CHANGED:
          /* We are only interested in state-changed messages from the pipeline */
          // 这里我们只关注pipeline 的状态变化消息
          if (GST_MESSAGE_SRC (msg) == GST_OBJECT (data.pipeline)) {
            GstState old_state, new_state, pending_state;
            gst_message_parse_state_changed (msg, &old_state, &new_state, &pending_state);
            g_print ("\nPipeline state changed from %s to %s:\n",
                gst_element_state_get_name (old_state), gst_element_state_get_name (new_state));
            // 打印audio_convert元素src pad capabilities
            print_pad_capabilities(data.audio_convert, "src");

在source元素解析到audio数据时，会产生sometimes pad，此时我们打印此pad的capabilities

GstCaps *caps = gst_pad_get_current_caps (new_pad);
  g_print("-----------------------uridecodebin audio sometimes pad-------------------------\n");
  print_caps (caps, "      ");
  gst_caps_unref (caps);
  g_print("-----------------------uridecodebin audio sometimes pad-------------------------\n");

编译并执行demo.c，会有下面日志：

Obtained request pad src_1 for audio branch.
Obtained request pad src_2 for video branch.

Pipeline state changed from NULL to READY:
Caps for the src pad:
      audio/x-raw
                   rate: [ 1, 2147483647 ]
                 format: { (string)S8, (string)U8, (string)S16LE, (string)S16BE, (string)U16LE, (string)U16BE, (string)S24_32LE, (string)S24_32BE, (string)U24_32LE, (string)U24_32BE, (s
tring)S32LE, (string)S32BE, (string)U32LE, (string)U32BE, (string)S24LE, (string)S24BE, (string)U24LE, (string)U24BE, (string)S20LE, (string)S20BE, (string)U20LE, (string)U20BE, (string
)S18LE, (string)S18BE, (string)U18LE, (string)U18BE, (string)F32LE, (string)F32BE, (string)F64LE, (string)F64BE }
               channels: [ 1, 2147483647 ]
                 layout: { (string)interleaved, (string)non-interleaved }
Received new pad 'src_0' from 'source':
It has type 'video/x-raw' which is not raw video. Ignoring.
Received new pad 'src_1' from 'source':
Link succeeded (type 'audio/x-raw').
-----------------------uridecodebin audio sometimes pad-------------------------
      audio/x-raw
                 format: F32LE
                 layout: interleaved
                   rate: 48000
               channels: 2
           channel-mask: 0x0000000000000003
-----------------------uridecodebin audio sometimes pad-------------------------

Pipeline state changed from READY to PAUSED:
Caps for the src pad:
      audio/x-raw
                   rate: 48000
                 format: S16LE
               channels: 2
                 layout: interleaved
           channel-mask: 0x0000000000000003

Pipeline state changed from PAUSED to PLAYING:
Caps for the src pad:
      audio/x-raw
                   rate: 48000
                 format: S16LE
               channels: 2
                 layout: interleaved
           channel-mask: 0x0000000000000003
End-Of-Stream reached.

(uridecodebin2audioconvert:50010): GStreamer-CRITICAL **: 14:21:19.751: gst_mini_object_unref: assertion 'GST_MINI_OBJECT_REFCOUNT_VALUE (mini_object) > 0' failed

分析日志，audio_convert元素的src capability最终固定为

audio/x-raw
                   rate: 48000
                 format: S16LE
               channels: 2
                 layout: interleaved
           channel-mask: 0x0000000000000003

source元素的sometimes pad capability为

audio/x-raw
                 format: F32LE
                 layout: interleaved
                   rate: 48000
               channels: 2
           channel-mask: 0x0000000000000003

这两个capability format 不同，audio_convert元素把 sometimes pad capability转换为自己需要的src pad capability。

audio_convert元素的src pad capability是如何产生的呢？audio_convert与后面audio相关元素(即audio_resample, visual)pad capabilities交集(可通过gst-inspect-1.0自行查看)为

audio/x-raw
                 format: S16LE
                 layout: interleaved
                   rate: [ 8000, 96000 ]
               channels: 2
           channel-mask: 0x0000000000000003

与source元素的sometimes pad capability交集如下，两者format不同，audio_convert元素需要把format: F32LE转换为format: S16LE

audio/x-raw
                   rate: 48000
                 
               channels: 2
                 layout: interleaved
           channel-mask: 0x0000000000000003

其他分析

gstreamer pad capabilities介绍

pipeline2

若修改pipeline1中元素位置，变为pipeline2，source元素设置的多媒体源仍然为“https://www.freedesktop.org/software/gstreamer-sdk/data/media/sintel_trailer-480p.webm”，pipeline2会运行失败：source元素与tee元素连接失败。为什么呢？

tee后面audio相关元素(即audio_convert, audio_resample, visual)pad capabilities交集(可通过gst-inspect-1.0自行查看)为

audio/x-raw
                 format: S16LE
                 layout: interleaved
                   rate: [ 8000, 96000 ]
               channels: 2
           channel-mask: 0x0000000000000003

与source元素sometimes pad capability(它的format 为F32LE)的format不同，因此两者连接失败。

通过对pipeline1的分析，我们知道visual元素的 sink pad capabilities会影响到audio_convert的src pad capabilities，其中的实现机制是什么？下节我们会介绍不同元素pad间协商机制

capabilities gstreamer streamer stream
上一篇: 努力一个月，达到兴趣认证标准了，立马申请，可是今天收到通知，我的兴趣认证申请失败了，心情不美丽了。正好周末，出去散散心，
下一篇: HTML5 form表单 调查问卷制作HTML 表格和表单综合应用技术练习：制作调查问卷（内含超详细代码解读）

继续阅读