ubuntu+systemtap進行Linux核心和使用者空間開發測試1 Systemtap安裝2 Hello World基本測試3 核心函數測試4 核心子產品測試5 使用者态監控Utrace6 Systemtap加載KO異常，Unknown symbol inmodule7 error: implicit declaration offunction ‘unregister_uprobe’8 參考資料

進行linux核心和使用者空間開發測試

sailor_forever  sailing_9806#163.com

(本原創文章發表于sailor_forever 的個人blog，未經本人許可，不得用于商業用途。任何個人、媒體、其他網站不得私自抄襲；網絡媒體轉載請注明出處，增加原文連結，否則屬于侵權行為。如有任何問題，請留言或者發郵件給sailing_9806＃163.com)

 【摘要】本文主要介紹在ubuntu平台 + 自定義核心上如何安裝systemtap工具包及解決編譯加載運作過程中的一些問題，如何利用systemtap工具監測分析核心函數，核心子產品及使用者态程式。

【關鍵字】ubuntu, systemtap, utrace, backtrace, dtrace,callstack

systemtap 是學習核心一個必不可少的工具，它不僅可以偵測核心空間的相關資訊，還可以偵測使用者空間的資訊，是研究核心源代碼、優化系統性能及調試診斷系統問題的一個必備工具。相關原理資訊可參見附錄及“linux 下的一個全新的性能測量和調式診斷工具 systemtap, 第 3 部分:

systemtap”

用系統工具自動安裝systemtap時，使用者不用考慮各個軟體包的依賴關系，系統會自動根據目前核心版本下載下傳相應的軟體包。

dd@ubuntu:/mnt/hgfs/systemtap/kvm$ sudoapt-get install systemtap

。。

the following extra packages will beinstalled:

  libdw1 systemtap-commonsystemtap-runtime

the following new packages will beinstalled:

 libdw1 systemtap systemtap-common systemtap-runtime

。

get:1 http://us.archive.ubuntu.com/ubuntu/quantal-updates/main libdw1 i386 0.153-1ubuntu1.1 [216 kb]

如上可知，系統自動安裝了相關的4個軟體包libdw1 systemtap-common systemtap-runtime

有時候系統自動安裝的不一定能用，這個時候需要更新，此時隻能通過下載下傳源代碼自己編譯

首先要解除安裝系統之前安裝的相關package，避免後續帶來其他副作用。

dd@ubuntu:/mnt/hgfs/systemtap$ sudo apt-getremove systemtap systemtap-common systemtap-runtime

removing systemtap ...

removing systemtap-common ...

removing systemtap-runtime ...

processing triggers for man-db ...

下載下傳相應版本的源碼包。源碼包并不是越新越好，因為他們之間有一定的依賴關系，最簡單的是按照核心源碼包釋出的時間來選擇systemtap。

./configure  --with-elfutils=../elfutils-0.156  --prefix=/usr/local

。。。。。

configure: ./configure  ‘--with-elfutils=../elfutils-0.156‘‘--prefix=/usr/local‘ --prefix=/home/dd/systemtap-2.0-4155

configure: running systemtap uninstalled, entirely out of the buildtree,

configure: is not supported.

重新開機系統之後再執行，仍然如此，不知是否有其他副作用

dd@ubuntu:/mnt/hgfs/systemtap/systemtap-2.0$sudo make

ln -fs libdw.so libdw.so.1

ln: failed to create symbolic link `libdw.so.1‘: operation notsupported

make[4]: *** [libdw.so] error 1

windows不支援符号連結，是以system包不能放在windows目錄編譯

dd@ubuntu:~/systemtap/systemtap-2.0$ sudomake

。。。

m4 -di386 -ddisassembler../../../elfutils-0.156/libcpu/defs/i386 > i386_defst

/bin/bash: m4: command not found

system調試核心或者application，必須有debuginfo，是以必須安裝核心debug symbol。具體參見附錄。

systemtap安裝完畢後，進行簡單的測試。如果能列印出helloworld則認為基本功能ok。

dd@ubuntu:/usr/src$ sudo stap -ve ‘probebegin { log("hello world") exit() }‘

pass 1: parsed user script and 81 libraryscript(s) using 22640virt/13560res/2192shr kb, in 80usr/10sys/93real ms.

pass 2: analyzed script: 1 probe(s), 2function(s), 0 embed(s), 0 global(s) using 22904virt/14088res/2264shr kb, in0usr/0sys/3real ms.

pass 3: translated to c into"/tmp/stapngferv/stap_202537bef6262d2233e5759ff826c9d3_744_src.c"using 22904virt/14364res/2508shr kb, in 0usr/0sys/0real ms.

pass 4: compiled c into"stap_202537bef6262d2233e5759ff826c9d3_744.ko" in4760usr/900sys/6360real ms.

pass 5: starting run.

hello world

pass 5: run completed in 0usr/20sys/359realms.

dd@ubuntu:/usr/src$ sudo stap -e ‘probekernel.function("sys_open") {log("hello world") exit()}‘

dd@ubuntu:/mnt/hgfs/systemtap/example$ cat./timer.stp

#!/usr/bin/stap 

probekernel.function("do_timer").return { 

print_backtrace(); 

printf("\n"); 

exit(); 

}

dd@ubuntu:/mnt/hgfs/systemtap/example$ sudostap ./timer.stp

returning from:  0xc1091520 : do_timer+0x0/0x40 [kernel]

returning to  : 0xc1097e4a : tick_do_update_jiffies64+0xca/0x130 [kernel]

 0xc1098074 : tick_sched_timer+0xc4/0xd0[kernel]

 0xc1068f70 : __run_hrtimer+0x70/0x190 [kernel]

 0xc1069be7 : hrtimer_interrupt+0xe7/0x2a0 [kernel]

 0xc15d0769 :smp_apic_timer_interrupt+0x59/0x8d [kernel]

 0xc15c9815 : apic_timer_interrupt+0x31/0x38[kernel]

 0xc1036f35 : native_safe_halt+0x5/0x10[kernel]

 0xc1017b66 : default_idle+0x46/0x190 [kernel]

 0xc1018716 : cpu_idle+0xb6/0xe0 [kernel]

 0xc159f715 : rest_init+0x5d/0x68 [kernel]

 0xc18ae9be : start_kernel+0x35d/0x363 [kernel]

 0xc18ae303 : i386_start_kernel+0xa6/0xad[kernel]

dd@ubuntu:/mnt/hgfs/systemtap/kvm$ sudostap ./vmx_inject_irq.stp

[sudo] password for dd:

semantic error: while resolving probepoint: identifier ‘kernel‘ at ./vmx_inject_irq.stp:1:7

       source: probe kernel.function("vmx_inject_irq") {

                      ^

semantic error: no match

pass 2: analysis failed.  try again with another ‘--vp 01‘ option.

dd@ubuntu:/mnt/hgfs/systemtap/kvm$ cat/proc/kallsyms | grep vmx_inject_irq

00000000 t vmx_inject_irq      [kvm_intel]

kvm_intel以子產品形式加載，其symbol不能以kernel.function形式通路。

ubuntu平台上需要修正動态module的debug info資訊

dd@ubuntu:/mnt/hgfs/systemtap/kvm$ catfix-module.sh

#!/bin/sh

for file in `find /usr/lib/debug -name‘*.ko‘ -print`

do

         buildid=`eu-readelf-n $file| grep build.id: | awk ‘{print $3}‘`

         dir=`echo$buildid | cut -c1-2`

         fn=`echo$buildid | cut -c3-`

         mkdir-p /usr/lib/debug/.build-id/$dir

         ln-s $file /usr/lib/debug/.build-id/$dir/$fn

         ln-s $file /usr/lib/debug/.build-id/$dir/${fn}.debug

done

dd@ubuntu:/mnt/hgfs/systemtap/kvm$ sudostap -l ‘module("*kvm*").function("*")‘ >kvm-module-sysm.log

dd@ubuntu:/mnt/hgfs/systemtap/kvm$ catkvm-module-sysm.log | grep vmx_inject_irq

module("kvm_intel").function("vmx_inject_irq@/build/buildd/linux-3.5.0/arch/x86/kvm/vmx.c:4036")

dd@ubuntu:/mnt/hgfs/systemtap/kvm$ catvmx_inject_irq.stp

probemodule("kvm_intel").function("vmx_inject_irq") {

printf("-------------------------------------\n")

print_backtrace()

很多符号都無法解析啊？？？？？？？ 根本原因待分析。

-------------------------------------

 0xf86e4230 : vmx_inject_irq+0x0/0xe0[kvm_intel]

 0xf998e694 [kvm]

 0xc10654cd (inexact)

 0xf997b25b [kvm] (inexact)

 0xf998f11f [kvm] (inexact)

 0xf997922b [kvm] (inexact)

 0xc1075dee (inexact)

 0xc1075ebf (inexact)

 0xc1099a12 (inexact)

 0xc109be08 (inexact)

 0xf9978dd0 [kvm] (inexact)

 0xc115e8fa (inexact)

 0xc1069c65 (inexact)

 0xc109c120 (inexact)

 0xc115ee88 (inexact)

 0xc15cff5f (inexact)

 0xc15c0000 (inexact)

使用者态監控需要utrace支援，但utrace預設隻有redhat系列的kernel支援，如fedora等。其他發行版linux需要patch

systemtap的readme中有關于utrace支援的描述：

- consider applying the utrace kernelpatches, if you wish to probe

 user-space applications. http://sourceware.org/systemtap/wiki/utrace

  orif your kernel is near 3.5, apply the uprobes and related patches

 (see news).  or if your kernel is>= 3.5, enjoy the built-in uprobes.

- build the kernel using your normalprocedures.  enable

 config_debug_info, config_kprobes, config_relay, config_debug_fs,

 config_modules, config_module_unload, config_utrace if able

- % make modules_install install headers_install

- boot into the kernel.

ubuntu 要支援utrace有兩種方式：

1. 自己給 ubuntu 自帶的老 kernel 應用 utrace 更新檔，并重新編譯。但好像redhat提供的patch不能通路了

http://people.redhat.com/roland/utrace/

ubuntu 11.10 (oneiric)上編譯帶utrace更新檔的核心

2. 将 kernel 更新到官方最新的 3.5 或以上的版本。最新的kernel 預設包含了 uprobes 機制，不再需要 utrace 更新檔了。

kernel hacking  --->

[*] tracers --->

[*]  enable uprobes-based dynamic events

新的核心裡已經沒有選項config_utrace，選擇utrace相關的選項即可，包括config_arch_supports_uprobesconfig_uprobes config_uprobe_event config_kprobes config_debug_fs 等

dd@ubuntu:/mnt/hgfs/systemtap/example$ cat/boot/config-3.5.0-17-generic | grep uprobe

config_arch_supports_uprobes=y

config_uprobes=y

config_uprobe_event=y

dd@ubuntu:/mnt/hgfs/systemtap/example$ cat/boot/config-3.5.0-17-generic | grep kprobe

config_kprobes=y

config_have_kprobes=y

# config_kprobes_sanity_test is not set

config_kprobe_event=y

如果不是必須用某一個版本，建議直接更新到3.5以後的核心版本，如ubuntu 12.10

首先寫了一個使用者态的測試函數，可參考如下

utrace調試的程式需要在編譯時加 -g

選項

dd@ubuntu:/mnt/hgfs/systemtap/example$./utrace-test

obtained 8 stack frames.nm

./utrace-test() [0x80485c1]

[0xb77a3400]

./utrace-test() [0x804869f]

./utrace-test() [0x80486ac]

./utrace-test() [0x80486b6]

./utrace-test() [0x80486eb]

/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb75ff4d3]

./utrace-test() [0x80484b1]

有函數調用棧，但是沒有函數名稱，莫非是編譯優化導緻無符号表？經查需要傳遞(-rdynamic)，-rdynamic可用來通知連結器将所有符号添加到動态符号表中

dd@ubuntu:/mnt/hgfs/systemtap/example$ gcc-g -rdynamic utrace-test.c -o utrace-test

obtained 7 stack frames.

./utrace-test(print_trace+0x1f) [0x80487ab]

./utrace-test(dummy_function_3+0xb)[0x8048840]

./utrace-test(test2+0xb) [0x804884d]

./utrace-test(test1+0xb) [0x804885a]

./utrace-test(main+0xb) [0x8048867]

/lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb75f44d3]

./utrace-test() [0x80486c1]

ok，能夠正确列印出函數調用順序，參考基準已經建立起來了。

如下測試顯示stap可以監測到待調試程式utrace-test的函數清單。

dd@ubuntu:/mnt/hgfs/systemtap/example$ sudostap -l ‘process("./utrace-test").function("*")‘

process("/mnt/hgfs/systemtap/example/utrace-test").function("dummy_function_3@/mnt/hgfs/systemtap/example/utrace-test.c:26")

process("/mnt/hgfs/systemtap/example/utrace-test").function("main@/mnt/hgfs/systemtap/example/utrace-test.c:39")

process("/mnt/hgfs/systemtap/example/utrace-test").function("print_trace@/mnt/hgfs/systemtap/example/utrace-test.c:7")

process("/mnt/hgfs/systemtap/example/utrace-test").function("test1@/mnt/hgfs/systemtap/example/utrace-test.c:34")

process("/mnt/hgfs/systemtap/example/utrace-test").function("test2@/mnt/hgfs/systemtap/example/utrace-test.c:30")

當任意一個函數被調用時，列印其函數名及參數

dd@ubuntu:/mnt/hgfs/systemtap/example$ catutrace.stp

probeprocess("./utrace-test").function("*") { 

printf("%s(%s)\n", probefunc(), $$parms);

實際檢測到的函數調用順序與實際比對

dd@ubuntu:/mnt/hgfs/systemtap/example$ sudostap ./utrace.stp

_start()

__libc_csu_init()

__i686.get_pc_thunk.bx()

_init()

frame_dummy()

register_tm_clones()

main()

test1()

test2()

dummy_function_3()

print_trace()

__do_global_dtors_aux()

deregister_tm_clones()

_fini()

利用print_backtrace列印監測點的函數調用棧。

dd@ubuntu:/mnt/hgfs/systemtap/example$ catutrace-stack.stp

probeprocess("./utrace-test").function("dummy_function_3"){ 

printf("%s(%s)\n", probefunc(),$$parms);

dd@ubuntu:/mnt/hgfs/systemtap/example$ sudostap ./utrace-stack.stp

semantic error: while resolving probepoint: identifier ‘process‘ at ./utrace.stp:3:7

       source: probeprocess("./utrace-test").function("dummy_function_3").return{ 

semantic error: process return probes not available withinode-based uprobes

pass 2: analysis failed.  try again with another ‘--vp 01‘ option.

去掉return，再執行

warning: no or bad debug frame hdr

warning: no binary search table for eh frame, doing slow linear searchfor stap_3fb2bcd721ab82a6de577773eff766b_10596

 0xf8887e10 [stap_3fb2bcd721ab82a6de577773eff766b_10596+0xbe10/0x0]

 0xf8888209[stap_3fb2bcd721ab82a6de577773eff766b_10596+0xc209/0x0]

 0xf88868b0[stap_3fb2bcd721ab82a6de577773eff766b_10596+0xa8b0/0x0]

 0xc113eb3f

 0xc104e51b (inexact)

 0xc161ac9d (inexact)

沒有列印出使用者态的調用堆棧，具體原因待分析。

加載動态編譯出來的module時失敗unknown symbol inmodule

dd@ubuntu:/mnt/hgfs/systemtap/kvm$ sudostap -ve ‘probe begin { log("hello world") exit() }‘

pass 4: compiled c into"stap_aeecf973d1b00387ff715fbb9a761ae9_688.ko" in6010usr/1080sys/10345real ms.

error inserting module‘/tmp/stapoo63sx/stap_aeecf973d1b00387ff715fbb9a761ae9_688.ko‘: unknown symbolin module

warning: /usr/bin/staprun exited withstatus: 1

pass 5: run completed in 0usr/0sys/11realms.

pass 5: run failed.  try again with another ‘--vp 00001‘ option.

dd@ubuntu:/mnt/hgfs/systemtap/kvm$ sudostap -e ‘probe kernel.function("sys_open") {log("helloworld") exit()}‘

error inserting module‘/tmp/stappq4mh5/stap_b3efd67e86a8df14579c7f419ddcd07f_826.ko‘: unknown symbolin module

pass 5: run failed.  try again with another ‘--vp 00001‘ option

檢視核心log，分析unknown symbol inmodule的意義，這種問題通常都是某個symbol無法找到。

dd@ubuntu:~/systemtap/systemtap-2.5$ dmesg| tail

…

[  76.855961] stap_880c6f2fc0076129ea6feb7db23da703_899:

unknown parameter `_stp_bufsize‘

“-k”參數保留運作過程中生産的檔案，查找線索

dd@ubuntu:~$ sudo stap -k -ve ‘probe begin{ log("hello world") exit() }‘

error inserting module‘/tmp/staplcmn6z/stap_4270.ko‘: unknown symbol in module

pass 5: run completed in 0usr/0sys/3realms.

keeping temporary directory"/tmp/staplcmn6z"

dd@ubuntu:~$ sudo nm/tmp/staplcmn6z/stap_4270.ko | grep _stp_bufsize

00000032 r __mod__stp_bufsize76

00000050 r __mod__stp_bufsizetype75

00000000 r __param__stp_bufsize

00000515 r __param_str__stp_bufsize

00000130 b _stp_bufsize

“--vp00005”啟動systemtap的調試選項獲得更多的log。 具體用法可參考

dd@ubuntu:/mnt/hgfs/systemtap/example$ sudostap --vp 00005 -ve‘probe begin{printf("hello world\n") exit()}‘

running /usr/local/bin/staprun -v -v -v -v -v-r /tmp/stapwlfjon/stap_333448f0b355beeb59e41a501a5421fd_817.ko

staprun:parse_modpath:386

staprun:insert_module:71 inserting module/tmp/stapwlfjon/stap_333448f0b355beeb59e41a501a5421fd_817.ko

staprun:insert_module:97module options: _stp_bufsize=0

staprun:insert_module:105 module pathcanonicalized to ‘/tmp/stapwlfjon/stap_333448f0b355beeb59e41a501a5421fd_817.ko‘

staprun:insert_module:183 modulestap_333448f0b355beeb59e41a501a5421f_6340 inserted from file/tmp/stapwlfjon/stap_333448f0b355beeb59e41a501a5421fd_817.ko

error:couldn‘t insert module ‘/tmp/stapwlfjon/stap_333448f0b355beeb59e41a501a5421fd_817.ko‘:unknown symbol in module

spawn waitpid result (0x100): 1

warning: /usr/local/bin/staprun exited withstatus: 1

pass 5: run completed in 0usr/0sys/2real ms.

pass 5: run failed.  [man error::pass5]

running rm -rf /tmp/stapwlfjon

spawn waitpid result (0x0): 0

removed temporary directory"/tmp/stapwlfjon"

dd@ubuntu:~/systemtap/systemtap-2.5$ grep -nr"_stp_bufsize" ./

binary file ./staprun/staprun matches

binary file ./staprun/staprun-staprun.omatches

./staprun/staprun.c:190:     /* add the _stp_bufsize option.  */

./staprun/staprun.c:192:                   "_stp_bufsize=%d",buffer_size))

./runtime/transport/transport.c:75:static int_stp_bufsize;

./runtime/transport/transport.c:76:module_param(_stp_bufsize,int, 0);

./runtime/transport/transport.c:77:module_parm_desc(_stp_bufsize,"buffer size");

./runtime/transport/transport.c:408:  if (_stp_bufsize) {

./runtime/transport/transport.c:409:         unsigned size = _stp_bufsize * 1024 *1024;

./runtime/transport/ring_buffer.c:83:       unsigned long buffer_size = _stp_bufsize* 1024 * 1024;

./runtime/transport/transport.h:48:static int_stp_bufsize;

dd@ubuntu:/mnt/hgfs/systemtap/example$ sudomodinfo stapomk4ac/stap_12809.ko [sudo] password for dd:

filename:       stapomk4ac/stap_12809.ko

parm:           _stp_bufsize:buffer size (int)

從上面的代碼看，子產品參數_stp_bufsize确實存在，且加載的過程沒有問題啊，很奇怪為什麼會fail。難道是版本不比對？嘗試解除安裝2.0版本，安裝2.5版本，依然有問題

于是嘗試分析其内在原理，看是否能真正fix或者workaround這個問題。

./runtime/transport/transport.c接收子產品參數_stp_bufsize，但是從代碼看并沒有特别的用處；另外通過“staprun:insert_module:97module options: _stp_bufsize=0”知道./staprun/staprun.c:192實際傳入的參數值是0，是以可去掉special_options參數試試

static int insert_stap_module(privilege_t*user_credentials)

{

       charspecial_options[128] = {0};

#if 0

        /* add the _stp_bufsize option.  */

        if (snprintf_chk(special_options,sizeof (special_options),

                        "_stp_bufsize=%d", buffer_size))

                return -1;

#endif

       stap_module_inserted= insert_module(modpath, special_options,

                                        modoptions,

                                        assert_stap_module_permissions,

                                        user_credentials);

       if (stap_module_inserted != 0)

                err("error insertingmodule ‘%s‘: %s\n", modpath, moderror(errno));

       returnstap_module_inserted;

重新編譯加載後，一切正常。

dd@ubuntu:/mnt/hgfs/systemtap/example$ sudostap -ve ‘probe process("./utrace-test").function("*"){printf("-------------------------------------\n") print_backtrace()printf("-------------------------------------\n")}‘

in file included from/tmp/stapri0noh/stap_db3a2cd3f06ff0a78a55d936a32da8b0_1325_src.c:184:0:

/usr/share/systemtap/runtime/uprobes-inode.c:in function ‘stp_inode_uprobes_unreg’:

/usr/share/systemtap/runtime/uprobes-inode.c:94:3:error: implicit declaration of function ‘unregister_uprobe’[-werror=implicit-function-declaration]

/usr/share/systemtap/runtime/uprobes-inode.c:in function ‘stp_inode_uprobes_reg’:

/usr/share/systemtap/runtime/uprobes-inode.c:107:3:error: implicit declaration of function ‘register_uprobe’

[-werror=implicit-function-declaration]

cc1: all warnings being treated as errors

make[1]: ***[/tmp/stapri0noh/stap_db3a2cd3f06ff0a78a55d936a32da8b0_1325_src.o] error 1

make: *** [_module_/tmp/stapri0noh] error 2

warning: make exited with status: 2

pass 4: compiled c into"stap_db3a2cd3f06ff0a78a55d936a32da8b0_1325.ko" in280usr/170sys/681real ms.

pass 4: compilation failed.  try again with another ‘--vp 0001‘ option.

stap fails with: "error: implicitdeclaration of function ‘unregister_uprobe’"

dd@ubuntu:/mnt/hgfs/systemtap$ stap -v

systemtap translator/driver (version 1.7/0.153 debian version1.7-1ubuntu1 (quantal))

copyright (c) 2005-2012 red hat, inc. andothers

this is free software; see the source forcopying conditions.

enabled features: avahi libsqlite3 nssboost_shared_ptr tr1_unordered_map nls

核心中沒有systemtap/runtime引用的api register_uprobe。在新版本的systemtap中已經fix了，是以需要下載下傳源代碼手動更新

dd@ubuntu:/mnt/hgfs/systemtap$ sudo grep"\suprobe.*register" /proc/kallsyms

c10fe1d0 t uprobe_register

c10fe430 t uprobe_unregister

更新到2.0版本之後，正常

dd@ubuntu:~/systemtap/systemtap-2.0$ stap-v

systemtap translator/driver (version2.0/0.156, non-git sources)

enabled features: tr1_unordered_map nls

systemtap wiki

a guide on how to install systemtap on anubuntu system

linux 下的一個全新的性能測量和調式診斷工具systemtap, 第 3 部分: systemtap

使用systemtap調試linux核心示例

user-space probing

dynamic tracing with dtrace and systemtap

systemtap : stap processing 5 stepsintroduce