FS線上一次Crashes分析定位過程-ldns庫問題
– by yine 2018-04-10 15:33:05
一、故障發生時間點
2018-04-10 09:54:07
二、堆棧檢視結果
warning: .dynamic section for "/usr/lib/x86_64-linux-gnu/librtmp.so.1" is not at the expected address (wrong library or version mismatch?)
warning: .dynamic section for "/usr/lib/libldns.so.1" is not at the expected address (wrong library or version mismatch?)
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/freeswitch -nc -nonat -nosql -u popo -g netease'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f2388bc9067 in __GI_raise ([email protected]=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt full
#0 0x00007f2388bc9067 in __GI_raise ([email protected]=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
resultvar = 0
pid = 28593
selftid = 20809
#1 0x00007f2388bca448 in __GI_abort () at abort.c:89
save_stage = 2
act = {__sigaction_handler = {sa_handler = 0x3030303030207078, sa_sigaction = 0x3030303030207078}, sa_mask = {__val = {3475143045726351408, 2314885530819502128, 2314885530818453536, 8319937555149627424, 746872325959545721,
3775530756625032759, 3631650816742404144, 3472329422401517619, 3467895374536122416, 2319406791620833328, 3761104034442405222, 2314885530819704883, 2314885530818453536, 2314885530818453536, 4069054363051241248,
139789281265312}}, sa_flags = 65, sa_restorer = 0x7f233a740700}
sigs = {__val = {32, 0 <repeats 15 times>}}
#2 0x00007f2388c071b4 in __libc_message ([email protected]=2, [email protected]=0x7f2388cf9cb3 "*** %s ***: %s terminated\n") at ../sysdeps/posix/libc_fatal.c:175
ap = {{gp_offset = 32, fp_offset = 32547, overflow_arg_area = 0x7f233a740710, reg_save_area = 0x7f233a7406a0}}
fd = 2
on_2 = <optimized out>
list = <optimized out>
nlist = <optimized out>
cp = <optimized out>
written = <optimized out>
#3 0x00007f2388c8caa7 in __GI___fortify_fail ([email protected]=0x7f2388cf9c4a "buffer overflow detected") at fortify_fail.c:31
No locals.
#4 0x00007f2388c8acc0 in __GI___chk_fail () at chk_fail.c:28
No locals.
#5 0x00007f2388c8ca17 in __fdelt_chk (d=<optimized out>) at fdelt_chk.c:25
No locals.
#6 0x00007f23822184c5 in ?? () from /usr/lib/libldns.so.1
No symbol table info available.
#7 0x0000000000000000 in ?? ()
No symbol table info available.
(gdb)
三、FS日志檢視結果
[email protected]:~/DATA/logs/freeswitch$ grep "8e660ca2-d28a-4f09-a6f7-260bd25b75f4" freeswitch.log
8e660ca2-d28a-4f09-a6f7-260bd25b75f4 2018-04-10 09:54:10.237472 [NOTICE] switch_channel.c:1104 New Channel sofia/internal/[email protected]:53 [8e660ca2-d28a-4f09-a6f7-260bd25b75f4]
8e660ca2-d28a-4f09-a6f7-260bd25b75f4 2018-04-10 09:54:10.357473 [INFO] mod_dialplan_xml.c:637 Processing test <test>->test in context default
8e660ca2-d28a-4f09-a6f7-260bd25b75f4 2018-04-10 09:54:10.377454 [NOTICE] switch_ivr.c:2172 Transfer sofia/internal/[email protected]:53 to enum[test@default]
[email protected]:~/DATA/logs/freeswitch$
四、問題定位
- 通過堆棧可以看出libldns庫，通過fs中的日志可以看到最後執行的一行是:mod_enum這個子產品下的enum指令後才crash，開始進行漫天的search，終于發現一些端倪;
- 首先發現有人在FS中報了這樣一個jira單子:https://freeswitch.org/jira/browse/FS-7624?attachmentViewMode=list
- FS作者向ldns庫作者提了這樣一個問題:https://www.nlnetlabs.nl/bugs-script/show_bug.cgi?id=678
- ldns作者做了這樣一個patch: https://www.nlnetlabs.nl/bugs-script/attachment.cgi?id=285&action=diff
五、問題解決
-
接作者所說增加宏定義,FD_SETSIZE 自己想要擴充的值
From your back trace I see that the crash happens in ldns_sock_wait which uses select to wait for a socket to become readable or writable. The maximum number of sockets fed to select is FD_SETSIZE which is 1024 by default. In the issue report I read that this crash only occurs when the number of file descriptors in use is more than 1024.
-
直接更新debian8上的ldns庫至1.7.0版本解決問題
https://git.nlnetlabs.nl/ldns/tree/Changelog?h=release-1.7.0&id=54822adfc9fffbe47107c1201df5bca917793fa4 中的bugfix #678: Use poll i.s.o. select to support > 1024 fds 這一條即是對本BUG的修複内容
- 但是1.7.0在debian8的發行版本裡沒有,最新的也隻有1.6.18,是以隻能自己編譯依賴
- 先進入/usr/lib/freeswitch/mod目錄下檢視mod_enum.so對ldns的依賴,如下:
可以看到,第二項就是對其的依賴。/usr/lib/freeswitch/mod# ldd mod_enum.so linux-vdso.so.1 (0x00007ffde5fc5000) libldns.so.1 => /usr/lib/libldns.so.1 (0x00007f1c4e9f6000) libfreeswitch.so.1 => /usr/lib/libfreeswitch.so.1 (0x00007f1c4e351000) libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f1c4e0f0000) libcrypto.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f1c4dcf4000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1c4dad7000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1c4d72c000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1c4d528000) libpq.so.5 => /usr/lib/x86_64-linux-gnu/libpq.so.5 (0x00007f1c4d2f7000) libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f1c4d0f2000) libsqlite3.so.0 => /usr/lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f1c4ce29000) libfreetype.so.6 => /usr/lib/x86_64-linux-gnu/libfreetype.so.6 (0x00007f1c4cb7f000) libcurl.so.4 => /usr/lib/x86_64-linux-gnu/libcurl.so.4 (0x00007f1c4c90b000) libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f1c4c69d000) libspeex.so.1 => /usr/lib/x86_64-linux-gnu/libspeex.so.1 (0x00007f1c4c484000) libspeexdsp.so.1 => /usr/lib/x86_64-linux-gnu/libspeexdsp.so.1 (0x00007f1c4c271000) libedit.so.2 => /usr/lib/x86_64-linux-gnu/libedit.so.2 (0x00007f1c4c038000) libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f1c4be01000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f1c4bbf9000) libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f1c4b9de000) libpng16.so.16 => /lib/x86_64-linux-gnu/libpng16.so.16 (0x00007f1c4b7ab000) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f1c4b4a0000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1c4b19f000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f1c4af89000) libodbc.so.2 => /usr/lib/x86_64-linux-gnu/libodbc.so.2 (0x00007f1c4ad21000) /lib64/ld-linux-x86-64.so.2 (0x00007f1c4ee5b000) libgssapi_krb5.so.2 => /usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f1c4aad6000) libldap_r-2.4.so.2 => /usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f1c4a884000) libidn.so.11 => /usr/lib/x86_64-linux-gnu/libidn.so.11 (0x00007f1c4a650000) librtmp.so.1 => /usr/lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f1c4a433000) libssh2.so.1 => /usr/lib/x86_64-linux-gnu/libssh2.so.1 (0x00007f1c4a20a000) libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f1c49f36000) libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f1c49d05000) libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f1c49b01000) liblber-2.4.so.2 => /usr/lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f1c498f2000) libbsd.so.0 => /lib/x86_64-linux-gnu/libbsd.so.0 (0x00007f1c496e2000) libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f1c494b8000) libltdl.so.7 => /usr/lib/x86_64-linux-gnu/libltdl.so.7 (0x00007f1c492ae000) libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f1c490a2000) libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f1c48e9e000) libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f1c48c87000) libsasl2.so.2 => /usr/lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f1c48a6b000) libgnutls-deb0.so.28 => /usr/lib/x86_64-linux-gnu/libgnutls-deb0.so.28 (0x00007f1c4874b000) libhogweed.so.2 => /usr/lib/x86_64-linux-gnu/libhogweed.so.2 (0x00007f1c4851c000) libnettle.so.4 => /usr/lib/x86_64-linux-gnu/libnettle.so.4 (0x00007f1c482ea000) libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f1c48067000) libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x00007f1c47d85000) libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f1c47b3f000) libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f1c4792b000) libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x00007f1c47719000) libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f1c47511000)
六、系統無污染替換方法
# 呼叫中心-媒體服務底層依賴子產品替換方法
## 下載下傳ldns源碼
1. http://www.linuxfromscratch.org/blfs/view/svn/basicnet/ldns.html
2. cd /home/popo/freeswitch/src
3. wget http://www.nlnetlabs.nl/downloads/ldns/ldns-1.7.0.tar.gz
4. wget http://www.openssl.org/source/openssl-1.1.0c.tar.gz
## 安裝openssl
1. cd /home/popo/freeswitch/bin && mkdir openssl-1.1.0c && mkdir ldns-1.7.0
2. 編譯openssl高版本:
./config --prefix=/home/popo/freeswitch/bin/openssl-1.1.0c/openssl --openssldir=/home/popo/freeswitch/bin/openssl-1.1.0c/ssl && make && make install
## 安裝ldns高版本庫
1. tar zxvf ldns-1.7.0.tar.gz && cd ldns-1.7.0
2. ./configure --prefix=/home/popo/freeswitch/bin/ldns-1.7.0 --with-ssl=/home/popo/freeswitch/bin/openssl-1.1.0c/openssl && make && make install
3. cd /home/popo/freeswitch/bin/ldns-1.7.0/lib
4. ln -s libldns.so.2.0.0 libldns.so.1
## 配置使用者環境變量
1. cd ~
2. vim .profile
3. 打開此檔案添加如下：
PATH=/home/popo/freeswitch/bin/openssl-1.1.0c/openssl/bin:$PATH
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/popo/freeswitch/bin/openssl-1.1.0c/openssl/lib
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/popo/freeswitch/bin/ldns-1.7.0/lib
export PATH LD_LIBRARY_PATH
4. . .profile 使生效
5. 校驗openssl是否生效: openssl version
6. 檢視環境變量是否生效: env
## 重新開機FS使其mod_enum子產品所依賴的ldns庫生效
1. sudo /etc/freeswitch restart
2. ldd /usr/lib/freeswitch/mod/mod_enum.so
## 回退方法
1. 删除環境變量 .profile 中的新增配置項
2. 重新開機FS複原庫依賴 sudo /etc/freeswitch restart
over!
![linux-svn解除安裝與安裝[圖]](data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=)