天天看點

系統診斷小技巧(10):ssh/sshd的調試模式

網絡應用的邏輯一般都是集中在兩端。是以端到端的排查思路是解決問題的重要思路。排查ssh問題也是如此。

那麼,怎麼讓ssh兩端輸出更多資訊呢?

如何啟動調試選項?

ssh和sshd都有配置選項和指令行選項來啟動調試。兩者的配置選項是一樣的,都是 LogLevel。

關于配置項的具體用法,請參考

ssh_config(5)

sshd_config(5)

。我們下面隻讨論指令行選項。

ssh的調試選項-v

給ssh指令行加上-v選項就能啟動其調試模式。我們對比下有無-v選項的差別。

沒有-v選項時,ssh嚴格遵循了“沉默是金”的訓導

root@localhost:~$ ssh -i ~/.ssh/id_rsa [email protected] exit
root@localhost:~$           

有-v選項時ssh輸出如下

root@localhost:~$ ssh -v -i ~/.ssh/id_rsa [email protected] exit
OpenSSH_7.6p1 Ubuntu-4, OpenSSL 1.0.2n  7 Dec 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 101.201.155.113 [101.201.155.113] port 22.
debug1: Connection established.
debug1: identity file /home/demo/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory
debug1: identity file /home/demo/.ssh/id_rsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000000
debug1: Authenticating to 101.201.155.113:22 as 'root'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes128-ctr MAC: [email protected] compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: [email protected] compression: none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(2048<3072<8192) sent
debug1: got SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: got SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: ssh-rsa SHA256:/CxKtMSkLDdqyBNlGyjmpKtOB5YiiHxQmN8veGnvlgA
debug1: Host '101.201.155.113' is known and matches the RSA host key.
debug1: Found key in /home/demo/.ssh/known_hosts:1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug1: Next authentication method: gssapi-with-mic
debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1000)

debug1: Unspecified GSS failure.  Minor code may provide more information
No Kerberos credentials available (default cache: FILE:/tmp/krb5cc_1000)

debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:90VKHu3ur7IZpuMwuS5NoJkyLs5vmmbb2dqNFrM4c64 /home/nerd/.ssh/id_aliyun_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: Authentication succeeded (publickey).
Authenticated to 101.201.155.113 ([101.201.155.113]:22).
debug1: channel 0: new [client-session]
debug1: Requesting [email protected]
debug1: Entering interactive session.
debug1: pledge: network
debug1: Sending environment.
debug1: Sending env LC_MEASUREMENT = en_US.UTF-8
debug1: Sending env LC_PAPER = en_US.UTF-8
debug1: Sending env LC_MONETARY = en_US.UTF-8
debug1: Sending env LANG = en_US.UTF-8
debug1: Sending env LC_NAME = en_US.UTF-8
debug1: Sending env LC_ADDRESS = en_US.UTF-8
debug1: Sending env LC_NUMERIC = en_US.UTF-8
debug1: Sending env LC_TELEPHONE = en_US.UTF-8
debug1: Sending env LC_IDENTIFICATION = en_US.UTF-8
debug1: Sending env LC_TIME = en_US.UTF-8
debug1: Sending command: exit
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype [email protected] reply 0
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 3640, received 2848 bytes, in 0.0 seconds
Bytes per second: sent 189533.1, received 148294.0
debug1: Exit status 0
root@localhost:~$           

可見,使用-v選項時,ssh會給出豐富的互動過程資訊。這對排查問題無疑大有幫助。

sshd的調試選項-d

給sshd加上-d選項既能啟動其調試模式。對比下

沒有-d選項的情形

ssh端的輸出

ali-4c3275957b8b:~ $ ssh -i ~/.ssh/id_aliyun_rsa [email protected] exit
root@localhost:~ $           

sshd端的輸出

root@iZ2zeecuf3v6yjfdp82unlZ:~#$(which sshd) -D
Could not load host key: /etc/ssh/ssh_host_ed25519_key
  C-c C-c
root@iZ2zeecuf3v6yjfdp82unlZ:~#           

有-d選項的情形

ssh端

root@localhost:~ $ ssh -i ~/.ssh/id_aliyun_rsa [email protected] exit
debug1: SELinux support disabled
debug1: PAM: reinitializing credentials
debug1: permanently_set_uid: 0/0
Environment:
  LANG=en_US.UTF-8
  USER=root
  LOGNAME=root
  HOME=/root
  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
  MAIL=/var/mail/root
  SHELL=/bin/bash
  SSH_CLIENT=xxx.xxx.xxx.xxx 1513 22
  SSH_CONNECTION=xxx.xxx.xxx.xxx 1513 xxx.xxx.xxx.xxx 22
root@localhost:~ $           

sshd端的情形

root@localhost:~ $(which sshd) -D -d
debug1: sshd version OpenSSH_7.2, OpenSSL 1.0.2g  1 Mar 2016
debug1: private host key #0: ssh-rsa SHA256:ljPn4MtPW06wd2I2raiiMjgIc8BkuawPLC/XS90+x2w
debug1: private host key #1: ssh-dss SHA256:2uuXiQWHJf28G8fCSL14aj+eDweQyQzoUgWrm4l6dLE
debug1: private host key #2: ecdsa-sha2-nistp256 SHA256:JWEiiREibEzjiT6L9lxnvIHJ+CI7DhRQMHCR/boLF5o
debug1: key_load_private: No such file or directory
debug1: key_load_public: No such file or directory
Could not load host key: /etc/ssh/ssh_host_ed25519_key
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-D'
debug1: rexec_argv[2]='-d'
debug1: Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 4 out 4 newsock 4 pipe -1 sock 7
debug1: inetd sockets after dupping: 3, 3
Connection from 106.11.34.13 port 1513 on 172.17.165.12 port 22
debug1: Client protocol version 2.0; client software version OpenSSH_7.6
debug1: match: OpenSSH_7.6 pat OpenSSH* compat 0x04000000
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.4
debug1: permanently_set_uid: 109/65534 [preauth]
debug1: list_hostkey_types: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug1: kex: algorithm: [email protected] [preauth]
debug1: kex: host key algorithm: ecdsa-sha2-nistp256 [preauth]
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none [preauth]
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none [preauth]
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
debug1: rekey after 134217728 blocks [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
debug1: rekey after 134217728 blocks [preauth]
debug1: SSH2_MSG_NEWKEYS received [preauth]
debug1: KEX done [preauth]
debug1: userauth-request for user root service ssh-connection method none [preauth]
debug1: attempt 0 failures 0 [preauth]
debug1: PAM: initializing for "root"
debug1: PAM: setting PAM_RHOST to "106.11.34.13"
debug1: PAM: setting PAM_TTY to "ssh"
debug1: userauth-request for user root service ssh-connection method publickey [preauth]
debug1: attempt 1 failures 0 [preauth]
debug1: userauth_pubkey: test whether pkalg/pkblob are acceptable for RSA SHA256:90VKHu3ur7IZpuMwuS5NoJkyLs5vmmbb2dqNFrM4c64 [preauth]
debug1: temporarily_use_uid: 0/0 (e=0/0)
debug1: trying public key file /root/.ssh/authorized_keys
debug1: fd 4 clearing O_NONBLOCK
debug1: matching key found: file /root/.ssh/authorized_keys, line 1 RSA SHA256:90VKHu3ur7IZpuMwuS5NoJkyLs5vmmbb2dqNFrM4c64
debug1: restore_uid: 0/0
Postponed publickey for root from 106.11.34.13 port 1513 ssh2 [preauth]
debug1: userauth-request for user root service ssh-connection method publickey [preauth]
debug1: attempt 2 failures 0 [preauth]
debug1: temporarily_use_uid: 0/0 (e=0/0)
debug1: trying public key file /root/.ssh/authorized_keys
debug1: fd 4 clearing O_NONBLOCK
debug1: matching key found: file /root/.ssh/authorized_keys, line 1 RSA SHA256:90VKHu3ur7IZpuMwuS5NoJkyLs5vmmbb2dqNFrM4c64
debug1: restore_uid: 0/0
debug1: do_pam_account: called
Accepted publickey for root from 106.11.34.13 port 1513 ssh2: RSA SHA256:90VKHu3ur7IZpuMwuS5NoJkyLs5vmmbb2dqNFrM4c64
debug1: monitor_child_preauth: root has been authenticated by privileged process
debug1: monitor_read_log: child log fd closed
debug1: PAM: establishing credentials
debug1: rekey after 134217728 blocks
debug1: rekey after 134217728 blocks
debug1: ssh_packet_set_postauth: called
debug1: Entering interactive session for SSH2.
debug1: server_init_dispatch_20
debug1: server_input_channel_open: ctype session rchan 0 win 2097152 max 32768
debug1: input_session_request
debug1: channel 0: new [server-session]
debug1: session_new: session 0
debug1: session_open: channel 0
debug1: session_open: session 0: link with channel 0
debug1: server_input_channel_open: confirm session
debug1: server_input_global_request: rtype [email protected] want_reply 0
debug1: server_input_channel_req: channel 0 request env reply 0
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req env
debug1: server_input_channel_req: channel 0 request exec reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req exec
Starting session: command for root from 106.11.34.13 port 1513 id 0
debug1: Received SIGCHLD.
debug1: session_by_pid: pid 8729
debug1: session_exit_message: session 0 channel 0 pid 8729
debug1: session_exit_message: release channel 0
debug1: session_by_channel: session 0 channel 0
debug1: session_close_by_channel: channel 0 child 0
Close session: user root from 106.11.34.13 port 1513 id 0
debug1: channel 0: free: server-session, nchannels 1
Received disconnect from 106.11.34.13 port 1513:11: disconnected by user
Disconnected from 106.11.34.13 port 1513
debug1: do_cleanup
debug1: PAM: cleanup
debug1: PAM: closing session
debug1: PAM: deleting credentials
debug1: audit_event: unhandled event 12
root@localhost:~#$           

但是,sshd是以守護程序方式運作的,要修改其啟動腳本增加選項嗎?

如何在前端啟動sshd?

的确可以通過修改啟動腳本或者有關配置檔案來增加-d選項。比如在CentOS上可以通過修改/etc/sysconfig/sshd來實作這個目的。

但是更好的方案是使用指令行參數的方式。這需要把sshd啟動到前端。

上面的測試為了突出重點,我們故意忽略了一些準備和善後步驟。現在我們補充齊全

root@localhost:~ netstat -ltpn # 1. 檢查sshd是否在運作
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      978/sshd
tcp        0      0 127.0.0.1:32000         0.0.0.0:*               LISTEN      918/java
root@localhost:~ service sshd stop # 2. 暫停sshd的運作
root@localhost:~ netstat -ltpn # 3. 确認sshd已經關閉
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:32000         0.0.0.0:*               LISTEN      918/java
root@localhost:~ $(which sshd) -D -d # 4. 前端啟動sshd,并且啟動調試模式
... ...           

ssh/sshd兩端排查的步驟

重要

關閉sshd前要考慮會不會把自己關在系統外。沒有辦法登入系統可不是個好消息。如果root登入系統隻能使用ssh,則請做好準備前,務必不要關閉sshd。

阿裡雲的執行個體可以在官網控制台通過VNC登入系統。使用VNC通道請提前準備好VNC密碼和系統的密碼登入方式,并且驗證VNC通道工作符合預期。

步驟

sshd端的步驟

# 1. 檢查sshd是否在運作
netstat -ltpn
# 2. 關閉sshd
service sshd stop
# 3. 檢查sshd是否已經關閉
netstat -ltpn
# 4. 啟動到前端并且啟動調試模式
$(which sshd) -D -d # -D確定sshd在前端運作
# 5. 排查、測試、驗證 ... ...
# 6. 重新啟動sshd
service sshd start           

ssh端的步驟

ssh -vvv login_name@sshd_host           

探索愉快。