在tomcat中安裝logback插件，此插件用于生成日志。該日志中包括很多資訊，我們的目的是将需要的資料進行整理，将其插入到mysql資料庫中，并将其做成定時任務執行。
一下是原日志檔案内容：
我們需要client,uniquecode,device,versioncode,interface,createtime這些字段
思路如下：
1.我們隻需要對包含接口和請求開始字段的行進行資料整理。
grep "personal/checkupdate.json請求開始" /home/logs/fight_cancer_interface/info.2014-05-19.log
2.我們要确定這些行中必須包括client,uniquecode,device,versioncode,interface,createtime這些字段。
grep "personal/checkupdate.json請求開始" /home/logs/fight_cancer_interface/info.2014-05-19.log|grep "client="|grep "uniquecode="|grep "device="|grep "client="|grep "versioncode="
3.需要将多餘的字段删除
grep "personal/checkupdate.json請求開始" /home/logs/fight_cancer_interface/info.2014-05-19.log|grep "client="|grep "uniquecode="|grep "device="|grep "client="|grep "versioncode="|awk -f"請求開始--:" '{print $1,$2}'|awk -f"info com.wyqj.fightcancer.intercepter.loginterceptor - /fight_cancer_interface/" '{print $1,$2}'|awk -f".json" '{print $1,$2}'|awk -f"[" '{print $1,"@@@###",$2}'|awk -f"]" '{print $1,"@@@###",$2}'|awk -f'@@@###' '{print $1,$3}'|sed 's/personal\/checkupdate/personal_checkupdate/g' | awk -f ",[[:digit:]][[:digit:]][[:digit:]]" '{print $1,$2}'|sed "s/ {/,/g"|sed "s/ personal/,personal/g"|sed "s/}//g"|awk -f"," '{print $6","$3","$5","$7","$2","$1");"}'|sed "s/,/\',\'/g"|sed "s/);/\');/g"|sed "s/^/\'/g"
4.将最前面加入資料庫插入語句
grep "personal/checkupdate.json請求開始" /home/logs/fight_cancer_interface/info.2014-05-18.log|grep "client="|grep "uniquecode="|grep "device="|grep "client="|grep "versioncode="|awk -f"請求開始--:" '{print $1,$2}'|awk -f"info com.wyqj.fightcancer.intercepter.loginterceptor - /fight_cancer_interface/" '{print $1,$2}'|awk -f".json" '{print $1,$2}'|awk -f"[" '{print $1,"@@@###",$2}'|awk -f"]" '{print $1,"@@@###",$2}'|awk -f'@@@###' '{print $1,$3}'|sed 's/personal\/checkupdate/personal_checkupdate/g' | awk -f ",[[:digit:]][[:digit:]][[:digit:]]" '{print $1,$2}'|sed "s/ {/,/g"|sed "s/ personal/,personal/g"|sed "s/}//g"|awk -f"," '{print $6","$3","$5","$7","$2","$1");"}'|sed "s/,/\',\'/g"|sed "s/);/\');/g"|sed "s/^/\'/g"|sed 's/^/insert into `fight_cancer`.`tb_statistics_base_info` (`client`,`uniquecode`,`device`,`versioncode`,`interface`,`createtime`) values (/g'|sed "s/client=//g"|sed "s/uniquecode=//g"|sed "s/device=//g" |sed "s/versioncode=//g"
5.将這些sql語句生成.sql檔案
grep "personal/checkupdate.json請求開始" /home/logs/fight_cancer_interface/info.2014-05-18.log|grep "client="|grep "uniquecode="|grep "device="|grep "client="|grep "versioncode="|awk -f"請求開始--:" '{print $1,$2}'|awk -f"info com.wyqj.fightcancer.intercepter.loginterceptor - /fight_cancer_interface/" '{print $1,$2}'|awk -f".json" '{print $1,$2}'|awk -f"[" '{print $1,"@@@###",$2}'|awk -f"]" '{print $1,"@@@###",$2}'|awk -f'@@@###' '{print $1,$3}'|sed 's/personal\/checkupdate/personal_checkupdate/g' | awk -f ",[[:digit:]][[:digit:]][[:digit:]]" '{print $1,$2}'|sed "s/ {/,/g"|sed "s/ personal/,personal/g"|sed "s/}//g"|awk -f"," '{print $6","$3","$5","$7","$2","$1");"}'|sed "s/,/\',\'/g"|sed "s/);/\');/g"|sed "s/^/\'/g"|sed 's/^/insert into `fight_cancer`.`tb_statistics_base_info` (`client`,`uniquecode`,`device`,`versioncode`,`interface`,`createtime`) values (/g'|sed "s/client=//g"|sed "s/uniquecode=//g"|sed "s/device=//g" |sed "s/versioncode=//g" > 2014-05-18.sql
6.将sql檔案生成資料庫
mysql -u使用者名 -p密碼 資料庫名 < 2014-05-18.sql
7.完善腳本
#!/bin/bash
#dates=`date +%y-%m-%d`
dates=`date -d"1 days ago" +%y-%m-%d`
path_log=/home/logs/fight_cancer_interface/info.$dates.log
path_sql=/home/sql/$dates.sql
grep "personal/checkupdate.json請求開始" $path_log|grep "client="|grep "uniquecode="|grep "device="|grep "client="|grep "versioncode="|awk -f"請求開始--:" '{print $1,$2}'|awk -f"info com.wyqj.fightcancer.intercepter.loginterceptor - /fight_cancer_interface/" '{print $1,$2}'|awk -f".json" '{print $1,$2}'|awk -f"[" '{print $1,"@@@###",$2}'|awk -f"]" '{print $1,"@@@###",$2}'|awk -f'@@@###' '{print $1,$3}'|sed 's/personal\/checkupdate/personal_checkupdate/g' | awk -f ",[[:digit:]][[:digit:]][[:digit:]]" '{print $1,$2}'|sed "s/ {/,/g"|sed "s/ personal/,personal/g"|sed "s/}//g"|awk -f"," '{print $6","$3","$5","$7","$2","$1");"}'|sed "s/,/\',\'/g"|sed "s/);/\');/g"|sed "s/^/\'/g"|sed 's/^/insert into `fight_cancer`.`tb_statistics_base_info` (`client`,`uniquecode`,`device`,`versioncode`,`interface`,`createtime`) values (/g'|sed "s/client=//g"|sed "s/uniquecode=//g"|sed "s/device=//g" |sed "s/versioncode=//g" > $path_sql
mysql -u使用者名 -p密碼 資料庫名 < $path_sql
8.将此腳本做成定時任務
crontab -e
[注]這裡我做了兩個定時任務，這裡需要注意的是，23點59分執行資料庫備份腳本，01點01分執行日志整理腳本。日志整理腳本必須在資料庫備份腳本執行之後執行。因為如果日志檔案過大，需要很長時間執行，如果在其還沒執行完就進行資料庫備份，可能會出現問題。
這個腳本需要熟練運用sed和awk，通過此腳本也看出了sed和awk功能的強大。我之前嘗試使用cut指令進行資料分割，都沒能如願。希望以後能更深入地學習sed和awk，并能夠掌握sed和awk程式設計。