å ¥ä¾µæ£æµç³»ç»-æµéåæ
ââhttp://www.icir.org/vern/papers/bro-CN99.htmlââ
å ¶ä»å类项ç®SuricataãSnortÂ
Zeek (Bro) æ¯ä¸æ¬¾å¤§åé¼é¼çå¼æºç½ç»å®å ¨åæå·¥å ·ãéè¿ Zeek å¯ä»¥çæµç½ç»æµéä¸çå¯çæ´»å¨ï¼éè¿ Zeek çèæ¬å¯ä»¥å®ç°çµæ´»çåæåè½ï¼å¯æ¯å®ç°å¤ç§åè®®çå¼ç¸æºç¨çåæãæ¬æ主è¦æ¯å° Zeek ç»å被å¨æ«æå¨çä¸äºå®è·µçä»ç»ï¼ä»¥å Zeek é¨ç½²ç踩è¿çä¸äºåã
å®è£
Zeek çå®è£ è¿æ¯æ¯è¾ç®åçï¼ç¬è 主è¦æ¯å¨ Mac ä¸ä»¥å Linux ä¸å®è£ ãè¿ä¸¤ä¸ªæä½ç³»ç»çå®è£ æ¹å¼è¿æ¯æ¯è¾ç±»ä¼¼çãå¯¹äº Linux èè¨ï¼éè¦å®è£ ä¸äºä¾èµå ï¼
sudo yum install cmake make gcc gcc-c++ flex bison libpcap-devel openssl-devel python-devel swig zlib-devel
è¿éææéå°ä¸ä¸ªé®é¢å°±æ¯å¯è½ä½ ç Redhat éåæºéé¢æ²¡æå å« libpcap-develï¼å 为è¿ä¸ªå å¨å¯éçèå´å ï¼èå ç½çæå¡å¨å没æäºèç½è¿æ¥ãå¯ä»¥éè¿æå·¥ä¸è½½ç¸åºçæ¬ç libpcap 以å libpcap-devel å³å¯ã
Mac ä¸éè¦çä¾èµæ´å°ä¸ç¹ï¼é¦å éè¦ç¡®ä¿å®è£ äº xcode-selectï¼å¦æ没æå®è£ ï¼å¯ä»¥éè¿ xcode-select --install æ¥è¿è¡å®è£ ãMac ä¸åªéè¦å®è£ ä¾èµ cmake, swig, openssl, bison å³å¯ï¼å¯ä»¥éè¿ Homebrew æ¥è¿è¡å®è£ ã
ä¾èµå å®è£ å®æ¯ä¹åå°±å¯ä»¥å®è£ Zeekï¼å ¶å®æ¯å¯ä»¥éè¿å 管çå·¥å ·æ¥è¿è¡å®è£ çï¼ä¸è¿è¿éææ¨è使ç¨åºäºæºç çå®è£ æ¹å¼ï¼å®è£ æ¯è¾ç®åèä¸è¿å®¹æææ¥é®é¢ãä» Zeek ç Github ââReleaseââ å³å¯ä¸è½½æºç å ï¼ç®åæå®è£ çæ¯ 3.0.0 çæ¬ï¼æ³¨æä¸ç¹æ¯ï¼å¦æ使ç¨ææ°ççæ¬ï¼å¯è½éè¦ 7.0 以ä¸çæ¬ç cmakeï¼å 为éè¦ C++17 çè¯è¨ç¹æ§ãèä¸è¬éåæºé»è®¤ç cmake çæ¬æ¯4+çæ¬ï¼æ以å¦æä½ çæå¡å¨ä¹æ æ³ä¸äºèç½ï¼å»ºè®®å¯ä»¥å®è£ 3.0.0 çæ¬ã
./configure & make & make install
å®è£ 使ç¨ä¸é¢çå½ä»¤å°±å¯ä»¥äºï¼ä¸è¿ make çæ¶é´è¿æ¯æ¯è¾é¿çï¼è¿ä¸ªåå³äºä½ æºå¨çæ§è½ï¼ä¸è¿ä¸è¬å®è£ è¿æ¯éè¦å个å°æ¶å°ä¸ä¸ªå°æ¶ï¼è¿ä¹æ¯å 为 C++ ç¼è¯é度æ¯è¾æ ¢çåå ã
é群å®è£
é群å®è£ çæ¹å¼ååæºçæ¹å¼ä¸å¤ªä¸æ ·ãä¹åå¨æµè¯ç¯å¢ä½¿ç¨çé½æ¯åæºæ¨¡å¼ï¼é群åå¯ä»¥ç®¡çå¤ä¸ªå®ä¾ï¼åæ¥æä¹å°è¯äºéè¿é群çæ¹å¼æ¥è¿è¡å®è£ ãå¦æéè¦é ç½®é群çè¯ï¼å»ºè®®å®è£ PF_RINGï¼PF_RING å¯ä»¥å éç½ç»å çé度ãå¯¹äº Zeek é群ä¸çæ¯ä¸ª worker é½æ¯éè¦å®è£ PF_RINGï¼ä½åªéè¦å¨ manager ä¸å®è£ Zeek å°±å¯ä»¥äºï¼å¯ä»¥éè¿ zeekctl å¨å ¶å® worker ä¸å®è£ Zeekãä¸è¿éè¦ç¡®ä¿å¯ä»¥éè¿ ssh å°å ¶å® woker æºå¨ä¸ï¼å¯ä»¥éè¿å ¬é¥çå½¢å¼æ¥å®ç°ï¼å° manager çå ¬é¥æ¾å°å ¶å® worker ç authorized_keys ä¸ã
PF_RING çå®è£ æ¥éª¤ç¸å¯¹æ¥è¯´å¤äºä¸äºï¼ä½ä¹æ¯æç §è¯´æå®è£ å³å¯ãåä¸é¢çåæºå®è£ æ¹å¼ä¸åçæ¯é群å®è£ çæ¹å¼çæ¶åï¼å®è£ Zeek éè¦é ç½®åç¼ã
å®è£ PF_RING
tar xvzf PF_RING-5.6.2.tar.gz
cd PF_RING-5.6.2/userland/lib
./configure --prefix=/opt/pfring
make install
cd ../libpcap
./configure --prefix=/opt/pfring
make install
cd ../tcpdump-4.1.1
./configure --prefix=/opt/pfring
make install
cd ../../kernel
make
make install
modprobe pf_ring enable_tx_capture=0 min_num_slots=32768
å®è£ Zeek
./configure --with-pcap=/opt/pfring
make
make install
ç¡®ä¿ Zeek æ£ç¡®å ³èå°äº PF_RING ä¸ç libpcap åºä¸
ldd /usr/local/zeek/bin/zeek | grep pcap
libpcap.so.1 => /opt/pfring/lib/libpcap.so.1 (0x00007fa6d7d24000)
æ¥çå°±æ¯éè¿ PF_RING æ¥è¿è¡ Zeekctl çé ç½®ï¼Zeek çå®è£ è·¯å¾ä¸è¬é½å¨ /usr/local/zeekãéè¿ /usr/local/zeek/etc/node.cfg æ¥è¿è¡é群ç»ç¹çé ç½®ï¼å¨é群é ç½®ä¸ï¼manager, proxy 以å worker æ¯å¿ é¡»çï¼å¦æä¸è®¾ç½® loggerï¼é»è®¤å° manager ä½ä¸º loggerã
[worker-1]
type=worker
host=10.0.0.50
interface=eth0
lb_method=pf_ring
lb_procs=10
pin_cpus=2,3,4,5,6,7,8,9,10,11
æ¥ä¸æ¥åªéè¦éè¿ zeekctl install å°±ä¼å¨å ¶å®å®ä¾ä¸æ¥è¿è¡å®è£ äºãå¦æå®è£ è¿ç¨ä¸åºç°äºé®é¢ï¼å¯ä»¥éè¿ zeekctl diag woker-1 æ¥ææ¥å ·ä½çåå ã
Zeek ç»å被å¨æ«æå¨çç©æ³
ä¸é¢è®²çé½æ¯ Zeek çå®è£ ï¼ä¸é¢èä¸ä¸ Zeek å被å¨æ«æå¨çç»åã被å¨æ«æå¨çææå¾å¾åå³äºæµéçè´¨éåæ°éï¼å¨æ们çå®é å®è·µä¸ï¼åç°éè¿ Zeek è·åçæµéå æ们被å¨æ«æå¨æµè¯æµéçç»å¤§ä¸é¨åãZeek å¯¹äº http 解æçæ¥å¿é½ä¼åå¨å¨ /usr/local/zeek/logs ä¸ãå¦æ Zeek æ¯å¯å¨ç¶æï¼é£ä¹ http.log çè·¯å¾ä¼å¨ /usr/local/zeel/logs/current ä¸ï¼èåå²æ¥å¿åä¼è¢«æå ãå¦æä½¿ç¨ Zeek å»æè·æµéçæ¶åï¼æ¥å¿å¾å¾ä¼å å¾å¤§çåå¨ï¼æ以è¦è®°å¾ä¿®æ¹ Zeek æ¥å¿çåå¨è·¯å¾ï¼å¦åå¾å®¹æå°±æç³»ç»çå¡æ»¡ã
éè¿èæ¬èªå®ä¹ http.log
http.log ä¸å ¶å®å·²ç»å å«äºä¸°å¯çå段ï¼å¸¸è§çä¸äºå段å¦ä¸ï¼
# ts uid orig_h orig_p resp_h resp_p
1311627961.8 HSH4uV8KVJg 192.168.1.100 52303 192.150.187.43 80
ä¸è¿éé¢è¿æä¸äºä¿¡æ¯æ¯ç¼ºå¤±çï¼æ¯å¦ä¸äº http 请æ±å¤´ä»¥å POST 请æ±ç请æ±ä½ï¼ä¸ºäºæ·»å è¿äºå段ï¼å¯ä»¥éè¿èªå®ä¹ Zeek èæ¬æ¥å®ç°ï¼Zeek èæ¬çè½åççé常强大ï¼éè¿èæ¬å ¶å®æå¾å¤æ´é«çº§çç©æ³ã
æ·»å 请æ±å¤´
@load base/protocols/http/main
module HTTP;
export {
redef record Info += {
header_host: string &log &optional;
header_accept: string &log &optional;
header_accept_charset: string &log &optional;
header_accept_encoding: string &log &optional;
header_accept_language: string &log &optional;
header_accept_ranges: string &log &optional;
header_authorization: string &log &optional;
header_connection: string &log &optional;
header_cookie: string &log &optional;
header_content_length: string &log &optional;
header_content_type: string &log &optional;
};
}
event http_header(c: connection, is_orig: bool, name: string, value: string) &priority=3
{
if ( ! c?$http )
return;
if ( is_orig )
{
if ( log_client_header_names )
{
switch ( name ) {
case "HOST":
c$http$header_host = value;
break;
case "ACCEPT":
c$http$header_accept = value;
break;
case "ACCEPT-CHARSET":
c$http$header_accept_charset = value;
break;
case "ACCEPT-ENCODING":
c$http$header_accept_encoding = value;
break;
case "ACCEPT-LANGUAGE":
c$http$header_accept_language = value;
break;
case "ACCEPT-RANGES":
c$http$header_accept_ranges = value;
break;
case "AUTHORIZATION":
c$http$header_authorization = value;
break;
case "CONNECTION":
c$http$header_connection = value;
break;
case "COOKIE":
c$http$header_cookie = value;
break;
case "CONTENT-LENGTH":
c$http$header_content_length = value;
break;
case "CONTENT-TYPE":
c$http$header_content_type = value;
break;
}
}
}
}
æ·»å POST 请æ±ä½
export {
## The length of POST bodies to extract.
const http_post_body_length = 200 &redef;
}
redef record HTTP::Info += {
postdata: string &log &optional;
};
event log_post_bodies(f: fa_file, data: string)
{
for ( cid in f$conns )
{
local c: connection = f$conns[cid];
if ( ! c$http?$postdata )
c$http$postdata = "";
# If we are already above the captured size here, just return.
if ( |c$http$postdata| > http_post_body_length )
return;
c$http$postdata = c$http$postdata + data;
if ( |c$http$postdata| > http_post_body_length )
{
c$http$postdata = c$http$postdata[0:http_post_body_length] + "...";
}
}
}
event file_over_new_connection(f: fa_file, c: connection, is_orig: bool)
{
if ( is_orig && c?$http && c$http?$method && c$http$method == "POST" )
{
Files::add_analyzer(f, Files::ANALYZER_DATA_EVENT, [$stream_event=log_post_bodies]);
}
}
éè¿ä¸è¿°çèæ¬å°±å¯ä»¥æ·»å ä¸äºè¯·æ±å¤´ä»¥å POST 请æ±ç请æ±ä½ï¼å®æ´çèæ¬å¯ä»¥åè ââhttp-customââãèæ¬ç¼åå®æ¯ï¼éè¦éè¿ zeekctl é¨ç½²æè½çæï¼æ¥éª¤ä¹é常ç®åã
mv http-custom /usr/local/bro/share/bro/base/protocols
echo '@load base/protocols/http-custom' >> /usr/local/bro/share/bro/site/local.bro
zeekctl deploy
对äºè¢«å¨æ«æå¨ï¼æ们ç®åçæ¹æ¡æ¯éè¿ Filebeat å»ééæ¥å¿ç¶åè¾åºç» Logstash åå¤çï¼å¤çå®æ¯ä¹ååè¾åºå° Kafkaã
Filebeat å Logstash éç¨äºå¤ç§åºæ¯ï¼å¨æ¥å¸¸çåç§æ¥å¿ééåºæ¯é½è½æ´¾ä¸ç¨åºãéè¿ Logstash å¯ä»¥å®ææ¥å¿çµæ´»çå¤çï¼å 为 Logstash éé¢å å«äºåç§ä¸°å¯çæ件ï¼å ä¹å¯ä»¥å®æ对äºæ¥å¿çä»»ä½æä½ãæ¯å¦ä¸ºäºä¿è¯ POST 请æ±ä½ä¿è¯ä¼ è¾çæ£ç¡®æ§ï¼å¯ä»¥éè¿ base64 æ¥è¿è¡ç¼ç ãéè¿Â ââlogstash-filter-base64ââ å¯ä»¥éåå°å®ç°å段çç¼ç æè 解ç ãéè¿ filter ä¸ç mutate æ件å¯ä»¥å¢å å段æè å é¤å段ã
base64 {
field => "postdata"
action => "encode"
}
éè¿è¿ç§æ¹æ¡è¿æä¸ä¸ªä¼å¿å°±æ¯æ们è¿å¯ä»¥å°æ们çæ¥å¿è¾åºå°å«çå°æ¹ï¼æ¯å¦ esï¼è¿ä¸ªä¹å¯ä»¥æ¹ä¾¿åç»ææ¥æ¥å¿é®é¢ã
ä¸è¿æå¨åé¢ååç°äºä¸ç§æ°çæ¹æ¡ï¼å¯ä»¥éè¿ Zeek çæ件ï¼å° http.log ç´æ¥è¾åºå° Kafkaï¼è¿ä¸ªæ¹æ¡çä¼ç¹ä¸»è¦æ¯æ´é«æï¼åæ¶ä¹èçäºä¸äºææ¬ï¼æ¯ç« Logstash éè¦çæºå¨æ§è½è¿æ¯æ¯è¾å¤§çã对äºè¿ä¸ªæ¹æ¡ä¸»è¦æ¯ä¸¤ä¸ªé®é¢ï¼ç¬¬ä¸ä¸ªé®é¢æ¯é¦å éè¦å¤ç好æ¥å¿çæ ¼å¼ï¼è¿æ ·ä¿è¯åç»å¤çå°ä¾¿å©æ§ï¼ç¬¬äºä¸ªé®é¢æ¯å¦ä½å°æ¥å¿ç´æ¥ä» Zeek è¾åºå° Kafkaãå ¶å®ææ¯å 解å³äºç¬¬ä¸ä¸ªé®é¢å解å³ç¬¬äºä¸ªé®é¢çï¼å 为第äºä¸ªé®é¢çå¤ççæ¹å¼æ´çµæ´»ï¼å¾çäº Zeek èæ¬ç便å©æ§ï¼è¯å®æ¯å¯ä»¥å®ç°çã
ââmetron-bro-plugin-kafkaââÂ æ¯ Apache å®æ¹çä¸ä¸ª Bro çæ件ï¼ä¸è¿å 为 Zeek3.0.0 æ¯å¯ä»¥å ¼å®¹çï¼æ以è¿ä¸ªæ件æ¯å¯ä»¥ä½¿ç¨çãè¿ä¸ªæ件æ两ç§å®è£ æ¹å¼ï¼ä¸ç§æ¯éè¿ bro-pkg (Bro çå®æ¹å 管çå·¥å ·)æ¥è¿è¡å®è£ ï¼å¦å¤ä¸ç§åæ¯éè¿æå·¥å®è£ ãç±äºç½ç»çåå ï¼ææ´æ¨è使ç¨æå·¥å®è£ çæ¹å¼ï¼æå°è¯éè¿ bro-pkg çæ¹å¼æ¥è¿è¡å®è£ ï¼é度ç¹å«æ ¢ã
å®è£ librdkafka
curl -L https://github.com/edenhill/librdkafka/archive/v0.11.5.tar.gz | tar xvzcd librdkafka-0.11.5/
./configure --enable-sasl
make
sudo make install
å®è£ æ件
./configure --bro-dist=$BRO_SRC
make
sudo make install
è¿éæä¸ä¸ªåå°±æ¯å®è£ ææ¡£æ ¹æ¬å°±æ²¡æ说 $BRO_SRC æ¯åªä¸ªè·¯å¾ï¼æ以å®è£ çæ¶åæ»æ¯æ¥éï¼åæ¥æå¼æ¸ æ¥è¿ä¸ªè·¯å¾å ¶å®å°±æ¯å½å Zeek 解ååçè·¯å¾ï¼å³ Zeek å®è£ å çè·¯å¾ã
éªè¯ç»æ
zeek -N Apache::Kafka
Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3)
æ¥çå°±æ¯å° http çæ¥å¿è¿è¡å¤çï¼å 为å¨åå§ç http.log ä¸æè¿å¤å段æ¯æ们并ä¸éè¦çãå¨ç 究äºå®æ¹ææ¡£ä¹åï¼å¯ä»¥éè¿Â ââFiltersââ å¯ä»¥å®ä¹ä¸ä¸ªæ°çæ¥å¿æ件ï¼å¯ä»¥æ·è´å ¶å®çæ¥å¿è¾åºå°æ°çæ件ï¼å¯ä»¥èªå®ä¹å段ï¼æ¹å¼æ¯è¾çµæ´»ãå¦å¤è¿å¯ä»¥éè¿ Writer å¯ä»¥å°æ¥å¿åå ¥å° sqlite æ°æ®åºä¸ãä¸è¿ï¼è¿éæ们主è¦æ¯éè¿æ件å°æ¥å¿åå ¥å° Kafkaã
æ们çç®æ æ¯è·å http.log ä¸çé¨åå段ï¼æ以å¯ä»¥éè¿ Filters æ¥å®ç°æ¥å¿æ件çå¤å¶å¹¶ä¸å¯¹æ¥å¿å段è¿è¡è¿æ»¤ï¼åºäº KafkaWriter å°æ¥å¿æ件ç´æ¥åå ¥å° Kafka ä¸ã为äºå®ä¹ Filterï¼å¨Â â
â/usr/local/zeek/share/zeek/base/protocols/http/main.zeekâ
â ç zeek_init å½æ°ä¸è¿è¡å®ä¹ï¼
event zeek_init() &priority=5
{
Log::create_stream(HTTP::LOG, [$columns=Info, $ev=log_http, $path="http"]);
Analyzer::register_for_ports(Analyzer::ANALYZER_HTTP, ports);
local filter: Log::Filter =
[
$name="kafka-http",
$include=set("host","id.resp_p","uri"),
$writer=Log::WRITER_KAFKAWRITER
];
Log::add_filter(HTTP::LOG, filter);
}
redef Kafka::topic_name = "bro-test";
redef Kafka::kafka_conf = table(
["metadata.broker.list"] = "127.0.0.1:9092"
);