網絡爬蟲學習第五彈：lxml庫的使用

lxml庫使用

我們可以利用python中的lxml庫來使用Xpath對HTML文檔進行搜尋。

選取節點：

nodename 選取此節點的所有子節點。

/ 從根節點選取。

// 從比對選擇的目前節點選擇文檔中的節點，而不考慮它們的位置。

. 選取目前節點。

… 選取目前節點的父節點。

@ 選取屬性。

from lxml import etree

text='''
<div>
<ul>
<li class="item-0"><a href="link1.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" >first item<a/></li>
<li class="item-1"><a href="link2.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" >second item<a/></li>
<li class="item-inactive"><a href="link3.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" >third item<a/></li>
<li class="item-1"><a href="link4.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" >fourth item</a></li>
<li class="item-0"><a href="link5.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" >fifth item</a>
</ul>
</div>
'''
# 利用lxml庫的etree子產品調用HTML類将html文本進行初始化，構造一個Xpath解析對象。
html=etree.HTML(text) 
result=etree.tostring(html) # 調用tostring()的方法可以輸出修正後的HTML代碼
print(type(result))
print(result.decode('utf-8')) # 結果是bytes類型，利用decode()将其轉化為str類型

#以下輸出結果中，殘缺的html文本被補全，并且自動添加了body、html節點

<class 'bytes'>
<html><body><div>
<ul>
<li class="item-0"><a href="link1.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" >first item</a><a/></li>
<li class="item-1"><a href="link2.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" >second item</a><a/></li>
<li class="item-inactive"><a href="link3.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" >third item</a><a/></li>
<li class="item-1"><a href="link4.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" >fourth item</a></li>
<li class="item-0"><a href="link5.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" >fifth item</a>
</li></ul>
</div>
</body></html>

# 也可以直接讀取檔案進行解析
from lxml import etree

html=etree.parse('./test.html',etree.HTMLParser())
result=etree.tostring(html)
print(result.decode('utf-8'))

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html><body><p>from lxml import etree&#13;
text = '''&#13;
</p><div>&#13;
    <ul>&#13;
         <li class="item-0"><a href="link1.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" >first item</a></li>&#13;
         <li class="item-1"><a href="link2.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" >second item</a></li>&#13;
         <li class="item-inactive"><a href="link3.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" >third item</a></li>&#13;
         <li class="item-1"><a href="link4.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" >fourth item</a></li>&#13;
         <li class="item-0"><a href="link5.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" >fifth item</a>&#13;
     </li></ul>&#13;
 </div>&#13;
'''&#13;
html = etree.HTML(text)&#13;
result = etree.tostring(html)&#13;
print(result)</body></html>

所有節點

一般用 // 開頭的XPath規則來選擇所有符合要求的節點。

from lxml import etree
'''這裡使用*來比對所有節點，整個html文本的所有節點都會被擷取。
   傳回一個清單，每個元素是Element類型，其後跟了節點的名稱。'''
html=etree.parse('./test.html',etree.HTMLParser())
result=html.xpath('//*') 
print(result)

[<Element html at 0x280b7673808>, <Element body at 0x280b764be48>, <Element p at 0x280b764bb88>, <Element div at 0x280b75d54c8>, <Element ul at 0x280b76736c8>, <Element li at 0x280b76738c8>, <Element a at 0x280b76735c8>, <Element li at 0x280b7673308>, <Element a at 0x280b7673848>, <Element li at 0x280b7673908>, <Element a at 0x280b7673508>, <Element li at 0x280b7673448>, <Element a at 0x280b7673408>, <Element li at 0x280b7673a48>, <Element a at 0x280b7673cc8>]

from lxml import etree

html=etree.parse('./test.html',etree.HTMLParser())
# 選取所有li節點，可以使用//，直接加上節點名稱即可，調用時直接使用xpath()方法
result=html.xpath('//li')
print(result)

[<Element li at 0x280b7626d08>, <Element li at 0x280b7673a08>, <Element li at 0x280b7673f48>, <Element li at 0x280b7673f88>, <Element li at 0x280b7673fc8>]

子節點

通過 / 或 // 來查找元素的節點或子孫節點

from lxml import etree

html=etree.parse('./test.html',etree.HTMLParser())
result=html.xpath('//li/a') # 查找li節點中的所有直接a子節點
print(result)

[<Element a at 0x280b75d5408>, <Element a at 0x280b7673f08>, <Element a at 0x280b7673dc8>, <Element a at 0x280b76735c8>, <Element a at 0x280b7673308>]

from lxml import etree

html=etree.parse('./test.html',etree.HTMLParser())
result=html.xpath('//ul//a') # 查找ul節點下的所有a節點。
print(result)
# 如果用//ul/a就不會傳回結果，因為a不是ul的直接節點

[<Element a at 0x280b764b448>, <Element a at 0x280b7673e48>, <Element a at 0x280b7673d08>, <Element a at 0x280b7673e08>, <Element a at 0x280b7673408>]

父節點

當我們知道了子節點後，我們可以用 … 來查找父節點

from lxml import etree

html=etree.parse('./test.html',etree.HTMLParser())
# 查找順序：<a href='link4.html'>-->(直接父節點<li>)-->直接屬性class的内容
result=html.xpath('//a[@href="link4.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" ]/../@class')
print(result)

['item-1']

from lxml import etree

html=etree.parse('./test.html',etree.HTMLParser())
result=html.xpath('//a[@href="link4.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" ]/parent::*/@class') #也可以用parent::來擷取
print(result)

['item-1']

屬性比對

我們可以通過@符号進行屬性過濾

from lxml import etree

html=etree.parse('./test.html',etree.HTMLParser())
result=html.xpath('//li[@class="item-0"]') # 選取屬性class為'item-0'的li節點
print(result)

[<Element li at 0x280b7671bc8>, <Element li at 0x280b7673f08>]

文本擷取

利用text()方法擷取節點中文本

from lxml import etree
# 擷取所有<li class='item-0'>節點的文本,法一：
html=etree.parse('./test.html',etree.HTMLParser())
# 因為文本内容實際在a節點内，是以先要進入a節點，再用text()方法進行提取文本
result=html.xpath('//li[@class="item-0"]/a/text()')
print(result)

['first item', 'fifth item']

from lxml import etree
# 擷取所有<li class='item-0'>節點的文本,法二：
html=etree.parse('./test.html',etree.HTMLParser())
#使用//擷取其節點下所有文本内容，其中\r\n是補全後li節點内的内容
result=html.xpath('//li[@class="item-0"]//text()')
print(result)

['first item', 'fifth item', '\r\n     ']

屬性擷取

同樣用@可以用來擷取屬性

from lxml import etree

html=etree.parse('./test.html',etree.HTMLParser())
result=html.xpath('//li/a/@href') # 比對所有li節點下a節點的href屬性
print(result)

['link1.html', 'link2.html', 'link3.html', 'link4.html', 'link5.html']

屬性多值比對

當某些節點的屬性包含多個值時，就無法單純的用xpath()方法比對，此時需要用contains()方法，第一個參數傳入屬性名稱，第二個參數傳入屬性值。

from lxml import etree

text='''
<li class="li li-first"><a href="link.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" >first item</a></li>
'''
html=etree.HTML(text)
result=html.xpath('//li[contains(@class,"li")]/a/text()')
print(result)

['first item']

多屬性比對

當多個屬性确定一個節點時，我們需要同時比對多個屬性，此時可以用運算符 and 來連接配接

from lxml import etree

text='''
<li class="li li-first" name="item"><a href="link.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" >first item</a></i>
'''
html=etree.HTML(text)
result=html.xpath('//li[contains(@class,li) and @name="item"]/a/text()')
print(result)

['first item']

按序選擇

from lxml import etree

text='''
<div>
    <ul>
         <li class="item-0"><a href="link1.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" >first item</a></li>
         <li class="item-1"><a href="link2.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" >second item</a></li>
         <li class="item-inactive"><a href="link3.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" ><span class="bold">third item</span></a></li>
         <li class="item-1"><a href="link4.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" >fourth item</a></li>
         <li class="item-0"><a href="link5.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" >fifth item</a></li>
     </ul>
 </div>
'''
html=etree.HTML(text)
result=html.xpath('//li[1]/a/text()') # 查找第一個li節點内的文本
print(result)
result=html.xpath('//li[last()]/a/text()') # 查找最後一個節點内的文本
print(result)
result=html.xpath('//li[position()<3]/a/text()') # 查找位置小于3的文本
print(result)
result=html.xpath('//li[last()-2]/a/text()') # 查找倒數第三位的節點
print(result)

['first item']
['fifth item']
['first item', 'second item']
[]

節點軸選擇

ancestor：比對目前節點的所有祖先節點

attribute：比對目前節點的所有屬性值

child：比對目前節點的所有子節點

decendant：比對目前節點的所有子孫節點

following：比對目前節點之後的所有節點

following-sibling：比對目前節點之後的所有同級節點

from lxml import etree

text='''
<div>
    <ul>
         <li class="item-0"><a href="link1.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" ><span>first item</span></a></li>
         <li class="item-1"><a href="link2.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" >second item</a></li>
         <li class="item-inactive"><a href="link3.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" ><span class="bold">third item</span></a></li>
         <li class="item-1"><a href="link4.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" >fourth item</a></li>
         <li class="item-0"><a href="link5.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" >fifth item</a></li>
     </ul>
 </div>
'''
html=etree.HTML(text)
result=html.xpath('//li[1]/ancestor::*') # 調用ancestor軸，比對第一個li節點的所有祖先節點
print(result)
result=html.xpath('//li[1]/ancestor::div') # 調用ancestor軸，比對第一個li節點的祖先節點中的div節點
print(result)
result=html.xpath('//li[1]/attribute::*') # 調用attribute軸，比對第一個li節點的所有屬性值
print(result)
result=html.xpath('//li[1]/child::a[@href="link1.html" target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow"  target="_blank" rel="external nofollow" ]') # 調用child軸，比對第一個li節點的中特定屬性的a節點
print(result)
result=html.xpath('//li[1]/descendant::span') # 調用descendant軸，比對第一個節點的所有子孫節點中的span節點
print(result)
result=html.xpath('//li[1]/following::*[2]') # 調用following軸，比對第一個節點的後續節點的第二個節點
print(result)
result=html.xpath('//li[1]/following-sibling::*') # 調用following-sibling軸，擷取第一個li節點的所有同級節點
print(result)

[<Element html at 0x280b765ee08>, <Element body at 0x280b7673408>, <Element div at 0x280b767c708>, <Element ul at 0x280b767cdc8>]
[<Element div at 0x280b767c708>]
['item-0']
[<Element a at 0x280b7687cc8>]
[<Element span at 0x280b7680208>]
[<Element a at 0x280b767c708>]
[<Element li at 0x280b7673408>, <Element li at 0x280b7680208>, <Element li at 0x280b7680248>, <Element li at 0x280b7680348>]

參考：崔慶才《python3網絡爬蟲開發實戰》

網絡爬蟲學習第五彈：lxml庫的使用

lxml庫使用

所有節點

子節點

父節點

屬性比對

文本擷取

屬性擷取

屬性多值比對

多屬性比對

按序選擇

節點軸選擇

繼續閱讀

來自python的【條件控制/語句循環/break/continue/else/pass】一、條件控制二、語句循環

無法解析的外部符号 wmain，該符号在函數 "void cdecl mainCRTStartupHelper(struct HINSTANCE *,unsigned short con......

TestLink導出用例轉換工具(XML2Excel)

YAML簡介和PyYAML安全操作YAML支援的類型YAML的優點：yaml的基本文法python操作

Small tricks

libsvm for python 安裝

學習軟體測試基礎測試第七天

Zeppelin 配置通路 REST APIApache Zeppelin Configuration REST API

【Torch】最簡潔logging使用指南

27. Remove Element(清單)題目代碼

Cloud Studio初體驗

使用 ctypes 進行 Python 和 C 的混合程式設計

【python】【資料處理】畫多元資料分布圖

【python】netconf協定對接管理裝置

「Python 網絡自動化」NETCONF —— Python 使用 NETCONF 管理配置 H3C 網絡裝置

在python中建立excel并寫入