Python爬虫 | 爬虫工具：BeautifulSoup

示例网站：名诗词句网

链接：https://www.shicimingju.com/

一、引用 BeautifulSoup以及requests包

from bs4 import BeautifulSoup
import requests
           

二、 获取目录页面的章节名称及其地址

headers = {
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4292.2 Safari/537.36'
}
url = 'https://www.shicimingju.com/book/sanguoyanyi.html'
page_text = requests.get(url=url,headers=headers).text
soup = BeautifulSoup(page_text,'lxml')
a_list = soup.select('.book-mulu > ul >li > a')
# 展示前十条数据
print(a_list[0:10])
           

[
	<a href="/book/sanguoyanyi/1.html" target="_blank" rel="external nofollow" >第一回·宴桃园豪杰三结义  斩黄巾英雄首立功</a>, 
	<a href="/book/sanguoyanyi/2.html" target="_blank" rel="external nofollow" >第二回·张翼德怒鞭督邮    何国舅谋诛宦竖</a>, 
	<a href="/book/sanguoyanyi/3.html" target="_blank" rel="external nofollow" >第三回·议温明董卓叱丁原  馈金珠李肃说吕布</a>, 
	<a href="/book/sanguoyanyi/4.html" target="_blank" rel="external nofollow" >第四回·废汉帝陈留践位    谋董贼孟德献刀</a>, 
	<a href="/book/sanguoyanyi/5.html" target="_blank" rel="external nofollow" >第五回·发矫诏诸镇应曹公  破关兵三英战吕布</a>, 
	<a href="/book/sanguoyanyi/6.html" target="_blank" rel="external nofollow" >第六回·焚金阙董卓行凶    匿玉玺孙坚背约</a>, 
	<a href="/book/sanguoyanyi/7.html" target="_blank" rel="external nofollow" >第七回·袁绍磐河战公孙    孙坚跨江击刘表</a>, 
	<a href="/book/sanguoyanyi/8.html" target="_blank" rel="external nofollow" >第八回·王司徒巧使连环计  董太师大闹凤仪亭</a>, 
	<a href="/book/sanguoyanyi/9.html" target="_blank" rel="external nofollow" >第九回·除暴凶吕布助司徒  犯长安李傕听贾诩</a>, 
	<a href="/book/sanguoyanyi/10.html" target="_blank" rel="external nofollow" >第一十回·勤王室马腾举义    报父仇曹操兴师</a>
]
           

三、获取章节的文本内容

with open('三国演义.txt','w',encoding='utf-8') as f:
    for a in a_list:
        title = a.string
        detail_url = 'https://www.shicimingju.com' + a['href']
        page_teat_detail = requests.get(url = detail_url,headers=headers).text
        detail_soup = BeautifulSoup(page_teat_detail,'lxml')
        content = detail_soup.find('div',class_='chapter_content').text
        f.write(title + content)
           

四、 爬取数据展示

第一章

最后一章