天天看點
傳回

scrapy爬取小說整站,并下載下傳圖檔儲存到本地,擷取本地位址存儲到對應得小說

首先我們來看看代碼吧,有些重點,我會單獨列出來

1: 建立項目scrapy startproject books

2: 建立spider檔案    scrapy genspider book quanshuwang.com            注: 本次項目案例爬取的是全書網

3: 更換目标的完整位址     http://www.quanshuwang.com

4: 我先說一下我這次的思路,本次爬取建立了資料庫,資料庫中有四張表

分别是:   分類表     小說介紹表      章節表     内容表

我們來看看都是怎麼實作的吧

import scrapy
from ..items import BooksItem


class BookSpider(scrapy.Spider):
    name = 'book'
    allowed_domains = ['quanshuwang.com']
    start_urls = ['http://www.quanshuwang.com']

    def __init__(self):
        self.count = 0

    def parse(self, response):
        # 擷取所有分類位址,目的所有分類下的所有小說
        classify_list_title = response.xpath("//nav[@class='channel-nav']//li/a/text()").extract()[:-1]
        for classify_title in classify_list_title:
            classify_title = classify_title                  # 擷取分類名稱
        classify_list_url = response.xpath("//nav[@class='channel-nav']//li/a/@href").extract()[:-1]
        for classify_url in classify_list_url:
            url = classify_url
            books_id = url.split("/")[-1].split("_")[-2]                  # 擷取分類id，目的關聯每部小說對應的分類
            yield scrapy.Request(url=url, callback=self.parse2, dont_filter=True, meta={'each_url': url})

第一部分主要擷取,分類位址,以及分類的id     目的:一對多,一個分類下有很多小說,我們拿到這個分類主要是對應每個分類下的所有小說,這裡沒有使用外鍵

def parse2(self, response):
        # 擷取小說位址,目的所有小說詳情資訊
        books_list_url = response.xpath('//ul[@class="seeWell cf"]/li/a/@href').extract()
        for books_url in books_list_url:
            url = books_url
            yield scrapy.Request(url=url, callback=self.parse3, dont_filter=True, meta={'each_url': url})
        next_page = response.xpath('//a[@class="next"]/@href').extract_first()
        if next_page is not None:
            next_page = response.urljoin(next_page)

            yield scrapy.Request(next_page, callback=self.parse2)

第二部分,我們隻擷取了小說詳情位址,因為我們不在這裡拿小說資訊

def parse3(self, response):
        item = BooksItem()
        # 擷取自己所需要的資料
        item['books_title'] = response.xpath("//div[@class='b-info']/h1/text()").extract_first()  # 名稱

        item['books_author'] = response.xpath("//dl[@class='bookso']/dd/text()").extract_first()  # 作者

        item['books_status'] = response.xpath("//dl/dd/text()").extract_first()  # 狀态

        books_introduce = response.xpath("//div[@id='waa']/text()").extract_first().split("介紹:")[-1].split(",")[0]  # 介紹
        item['books_introduce'] = books_introduce
        front_image_path = response.xpath("//a[@class='l mr11']/img/@src").extract_first()       # 圖檔位址
        # list = []
        # list.append(front_image_path)
        item['front_image_path'] = [front_image_path]
        item['books_classify_id'] = response.xpath("//div[@class='main-index']/a[2]/@href").extract_first().split("/")[-1].split("_")[-2]     #分類與小說關聯id
        item['books_chapter_id'] = response.xpath("//div[@class='b-oper']/a[1]/@href").extract_first().split("/")[-1]  # 小說與章節關聯id
        yield item
        section_list_url = response.xpath("//div[@class='b-oper']/a[1]/@href").extract()       # 章節清單位址
        for section_url in section_list_url:
            url = section_url
            yield scrapy.Request(url=url, callback=self.parse4, dont_filter=True, meta={'each_url': url})

第三部分,我們主要擷取小說自己所需要的内容

def parse4(self, response):
        # each_url = response.meta['each_url']
        # 擷取每章節名稱
        books_section_title = response.xpath("//div[@class='clearfix dirconone']/li/a/text()").extract()
        for section_title in books_section_title:
            chapter_title = section_title               # 擷取章節

        books_content_list_url = response.xpath("//div[@class='clearfix dirconone']/li/a/@href").extract()
        for book_content_url in books_content_list_url:
            url = book_content_url
            books_content_id = url.split("/")[-1].split(".")[-2]                    # 擷取小說章節對應内容章節

            yield scrapy.Request(url=url, callback=self.parse5, dont_filter=True, meta={'each_url': url})

第四部分: 主要是擷取小說章節名稱,以及内容位址

def parse5(self, response):
        each_url = response.meta['each_url']
        books_content_id = each_url.split('/')[-1].split(".")[-2]               # 内容id對應小說章節
        books_content_list = response.xpath("//div[@class='mainContenr']/text()").extract()
        content = ""
        for books_content in books_content_list:
            content += books_content
            content = content

最後一部分: 主要擷取小說内容

以上就是所有spider中所有代碼, 最後會來個完整的

接下我們看一下settings.py配置

# -*- coding: utf-8 -*-

# Scrapy settings for books project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     https://doc.scrapy.org/en/latest/topics/settings.html
#     https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
#     https://doc.scrapy.org/en/latest/topics/spider-middleware.html

import os
import sys

# sys.path.append(os.path.dirname(os.path.abspath('.')))
# os.environ['DJANGO_SETTINGS_MODULE'] = 'BooksAdmin.settings'

# import django
#
# django.setup()


BOT_NAME = 'books'

SPIDER_MODULES = ['books.spiders']
NEWSPIDER_MODULE = 'books.spiders'


# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'books (+http://www.yourdomain.com)'

# Obey robots.txt rules
ROBOTSTXT_OBEY = False

# Configure maximum concurrent requests performed by Scrapy (default: 16)
CONCURRENT_REQUESTS = 32

# Configure a delay for requests for the same website (default: 0)
# See https://doc.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
# DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16

# Disable cookies (enabled by default)
#COOKIES_ENABLED = False

# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False

# Override the default request headers:
# DEFAULT_REQUEST_HEADERS = {
#     'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
#     'Accept-Language': 'en',
#     'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36'
# }

# Enable or disable spider middlewares
# See https://doc.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
#    'books.middlewares.BooksSpiderMiddleware': 543,
#}

# Enable or disable downloader middlewares
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
# DOWNLOADER_MIDDLEWARES = {
#    'books.middlewares.BooksDownloaderMiddleware': 543,
# }

# Enable or disable extensions
# See https://doc.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
#    'scrapy.extensions.telnet.TelnetConsole': None,
#}

# Configure item pipelines
# See https://doc.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
    'books.pipelines.BooksPipeline': 300,
    'scrapy.pipelines.images.ImagesPipeline': 2,
    'books.pipelines.ArticleImagePipeline': 1,
}


IMAGES_URLS_FIELD = "front_image_path"        # image_url是在items.py中配置的網絡爬取得圖檔位址
#配置儲存本地的位址
project_dir = os.path.abspath(os.path.dirname(__file__))  #擷取目前爬蟲項目的絕對路徑
IMAGES_STORE = os.path.join(project_dir, 'images')  #組裝新的圖檔路徑
IMAGES_MIN_HEIGHT = 100                 #設定下載下傳圖檔的最小高度
IMAGES_MIN_WIDTH = 100                  #設定下載下傳圖檔的最小寬度
IMAGES_EXPIRES = 90  #90天内抓取的都不會被重抓
# Enable and configure the AutoThrottle extension (disabled by default)
# See https://doc.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False

# Enable and configure HTTP caching (disabled by default)
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
HTTPCACHE_ENABLED = True
HTTPCACHE_EXPIRATION_SECS = 0
HTTPCACHE_DIR = 'httpcache'
HTTPCACHE_IGNORE_HTTP_CODES = []
HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

都有注釋

接着item.py

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# https://doc.scrapy.org/en/latest/topics/items.html

import scrapy


class BooksItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    books_title = scrapy.Field()
    books_author = scrapy.Field()
    books_status = scrapy.Field()
    books_introduce = scrapy.Field()
    images_url = scrapy.Field()
    front_image_path = scrapy.Field()
    print("*****************************///", front_image_path)
    books_classify_id = scrapy.Field()
    books_chapter_id = scrapy.Field()
    pass

這裡我寫了小說資訊,其他的跟這個一樣

我們主要來看一下圖檔下載下傳以及擷取圖檔本地位址怎麼擷取

front_image_path = response.xpath("//a[@class='l mr11']/img/@src").extract_first()       # 圖檔位址
   
item['front_image_path'] = [front_image_path]

這是spider擷取圖檔的代碼,有兩種寫法,一種是上面這種,還有一種就是下面這種

front_image_path = response.xpath("//a[@class='l mr11']/img/@src").extract_first()       # 圖檔位址
list = []
list.append(front_image_path)
item['front_image_path'] = [list]

以上兩種方法都可以,第一種比較簡單

接下載下傳我們看看settings.py怎麼配置的

ITEM_PIPELINES = {
    'books.pipelines.BooksPipeline': 300,
    'scrapy.pipelines.images.ImagesPipeline': 2,
}


IMAGES_URLS_FIELD = "front_image_path"        # front_image_path是在items.py中配置的網絡爬取得圖檔位址
#配置儲存本地的位址
project_dir = os.path.abspath(os.path.dirname(__file__))  #擷取目前爬蟲項目的絕對路徑
IMAGES_STORE = os.path.join(project_dir, 'images')  #組裝新的圖檔路徑
IMAGES_MIN_HEIGHT = 100                 #設定下載下傳圖檔的最小高度
IMAGES_MIN_WIDTH = 100                  #設定下載下傳圖檔的最小寬度
IMAGES_EXPIRES = 90  #90天内抓取的都不會被重抓

重點: IMAGES_URLS_FIELD = "front_image_path"        # front_image_path是在items.py中配置的網絡爬取得圖檔位址一緻

#配置儲存本地的位址

project_dir = os.path.abspath(os.path.dirname(__file__))  #擷取目前爬蟲項目的絕對路徑

IMAGES_STORE = os.path.join(project_dir, 'images')  #組裝新的圖檔路徑

以上配置圖檔下載下傳就沒問題了,scrapy crawl book運作看一下

最後一個重點:  我們擷取本地圖檔的位址,我們要重寫pipeline

pipelines.py

#既然要重寫，記得提前引入
from scrapy.pipelines.images import ImagesPipeline

# 重載ImagePipeline中的item_completed方法，擷取下載下傳位址
class ArticleImagePipeline(ImagesPipeline):        
    def item_completed(self, results, item, info):
        for ok, value in results:        #通過斷點可以看到圖檔路徑存在results内
            images_url = value['path']                #将路徑儲存在item中傳回
            item['images_url'] = images_url
        return item

最後在items.py中會有一個

images_url = scrapy.Field()

注: 一定要在settings.py中設定

'books.pipelines.ArticleImagePipeline': 1,

不然擷取不到

完整代碼

spider.py

# -*- coding: utf-8 -*-
import scrapy
from ..items import BooksItem


class BookSpider(scrapy.Spider):
    name = 'book'
    allowed_domains = ['quanshuwang.com']
    start_urls = ['http://www.quanshuwang.com']

    def __init__(self):
        self.count = 0

    def parse(self, response):
        # 擷取所有分類位址,目的所有分類下的所有小說
        classify_list_title = response.xpath("//nav[@class='channel-nav']//li/a/text()").extract()[:-1]
        for classify_title in classify_list_title:
            classify_title = classify_title                  # 擷取分類名稱
        classify_list_url = response.xpath("//nav[@class='channel-nav']//li/a/@href").extract()[:-1]
        for classify_url in classify_list_url:
            url = classify_url
            books_id = url.split("/")[-1].split("_")[-2]                  # 擷取分類id，目的關聯每部小說對應的分類
            yield scrapy.Request(url=url, callback=self.parse2, dont_filter=True, meta={'each_url': url})

    def parse2(self, response):
        # 擷取小說位址,目的所有小說詳情資訊
        books_list_url = response.xpath('//ul[@class="seeWell cf"]/li/a/@href').extract()
        for books_url in books_list_url:
            url = books_url
            yield scrapy.Request(url=url, callback=self.parse3, dont_filter=True, meta={'each_url': url})
        next_page = response.xpath('//a[@class="next"]/@href').extract_first()
        if next_page is not None:
            next_page = response.urljoin(next_page)

            yield scrapy.Request(next_page, callback=self.parse2)

    def parse3(self, response):
        item = BooksItem()
        # 擷取自己所需要的資料
        item['books_title'] = response.xpath("//div[@class='b-info']/h1/text()").extract_first()  # 名稱

        item['books_author'] = response.xpath("//dl[@class='bookso']/dd/text()").extract_first()  # 作者

        item['books_status'] = response.xpath("//dl/dd/text()").extract_first()  # 狀态

        books_introduce = response.xpath("//div[@id='waa']/text()").extract_first().split("介紹:")[-1].split(",")[0]  # 介紹
        item['books_introduce'] = books_introduce
        front_image_path = response.xpath("//a[@class='l mr11']/img/@src").extract_first()       # 圖檔位址
        # list = []
        # list.append(front_image_path)
        item['front_image_path'] = [front_image_path]
        item['books_classify_id'] = response.xpath("//div[@class='main-index']/a[2]/@href").extract_first().split("/")[-1].split("_")[-2]     #分類與小說關聯id
        item['books_chapter_id'] = response.xpath("//div[@class='b-oper']/a[1]/@href").extract_first().split("/")[-1]  # 小說與章節關聯id
        yield item
        section_list_url = response.xpath("//div[@class='b-oper']/a[1]/@href").extract()       # 章節清單位址
        for section_url in section_list_url:
            url = section_url
            yield scrapy.Request(url=url, callback=self.parse4, dont_filter=True, meta={'each_url': url})

    def parse4(self, response):
        # each_url = response.meta['each_url']
        # 擷取每章節名稱
        books_section_title = response.xpath("//div[@class='clearfix dirconone']/li/a/text()").extract()
        for section_title in books_section_title:
            chapter_title = section_title               # 擷取章節

        books_content_list_url = response.xpath("//div[@class='clearfix dirconone']/li/a/@href").extract()
        for book_content_url in books_content_list_url:
            url = book_content_url
            books_content_id = url.split("/")[-1].split(".")[-2]                    # 擷取小說章節對應内容章節

            yield scrapy.Request(url=url, callback=self.parse5, dont_filter=True, meta={'each_url': url})

    def parse5(self, response):
        each_url = response.meta['each_url']
        books_content_id = each_url.split('/')[-1].split(".")[-2]               # 内容id對應小說章節
        books_content_list = response.xpath("//div[@class='mainContenr']/text()").extract()
        content = ""
        for books_content in books_content_list:
            content += books_content
            content = content

settings.py

# -*- coding: utf-8 -*-

# Scrapy settings for books project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     https://doc.scrapy.org/en/latest/topics/settings.html
#     https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
#     https://doc.scrapy.org/en/latest/topics/spider-middleware.html

import os
import sys

# sys.path.append(os.path.dirname(os.path.abspath('.')))
# os.environ['DJANGO_SETTINGS_MODULE'] = 'BooksAdmin.settings'

# import django
#
# django.setup()


BOT_NAME = 'books'

SPIDER_MODULES = ['books.spiders']
NEWSPIDER_MODULE = 'books.spiders'


# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'books (+http://www.yourdomain.com)'

# Obey robots.txt rules
ROBOTSTXT_OBEY = False

# Configure maximum concurrent requests performed by Scrapy (default: 16)
CONCURRENT_REQUESTS = 32

# Configure a delay for requests for the same website (default: 0)
# See https://doc.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
# DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16

# Disable cookies (enabled by default)
#COOKIES_ENABLED = False

# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False

# Override the default request headers:
# DEFAULT_REQUEST_HEADERS = {
#     'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
#     'Accept-Language': 'en',
#     'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36'
# }

# Enable or disable spider middlewares
# See https://doc.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
#    'books.middlewares.BooksSpiderMiddleware': 543,
#}

# Enable or disable downloader middlewares
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
# DOWNLOADER_MIDDLEWARES = {
#    'books.middlewares.BooksDownloaderMiddleware': 543,
# }

# Enable or disable extensions
# See https://doc.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
#    'scrapy.extensions.telnet.TelnetConsole': None,
#}

# Configure item pipelines
# See https://doc.scrapy.org/en/latest/topics/item-pipeline.html
ITEM_PIPELINES = {
    'books.pipelines.BooksPipeline': 300,
    'scrapy.pipelines.images.ImagesPipeline': 2,
    'books.pipelines.ArticleImagePipeline': 1,
}


IMAGES_URLS_FIELD = "front_image_path"        # front_image_path是在items.py中配置的網絡爬取得圖檔位址
#配置儲存本地的位址
project_dir = os.path.abspath(os.path.dirname(__file__))  #擷取目前爬蟲項目的絕對路徑
IMAGES_STORE = os.path.join(project_dir, 'images')  #組裝新的圖檔路徑
IMAGES_MIN_HEIGHT = 100                 #設定下載下傳圖檔的最小高度
IMAGES_MIN_WIDTH = 100                  #設定下載下傳圖檔的最小寬度
IMAGES_EXPIRES = 90  #90天内抓取的都不會被重抓
# Enable and configure the AutoThrottle extension (disabled by default)
# See https://doc.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False

# Enable and configure HTTP caching (disabled by default)
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
HTTPCACHE_ENABLED = True
HTTPCACHE_EXPIRATION_SECS = 0
HTTPCACHE_DIR = 'httpcache'
HTTPCACHE_IGNORE_HTTP_CODES = []
HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

items.py

# -*- coding: utf-8 -*-

# Define here the models for your scraped items
#
# See documentation in:
# https://doc.scrapy.org/en/latest/topics/items.html

import scrapy


class BooksItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    books_title = scrapy.Field()
    books_author = scrapy.Field()
    books_status = scrapy.Field()
    books_introduce = scrapy.Field()
    images_url = scrapy.Field()
    front_image_path = scrapy.Field()
    print("*****************************///", front_image_path)
    books_classify_id = scrapy.Field()
    books_chapter_id = scrapy.Field()
    pass

pipelines.py

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html
#既然要重寫，記得提前引入
from scrapy.pipelines.images import ImagesPipeline


class BooksPipeline(object):
    def process_item(self, item, spider):
        # print('打開資料庫')
        # item.save()  # 資料将會自動添加到指定的表
        # print('關閉資料庫')
        return item


class ArticleImagePipeline(ImagesPipeline):
    def item_completed(self, results, item, info):
        for ok, value in results:
            images_url = value['path']
            item['images_url'] = images_url
        return item

以上就是所有代碼

給大家看一下項目結構吧

ide html 下載下傳圖檔
上一篇: Autosar 網絡管理總結
下一篇: PHY晶片外圍配置說明

繼續閱讀