爬蟲必備良藥（一）——設定随機User-Agent

前言

最近在自學爬蟲，琢磨各種防封、防反爬蟲措施，于是乎想自己收集一些user-agent用來做随機header，偶然發現github上有一個專門提供随機ua的庫，遂記錄之

GitHub

名稱：fake-useragent

Demo

setting.py

DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddleware.useragent.UserAgentMiddleware': None,
    'ArticleSpider.middlewares.RandomUserAgentMiddleware': 543,
}
RANDOM_UA_TYPE="random"
           

middlewares.py
           

from fake_useragent import UserAgent

#####################################
    #這裡都是scrapy自動生成的代碼#
#####################################


class RandomUserAgentMiddleware(object):
    #設定随機User-Agent

    def __init__(self, crawler):
        super(RandomUserAgentMiddleware,self).__init__()
        self.ua=UserAgent()
        self.ua_type=crawler.settings.get("RANDOM_UA_TYPE","random")

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler)

    def process_request(self, request, spider):

        def get_ua():
            return getattr(self.ua,self.ua_type)

        request.headers.setdefault('User-Agent',get_ua())