中間件位于引擎與下載下傳器、引擎與spider之間，是處理scrapy中兩個重要對象Request、Response及資料資料對象Item的重要的擴充。
那麼中間件分類兩類就不難了解了，其中一類在引擎與下載下傳器之間我們可以稱之為下載下傳中間件、另一個在引擎與spider之間我們可以稱之為爬蟲中間件；下載下傳中間件和spider中間件都對Request、Response請求處理，根據位置不同，他們主要負責的職能也不同。
spider中間件（主職過濾）對Request、Response的主要作用在過濾，可以對特定路徑的URL請求丢棄、對特定頁面響應過濾、同時對一些不含有指定資訊的item過濾，當然pipeline也能實作item的過濾。
下載下傳中間件（主職加工）主要作用是加工，如給Request添加代理、添加UA、添加cookie，對Response傳回資料編碼解碼、壓縮解壓縮、格式化等預處理。
下面我們将從這兩個中間件出發來詳細講解其中奧妙。
spider中間件
在建立scrapy項目後會自動在middlewares.py檔案下生成一個spider中間件和下載下傳中間件模闆，檢視代碼：
class ProxyExampleSpiderMiddleware(object):
# Not all methods need to be defined. If a method is not defined,
# scrapy acts as if the spider middleware does not modify the
# passed objects.
@classmethod
def from_crawler(cls, crawler):
# This method is used by Scrapy to create your spiders.
s = cls()
crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
return s
def process_spider_input(self, response, spider):
# Called for each response that goes through the spider
# middleware and into the spider.
# Should return None or raise an exception.
return None
def process_spider_output(self, response, result, spider):
# Called with the results returned from the Spider, after
# it has processed the response.
# Must return an iterable of Request, dict or Item objects.
for i in result:
yield i
def process_spider_exception(self, response, exception, spider):
# Called when a spider or process_spider_input() method
# (from other spider middleware) raises an exception.
# Should return either None or an iterable of Response, dict
# or Item objects.
pass
def process_start_requests(self, start_requests, spider):
# Called with the start requests of the spider, and works
# similarly to the process_spider_output() method, except
# that it doesn’t have a response associated.
# Must return only requests (not items).
for r in start_requests:
yield r
def spider_opened(self, spider):
spider.logger.info('Spider opened: %s' % spider.name)
-
process_spider_input(response, spider)
-
process_spider_output(response, result, spider)
-
process_spider_exception(response, exception, spider)
-
from_crawler(cls, crawler)
- 讀取配置檔案中的參數進行中間件配置
看一個scrapy内置的spider中間件源碼
scrapy.spidermiddlewares.httperror.HttpErrorMiddleware 過濾出所有失敗(錯誤)的HTTP response，是以spider不需要處理這些reques
class HttpErrorMiddleware(object):
@classmethod
def from_crawler(cls, crawler):
return cls(crawler.settings)
def __init__(self, settings):
self.handle_httpstatus_all = settings.getbool('HTTPERROR_ALLOW_ALL')
self.handle_httpstatus_list = settings.getlist('HTTPERROR_ALLOWED_CODES')
def process_spider_input(self, response, spider):
if 200 <= response.status < 300: # common case
return
meta = response.meta
if 'handle_httpstatus_all' in meta:
return
if 'handle_httpstatus_list' in meta:
allowed_statuses = meta['handle_httpstatus_list']
elif self.handle_httpstatus_all:
return
else:
allowed_statuses = getattr(spider, 'handle_httpstatus_list', self.handle_httpstatus_list)
if response.status in allowed_statuses:
return
raise HttpError(response, 'Ignoring non-200 response')
def process_spider_exception(self, response, exception, spider):
if isinstance(exception, HttpError):
spider.crawler.stats.inc_value('httperror/response_ignored_count')
spider.crawler.stats.inc_value(
'httperror/response_ignored_status_count/%s' % response.status
)
logger.info(
"Ignoring response %(response)r: HTTP status code is not handled or not allowed",
{'response': response}, extra={'spider': spider},
)
return []
開啟自定義spider中間件方式，在配置檔案setting.py中添加命名為SPIDER_MIDDLEWARES的字典，其中key為下載下傳器路徑，value為優先級，數字越小越靠近引擎，process_spider_input()優先處理，數字越大越靠近spider，process_spider_output()優先處理
内置spider 中間件
SPIDER_MIDDLEWARES_BASE = {
# Engine side
'scrapy.spidermiddlewares.httperror.HttpErrorMiddleware': 50,
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware': 500,
'scrapy.spidermiddlewares.referer.RefererMiddleware': 700,
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware': 800,
'scrapy.spidermiddlewares.depth.DepthMiddleware': 900,
# Spider side
} 下載下傳中間件
下載下傳中間件的使用頻率遠高于spider中間件，他是我們設定反爬蟲措施的主要戰場，同樣在建立scrapy項目的同時會生成一個下載下傳中間件的模闆，如下：
class ProxyExampleDownloaderMiddleware(object):
# Not all methods need to be defined. If a method is not defined,
# scrapy acts as if the downloader middleware does not modify the
# passed objects.
@classmethod
def from_crawler(cls, crawler):
# This method is used by Scrapy to create your spiders.
s = cls()
crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
return s
def process_request(self, request, spider):
# Called for each request that goes through the downloader
# middleware.
# Must either:
# - return None: continue processing this request
# - or return a Response object
# - or return a Request object
# - or raise IgnoreRequest: process_exception() methods of
# installed downloader middleware will be called
return None
def process_response(self, request, response, spider):
# Called with the response returned from the downloader.
# Must either;
# - return a Response object
# - return a Request object
# - or raise IgnoreRequest
return response
def process_exception(self, request, exception, spider):
# Called when a download handler or a process_request()
# (from other downloader middleware) raises an exception.
# Must either:
# - return None: continue processing this exception
# - return a Response object: stops process_exception() chain
# - return a Request object: stops process_exception() chain
pass
def spider_opened(self, spider):
spider.logger.info('Spider opened: %s' % spider.name)
from_crawler(cls, crawler)
這個類方法通常是通路settings和signals的入口函數
process_request(self, request, spider)
當引擎将請求發送給下載下傳器之前調用，用于對request請求加工，傳回值為None、Request、Response、IgnoreRequest異常。
傳回為None時其他的下載下傳中間件的process_request方法執行，直到内置的一個下載下傳器方法傳回Request對象為止
當傳回Request對象後，其他的process_request方法不再執行，而是将翻譯的Request發往排程器排隊等待新一輪的process_request依次執行。
當傳回Response，這時更加簡單了，相當于告訴引擎我已經得到結果了，不需要其他process_request執行，接下來依次執行process_resposne
-
process_response(request, response, spider)
當process_request傳回resposne對象後，該函數的流水線登場對resposne處理
若傳回Response對象，它會被下個中間件中的process_response()處理
若傳回Request對象，中間鍊停止，然後傳回的Request會被重新排程下載下傳
抛出IgnoreRequest，回調函數 Request.errback将會被調用處理，若沒處理，将會忽略
- 當然如果所有的中間件都沒意見那麼将交給引擎轉交給spider的解析函數
-
process_exception(request, exception, spider)
- 總結：下載下傳中間件，是兩條流水線，一條是process_request、一條是process_requests，這兩條流水線的最末端是内置的方法，當一個Request請求發來，将被篇process_request流水線上的工位逐個檢查處理，有的說缺少材料，補一份清單流水線重新檢查，有的說這個已經是response成品了不用檢查了，并貼上response标簽，有的說這個是殘次品按照殘次品處理，當然如果所有工位都沒有意見那麼最後一個工位的員工将按照最後流程貼上一個response成品标簽，進而進入Process_response流水線。
- 在process_resposne流水線上同上處理，如果有的工位說還是Request半成品，那麼傳回process_request流水線在加工；如果有的說确實是response産品，那麼給下一個工位逐一确定，直到最後一個内置工位确認确實是response産品，那麼交給引擎。
中間件就是這樣的流水線思想，我們自定義的中間件隻不過是流水線上的一個工位，當然你可以給這個工位配置設定順序和權限，可以直接給産品貼标簽，貼上不同的标簽，将進入不同的流水線再處理。
常用自定義中間件源碼：
user-agent中間件
from faker import Faker
class UserAgent_Middleware():
def process_request(self, request, spider):
f = Faker()
agent = f.firefox()
request.headers['User-Agent'] = agent 代理ip中間件
class Proxy_Middleware():
def process_request(self, request, spider):
try:
xdaili_url = spider.settings.get('XDAILI_URL')
r = requests.get(xdaili_url)
proxy_ip_port = r.text
request.meta['proxy'] = 'https://' + proxy_ip_port
except requests.exceptions.RequestException:
print('擷取訊代理ip失敗！')
spider.logger.error('擷取訊代理ip失敗！') scrapy中對接selenium
from scrapy.http import HtmlResponse
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from gp.configs import *
class ChromeDownloaderMiddleware(object):
def __init__(self):
options = webdriver.ChromeOptions()
options.add_argument('--headless') # 設定無界面
if CHROME_PATH:
options.binary_location = CHROME_PATH
if CHROME_DRIVER_PATH:
self.driver = webdriver.Chrome(chrome_options=options, executable_path=CHROME_DRIVER_PATH) # 初始化Chrome驅動
else:
self.driver = webdriver.Chrome(chrome_options=options) # 初始化Chrome驅動
def __del__(self):
self.driver.close()
def process_request(self, request, spider):
try:
print('Chrome driver begin...')
self.driver.get(request.url) # 擷取網頁連結内容
return HtmlResponse(url=request.url, body=self.driver.page_source, request=request, encoding='utf-8',
status=200) # 傳回HTML資料
except TimeoutException:
return HtmlResponse(url=request.url, request=request, encoding='utf-8', status=500)
finally:
print('Chrome driver end...') scrapy對接cookie中間件
class WeiBoMiddleWare(object):
def __init__(self, cookies_pool_url):
self.logging = logging.getLogger("WeiBoMiddleWare")
self.cookies_pool_url = cookies_pool_url
def get_random_cookies(self):
try:
response = requests.get(self.cookies_pool_url)
except Exception as e:
self.logging.info('Get Cookies failed: {}'.format(e))
else:
# 在中間件中，設定請求頭攜帶的Cookies值，必須是一個字典，不能直接設定字元串。
cookies = json.loads(response.text)
self.logging.info('Get Cookies success: {}'.format(response.text))
return cookies
@classmethod
def from_settings(cls, settings):
obj = cls(
cookies_pool_url=settings['WEIBO_COOKIES_URL']
)
return obj
def process_request(self, request, spider):
request.cookies = self.get_random_cookies()
return None
上面源碼來自（https://blog.csdn.net/BF02jgtRS00XKtCx/article/details/82141627、https://blog.csdn.net/qq_42336549/article/details/80991814）用以示例。
使用方法，在setting.py檔案中配置DOWNLOADERMIDDLEWARES 字典，鍵為中間件路徑，值是優先級，數字越小，越靠近引擎，數字越大越靠近下載下傳器，是以數字越小的，
process_request()
優先處理；數字越大的，
process_response()
優先處理.
内置下載下傳中間件如下：
最後再附上一個内置的下載下傳中間件源碼：
預設開啟，Scrapy将記錄所有在request(
Cookie
請求頭)發送的cookies及response接收到的cookies(
Set-Cookie
接收頭)。
class CookiesMiddleware(object):
"""This middleware enables working with sites that need cookies"""
def __init__(self, debug=False):
self.jars = defaultdict(CookieJar)
self.debug = debug
@classmethod
def from_crawler(cls, crawler):
if not crawler.settings.getbool('COOKIES_ENABLED'):
raise NotConfigured
return cls(crawler.settings.getbool('COOKIES_DEBUG'))
def process_request(self, request, spider):
if request.meta.get('dont_merge_cookies', False):
return
cookiejarkey = request.meta.get("cookiejar")
jar = self.jars[cookiejarkey]
cookies = self._get_request_cookies(jar, request)
for cookie in cookies:
jar.set_cookie_if_ok(cookie, request)
# set Cookie header
request.headers.pop('Cookie', None)
jar.add_cookie_header(request)
self._debug_cookie(request, spider)
def process_response(self, request, response, spider):
if request.meta.get('dont_merge_cookies', False):
return response
# extract cookies from Set-Cookie and drop invalid/expired cookies
cookiejarkey = request.meta.get("cookiejar")
jar = self.jars[cookiejarkey]
jar.extract_cookies(response, request)
self._debug_set_cookie(response, spider)
return response
def _debug_cookie(self, request, spider):
if self.debug:
cl = [to_native_str(c, errors='replace')
for c in request.headers.getlist('Cookie')]
if cl:
cookies = "\n".join("Cookie: {}\n".format(c) for c in cl)
msg = "Sending cookies to: {}\n{}".format(request, cookies)
logger.debug(msg, extra={'spider': spider})
def _debug_set_cookie(self, response, spider):
if self.debug:
cl = [to_native_str(c, errors='replace')
for c in response.headers.getlist('Set-Cookie')]
if cl:
cookies = "\n".join("Set-Cookie: {}\n".format(c) for c in cl)
msg = "Received cookies from: {}\n{}".format(response, cookies)
logger.debug(msg, extra={'spider': spider})
def _format_cookie(self, cookie):
# build cookie string
cookie_str = '%s=%s' % (cookie['name'], cookie['value'])
if cookie.get('path', None):
cookie_str += '; Path=%s' % cookie['path']
if cookie.get('domain', None):
cookie_str += '; Domain=%s' % cookie['domain']
return cookie_str
def _get_request_cookies(self, jar, request):
if isinstance(request.cookies, dict):
cookie_list = [{'name': k, 'value': v} for k, v in \
six.iteritems(request.cookies)]
else:
cookie_list = request.cookies
cookies = [self._format_cookie(x) for x in cookie_list]
headers = {'Set-Cookie': cookies}
response = Response(request.url, headers=headers)
return jar.make_cookies(response, request)
·END·
![TestLink導出用例轉換工具(XML2Excel)[圖]](data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACwAAAAAAQABAAACAkQBADs=)