網絡爬蟲Scrapy架構--騰訊社招資訊

一.建立Scrapy項目：

scrapy startproject Tencent
           

指令執行後，會建立一個Tencent檔案夾，結構如下

1. 修改items.py

# -*- coding: utf-8 -*-
import scrapy
class TencentItem(scrapy.Item):
  # 職位名
  positionname = scrapy.Field()
  # 詳情連接配接
  positionlink = scrapy.Field()
  # 職位類别
  positionType = scrapy.Field()
  # 招聘人數
  peopleNum = scrapy.Field()
  # 工作地點
  workLocation = scrapy.Field()
  # 釋出時間
  publishTime = scrapy.Field()
           

2.編寫spider檔案

進入Tencent目錄，使用指令建立一個基礎爬蟲類：

# tencentPostion為爬蟲名，tencent.com為爬蟲作用範圍
scrapy genspider tencentPostion "tencent.com"
           

執行指令後會在spiders檔案夾中建立一個tencentPostion.py的檔案，現在開始對其編寫：

# -*- coding: utf-8 -*-
import scrapy
from tencent.items import TencentItem
class TencentpositionSpider(scrapy.Spider):
  """
  功能：爬取騰訊社招資訊
  """
  # 爬蟲名
  name = "tencentPosition"
  # 爬蟲作用範圍
  allowed_domains = ["tencent.com"]
  url = "http://hr.tencent.com/position.php?&start="
  offset = 0
  # 起始url
  start_urls = [url + str(offset)]
  def parse(self, response):
    for each in response.xpath("//tr[@class='even'] | //tr[@class='odd']"):
      # 初始化模型對象
      item = TencentItem()
      # 職位名稱
      item['positionname'] = each.xpath("./td[1]/a/text()").extract()[0]
      # 詳情連接配接
      item['positionlink'] = each.xpath("./td[1]/a/@href").extract()[0]
      # 職位類别
      item['positionType'] = each.xpath("./td[2]/text()").extract()[0]
      # 招聘人數
      item['peopleNum'] = each.xpath("./td[3]/text()").extract()[0]
      # 工作地點
      item['workLocation'] = each.xpath("./td[4]/text()").extract()[0]
      # 釋出時間
      item['publishTime'] = each.xpath("./td[5]/text()").extract()[0]
      yield item
    if self.offset < 1680:
      self.offset += 10
    # 每次處理完一頁的資料之後，重新發送下一頁頁面請求
    # self.offset自增10，同時拼接為新的url，并調用回調函數self.parse處理Response
    yield scrapy.Request(self.url + str(self.offset), callback = self.parse)
           

3.編寫pipelines檔案

# -*- coding: utf-8 -*-
import json
class TencentPipeline(object):
　　""" 
    功能：儲存item資料 
  """
  def __init__(self):
    self.filename = open("tencent.json", "w")
  def process_item(self, item, spider):
    text = json.dumps(dict(item), ensure_ascii = False) + ",\n"
    self.filename.write(text.encode("utf-8"))
    return item
  def close_spider(self, spider):
    self.filename.close()
           

4.settings檔案設定（主要設定内容）

# 設定請求頭部，添加url
DEFAULT_REQUEST_HEADERS = {
  "User-Agent" : "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0;",
  'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
}
# 設定item——pipelines
ITEM_PIPELINES = {
  'tencent.pipelines.TencentPipeline': 300,
}
           

二.執行爬蟲

# tencentPosition為爬蟲名
scrapy crawl tencentPosition
           

三.運作截圖

參考資料：

https://www.jb51.net/article/135846.htm