Scrapy 抓取圖檔

1. 先寫一個特殊的Item

class CSDNImgItem(scrapy.Item):
    image_urls = scrapy.Field()
    images = scrapy.Field()      

注意這個字段是寫死的image_urls 是圖檔的位址的一個數組，images記錄圖檔資訊不用管。

2.yield item

image_urls = response.css('#cnblogs_post_body img::attr("src")').extract()
if len(image_urls) > 0:
            imageItem = CSDNImgItem()
            imageItem['image_urls'] = image_urls
            yield imageItem      

這樣寫即可記得image_urls是一個數組

3. 圖檔下載下傳pipeline

from scrapy.pipelines.images import ImagesPipeline


class CSDNImgPipeline(ImagesPipeline):
    def get_media_requests(self, item, info):
        if 'image_urls' in item.keys():
            for image_url in item['image_urls']:
                last_name = image_url[image_url.rfind('/') + 1:len(image_url)]
                yield Request(image_url, meta={'name': last_name})

    def file_path(self, request, response=None, info=None):
        today = datetime.datetime.now().strftime('%Y%m%d')
        name = ''
        if '.' in request.meta['name']:
            name = request.meta['name'][0:request.meta['name'].rindex('.')]
        else:
            name = request.meta['name']
        result = "%s/%s.jpg" % (today, name)

        return result
        pass
    def item_completed(self, results, item, info):
        image_paths = [x['path'] for ok, x in results if ok]
        # if not image_paths:
        #     raise DropItem("Item contains no images")
        return item      

我這裡對不是圖檔的item沒有進行dropitem的處理，因為有可能是其他的文字item。然後重寫file_path，我這邊是以201804/123456.jpg 這樣的格式命名的。

4.圖檔下載下傳settings

ITEM_PIPELINES = 
    'tutorial.pipelines.CSDNImgPipeline':400,
}
IMAGES_STORE ='/Users/walle/PycharmProjects/imgSave/img'