天天看點

在Linux安裝python并運作,在linux下安裝并運作scrapyd

系統:centos7.4

安裝scrapyd:pip isntall scrapyd

因為我騰訊雲上是python2與python3并存的 是以我執行的指令是:pip3 isntall scrapyd

安裝後建立一個配置檔案:

sudo mkdir /etc/scrapyd

sudo vim /etc/scrapyd/scrapyd.conf

寫入如下内容:(給内容在https://scrapyd.readthedocs.io/en/stable/config.html可找到)

[scrapyd]

eggs_dir = eggs

logs_dir = logs

items_dir =

jobs_to_keep = 5

dbs_dir = dbs

max_proc = 0

max_proc_per_cpu = 10

finished_to_keep = 100

poll_interval = 5.0

bind_address = 0.0.0.0

http_port = 6800

debug = off

runner = scrapyd.runner

application = scrapyd.app.application

launcher = scrapyd.launcher.launcher

webroot = scrapyd.website.root

[services]

schedule.json = scrapyd.webservice.schedule

cancel.json = scrapyd.webservice.cancel

addversion.json = scrapyd.webservice.addversion

listprojects.json = scrapyd.webservice.listprojects

listversions.json = scrapyd.webservice.listversions

listspiders.json = scrapyd.webservice.listspiders

delproject.json = scrapyd.webservice.deleteproject

delversion.json = scrapyd.webservice.deleteversion

listjobs.json = scrapyd.webservice.listjobs

daemonstatus.json = scrapyd.webservice.daemonstatus

主要更改bind_address=0.0.0.0

建立檔案後執行指令啟動scrapyd:   (scrapyd > /dev/null &)  當想要記錄輸出日志時: (scrapyd > /root/scrapyd.log &)

坑1:當我執行完指令後報錯,說是找不到指令:

在Linux安裝python并運作,在linux下安裝并運作scrapyd

那是因為我系統上python2與3并存,是以找不到,這時應該做軟連接配接:

我的python3路徑:  /usr/local/python3

制作軟連接配接: ln -s /usr/local/python3/bin/scrapy  /usr/bin/scrapy

昨晚軟連接配接後,執行上邊指令,又報錯:

坑2:

在Linux安裝python并運作,在linux下安裝并運作scrapyd

這個好像是那個配置檔案的最後一行有問題,具體原因不大清楚,我将最後一行删除,再次重新執行,scrapyd就跑起來了

在Linux安裝python并運作,在linux下安裝并運作scrapyd