celery守護程序(生産伺服器進行部署)

celery守護程序(Shell 腳本方式)

django + redis + celery

redis同時作為broker和backend

一. 配置 worker

腳本名字 : celeryd

在 /etc/init.d/建立celeryd

指令使用方法 :
    /etc/init.d/celeryd {start|stop|restart|status}

配置檔案 :
    /etc/default/celeryd

編寫worker啟動腳本

此檔案直接粘貼到/etc/init.d/celeryd, 通用, 檔案來自celery官網：https://github.com/celery/celery/tree/master/extra/generic-init.d

#!/bin/sh -e
# ============================================
#  celeryd - Starts the Celery worker daemon.
# ============================================
#
# :Usage: /etc/init.d/celeryd {start|stop|force-reload|restart|try-restart|status}
# :Configuration file: /etc/default/celeryd (or /usr/local/etc/celeryd on BSD)
#
# See http://docs.celeryproject.org/en/latest/userguide/daemonizing.html#generic-init-scripts


### BEGIN INIT INFO
# Provides:          celeryd
# Required-Start:    $network $local_fs $remote_fs
# Required-Stop:     $network $local_fs $remote_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: celery task worker daemon
### END INIT INFO
#
#
# To implement separate init-scripts, copy this script and give it a different
# name.  That is, if your new application named "little-worker" needs an init,
# you should use:
#
#   cp /etc/init.d/celeryd /etc/init.d/little-worker
#
# You can then configure this by manipulating /etc/default/little-worker.
#
VERSION=10.1
echo "celery init v${VERSION}."
if [ $(id -u) -ne 0 ]; then
    echo "Error: This program can only be used by the root user."
    echo "       Unprivileged users must use the 'celery multi' utility, "
    echo "       or 'celery worker --detach'."
    exit 1
fi

origin_is_runlevel_dir () {
    set +e
    dirname $0 | grep -q "/etc/rc.\.d"
    echo $?
}

# Can be a runlevel symlink (e.g., S02celeryd)
if [ $(origin_is_runlevel_dir) -eq 0 ]; then
    SCRIPT_FILE=$(readlink "$0")
else
    SCRIPT_FILE="$0"
fi
SCRIPT_NAME="$(basename "$SCRIPT_FILE")"

DEFAULT_USER="celery"
DEFAULT_PID_FILE="/var/run/celery/%n.pid"
DEFAULT_LOG_FILE="/var/log/celery/%n%I.log"
DEFAULT_LOG_LEVEL="INFO"
DEFAULT_NODES="celery"
DEFAULT_CELERYD="-m celery worker --detach"

if [ -d "/etc/default" ]; then
    CELERY_CONFIG_DIR="/etc/default"
else
    CELERY_CONFIG_DIR="/usr/local/etc"
fi

# 配置檔案的位置
CELERY_DEFAULTS=${CELERY_DEFAULTS:-"$CELERY_CONFIG_DIR/${SCRIPT_NAME}"}

# Make sure executable configuration script is owned by root
_config_sanity() {
    local path="$1"
    local owner=$(ls -ld "$path" | awk '{print $3}')
    local iwgrp=$(ls -ld "$path" | cut -b 6)
    local iwoth=$(ls -ld "$path" | cut -b 9)

    if [ "$(id -u $owner)" != "0" ]; then
        echo "Error: Config script '$path' must be owned by root!"
        echo
        echo "Resolution:"
        echo "Review the file carefully, and make sure it hasn't been "
        echo "modified with mailicious intent.  When sure the "
        echo "script is safe to execute with superuser privileges "
        echo "you can change ownership of the script:"
        echo "    $ sudo chown root '$path'"
        exit 1
    fi

    if [ "$iwoth" != "-" ]; then  # S_IWOTH
        echo "Error: Config script '$path' cannot be writable by others!"
        echo
        echo "Resolution:"
        echo "Review the file carefully, and make sure it hasn't been "
        echo "modified with malicious intent.  When sure the "
        echo "script is safe to execute with superuser privileges "
        echo "you can change the scripts permissions:"
        echo "    $ sudo chmod 640 '$path'"
        exit 1
    fi
    if [ "$iwgrp" != "-" ]; then  # S_IWGRP
        echo "Error: Config script '$path' cannot be writable by group!"
        echo
        echo "Resolution:"
        echo "Review the file carefully, and make sure it hasn't been "
        echo "modified with malicious intent.  When sure the "
        echo "script is safe to execute with superuser privileges "
        echo "you can change the scripts permissions:"
        echo "    $ sudo chmod 640 '$path'"
        exit 1
    fi
}

if [ -f "$CELERY_DEFAULTS" ]; then
    _config_sanity "$CELERY_DEFAULTS"
    echo "Using config script: $CELERY_DEFAULTS"
    # 加載配置檔案的資訊到該腳本程式中
    . "$CELERY_DEFAULTS"
fi

# Sets --app argument for CELERY_BIN
CELERY_APP_ARG=""
if [ ! -z "$CELERY_APP" ]; then
    CELERY_APP_ARG="--app=$CELERY_APP"
fi

# Options to su
# can be used to enable login shell (CELERYD_SU_ARGS="-l"),
# or even to use start-stop-daemon instead of su.
CELERYD_SU=${CELERY_SU:-"su"}
CELERYD_SU_ARGS=${CELERYD_SU_ARGS:-""}

CELERYD_USER=${CELERYD_USER:-$DEFAULT_USER}

# Set CELERY_CREATE_DIRS to always create log/pid dirs.
CELERY_CREATE_DIRS=${CELERY_CREATE_DIRS:-0}
CELERY_CREATE_RUNDIR=$CELERY_CREATE_DIRS
CELERY_CREATE_LOGDIR=$CELERY_CREATE_DIRS
if [ -z "$CELERYD_PID_FILE" ]; then
    CELERYD_PID_FILE="$DEFAULT_PID_FILE"
    CELERY_CREATE_RUNDIR=1
fi
if [ -z "$CELERYD_LOG_FILE" ]; then
    CELERYD_LOG_FILE="$DEFAULT_LOG_FILE"
    CELERY_CREATE_LOGDIR=1
fi

CELERYD_LOG_LEVEL=${CELERYD_LOG_LEVEL:-${CELERYD_LOGLEVEL:-$DEFAULT_LOG_LEVEL}}
CELERY_BIN=${CELERY_BIN:-"celery"}
CELERYD_MULTI=${CELERYD_MULTI:-"$CELERY_BIN multi"}
CELERYD_NODES=${CELERYD_NODES:-$DEFAULT_NODES}

export CELERY_LOADER

if [ -n "$2" ]; then
    CELERYD_OPTS="$CELERYD_OPTS $2"
fi

CELERYD_LOG_DIR=`dirname $CELERYD_LOG_FILE`
CELERYD_PID_DIR=`dirname $CELERYD_PID_FILE`

# Extra start-stop-daemon options, like user/group.
if [ -n "$CELERYD_CHDIR" ]; then
    DAEMON_OPTS="$DAEMON_OPTS --workdir=$CELERYD_CHDIR"
fi


check_dev_null() {
    if [ ! -c /dev/null ]; then
        echo "/dev/null is not a character device!"
        exit 75  # EX_TEMPFAIL
    fi
}


maybe_die() {
    if [ $? -ne 0 ]; then
        echo "Exiting: $* (errno $?)"
        exit 77  # EX_NOPERM
    fi
}

create_default_dir() {
    if [ ! -d "$1" ]; then
        echo "- Creating default directory: '$1'"
        mkdir -p "$1"
        maybe_die "Couldn't create directory $1"
        echo "- Changing permissions of '$1' to 02755"
        chmod 02755 "$1"
        maybe_die "Couldn't change permissions for $1"
        if [ -n "$CELERYD_USER" ]; then
            echo "- Changing owner of '$1' to '$CELERYD_USER'"
            chown "$CELERYD_USER" "$1"
            maybe_die "Couldn't change owner of $1"
        fi
        if [ -n "$CELERYD_GROUP" ]; then
            echo "- Changing group of '$1' to '$CELERYD_GROUP'"
            chgrp "$CELERYD_GROUP" "$1"
            maybe_die "Couldn't change group of $1"
        fi
    fi
}


check_paths() {
    if [ $CELERY_CREATE_LOGDIR -eq 1 ]; then
        create_default_dir "$CELERYD_LOG_DIR"
    fi
    if [ $CELERY_CREATE_RUNDIR -eq 1 ]; then
        create_default_dir "$CELERYD_PID_DIR"
    fi
}

create_paths() {
    create_default_dir "$CELERYD_LOG_DIR"
    create_default_dir "$CELERYD_PID_DIR"
}

export PATH="${PATH:+$PATH:}/usr/sbin:/sbin"


_get_pidfiles () {
    # note: multi < 3.1.14 output to stderr, not stdout, hence the redirect.
    ${CELERYD_MULTI} expand "${CELERYD_PID_FILE}" ${CELERYD_NODES} 2>&1
}


_get_pids() {
    found_pids=0
    my_exitcode=0

    for pidfile in $(_get_pidfiles); do
        local pid=`cat "$pidfile"`
        local cleaned_pid=`echo "$pid" | sed -e 's/[^0-9]//g'`
        if [ -z "$pid" ] || [ "$cleaned_pid" != "$pid" ]; then
            echo "bad pid file ($pidfile)"
            one_failed=true
            my_exitcode=1
        else
            found_pids=1
            echo "$pid"
        fi

    if [ $found_pids -eq 0 ]; then
        echo "${SCRIPT_NAME}: All nodes down"
        exit $my_exitcode
    fi
    done
}


_chuid () {
    ${CELERYD_SU} ${CELERYD_SU_ARGS} "$CELERYD_USER" -c "$CELERYD_MULTI $*"
}


start_workers () {
    if [ ! -z "$CELERYD_ULIMIT" ]; then
        # 設定該程式打開檔案數
        ulimit -n $CELERYD_ULIMIT
    fi
    _chuid $* start $CELERYD_NODES $DAEMON_OPTS     \
                 --pidfile="$CELERYD_PID_FILE"      \
                 --logfile="$CELERYD_LOG_FILE"      \
                 --loglevel="$CELERYD_LOG_LEVEL"    \
                 $CELERY_APP_ARG                    \
                 $CELERYD_OPTS
}


dryrun () {
    (C_FAKEFORK=1 start_workers --verbose)
}


stop_workers () {
    _chuid stopwait $CELERYD_NODES --pidfile="$CELERYD_PID_FILE"
}


restart_workers () {
    _chuid restart $CELERYD_NODES $DAEMON_OPTS      \
                   --pidfile="$CELERYD_PID_FILE"    \
                   --logfile="$CELERYD_LOG_FILE"    \
                   --loglevel="$CELERYD_LOG_LEVEL"  \
                   $CELERY_APP_ARG                  \
                   $CELERYD_OPTS
}


kill_workers() {
    _chuid kill $CELERYD_NODES $DAEMON_OPTS --pidfile="$CELERYD_PID_FILE"
}


restart_workers_graceful () {
    echo "WARNING: Use with caution in production"
    echo "The workers will attempt to restart, but they may not be able to."
    local worker_pids=
    worker_pids=`_get_pids`
    [ "$one_failed" ] && exit 1

    for worker_pid in $worker_pids; do
        local failed=
        kill -HUP $worker_pid 2> /dev/null || failed=true
        if [ "$failed" ]; then
            echo "${SCRIPT_NAME} worker (pid $worker_pid) could not be restarted"
            one_failed=true
        else
            echo "${SCRIPT_NAME} worker (pid $worker_pid) received SIGHUP"
        fi
    done

    [ "$one_failed" ] && exit 1 || exit 0
}


check_status () {
    my_exitcode=0
    found_pids=0

    local one_failed=
    for pidfile in $(_get_pidfiles); do
        if [ ! -r $pidfile ]; then
            echo "${SCRIPT_NAME} down: no pidfiles found"
            one_failed=true
            break
        fi

        local node=`basename "$pidfile" .pid`
        local pid=`cat "$pidfile"`
        local cleaned_pid=`echo "$pid" | sed -e 's/[^0-9]//g'`
        if [ -z "$pid" ] || [ "$cleaned_pid" != "$pid" ]; then
            echo "bad pid file ($pidfile)"
            one_failed=true
        else
            local failed=
            kill -0 $pid 2> /dev/null || failed=true
            if [ "$failed" ]; then
                echo "${SCRIPT_NAME} (node $node) (pid $pid) is down, but pidfile exists!"
                one_failed=true
            else
                echo "${SCRIPT_NAME} (node $node) (pid $pid) is up..."
            fi
        fi
    done

    [ "$one_failed" ] && exit 1 || exit 0
}


case "$1" in
    start)
        check_dev_null
        check_paths
        start_workers
    ;;

    stop)
        check_dev_null
        check_paths
        stop_workers
    ;;

    reload|force-reload)
        echo "Use restart"
    ;;

    status)
        check_status
    ;;

    restart)
        check_dev_null
        check_paths
        restart_workers
    ;;

    graceful)
        check_dev_null
        restart_workers_graceful
    ;;

    kill)
        check_dev_null
        kill_workers
    ;;

    dryrun)
        check_dev_null
        dryrun
    ;;

    try-restart)
        check_dev_null
        check_paths
        restart_workers
    ;;

    create-paths)
        check_dev_null
        create_paths
    ;;

    check-paths)
        check_dev_null
        check_paths
    ;;

    *)
        echo "Usage: /etc/init.d/${SCRIPT_NAME} {start|stop|restart|graceful|kill|dryrun|create-paths}"
        exit 64  # EX_USAGE
    ;;
esac

exit 0

celeryd

改變權限:

chmod +x /etc/init.d/celeryd

編寫worker的配置檔案，位置：/etc/default/celeryd

# 節點名字【辨別作用】，日志也會以這個名字開頭，自定義即可，啟動多個的寫法：CELERYD_NODES="worker1 worker2 worker3" 或 CELERYD_NODES=10
CELERYD_NODES="worker1"

# 配置celery的位置，檢視位置：which celery
CELERY_BIN="/usr/local/bin/celery"

# 配置執行個體化app，一般指的建立app執行個體的檔案，django項目則配置項目名，完整的配置方法：CELERY_APP="proj.tasks:app"
# 這裡寫自己django的項目名字即可
CELERY_APP="mypro"

# 進入哪個workerr的目錄   絕對路徑: django項目比如結構是: /home/mypro/mypro/celery.py    那這裡填的就是如下所示
CELERYD_CHDIR="/home/mypro/mypro/"

# --time-limit ： 限制處理任務的時長
# --concurrency ： 設定最高的并發數
# 多個啟動worker,對每個woker進行單獨的配置方法：CELERYD_OPTS="--time-limit=300 -c 8 -c:worker2 4 -c:worker3 2 -Ofair:worker1"
CELERYD_OPTS=" --time-limit=300 --concurrency=8"

# 設定日志的級别：開發環境：DEBUG，生産環境：INFO
CELERYD_LOG_LEVEL="DEBUG"

# 設定存放目志位置
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"

# 設定啟動程式存放pid檔案
CELERYD_PID_FILE="/var/run/celery/%n.pid"

# 設定啟動程式的使用者，需要手動建立使用者群組
CELERYD_USER="celery"
CELERYD_GROUP="celery"

# 1：自動建立需要的目錄檔案并且設定運作程式所需的使用者群組，0：需要手動處理
CELERY_CREATE_DIRS=1

# 設定celery最大的檔案打開數
CELERYD_ULIMIT=65535

# 指明django的配置檔案
export DJANGO_SETTINGS_MODULE="settings"
# 導出路徑
export PYTHONPATH="$PYTHONPATH:/home/mypro"

建立celery使用者群組

useradd celery

啟動

[email protected]:/var/log# /etc/init.d/celeryd start
celery init v10.1.
Using config script: /etc/default/celeryd
- Creating default directory: '/var/log/celery'
- Changing permissions of '/var/log/celery' to 02755
- Changing owner of '/var/log/celery' to 'celery'
- Changing group of '/var/log/celery' to 'celery'
- Creating default directory: '/var/run/celery'
- Changing permissions of '/var/run/celery' to 02755
- Changing owner of '/var/run/celery' to 'celery'
- Changing group of '/var/run/celery' to 'celery'
celery multi v5.0.5 (singularity)
> Starting nodes...
	> [email protected]: OK

檢查啟動的程序

[email protected]:/var/log# ps -ef | grep python
root         590       1  0 18:06 ?        00:00:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
root         788       1  0 18:06 ?        00:00:00 /usr/bin/python3 /usr/share/unattended-upgrades/unattended-upgrade-shutdown --wait-for-signal
celery     2389       1  0 18:17 ?        00:00:03 /usr/bin/python3 -m celery --app=runyi --workdir=/home/runyi/runyi/runyi/ worker --pidfile=/var/run/celery/worker1.pid --logfile=/var/log/celery/worker1%I.log --loglevel=DEBUG --time-limit=300 --concurrency=8 -n [email protected] --executable=/usr/bin/python3
celery     2392    2389  0 18:17 ?        00:00:00 /usr/bin/python3 -m celery --app=runyi --workdir=/home/runyi/runyi/runyi/ worker --pidfile=/var/run/celery/worker1.pid --logfile=/var/log/celery/worker1%I.log --loglevel=DEBUG --time-limit=300 --concurrency=8 -n [email protected] --executable=/usr/bin/python3
celery     2393    2389  0 18:17 ?        00:00:00 /usr/bin/python3 -m celery --app=runyi --workdir=/home/runyi/runyi/runyi/ worker --pidfile=/var/run/celery/worker1.pid --logfile=/var/log/celery/worker1%I.log --loglevel=DEBUG --time-limit=300 --concurrency=8 -n [email protected] --executable=/usr/bin/python3
celery     2394    2389  0 18:17 ?        00:00:00 /usr/bin/python3 -m celery --app=runyi --workdir=/home/runyi/runyi/runyi/ worker --pidfile=/var/run/celery/worker1.pid --logfile=/var/log/celery/worker1%I.log --loglevel=DEBUG --time-limit=300 --concurrency=8 -n [email protected] --executable=/usr/bin/python3
celery     2395    2389  0 18:17 ?        00:00:00 /usr/bin/python3 -m celery --app=runyi --workdir=/home/runyi/runyi/runyi/ worker --pidfile=/var/run/celery/worker1.pid --logfile=/var/log/celery/worker1%I.log --loglevel=DEBUG --time-limit=300 --concurrency=8 -n [email protected] --executable=/usr/bin/python3
celery     2396    2389  0 18:17 ?        00:00:00 /usr/bin/python3 -m celery --app=runyi --workdir=/home/runyi/runyi/runyi/ worker --pidfile=/var/run/celery/worker1.pid --logfile=/var/log/celery/worker1%I.log --loglevel=DEBUG --time-limit=300 --concurrency=8 -n [email protected] --executable=/usr/bin/python3
celery     2397    2389  0 18:17 ?        00:00:00 /usr/bin/python3 -m celery --app=runyi --workdir=/home/runyi/runyi/runyi/ worker --pidfile=/var/run/celery/worker1.pid --logfile=/var/log/celery/worker1%I.log --loglevel=DEBUG --time-limit=300 --concurrency=8 -n [email protected] --executable=/usr/bin/python3
celery     2398    2389  0 18:17 ?        00:00:00 /usr/bin/python3 -m celery --app=runyi --workdir=/home/runyi/runyi/runyi/ worker --pidfile=/var/run/celery/worker1.pid --logfile=/var/log/celery/worker1%I.log --loglevel=DEBUG --time-limit=300 --concurrency=8 -n [email protected] --executable=/usr/bin/python3
celery     2399    2389  0 18:17 ?        00:00:00 /usr/bin/python3 -m celery --app=runyi --workdir=/home/runyi/runyi/runyi/ worker --pidfile=/var/run/celery/worker1.pid --logfile=/var/log/celery/worker1%I.log --loglevel=DEBUG --time-limit=300 --concurrency=8 -n [email protected] --executable=/usr/bin/python3
root        3215    2048  0 18:37 pts/2    00:00:00 grep --color=auto python

二. 配置 celerybeat

腳本名字 : celerybeat

在 /etc/init.d/建立celerybeat

指令使用方法 :
    /etc/init.d/celerybeat  {start|stop|restart|status}

配置檔案 :
    /etc/default/celerybeat

編寫beat啟動腳本

同樣來自官網

#!/bin/sh -e
# =========================================================
#  celerybeat - Starts the Celery periodic task scheduler.
# =========================================================
#
# :Usage: /etc/init.d/celerybeat {start|stop|force-reload|restart|try-restart|status}
# :Configuration file: /etc/default/celerybeat or /etc/default/celeryd
#
# See http://docs.celeryproject.org/en/latest/userguide/daemonizing.html#generic-init-scripts

### BEGIN INIT INFO
# Provides:          celerybeat
# Required-Start:    $network $local_fs $remote_fs
# Required-Stop:     $network $local_fs $remote_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: celery periodic task scheduler
### END INIT INFO

# Cannot use set -e/bash -e since the kill -0 command will abort
# abnormally in the absence of a valid process ID.
#set -e
VERSION=10.1
echo "celery init v${VERSION}."

if [ $(id -u) -ne 0 ]; then
    echo "Error: This program can only be used by the root user."
    echo "       Unpriviliged users must use 'celery beat --detach'"
    exit 1
fi

origin_is_runlevel_dir () {
    set +e
    dirname $0 | grep -q "/etc/rc.\.d"
    echo $?
}

# Can be a runlevel symlink (e.g., S02celeryd)
if [ $(origin_is_runlevel_dir) -eq 0 ]; then
    SCRIPT_FILE=$(readlink "$0")
else
    SCRIPT_FILE="$0"
fi
SCRIPT_NAME="$(basename "$SCRIPT_FILE")"

# /etc/init.d/celerybeat: start and stop the celery periodic task scheduler daemon.

# Make sure executable configuration script is owned by root
_config_sanity() {
    local path="$1"
    local owner=$(ls -ld "$path" | awk '{print $3}')
    local iwgrp=$(ls -ld "$path" | cut -b 6)
    local iwoth=$(ls -ld "$path" | cut -b 9)

    if [ "$(id -u $owner)" != "0" ]; then
        echo "Error: Config script '$path' must be owned by root!"
        echo
        echo "Resolution:"
        echo "Review the file carefully, and make sure it hasn't been "
        echo "modified with mailicious intent.  When sure the "
        echo "script is safe to execute with superuser privileges "
        echo "you can change ownership of the script:"
        echo "    $ sudo chown root '$path'"
        exit 1
    fi

    if [ "$iwoth" != "-" ]; then  # S_IWOTH
        echo "Error: Config script '$path' cannot be writable by others!"
        echo
        echo "Resolution:"
        echo "Review the file carefully, and make sure it hasn't been "
        echo "modified with malicious intent.  When sure the "
        echo "script is safe to execute with superuser privileges "
        echo "you can change the scripts permissions:"
        echo "    $ sudo chmod 640 '$path'"
        exit 1
    fi
    if [ "$iwgrp" != "-" ]; then  # S_IWGRP
        echo "Error: Config script '$path' cannot be writable by group!"
        echo
        echo "Resolution:"
        echo "Review the file carefully, and make sure it hasn't been "
        echo "modified with malicious intent.  When sure the "
        echo "script is safe to execute with superuser privileges "
        echo "you can change the scripts permissions:"
        echo "    $ sudo chmod 640 '$path'"
        exit 1
    fi
}

scripts=""

if test -f /etc/default/celeryd; then
    scripts="/etc/default/celeryd"
    _config_sanity /etc/default/celeryd
    . /etc/default/celeryd
fi

EXTRA_CONFIG="/etc/default/${SCRIPT_NAME}"
if test -f "$EXTRA_CONFIG"; then
    scripts="$scripts, $EXTRA_CONFIG"
    _config_sanity "$EXTRA_CONFIG"
    . "$EXTRA_CONFIG"
fi

echo "Using configuration: $scripts"

CELERY_BIN=${CELERY_BIN:-"celery"}
DEFAULT_USER="celery"
DEFAULT_PID_FILE="/var/run/celery/beat.pid"
DEFAULT_LOG_FILE="/var/log/celery/beat.log"
DEFAULT_LOG_LEVEL="INFO"
DEFAULT_CELERYBEAT="$CELERY_BIN"

CELERYBEAT=${CELERYBEAT:-$DEFAULT_CELERYBEAT}
CELERYBEAT_LOG_LEVEL=${CELERYBEAT_LOG_LEVEL:-${CELERYBEAT_LOGLEVEL:-$DEFAULT_LOG_LEVEL}}

CELERYBEAT_SU=${CELERYBEAT_SU:-"su"}
CELERYBEAT_SU_ARGS=${CELERYBEAT_SU_ARGS:-""}

# Sets --app argument for CELERY_BIN
CELERY_APP_ARG=""
if [ ! -z "$CELERY_APP" ]; then
    CELERY_APP_ARG="--app=$CELERY_APP"
fi

CELERYBEAT_USER=${CELERYBEAT_USER:-${CELERYD_USER:-$DEFAULT_USER}}

# Set CELERY_CREATE_DIRS to always create log/pid dirs.
CELERY_CREATE_DIRS=${CELERY_CREATE_DIRS:-0}
CELERY_CREATE_RUNDIR=$CELERY_CREATE_DIRS
CELERY_CREATE_LOGDIR=$CELERY_CREATE_DIRS
if [ -z "$CELERYBEAT_PID_FILE" ]; then
    CELERYBEAT_PID_FILE="$DEFAULT_PID_FILE"
    CELERY_CREATE_RUNDIR=1
fi
if [ -z "$CELERYBEAT_LOG_FILE" ]; then
    CELERYBEAT_LOG_FILE="$DEFAULT_LOG_FILE"
    CELERY_CREATE_LOGDIR=1
fi

export CELERY_LOADER

if [ -n "$2" ]; then
    CELERYBEAT_OPTS="$CELERYBEAT_OPTS $2"
fi

CELERYBEAT_LOG_DIR=`dirname $CELERYBEAT_LOG_FILE`
CELERYBEAT_PID_DIR=`dirname $CELERYBEAT_PID_FILE`

# Extra start-stop-daemon options, like user/group.

CELERYBEAT_CHDIR=${CELERYBEAT_CHDIR:-$CELERYD_CHDIR}
if [ -n "$CELERYBEAT_CHDIR" ]; then
    DAEMON_OPTS="$DAEMON_OPTS --workdir=$CELERYBEAT_CHDIR"
fi


export PATH="${PATH:+$PATH:}/usr/sbin:/sbin"

check_dev_null() {
    if [ ! -c /dev/null ]; then
        echo "/dev/null is not a character device!"
        exit 75  # EX_TEMPFAIL
    fi
}

maybe_die() {
    if [ $? -ne 0 ]; then
        echo "Exiting: $*"
        exit 77  # EX_NOPERM
    fi
}

create_default_dir() {
    if [ ! -d "$1" ]; then
        echo "- Creating default directory: '$1'"
        mkdir -p "$1"
        maybe_die "Couldn't create directory $1"
        echo "- Changing permissions of '$1' to 02755"
        chmod 02755 "$1"
        maybe_die "Couldn't change permissions for $1"
        if [ -n "$CELERYBEAT_USER" ]; then
            echo "- Changing owner of '$1' to '$CELERYBEAT_USER'"
            chown "$CELERYBEAT_USER" "$1"
            maybe_die "Couldn't change owner of $1"
        fi
        if [ -n "$CELERYBEAT_GROUP" ]; then
            echo "- Changing group of '$1' to '$CELERYBEAT_GROUP'"
            chgrp "$CELERYBEAT_GROUP" "$1"
            maybe_die "Couldn't change group of $1"
        fi
    fi
}

check_paths() {
    if [ $CELERY_CREATE_LOGDIR -eq 1 ]; then
        create_default_dir "$CELERYBEAT_LOG_DIR"
    fi
    if [ $CELERY_CREATE_RUNDIR -eq 1 ]; then
        create_default_dir "$CELERYBEAT_PID_DIR"
    fi
}


create_paths () {
    create_default_dir "$CELERYBEAT_LOG_DIR"
    create_default_dir "$CELERYBEAT_PID_DIR"
}

is_running() {
    pid=$1
    ps $pid > /dev/null 2>&1
}

wait_pid () {
    pid=$1
    forever=1
    i=0
    while [ $forever -gt 0 ]; do
        if ! is_running $pid; then
            echo "OK"
            forever=0
        else
            kill -TERM "$pid"
            i=$((i + 1))
            if [ $i -gt 60 ]; then
                echo "ERROR"
                echo "Timed out while stopping (30s)"
                forever=0
            else
                sleep 0.5
            fi
        fi
    done
}


stop_beat () {
    echo -n "Stopping ${SCRIPT_NAME}... "
    if [ -f "$CELERYBEAT_PID_FILE" ]; then
        wait_pid $(cat "$CELERYBEAT_PID_FILE")
    else
        echo "NOT RUNNING"
    fi
}

_chuid () {
    ${CELERYBEAT_SU} ${CELERYBEAT_SU_ARGS} \
        "$CELERYBEAT_USER" -c "$CELERYBEAT $*"
}

start_beat () {
    echo "Starting ${SCRIPT_NAME}..."
    _chuid $CELERY_APP_ARG $DAEMON_OPTS beat --detach \
                 --pidfile="$CELERYBEAT_PID_FILE"      \
                 --logfile="$CELERYBEAT_LOG_FILE"      \
                 --loglevel="$CELERYBEAT_LOG_LEVEL"    \
                 $CELERYBEAT_OPTS
}


check_status () {
    local failed=
    local pid_file=$CELERYBEAT_PID_FILE
    if [ ! -e $pid_file ]; then
        echo "${SCRIPT_NAME} is down: no pid file found"
        failed=true
    elif [ ! -r $pid_file ]; then
        echo "${SCRIPT_NAME} is in unknown state, user cannot read pid file."
        failed=true
    else
        local pid=`cat "$pid_file"`
        local cleaned_pid=`echo "$pid" | sed -e 's/[^0-9]//g'`
        if [ -z "$pid" ] || [ "$cleaned_pid" != "$pid" ]; then
            echo "${SCRIPT_NAME}: bad pid file ($pid_file)"
            failed=true
        else
            local failed=
            kill -0 $pid 2> /dev/null || failed=true
            if [ "$failed" ]; then
                echo "${SCRIPT_NAME} (pid $pid) is down, but pid file exists!"
                failed=true
            else
                echo "${SCRIPT_NAME} (pid $pid) is up..."
            fi
        fi
    fi

    [ "$failed" ] && exit 1 || exit 0
}


case "$1" in
    start)
        check_dev_null
        check_paths
        start_beat
    ;;
    stop)
        check_paths
        stop_beat
    ;;
    reload|force-reload)
        echo "Use start+stop"
    ;;
    status)
        check_status
    ;;
    restart)
        echo "Restarting celery periodic task scheduler"
        check_paths
        stop_beat && check_dev_null && start_beat
    ;;
    create-paths)
        check_dev_null
        create_paths
    ;;
    check-paths)
        check_dev_null
        check_paths
    ;;
    *)
        echo "Usage: /etc/init.d/${SCRIPT_NAME} {start|stop|restart|create-paths|status}"
        exit 64  # EX_USAGE
    ;;
esac

exit 0

編寫beat的配置檔案

# 配置celery的位置，檢視位置：which celery
CELERY_BIN="/usr/local/bin/celery"

# 配置執行個體化app，一般指的建立app執行個體的檔案，django項目則配置項目名，完整的配置方法：CELERY_APP="proj.tasks:app"
CELERY_APP="mypro"

# Django項目必須配置
export DJANGO_SETTINGS_MODULE='mypro.settings'

# 進入哪個worker的目錄,   絕對路徑: django項目比如結構是: /home/mypro/mypro/celery.py    那這裡填的就是如下所示
CELERYBEAT_CHDIR=" /home/mypro/mypro/"

# 額外的參數，即是beat序列化存放的位置
CELERYBEAT_OPTS="--schedule=/var/run/celery/celerybeat-schedule"

# PID檔案的完整路徑
CELERYBEAT_PID_FILE=/var/run/celery/celerybeatd.pid

# 日志檔案的完整路徑
CELERYBEAT_LOG_FILE=/var/log/celery/celerybeatd.log

# 要使用的日志級别
CELERYBEAT_LOG_LEVEL=INFO

# 運作使用者
CELERYBEAT_USER=celery

# 運作組
CELERYBEAT_GROUP=celery
export DJANGO_SETTINGS_MODULE="settings"
export PYTHONPATH="$PYTHONPATH:/home/mypro"

啟動celeryd beat服務

[email protected]: /etc/init.d/celerybeat start
celery init v10.1.
Using configuration: /etc/default/celeryd, /etc/default/celerybeat
Starting celerybeat...

檢查beat程序

[email protected]:/var/log/celery# ps -ef | grep python | grep beat
runyi      50495       1  0 12:09 ?        00:00:01 /usr/bin/python3 /usr/local/bin/celery --app=runyi --workdir=/home/runyi/runyi/runyi/ beat --detach --pidfile=/var/run/celery/celerybeatd.pid --logfile=/var/log/celery/celerybeatd.log --loglevel=INFO --schedule=/var/run/celery/celerybeat-schedule
[email protected]:/var/log/celery#

配置完成

三. 注意事項

我全部以root使用者來啟動或停止worke或者celerybeat

故障排除

如果您無法讓入詞腳本工作，則應嘗試以冗長的模式運作它們：
```
sh -x /etc/init.d/celeryd start
           
```
這可以揭示為什麼服務不會啟動的提示。

如果從業人員以"OK"開頭，但之後幾乎立即退出，并且日志檔案中沒有證據，則可能存在錯誤，但由于 daemons 标準輸出已關閉，您将無法在任何地方看到它們。對于這種情況，您可以使用環境變量跳過"大化"步驟：C_FAKEFORK
```
C_FAKEFORK=1 sh -x /etc/init.d/celeryd start
           
```
現在你應該能夠看到錯誤。

同理也可以用來啟動celerybeat:

sh -x /etc/init.d/celerybeat start

或者:

C_FAKEFORK=1 sh -x /etc/init.d/celerybeat start

當使用指令關閉worker不起作用是,你或許應該試試這行指令:
```
/etc/default/celeryd kill
           
```

四. 關于django新增異步代碼以及新增定時任務

如果django新增了異步代碼以及新增定時任務,那麼不僅僅要重新啟動django項目,還需要重新啟動worker和celerybeat

/etc/init.d/celeryd restart
/etc/init.d/celerybeat restart

五:

在配置的過程中遇到了許多問題,都一一記錄在我的其他celery文章中,有問題可以去找找, 懶得貼了

抱拳

參考文獻: https://www.cnblogs.com/ygbh/p/13638644.html

celery守護程序(生産伺服器進行部署)

celery守護程序(Shell 腳本方式)

一. 配置 worker

二. 配置 celerybeat

三. 注意事項

我全部以root使用者來啟動或停止worke或者celerybeat

四. 關于django新增異步代碼以及新增定時任務

五:

繼續閱讀

Apache (You don't have permission to access / on this server.）

debian9更新4.9.0核心到4.19.2核心過程

centOS7 配置 vsftpd 虛拟使用者及權限Vsftpd配置虛拟使用者及權限

linux-svn解除安裝與安裝

vsftp虛拟多使用者多權限一鍵部署腳本

Ubuntu14.04 LTS下安裝mongodb

httpd服務的部署、啟動、配置和簡單優化一、部署二、啟動三、配置檔案

配置網頁内容通路

手動安裝Intel network I217-LM網卡的Linux驅動

禁止ubuntu系統彈出報錯界面

Ubuntu Linux下Apache的配置檔案

samba伺服器的功能

【Linux】UDP廣播封包接收速率問題

Linux裝置模型（中）之上層容器

PowerPC平台 Linux移植三