Run Celery Workers with Supervisor

To run Celery workers with Supervisor, install Supervisor, add a [program:celery] section that runs celery -A proj worker -l info, set autostart=true, autorestart=true, and stopasgroup=true/killasgroup=true so the whole worker process group shuts down cleanly, then load it with sudo supervisorctl reread && sudo supervisorctl update. Supervisor then daemonizes the worker, restarts it automatically if it crashes, and centralizes its logs.

A Celery worker is just a long-running Python process. In development you start it by hand, but in production it needs to start on boot, come back after a crash or an out-of-memory kill, and write logs somewhere you can find them. Supervisor is a lightweight process control system that does exactly this for any foreground program, which is why it has been a popular way to keep Celery alive for years. This guide covers the modern Celery 5.x setup and shows where systemd and containers now fit instead.

Key takeaways

• Run the worker in the foreground (celery -A proj worker -l info) and let Supervisor handle backgrounding, restarts, and logging — never daemonize Celery itself.
• Always set stopasgroup=true and killasgroup=true: the prefork pool spawns child processes, and without these the children are orphaned on restart.
• Give long tasks room to finish by setting stopwaitsecs higher than your longest task; Supervisor's default SIGTERM triggers Celery's graceful warm shutdown.
• Run Celery Beat as a separate [program:celerybeat] and only ever run one Beat instance.
• In 2026, systemd ships on every modern Linux and is the simplest choice for a single host; reach for Supervisor when you want one tool managing many apps, and skip both inside Docker/Kubernetes.

Why use a process manager for Celery?

If you run celery -A proj worker in a terminal, the worker dies the moment that session ends. A process manager solves four production problems at once:

• Daemonization — the worker runs detached from any terminal and survives logout.
• Auto-restart — if the worker crashes, hits an unhandled exception, or is killed by the OOM killer, it is brought back automatically.
• Start on boot — workers come up with the server, no manual step.
• Centralized logging — stdout and stderr are captured to rotating log files.

Celery's own docs are explicit that you should not use Celery's deprecated --detach daemonization in production; instead, run the worker as a normal foreground process and let an external supervisor manage it. Supervisor is one such supervisor; systemd is the other (more on that below).

How do you install Supervisor?

On Debian/Ubuntu the cleanest option is the system package, which installs Supervisor as a systemd-managed service with a drop-in config directory at /etc/supervisor/conf.d/. Alternatively, install it into your project's virtualenv with pip (handy for self-contained deploys).

# Option A: system package (recommended on Debian/Ubuntu)
sudo apt update && sudo apt install -y supervisor
sudo systemctl enable --now supervisor

# Drop your program files here; they are auto-included:
#   /etc/supervisor/conf.d/*.conf

# Option B: install into a virtualenv with pip
pip install supervisor
echo_supervisord_conf | sudo tee /etc/supervisord.conf

If you use Option B, add an [include] section to /etc/supervisord.conf so your per-app program files are picked up:

; /etc/supervisord.conf
[include]
files = /etc/supervisor/conf.d/*.conf

A minimal Celery 5.x app

The config below assumes a standard Celery 5.x app exposed as proj.celery:app. For a Django project, proj/celery.py typically looks like this — note the modern celery -A proj worker invocation referenced everywhere in this guide (the old celery worker -A proj ordering is gone).

# proj/celery.py
import os
from celery import Celery

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "proj.settings")

app = Celery("proj")
# Pull CELERY_* keys from Django settings
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()

# Non-Django apps instead pass the broker directly:
#   app = Celery("proj", broker="redis://localhost:6379/0")

How do you write a Supervisor config for a Celery worker?

Create /etc/supervisor/conf.d/celery.conf. Use absolute paths (Supervisor does not read your shell profile), point command at the virtualenv's celery binary, and run with a dedicated unprivileged user — never run workers as root.

; /etc/supervisor/conf.d/celery.conf
[program:celery]
command=/home/deploy/proj/venv/bin/celery -A proj worker -l info --concurrency=4
directory=/home/deploy/proj
user=deploy
numprocs=1
stdout_logfile=/var/log/celery/worker.log
stderr_logfile=/var/log/celery/worker.log
autostart=true
autorestart=true
startsecs=10
; give running tasks time to finish before SIGKILL
stopwaitsecs=600
; SIGTERM/SIGKILL the WHOLE process group, not just the parent —
; the prefork pool's child workers are otherwise orphaned on restart
stopasgroup=true
killasgroup=true
; start the worker before beat
priority=998

What do the key directives mean?

Directive	What it does	Why it matters for Celery
command	The foreground process to run	Use celery -A proj worker, not --detach
directory	Working directory before exec	So relative imports/paths resolve
user	Drop privileges to this user	Workers should never run as root
autostart	Start when supervisord starts	Brings workers up on boot
autorestart	Restart on unexpected exit	Recovers from crashes and OOM kills
startsecs	Seconds up before "running"	Avoids restart loops on fast crashes
stopwaitsecs	Grace period before SIGKILL	Lets long tasks finish (warm shutdown)
stopasgroup	Send stop signal to the group	Reaches prefork child processes
killasgroup	Send SIGKILL to the group	Prevents orphaned worker children

The stopasgroup / killasgroup pair is the single most important detail people miss. Celery's default prefork pool runs N child processes; without group signalling, a restart kills only the parent and leaves zombie children holding tasks and memory.

How do you run Celery Beat under Supervisor?

Celery Beat is the scheduler that pushes periodic tasks onto the queue. It is a singleton — running two Beat processes double-fires every scheduled task — so give it its own program and numprocs=1.

; /etc/supervisor/conf.d/celerybeat.conf
[program:celerybeat]
command=/home/deploy/proj/venv/bin/celery -A proj beat -l info --schedule=/var/lib/celery/beat-schedule
directory=/home/deploy/proj
user=deploy
numprocs=1
stdout_logfile=/var/log/celery/beat.log
stderr_logfile=/var/log/celery/beat.log
autostart=true
autorestart=true
startsecs=10
; start beat after the worker
priority=999

How do you apply and manage Supervisor configs?

After adding or editing a .conf file, tell Supervisor to re-scan and apply it. reread parses config changes, update starts new programs and restarts changed ones — you rarely need to bounce the whole supervisord.

sudo supervisorctl reread          # pick up new/changed config files
sudo supervisorctl update          # apply: start new, restart changed programs
sudo supervisorctl status          # show state of every program

sudo supervisorctl restart celery  # restart just the worker
sudo supervisorctl stop celerybeat # stop the scheduler
sudo supervisorctl tail -f celery  # stream the worker's stdout log
sudo supervisorctl tail -f celery stderr

A note on graceful (warm) shutdown

When Supervisor stops a program it sends stopsignal (default SIGTERM). Celery treats SIGTERM as a warm shutdown: it stops accepting new tasks and waits for in-flight tasks to finish. Supervisor waits stopwaitsecs for that to happen, then escalates to SIGKILL. So set stopwaitsecs comfortably above your longest-running task, otherwise long tasks get hard-killed mid-execution and may be lost (or re-run if acks_late is on). To watch what your workers are doing in real time, pair this setup with Flower for live task and queue monitoring.

Supervisor vs systemd vs Docker/Kubernetes: which should you use?

Supervisor is no longer the only sensible answer. Since systemd is the default init system on essentially every modern Linux distribution, it can supervise Celery with no extra package. In containerized deploys you usually don't run a process manager at all — the orchestrator is the supervisor. Here is how to choose.

Approach	Use it when	Pros	Trade-offs
Supervisor	One host runs several apps/workers you want managed from one place; you want a simple, distro-agnostic config	Easy .conf files, friendly supervisorctl, per-program logs, works inside minimal images	Extra dependency; another daemon to install and keep running
systemd	A single VM/bare-metal host on a modern distro; you want zero extra packages and journald logs	Built in, boots reliably, journalctl logging, cgroup resource limits	Unit syntax is fiddly; less convenient for many similar programs
Docker / Kubernetes	You already containerize; you need horizontal scaling and rolling deploys	Orchestrator restarts/scales pods; declarative, reproducible	Don't run Supervisor in the container — run the worker as PID 1 / the container's main process

Rule of thumb: new single-server deploys can just use systemd; keep Supervisor when you already rely on it or want one tool fronting many apps; in Kubernetes, make celery -A proj worker -l info the container command and let a Deployment handle restarts and scaling.

The equivalent systemd unit

For comparison, the same worker as a systemd service. KillSignal=SIGTERM plus TimeoutStopSec gives you the same warm-shutdown behaviour as Supervisor's stopwaitsecs.

# /etc/systemd/system/celery.service
[Unit]
Description=Celery worker for proj
After=network.target redis.service

[Service]
Type=simple
User=deploy
WorkingDirectory=/home/deploy/proj
ExecStart=/home/deploy/proj/venv/bin/celery -A proj worker -l info
Restart=always
KillSignal=SIGTERM
TimeoutStopSec=600

[Install]
WantedBy=multi-user.target

# Enable and start:
#   sudo systemctl daemon-reload
#   sudo systemctl enable --now celery

Supervisor isn't only for Celery — the same pattern keeps any long-running process alive, as covered in daemonize any command with Supervisor. If you run Celery alongside Django, our notes on Django and Celery best practices cover broker choice, task design, and idempotency. With 12+ years and 50+ delivered projects, MicroPyramid has shipped and kept Celery pipelines running across many production stacks; if you'd rather hand off the ops, see our server maintenance services.

Frequently Asked Questions

Should I use Supervisor or systemd for Celery in 2026?

For a brand-new deploy on a single modern Linux host, systemd is usually the simplest choice because it ships with the OS, starts on boot, and logs to journald with no extra package. Choose Supervisor when you want one tool managing several apps from friendly .conf files and a single supervisorctl, or when you're standardizing across distros. Both run the worker as a foreground process and restart it on failure.

Why do I need stopasgroup and killasgroup for Celery?

Celery's default prefork pool runs several child worker processes under one parent. Without stopasgroup=true and killasgroup=true, Supervisor signals only the parent on stop or restart, leaving the children orphaned — they keep running, hold memory, and may process tasks no one is tracking. Setting both sends the signal to the entire process group, so the whole worker shuts down cleanly.

How do I run Celery Beat with Supervisor?

Give Beat its own [program:celerybeat] section that runs celery -A proj beat -l info with numprocs=1. Beat is a singleton scheduler: running two instances double-fires every periodic task. Keep it separate from the worker so you can restart each independently, and start the worker first using priority.

How do I reload a Supervisor config without restarting everything?

Edit the program's .conf file, then run sudo supervisorctl reread to parse the changes and sudo supervisorctl update to apply them. update only starts new programs and restarts the ones whose config actually changed — it won't bounce unrelated programs or the supervisord daemon itself.

Can I run multiple Celery workers under one Supervisor config?

Yes. Either set numprocs greater than one with a process_name=%(program_name)s_%(process_num)02d template, or define multiple [program:...] sections (for example one per queue, each with its own -Q queue_name flag). Separate sections are clearer when workers consume different queues or need different concurrency.

Do I still need Supervisor when running Celery in Docker?

Generally no. In a container, run celery -A proj worker -l info as the container's main process (PID 1) and let Docker's restart policy or Kubernetes restart and scale it. Running Supervisor inside a container adds a process manager the orchestrator is already providing, so it's usually an anti-pattern unless you must run multiple processes in one image.