Use FastAPI BackgroundTasks for small post-response work that can safely share the web process and tolerate loss or separate reconciliation. Use Celery when discrete jobs need independent workers, routing, retries, and horizontal scaling. Use a durable workflow when business execution spans many steps, waits, approvals, schedules, or deployments. The correct choice follows persistence and failure semantics, not the number of lines in the task function.

Why is "run it in the background" an incomplete requirement?

Three different problems are commonly called background jobs:

  1. post-response cleanup inside the web application;
  2. asynchronous work executed by independent workers;
  3. persistent business workflows that cross systems and time.

Their only shared property is that the HTTP request does not wait for completion. Their reliability contracts are different. The FastAPI Background Tasks documentation says tasks run after a response is returned and recommends larger tools such as Celery for heavy computation that does not need the same process memory. It does not promise durable enqueueing, crash recovery, or exactly-once execution.

Before choosing a tool, answer five questions:

  • After the API returns 202, may the task disappear?
  • If the application exits one millisecond later, what recovers the work?
  • What happens when the same task executes twice?
  • How long can it wait, and can it cross a deployment or a day?
  • How does a caller query status, cancel, retry, and understand failure?

Without those answers, asynchronous execution merely hides failure.

How do the three models differ?

Dimension FastAPI BackgroundTasks Celery task queue Durable workflow
Execution Web application process Independent workers, possibly on other hosts Workflow workers plus persistent state
Durable enqueue No separate durable queue by default Depends on broker and publish configuration Workflow history or state is normally persisted
Typical duration Small post-response action Discrete jobs from seconds to hours Multi-step work, waits, schedules, or days
Retry Application-defined Task retries, routing, and backoff Step-level retry, timers, signals, and recovery
Composition Ordered callbacks with limited semantics Canvas primitives such as chains and groups State machine, DAG, signal, and approval models
Deployment coupling Dies with the API process API and workers are decoupled Execution is separated from workflow state
Operational cost Low Medium: broker, workers, monitoring Medium to high: workflow service and model
Dominant risk Response succeeds while work is lost Duplicate delivery, poison jobs, queue backlog History compatibility and incorrect orchestration

"Workflow" is a category, not one vendor. It means business progress is explicitly modeled and persisted so execution can resume from a committed state. A homegrown state machine, DAG scheduler, or durable-execution platform still needs its guarantees verified.

When is BackgroundTasks the right choice?

It is appropriate when work is short, uses resources already available to the application, has low failure impact, and can be repaired by another mechanism or safely retried by the user. Examples include:

  • writing a non-critical auxiliary audit record;
  • best-effort cache warming;
  • sending a notification after the authoritative business state is already durable;
  • deleting a short-lived local temporary file.

A minimal example:

from fastapi import BackgroundTasks, FastAPI, status

app = FastAPI()


def warm_cache(item_id: str) -> None:
    cache_service.warm(item_id)


@app.post("/items/{item_id}/refresh", status_code=status.HTTP_202_ACCEPTED)
async def refresh_item(item_id: str, tasks: BackgroundTasks):
    tasks.add_task(warm_cache, item_id)
    return {"accepted": True, "itemId": item_id}

The example only expresses post-response invocation. It adds no durable execution semantics. When the client receives 202, warm_cache may not have started. A rolling restart, host failure, or exception can prevent completion.

The Starlette Background Tasks documentation also states that multiple tasks execute in order and that a raised exception prevents later tasks from executing. A sequence of mandatory business side effects therefore does not become atomic or reliable by adding each function to one BackgroundTasks object.

What are the most dangerous BackgroundTasks mistakes?

Moving the business commit after the response

An unsafe design:

@app.post("/orders", status_code=202)
async def create_order(payload, tasks: BackgroundTasks):
    tasks.add_task(save_order_and_charge, payload)
    return {"accepted": True}

The API says the request was accepted before an order exists. If the process exits, there is no durable intent to recover. Commit an order or job record first, then dispatch processing.

Running blocking work in an async function

async def does not automatically make CPU-intensive code or a synchronous SDK non-blocking. Blocking the event loop degrades every request on the worker. A small synchronous callback can use an appropriate thread path; image processing, model inference, large file conversion, and heavy computation normally belong outside the web process.

Capturing request-scoped objects

The response may close request transactions, temporary files, database sessions, or context variables. Pass stable identifiers to the background function and reacquire resources. Do not capture an ORM object whose session is about to close.

Treating a log line as failure handling

If there is no job record, metric, or alert, the user sees 202 and the team finds the failure only by accident. Business-relevant work needs queryable state and an accountable failure owner.

When should a service move to Celery?

Celery is a typical next step when work needs separate processes, multiple workers, queue routing, concurrency controls, or standardized retries. The Celery 5.6 Tasks guide emphasizes idempotent task design. Depending on acknowledgment and worker-failure behavior, a message may execute again.

Good candidates include:

  • email, webhooks, bulk imports, and document conversion;
  • CPU or memory workloads that must not affect API latency;
  • routing by tenant, priority, region, or hardware;
  • controlled retry, backoff, and soft or hard time limits;
  • independent worker scaling and maintenance windows.

A safe FastAPI-to-Celery boundary passes a stable identifier:

@app.post("/exports", status_code=202)
async def create_export(request: ExportRequest):
    job = await jobs.create(
        kind="export",
        state="queued",
        input_ref=request.input_ref,
    )
    build_export.apply_async(
        args=[str(job.id)],
        task_id=str(job.dispatch_id),
    )
    return {"jobId": str(job.id), "state": "queued"}

This example omits an important transaction boundary. If the database commits and broker publication fails, the job can remain queued forever. If the message is published before a transaction rolls back, the worker cannot find its job. Use a transactional outbox, reliable dispatch table, or equivalent reconciliation mechanism to connect committed business intent with eventual message visibility.

How do you make a Celery task safe to retry?

The foundation is idempotent side effects, not an autoretry_for decorator.

Use a business idempotency key

Create a stable key for each external effect, such as export:{job_id}:upload or invoice:{invoice_id}:send. Claim it atomically before execution and persist the result. A replay reads the existing result rather than charging, sending, or creating again.

Separate transient from permanent failures

  • timeouts, 429 responses, and temporary 5xx failures: retry a bounded number of times with jitter;
  • invalid input, denied permission, and permanently absent resources: fail immediately;
  • unknown failures: quarantine or require review rather than retrying forever.

Bound time and resources

Every external call needs connection and read timeouts. Limit task duration, memory, and input size. After a timeout, reconcile whether the downstream service committed; "the worker did not receive a response" does not mean "the remote operation did not happen."

Protect transitions with compare-and-swap

Use revision or conditional updates when workers move a job from queued to running, succeeded, or failed. A late attempt must not overwrite a newer execution. The same principle protects publication state in this FastAPI, MongoDB, and Redis bilingual publishing guide.

When is a task queue still not enough?

Celery Canvas can compose chains, groups, and chords. When a business process includes long waits, external approval, human input, timers, compensation, and several systems, however, a task chain can scatter state across the broker, result backend, and callbacks.

Signals that favor a durable workflow include:

  • one run lasts hours or days;
  • processing pauses for a webhook, user approval, or scheduled time;
  • each step has different retry, timeout, and compensation policy;
  • recovery must resume from step seven rather than replaying everything;
  • the product needs a complete progress view and manual intervention;
  • the business requires an auditable state history.

Temporal's durable execution documentation explains its event-history model for resuming workflow execution. Other engines use different designs. Do not infer identical guarantees from the word durable; evaluate state storage, deterministic-code constraints, version compatibility, retries, signals, and disaster recovery for the chosen system.

What should a reliable HTTP 202 API look like?

HTTP 202 means processing was accepted, not completed successfully. Model the job as a first-class resource:

POST /exports
Idempotency-Key: 8b3f...

202 Accepted
Location: /exports/job_123

{"jobId":"job_123","state":"queued"}

A status response might be:

{
  "jobId": "job_123",
  "state": "running",
  "revision": 4,
  "progress": {"completed": 18, "total": 40},
  "result": null,
  "error": null
}

Important rules:

  1. bind Idempotency-Key to the authenticated caller, route, and canonical input;
  2. return the same jobId when the same intent is replayed;
  3. allow only explicit state transitions;
  4. treat progress as a hint, not a final result;
  5. expose a stable error code and actionable message without internal stacks;
  6. implement cancellation as a conditional transition, not merely a process kill;
  7. expire results intentionally and enforce access control.

A real-time UI can use the stream design in this FastAPI SSE production guide, but the SSE connection cannot be the source of truth. After disconnecting, a client must reconstruct the view from the job resource.

Where should scheduled jobs run?

A schedule is a trigger, not a reliability model.

  • a single-instance, lossy maintenance action may use a simple scheduler;
  • Celery Beat can periodically publish Celery tasks, but only one scheduler may own a given schedule;
  • multi-step runs with dependencies or backfills fit a DAG or workflow;
  • a cloud timer should call an idempotent API rather than contain the entire business process in a shell command.

A recurring automation can be productized in a workflow tool such as Cronova. Independently verify duplicate-trigger deduplication, missed-window recovery, time-zone and DST behavior, and how historical runs behave after task definitions change.

How can a team migrate away from BackgroundTasks incrementally?

Phase 1: introduce a Job model

Even if execution remains in-process, persist queued, running, succeeded, and failed, along with an input reference, idempotency key, and revision. This stabilizes UI, audit, and recovery contracts first.

Phase 2: extract the executor

Make the executor accept only job_id. It loads inputs, atomically claims work, and writes state. It must not depend on a FastAPI Request, mutable global, or uncommitted transaction.

Phase 3: replace dispatch

Replace background_tasks.add_task(run_job, job_id) with Celery publication or workflow start. The API and Job contract remain stable, concentrating migration risk in dispatch.

Phase 4: add an outbox and reconciliation

Regularly find records where the database committed but dispatch did not happen, execution timed out, or completion events are missing. Recovery must be idempotent and reuse the original job and revision semantics.

What observability is minimally necessary?

Every model should expose low-sensitivity control-plane data:

  • job_id, task_id, or workflow_id;
  • task kind, queue, attempt, state, and duration;
  • retry reason code and next scheduled time;
  • queue latency and execution latency;
  • worker heartbeat and backlog depth;
  • final failure owner and resolution.

Do not log complete payloads, prompts, uploaded files, credentials, or third-party responses by default. Trace propagation should correlate the API, broker, and worker without putting user data in span names or high-cardinality labels.

Common failure modes

The API returns 202 and the task disappears

Critical work was probably placed in an in-process callback, or a database/broker dual write lacks an outbox. First locate the persistent Job, then its dispatch record. Without a Job, reliable recovery is impossible.

A notification or charge happens twice

Retries are an expected distributed-systems behavior. The root cause is a side effect without an idempotency boundary. Use business keys, unique constraints, and downstream idempotency keys.

The queue grows while worker CPU stays low

Possible causes include slow dependencies, inappropriate prefetch, a poison task retry loop, lock contention, or incorrect routing. Separate queue latency, active/reserved counts, external-call duration, and retry distributions.

Old workflows cannot resume after deployment

Long workflows cross code versions. Version workflow definitions, preserve compatibility with historical events, or use an explicit migration or continue-as-new strategy. Do not assume every run restarts on current code.

FAQ

Does BackgroundTasks retry automatically?

It provides no independent durable-queue retry guarantee. The application must observe and handle exceptions, and a terminated process has no automatic recovery contract.

Does Celery guarantee exactly once?

Do not design it as an exactly-once side-effect executor. Acknowledgment behavior, worker failure, publish retries, and downstream timeouts can create duplicates or uncertain outcomes. Design idempotent tasks and reconciliation with at-least-once delivery in mind.

Does using Redis mean a project should use Celery?

No. Redis can be a Celery broker, cache, Stream store, or part of another queue design. Sharing a component does not make the semantics equivalent. Define recovery and execution contracts first.

Should a small team start with a workflow engine?

If the actual business process waits across days, approvals, and many recoverable steps, explicit workflow state may be simpler than callback maintenance. A short email or cache warmup does not justify that operational surface.

Final decision checklist

  • [ ] Critical intent is durable before returning 202.
  • [ ] Crash-recovery ownership is explicit.
  • [ ] Duplicate execution cannot duplicate side effects.
  • [ ] Database and broker dual writes have an outbox or reconciliation.
  • [ ] Retry, timeout, cancellation, and permanent failure are distinct.
  • [ ] Users can query a stable Job resource.
  • [ ] Long waits and approvals use explicit workflow state.
  • [ ] Schedules define time zone, backfill, and single-owner behavior.
  • [ ] Workers, queues, and dependencies are observable.
  • [ ] Logs exclude credentials and sensitive payloads.

The simplest reliable rule is: use BackgroundTasks for disposable post-response work, Celery for independently executed retryable jobs, and durable workflows for long-lived state and coordination. Persist every critical intent before returning HTTP 202.