Amazon SQS: Message Queue Processing with Python (boto3)

Amazon Simple Queue Service (SQS) is a fully managed message queue that lets one part of your system hand work to another without the two being online at the same time. A producer drops a message on the queue; one or more consumers pull it off, do the work, and delete it. Because the queue sits in the middle, you get four things almost for free: decoupling (services don't call each other directly), buffering (traffic spikes pile up in the queue instead of crushing a downstream service), resilience (if a consumer crashes mid-work the message reappears and is retried), and independent scaling (add more consumers when the backlog grows).

SQS comes in two flavours. Standard queues give you near-unlimited throughput with at-least-once delivery and best-effort ordering - fast, cheap, but you may occasionally see a message twice or out of order. FIFO queues give you strict ordering and exactly-once processing within a message group, at a lower (but still high) throughput. Pick Standard unless duplicates or order would actually break your business logic.

This guide covers both queue types and shows production-grade Python with the current boto3 (Python 3.13): creating queues, producing single and batched messages, consuming with long polling and the visibility-timeout contract, the modern SQS-to-Lambda consumer, dead-letter queues for poison messages, and how SQS compares to SNS, EventBridge and Kinesis.

Standard vs FIFO queues

The queue type is the single most important decision and it is fixed at creation time - you cannot convert one into the other.

	Standard queue	FIFO queue
Ordering	Best-effort	Strict, ordered per message group
Delivery	At-least-once (duplicates possible)	Exactly-once processing
Throughput	Nearly unlimited	300 msg/s, or 3,000 with batching, per API action (much higher with high-throughput mode)
Deduplication	None	5-minute dedup window (content-based or explicit dedup id)
Queue name	Any	Must end in .fifo
Use when	Maximum throughput and you can tolerate duplicates / reordering	Order matters and a message must be processed once

A practical rule: if you are processing analytics events, sending emails, or resizing images, Standard is fine - just make your consumer idempotent. If you are applying financial transactions or state changes that must happen in order (per account, per order), reach for FIFO and use the account/order id as the MessageGroupId.

Creating a queue with boto3

Use the boto3 client rather than the resource API - it maps directly to the SQS API and is what you will see in the AWS docs. In production, leave credentials to the IAM role (on Lambda, ECS, or EC2) and only set region_name; never hard-code access keys.

import boto3

# Credentials & region come from the environment or the attached IAM role.
sqs = boto3.client("sqs", region_name="us-east-1")

# Standard queue
standard = sqs.create_queue(
    QueueName="orders",
    Attributes={
        "VisibilityTimeout": "60",            # seconds a consumer holds a message
        "MessageRetentionPeriod": "345600",   # 4 days (1 min - 14 days)
        "ReceiveMessageWaitTimeSeconds": "20",  # enable long polling on the queue
        "SqsManagedSseEnabled": "true",        # server-side encryption at rest
    },
)
queue_url = standard["QueueUrl"]

# FIFO queue - name MUST end in .fifo
fifo = sqs.create_queue(
    QueueName="orders.fifo",
    Attributes={
        "FifoQueue": "true",
        "ContentBasedDeduplication": "true",  # derive dedup id from the body hash
        "VisibilityTimeout": "60",
    },
)

Producing messages

A message body is plain text up to 256 KB (JSON is the common choice). Attach message attributes for metadata you want consumers to filter or route on without parsing the body. Always batch when you can: send_message_batch ships up to 10 messages in one network call, cutting both latency and request cost.

# Single message with a typed attribute (Standard queue)
sqs.send_message(
    QueueUrl=queue_url,
    MessageBody='{"order_id": 1001, "total": 49.90}',
    MessageAttributes={
        "event_type": {"DataType": "String", "StringValue": "order.created"},
    },
)

# Batch send - up to 10 entries, each needs a batch-local unique Id
resp = sqs.send_message_batch(
    QueueUrl=queue_url,
    Entries=[
        {"Id": "1", "MessageBody": '{"order_id": 1002}'},
        {"Id": "2", "MessageBody": '{"order_id": 1003}'},
    ],
)
print("sent:", resp.get("Successful"), "failed:", resp.get("Failed"))

# FIFO requires a MessageGroupId (the ordering scope). The dedup id is
# optional when ContentBasedDeduplication is enabled on the queue.
sqs.send_message(
    QueueUrl=fifo["QueueUrl"],
    MessageBody='{"order_id": 1004}',
    MessageGroupId="customer-42",          # messages in this group stay ordered
    # MessageDeduplicationId="order-1004", # needed if dedup is not content-based
)

Consuming: long polling and the visibility-timeout contract

Receiving is where most SQS bugs live, so understand the contract first.

Long polling. Set WaitTimeSeconds (1-20) on every receive. The call waits until a message arrives or the timeout elapses, instead of returning instantly empty. This slashes the number of empty receives (and therefore request cost) and reduces latency. Short polling (WaitTimeSeconds=0) is almost never the right choice.

Visibility timeout. When you receive a message it is not deleted - it becomes invisible to other consumers for the duration of VisibilityTimeout. You must delete it yourself once you have processed it successfully. If your code crashes or simply doesn't delete in time, the message becomes visible again and another consumer retries it. Set the timeout longer than your worst-case processing time, or extend it mid-flight with change_message_visibility for long jobs. This is exactly why Standard-queue consumers must be idempotent: a slow handler can cause the same message to be processed twice.

import json


def process(message):
    """Your business logic. Must be idempotent on Standard queues."""
    body = json.loads(message["Body"])
    print("handling", body)


def consume(queue_url):
    while True:
        resp = sqs.receive_message(
            QueueUrl=queue_url,
            MaxNumberOfMessages=10,        # batch receive (1-10)
            WaitTimeSeconds=20,            # long polling: wait up to 20s
            VisibilityTimeout=60,          # hide messages while we work
            MessageAttributeNames=["All"],
        )
        messages = resp.get("Messages", [])
        if not messages:
            continue  # long poll returned empty; loop again

        to_delete = []
        for msg in messages:
            try:
                process(msg)
                to_delete.append(
                    {"Id": msg["MessageId"], "ReceiptHandle": msg["ReceiptHandle"]}
                )
            except Exception as exc:  # noqa: BLE001
                # Don't delete: the message reappears after VisibilityTimeout
                # and is retried, eventually landing in the DLQ if it keeps failing.
                print("failed, will retry:", exc)

        if to_delete:
            # Delete only what we handled, in one batched call.
            sqs.delete_message_batch(QueueUrl=queue_url, Entries=to_delete)

The modern consumer: SQS to Lambda

For most workloads you should not run a polling loop at all. An event source mapping lets Lambda poll SQS for you, scale the number of concurrent executions with the backlog, and invoke your function with a batch of records. You write only the handler.

The one feature to always enable is partial batch responses. Configure the mapping with FunctionResponseTypes=["ReportBatchItemFailures"] and return the ids of only the records that failed. Without it, a single bad record forces the entire batch to be retried (and re-processed), which is both wasteful and a duplicate-processing hazard.

import json


def handle(body):
    """Idempotent business logic for one record."""
    ...


def lambda_handler(event, context):
    """SQS-triggered consumer using partial batch responses.

    Set the event source mapping's FunctionResponseTypes to
    ["ReportBatchItemFailures"] so only failed records are retried,
    not the whole batch. A record that keeps failing is moved to the
    DLQ once maxReceiveCount is exceeded.
    """
    failures = []
    for record in event["Records"]:
        try:
            handle(json.loads(record["body"]))
        except Exception:  # noqa: BLE001
            failures.append({"itemIdentifier": record["messageId"]})
    return {"batchItemFailures": failures}

Dead-letter queues and redrive

A dead-letter queue (DLQ) is an ordinary SQS queue that catches messages your consumer can never process - a malformed payload, a row that no longer exists, a permanent downstream error. You attach a redrive policy to the source queue: after a message has been received maxReceiveCount times without being deleted, SQS automatically moves it to the DLQ. This stops a single "poison" message from being retried forever and blocking the queue.

Operate the DLQ as an alarm and a recovery tool: alert on ApproximateNumberOfMessagesVisible > 0, inspect the failures, fix the bug or data, then use DLQ redrive (console or the start_message_move_task API) to send the messages back to the source queue for reprocessing.

import json

# 1. Create the dead-letter queue and read its ARN.
dlq_url = sqs.create_queue(QueueName="orders-dlq")["QueueUrl"]
dlq_arn = sqs.get_queue_attributes(
    QueueUrl=dlq_url, AttributeNames=["QueueArn"]
)["Attributes"]["QueueArn"]

# 2. Point the source queue at the DLQ after 5 failed receives.
sqs.set_queue_attributes(
    QueueUrl=queue_url,
    Attributes={
        "RedrivePolicy": json.dumps(
            {"deadLetterTargetArn": dlq_arn, "maxReceiveCount": 5}
        )
    },
)

Delay queues, large payloads and idempotency

Delay queues

Set DelaySeconds (0-900) on a queue, or DelaySeconds per message, to make new messages invisible for a short window after they are sent - handy for spacing out work or giving an upstream write time to settle. Note this is different from visibility timeout, which applies after a message is received.

Messages larger than 256 KB

The hard limit on a message is 256 KB. For bigger payloads, store the blob in S3 and put only the object key on the queue, or use the SQS Extended Client (Java) which does this transparently. In Python, uploading to S3 and sending the key is a few lines and keeps your messages small and cheap.

Idempotency on Standard queues

Because Standard delivery is at-least-once, design every consumer to be safe to run twice on the same message. The usual pattern is a deduplication store: write the MessageId (or a business key) to DynamoDB with a conditional put, and skip the work if it already exists. Our event-driven Lambda guide walks through this idempotency pattern with DynamoDB end to end.

Error handling, retries and monitoring

Retries are built in: failing to delete a message is the retry signal, and maxReceiveCount plus the DLQ bounds how many times it happens. For transient downstream errors, let the message return after the visibility timeout rather than retrying tightly in-process; for genuinely poison messages, let them flow to the DLQ. When you do retry in code (for example around a flaky API call inside one handler), use exponential backoff with jitter.

Monitoring comes from a handful of CloudWatch metrics. Watch ApproximateNumberOfMessagesVisible (the backlog - if it climbs, add consumers or raise Lambda concurrency), ApproximateAgeOfOldestMessage (the single best signal that consumers are stuck or too slow), NumberOfMessagesSent / NumberOfMessagesDeleted, and any non-empty DLQ. Alarm on a growing oldest-message age and on messages in DLQ > 0. For deeper Lambda-consumer hardening - cold starts, batch sizing, observability - see our AWS Lambda best practices guide.

SQS vs SNS vs EventBridge vs Kinesis

These services overlap, but each has a clear lane:

• SQS - point-to-point work queue: one message is processed by one consumer, with retries and a DLQ. Use it to decouple and buffer asynchronous work (jobs, orders, image processing).
• SNS - pub/sub fan-out: one message is pushed to many subscribers at once. The classic pattern is SNS to many SQS queues, so each downstream service gets its own buffered copy.
• EventBridge - event bus with content-based routing, schemas, SaaS integrations and scheduling. Reach for it when many event types must be routed by rules to many targets, not just queued.
• Kinesis - ordered, replayable streams for high-volume analytics and multiple independent readers that each track their own position. Use it for clickstreams and telemetry, not for a task queue.

Many production systems combine them - SNS fans an event out to several SQS queues, and Lambda drains each queue. There is no single "best"; match the tool to whether you need a queue, a fan-out, a router, or a stream.

Build it with MicroPyramid

MicroPyramid has spent 12+ years and 50+ delivered projects building event-driven systems on AWS for startups and enterprises. We design SQS/SNS/EventBridge topologies, write the Python consumers, set sane visibility timeouts and DLQs, and wire up CloudWatch alarms so the queue runs itself in production. If you want a second pair of hands, our AWS consulting services, cloud migration services and Python development services cover everything from a first queue to a full re-platform.

Frequently Asked Questions

What is the difference between SQS Standard and FIFO queues?

Standard queues give near-unlimited throughput with at-least-once delivery and best-effort ordering, so you may occasionally see a duplicate or an out-of-order message. FIFO queues guarantee strict ordering within a message group and exactly-once processing, at up to 300 messages/second (3,000 with batching, more in high-throughput mode). Choose Standard for maximum throughput when your consumer is idempotent; choose FIFO when order and no-duplicates are non-negotiable.

How does the SQS visibility timeout work?

When a consumer receives a message, SQS does not delete it - it hides the message from other consumers for the visibility-timeout period. Your code must explicitly delete the message after processing it. If processing fails or runs past the timeout, the message becomes visible again and is retried. Set the timeout longer than your worst-case processing time, or extend it mid-flight with change_message_visibility.

Should I use short polling or long polling?

Use long polling. Set WaitTimeSeconds to 20 on your receive calls (or ReceiveMessageWaitTimeSeconds on the queue) so the request waits for a message instead of returning immediately. Long polling dramatically reduces empty receives, lowers request cost, and cuts latency. Short polling (WaitTimeSeconds=0) is rarely the right default.

What is a dead-letter queue and when should I use one?

A dead-letter queue (DLQ) is a separate SQS queue that captures messages a consumer repeatedly fails to process. You attach a redrive policy with maxReceiveCount to the source queue; once a message has been received that many times without being deleted, SQS moves it to the DLQ. This prevents a single poison message from being retried forever. Always configure a DLQ in production and alarm when it is non-empty.

How do I handle SQS messages larger than 256 KB?

The maximum message size is 256 KB. For larger payloads, store the data in S3 and put only the object key (and any small metadata) on the queue; the consumer reads the object from S3. The SQS Extended Client library automates this in Java, and the same pattern is a few lines of boto3 in Python.

Should I poll SQS myself or trigger Lambda from the queue?

For most workloads, use an SQS-to-Lambda event source mapping: Lambda polls the queue, scales with the backlog, and invokes your handler with a batch - no polling loop to run or operate. Enable partial batch responses (ReportBatchItemFailures) so only failed records are retried. Run your own polling loop only for long-lived workers, very high sustained throughput, or stateful processing where a managed consumer doesn't fit.