Skip to main content
The AWS SQS destination delivers structured pipeline output as messages to an Amazon SQS queue. It supports both standard and FIFO queues with built-in deduplication.

​
Configuration

destination:
  type: "sqs"
  queue_url: "https://sqs.us-east-1.amazonaws.com/123456789/my-queue.fifo"
  region: "us-east-1"
  access_key_id: "${AWS_ACCESS_KEY_ID}"
  secret_access_key: "${AWS_SECRET_ACCESS_KEY}"
  batch_size: 10
  message_group_id: ""

​
Fields

FieldTypeRequiredDefaultDescription
queue_urlstringyes—SQS queue URL
regionstringyes—AWS region
access_key_idstringyes—AWS access key
secret_access_keystringyes—AWS secret key
batch_sizeintno10Messages per batch (max 10)
message_group_idstringnoauto-derivedFIFO message group ID

​
FIFO Queue Detection

FIFO mode is automatically activated when either condition is met:
  • The queue_url ends with .fifo
  • A message_group_id value is explicitly set
When FIFO mode is active, the destination sets the MessageGroupId and MessageDeduplicationId on every message.

​
Message Format

Each message is a JSON object containing the project, stream name, and the record payload: 
{
  "project": "analytics-prod",
  "stream": "hl_fills",
  "record": {
    "block_number": 12345,
    "...": "..."
  }
}

​
Message Attributes

Each SQS message includes the following message attributes:
AttributeTypeDescription
projectStringProject name
streamStringStream name
block_numberStringBlock number, if present in the record

​
Deduplication

For FIFO queues, the deduplication ID is deterministically derived from source metadata using the format: 
{topic}-{partition}-{offset}
This ensures that retried deliveries do not produce duplicate messages in the queue.