The Kafka destination forwards raw bytes from the pipeline source directly to a target Kafka topic. Unlike other destinations, Kafka operates in Raw pipeline mode - no normalization, ABI decoding, or transforms are applied.
Configuration
destination:
type: "kafka"
bootstrap_servers: "broker1:9092,broker2:9092"
sasl:
mechanism: "SCRAM-SHA-512"
username: "${KAFKA_USER}"
password: "${KAFKA_PASSWORD}"
output_topic: "my-output-topic"
Fields
| Field | Type | Required | Description |
|---|
bootstrap_servers | string | yes | Target Kafka broker addresses |
sasl.mechanism | string | yes | SASL mechanism (e.g., SCRAM-SHA-512) |
sasl.username | string | yes | SASL username |
sasl.password | string | yes | SASL password |
output_topic | string | yes | Target topic name |
Key Behavior
- Raw passthrough - bytes from the source are forwarded without any processing. No normalization, ABI decoding, or transform stages run.
- Exactly-once delivery - the destination uses transactional producing to guarantee exactly-once semantics end to end.
- Validation at startup - if your pipeline configuration includes
abi or transforms sections alongside a Kafka destination, the pipeline will reject the configuration and fail to start.
ABI decoding and transform configurations are not compatible with the Kafka destination. Remove those sections from your pipeline definition before deploying.
