Each stage of the pipeline has defined behavior for error conditions. Transient errors are retried automatically; permanent errors either skip the offending record or fail the pipeline depending on severity.
Normalization
| Scenario | Behavior |
|---|
| Corrupt or unknown protobuf | Skip record, increment error counter, log warning. |
| Unknown event variant | Skip event, increment error counter, continue processing. |
ABI Decoding
| Scenario | Behavior |
|---|
| Matched signature, bad data | Skip entry, increment error counter, log warning. |
| Unmatched ABI entry | Skip or write to raw_* table, depending on the abi.unmatched config value. |
If abi.unmatched is set to raw, unmatched entries are written to a fallback table (e.g., raw_logs) rather than being dropped. This is useful for auditing or debugging incomplete ABIs.
| Scenario | Behavior |
|---|
| SQL runtime error | Fail pipeline. |
Transform failures are not retried. Fix the SQL in your configuration file and redeploy the pipeline.
Destination
Destination error handling varies by destination type. Transient failures are retried with exponential backoff. Permanent failures either skip the batch or fail the pipeline.
Database destinations (ClickHouse, Postgres)
| Scenario | Behavior |
|---|
| Transient write failure (timeout, connection reset) | Retry with exponential backoff. |
| Permanent write failure (schema mismatch, constraint violation) | Fail pipeline. |
Object Storage
| Scenario | Behavior |
|---|
| Upload failure | Retry with exponential backoff. |
Amazon SQS
| Scenario | Behavior |
|---|
| Transient failure (throttling, temporary unavailability) | Retry with exponential backoff. |
| Permanent failure (invalid queue, authorization error) | Fail pipeline. |
Webhook
| Scenario | Behavior |
|---|
| 2xx response | Success. |
| 429 (Too Many Requests) | Retry with exponential backoff. Respects Retry-After header if present. |
| 5xx (Server Error) | Retry with exponential backoff. |
| 4xx (Client Error, not 429) | Permanent failure. Skip batch. |
Return a Retry-After header with 429 responses to control the backoff interval. Without it, the pipeline uses its default exponential backoff schedule.
Monitoring
Pipeline error counters and status are available through the GoldRush Platform dashboard. Use the dashboard to:
- Track skipped record counts per stage
- Monitor retry rates and backoff patterns on destination writes
- Detect pipeline failures and view error details
- Set up alerts on error thresholds
When a pipeline fails due to a permanent error, it stops consuming new data and preserves its last checkpoint. After you fix the underlying issue, the pipeline resumes from the last successful checkpoint on restart.
