Scaling Transactions API with Time Buckets

Explore GoldRush's time buckets for efficient transaction retrieval. Utilize 15-minute intervals for precise, scalable data access.

Getting bulk transactions with time buckets

GoldRush just launched a brand new endpoint that allows you to get all transactions in bulk, simply by specifying a time bucket. This is useful for developers who want to get all granular transactions data before, during, or after a specified time interval, especially for high-volume wallet addresses.

What are time buckets?

Time buckets are 15-min windows locked in time, beginning on 1 Jan 1970 00:00:00 (Unix Epoch time). Time bucket 0 starts at 1970-01-01 00:00:00 inclusive and ends at 1970-01-01 00:14:59 inclusive. When you specify a time bucket, all transactions within that 15-min window are returned.

Calculating time buckets

Suppose you want to find all transactions between 2023-03-08 03:35:00 GMT and 2023-03-08 03:58:00 GMT for a particular address.

To calculate the time bucket, the following method can be used.

Convert the start time 2023-03-08 02:35:00 GMT to its unix time – 1678242900
Divide the unix time by 900 (number of seconds in 15 minutes)
1. 1678242900 / 900 = 1864714.3333
Floor this number to the nearest integer – 1864714

That's it! The time bucket to use is 1864714.

This will give you the 15-min window of 02:30:00 to 02:45:00 (as we've rounded the time bucket down).
To get the next 15 minutes (till 03:00:00), simply supply the next time-bucket 1864715.

To get all the transactions for an account with many transactions, multiple calls will be necessary as you will need to crawl all the time buckets. Thus, it is recommended that transactions are stored locally so that the same time-bucket and wallet combination isn't called more than once.

Time buckets and Paginations

The time bucket method can be thought of an alternate way to do paginations. The difference is that pages are densely packed with items, while some time buckets can be empty if there are no transactions within that bucket.

Why are time buckets needed?

The initial version of the transaction endpoint was built to show the last N transactions for a wallet for display purposes. But having rich, granular, historical transaction data proved to be much more useful for other advanced use cases like calculating risk scores and profit-loss statements to calculate your tax liabilities. These sorts of paginated historical transaction queries are particularly high-intensity on our backend.

The time bucket method will scale to wallets with millions of transactions and not time out.

Calling transactions with time buckets

The endpoint takes three path parameters: chainName, address and timeBucket:

Get bulk time bucket transactions for address (v3)

Commonly used to fetch all transactions including their decoded log events in a 15-minute time bucket interval.

When you call the API with a particular time bucket:

curl "https://api.covalenthq.com/v1/eth-mainnet/bulk/transactions/0x88e6...5640/1864685/"

A sample response is:

{
  "data": {
    "address": "0x95222290dd7278aa3ddd389cc1e1d165cc4bafe5",
    "updated_at": "2023-03-08T05:48:33.744864117Z",
    "next_update_at": "2023-03-08T05:53:33.744893744Z",
    "quote_currency": "USD",
    "chain_id": 1,
    "chain_name": "eth-mainnet",
    "complete": true,
    "links": {
      "prev": "https://api.covalenthq.com/v1/eth-mainnet/bulk/transactions/0x95222290dd7278aa3ddd389cc1e1d165cc4bafe5/1864714/",
      "next": "https://api.covalenthq.com/v1/eth-mainnet/bulk/transactions/0x95222290dd7278aa3ddd389cc1e1d165cc4bafe5/1864716/"
    },
    "items": [
      {...18 items},
      {...18 items},
      {...18 items},
      {...18 items},
      {...18 items}
    ]
  },
  "error": false,
  "error_message": null,
  "error_code": null
}

The most important keys in the response are:

links.prev - the link to the previous non-empty time bucket
links.next - the link to the next non-empty time bucket so that you don't have to crawl every single time bucket exhaustively
complete - boolean if the time bucket is in the past and you can be sure that the bucket has been fully filled. If complete is false, it means that the bucket is still being filled and you can fetch this bucket again to get the latest transactions (if any.)

Supported Chains

The time bucket extension is supported on all chains through the Unified API.

Sample Code

// For calculating time bucket

var epochInMilli = Date.parse("01 Jan 1970 00:12:00 GMT");
var epoch = epochInMilli / 1000;

var bucket = Math.floor(epoch / 900);

Building Web3 Wallets (Part 4) - Displaying Curve Liquidity Pools

How to Track Wallets or Transactions with the GoldRush API