Flat-file access is now live

TL;DR

• You can now generate temporary S3-compatible credentials from the dashboard.
• Use them with AWS CLI, boto3, or any S3-compatible client.
• You can list files, browse prefixes, and download objects directly.
• Credentials are read-only and currently expire after 4 hours.

Flat-file access is now available in CryptoHFTData.

If you want direct access to dataset files instead of downloading through a single REST URL, you can now generate temporary S3 credentials from the dashboard and use standard S3 tooling.

What you can do with it

With the new flat-file flow, you can:

• list bucket contents
• browse exchange, date, and hour prefixes
• download a single file directly
• copy a full date partition recursively
• plug CryptoHFTData into existing boto3 or S3-based workflows

This is useful when you want raw file access instead of a higher-level wrapper.

If you prefer the existing flows:

• use the REST API when you already know the exact file you want
• use the Python SDK when you want pandas DataFrames directly
• use flat-file access when you want direct object-store style access

How to get credentials

1. Open your dashboard
2. Go to the API Keys section
3. Generate or reveal your normal API key if needed
4. Press Generate S3 Credentials
5. Copy the returned values into AWS CLI, boto3, or your own S3 client

The dashboard shows:

• bucket
• endpoint
• region
• permission
• expiration time
• access key id
• secret access key
• session token

If the credentials expire, generate a new set from the dashboard.

Bucket layout

Object keys follow this pattern:

{exchange}/{YYYY-MM-DD}/{HH}/{symbol}_{data_type}.parquet.zst

Example:

binance_spot/2025-08-01/20/BTCUSDT_trades.parquet.zst

AWS CLI example

Export the values returned by the dashboard:

export AWS_ACCESS_KEY_ID="<TEMP_ACCESS_KEY_ID>"
export AWS_SECRET_ACCESS_KEY="<TEMP_SECRET_ACCESS_KEY>"
export AWS_SESSION_TOKEN="<TEMP_SESSION_TOKEN>"
export CHD_BUCKET="cryptohftdata-processed"
export CHD_ENDPOINT="<R2_ENDPOINT_FROM_DASHBOARD>"

List the bucket:

aws s3 ls "s3://$CHD_BUCKET" \
  --endpoint-url "$CHD_ENDPOINT" \
  --region auto

Download one file:

aws s3 cp "s3://$CHD_BUCKET/binance_spot/2025-08-01/20/BTCUSDT_trades.parquet.zst" ./BTCUSDT_trades.parquet.zst \
  --endpoint-url "$CHD_ENDPOINT" \
  --region auto

Download a full date prefix:

aws s3 cp "s3://$CHD_BUCKET/binance_spot/2025-08-01/" ./binance_spot_2025-08-01/ \
  --recursive \
  --endpoint-url "$CHD_ENDPOINT" \
  --region auto

boto3 example

You can also use the returned credentials directly in Python:

import boto3

s3 = boto3.client(
    "s3",
    endpoint_url=payload["endpoint"],
    region_name=payload["region"],
    aws_access_key_id=creds["access_key_id"],
    aws_secret_access_key=creds["secret_access_key"],
    aws_session_token=creds["session_token"],
)

objects = s3.list_objects_v2(
    Bucket=payload["bucket"],
    Prefix="binance_spot/2025-08-01/20/",
)

for item in objects.get("Contents", []):
    print(item["Key"], item["Size"])

Quick usage notes

• Credentials are temporary, so refresh them when they expire.
• The current permission is read-only.
• Keep --region auto when using AWS CLI against R2.
• Use the endpoint shown in your dashboard response.

Where to start

If you want to try it now:

1. Open the dashboard
2. Generate S3 credentials
3. Run the AWS CLI example above
4. Read the docs at Flat-File Access

That is the fastest path to direct flat-file access in CryptoHFTData.