Storage Management

Overview

The Storage Management endpoints let you list, retrieve, and delete files that have been saved to local storage by curl jobs. Files are organized using three directory structures — by-job, by-domain, and by-date — but all three paths point to the same physical files via symlinks. Use these endpoints to audit downloads, find files by domain or date, or clean up old data.

List saved files

`GET /api/v1/curl/storage`

Retrieve a paginated list of all files saved to storage from HTTP requests.

Storage Organization:

by-job/{uuid}/filename — Primary location, organized by storage UUID
by-domain/{domain}/{uuid} — Grouped by source domain
by-date/{YYYY-MM-DD}/{uuid} — Grouped by download date

Parameters

Name	In	Type	Required	Description
`page`	query	integer	No	1-based page number (optional)
`limit`	query	integer	No	Items per page (optional; current handler returns all items when omitted)

Response

{
  "items": [
    {
      "path": "by-job/550e8400-e29b-41d4-a716-446655440000/report.pdf",
      "size": 1048576,
      "created_at": "2024-01-15T10:30:00Z",
      "job_id": "550e8400-e29b-41d4-a716-446655440000",
      "url": "https://api.example.com/reports/q4.pdf"
    },
    {
      "path": "by-date/2024-01-15/660e8400-e29b-41d4-a716-446655440111",
      "size": 2048,
      "created_at": "2024-01-15T11:45:12Z",
      "job_id": "660e8400-e29b-41d4-a716-446655440111",
      "url": "https://cdn.example.com/data.json"
    }
  ],
  "meta": {
    "total": 2,
    "page": 1,
    "limit": 50
  }
}

{
  "error": "STORAGE_ERROR",
  "message": "Failed to read storage directory"
}

Error Code	Title	Description	Resolution
`STORAGE_ERROR`	Storage read failed	Unable to read storage directory or file metadata	Check storage directory exists and has read permissions

SDK usage

const result = await client.curl.storage.listIterator();
for await (const file of result) {
  console.log(file.path, file.size);
}

With pagination parameters:

const result = await client.curl.storage.listIterator({
  page: 1,
  limit: 50,
});

Download a saved file

`GET /api/v1/curl/storage/{path}`

Retrieve the contents of a file previously saved to storage. The file is returned as a binary stream with Content-Type: application/octet-stream, making it suitable for downloads of any file type.

Path Examples:

by-job/550e8400-e29b-41d4-a716-446655440000/report.pdf
by-domain/api.example.com/660e8400-e29b-41d4-a716-446655440111
by-date/2024-01-15/770e8400-e29b-41d4-a716-446655440222

Parameters

Name	In	Type	Required	Description
`path`	path	string	Yes	Relative path to file in storage (supports nested paths)

Response

The file body is returned as a binary stream. Example (text content shown for illustration; actual responses are raw bytes):

%PDF-1.4
%âãÏÓ
1 0 obj
<< /Type /Catalog /Pages 2 0 R >>
endobj
...

{
  "error": "FILE_NOT_FOUND",
  "message": "File not found"
}

Error Code	Title	Description	Resolution
`FILE_NOT_FOUND`	File does not exist	No file exists at the specified storage path	Verify file path using listStorage, check if file was deleted

{
  "error": "FILE_READ_ERROR",
  "message": "Failed to read file"
}

Error Code	Title	Description	Resolution
`FILE_READ_ERROR`	File read failed	File exists but cannot be read	Check file permissions and disk integrity

SDK usage

const file = await client.curl.storage.getFile({
  path: "by-job/550e8400-e29b-41d4-a716-446655440000/report.pdf",
});

Delete a saved file

`DELETE /api/v1/curl/storage/{path}`

Permanently delete a file from storage. This action cannot be undone.

Parameters

Name	In	Type	Required	Description
`path`	path	string	Yes	Relative path to file in storage

Response

{
  "ok": true,
  "deleted": "by-job/550e8400-e29b-41d4-a716-446655440000/report.pdf"
}

{
  "error": "FILE_NOT_FOUND",
  "message": "File not found"
}

Error Code	Title	Description	Resolution
`FILE_NOT_FOUND`	File does not exist	Cannot delete file that doesn’t exist	Verify file path using listStorage endpoint

{
  "error": "FILE_DELETE_ERROR",
  "message": "Failed to delete file"
}

Error Code	Title	Description	Resolution
`FILE_DELETE_ERROR`	File deletion failed	File exists but could not be deleted	Check file permissions and retry operation

SDK usage

await client.curl.storage.deleteFile({
  path: "by-job/550e8400-e29b-41d4-a716-446655440000/report.pdf",
});