Execution Archival
Every time an automation runs, it creates execution records in the database. Over weeks and months, these records add up. The archival service automatically moves old records to cold storage so your database stays fast and your storage costs stay low.
"Where do old execution records go?"
They are compressed and moved to long-term storage (S3 or MinIO), organized neatly by date. You do not lose the data — it simply moves somewhere cheaper.
Why Archival Matters
| Without archival | With archival |
|---|---|
| Database grows continuously | Database stays a manageable size |
| Queries slow down over time | Queries remain fast |
| Storage costs increase | Old records move to cheaper storage |
| No automatic cleanup | Runs on a schedule with no manual effort |
The Basic Idea
Two Archival Modes
The archival service supports two modes. You can use one or both at the same time.
Time-Window Mode (the default)
This is the standard mode that applies to all jobs automatically. It archives execution records that fall within a specific age range, while protecting recent records.
| Setting | What it controls | Example |
|---|---|---|
| Recent months to keep | How many months of recent data are protected | 2 months |
| Archive window (months) | How far back to look for records to archive | 6 months |
Example: Suppose recent months to keep is set to 2 and the archive window is 6 months. If archival runs in March:
Per-Job Count Mode (opt-in)
Some jobs run very frequently and create a large number of records in a short time. For these jobs, you can switch to count-based archival instead. This mode keeps the N most recent executions per job and archives the rest.
| Setting | What it controls | Default |
|---|---|---|
| Archival enabled | Turns on per-job count mode for this job | Off |
| Executions to keep | How many recent executions to keep | 100 |
Example: A job runs every 15 minutes and has 2,000 execution records. With executions to keep set to 100, the archival service keeps the 100 most recent runs and archives the other 1,900.
What Gets Archived
Each archived record includes the complete picture of that execution run.
| Data | What it contains |
|---|---|
| Execution record | Status, start time, end time, trigger details, and result |
| Step records | Every step that ran, including its input, output, status, and duration |
Nothing is left behind. When an execution is archived, all of its step details go with it.
Safety: How Your Data Is Protected
The archival service is designed to never lose your data. Several safety measures work together:
| Safety feature | How it works |
|---|---|
| Upload verification | After uploading, the service checks the file size to confirm the archive was stored correctly |
| Delete only after verification | Original records are removed from the database only after the archive is confirmed |
| Automatic retry | If an upload fails, the service retries up to 3 times with increasing wait times (2 seconds, 4 seconds, 8 seconds) |
| Batch processing | Records are processed in batches (default: 500 at a time) to prevent memory issues with large datasets |
| Distributed lock | Only one archival process can run at a time, even across multiple servers |
Enabling Per-Job Archival
To switch a specific job from the default time-window mode to per-job count mode:
- Open the job settings.
- Find the execution configuration section.
- Turn on Archival Enabled.
- Optionally set Executions to Keep to a number that makes sense for your job (the default is 100).
For high-frequency jobs that run every few minutes, switch to per-job count mode and set a retention count that covers your typical review window — this prevents the database from accumulating thousands of short-lived records before the monthly time-window archival runs.
| Job type | Suggested approach |
|---|---|
| Runs daily or less often | Time-window mode (default) is usually fine |
| Runs hourly | Consider per-job mode with 200-500 executions kept |
| Runs every few minutes | Per-job mode is recommended with a count that covers your review needs |
What Archived Data Looks Like
Archived files are stored in a date-organized structure in cold storage:
| Level | Example | Purpose |
|---|---|---|
| Folder by year | job-executions/year=2025/ | Groups all archives from one year |
| Folder by month | job-executions/year=2025/month=06/ | Groups archives from one month |
| Archive file | batch-20250615-001.jsonl.gz | A compressed file containing execution records |
Each archive file is in JSONL.GZ format — one complete execution record per line, compressed with GZIP. This format is compact and easy to process if you ever need to retrieve archived data.
Configuration Overview
| Setting | What it does | Default |
|---|---|---|
| Archival enabled (master switch) | Turns the entire archival service on or off | — |
| Schedule | When archival runs | Monthly |
| Recent months to keep | How many months of recent data to protect | — |
| Archive window (months) | How far back to look for archivable records | — |
| Executions to keep | Default number of recent executions to keep per job (for per-job mode) | 100 |
| Batch size | How many records to process at a time | 500 |
| Storage type | Where archives are stored | S3 or MinIO |
Common Questions
Does archival delete my data?
Archival never permanently deletes your data. Records are only removed from the main database after the archive file has been uploaded and verified in cold storage — your execution history is always preserved.
No. Archival moves your data to cold storage. The original records are only removed from the main database after the archive has been uploaded and verified.
Can I still access archived records?
Yes. Archived records are stored in compressed files in your configured S3 or MinIO storage. They can be retrieved and reviewed when needed.
What happens if an upload fails?
The service retries up to 3 times with increasing wait times. If all retries fail, the original records stay untouched in the database. Nothing is lost.
How often does archival run?
By default, it runs monthly. Your administrator can adjust the schedule to run more or less frequently.
Can two archival processes run at the same time?
No. A distributed lock ensures only one archival process runs at a time, even if your platform runs on multiple servers. This prevents duplicate work or conflicts.
Will archival slow down my automations?
No. Archival runs on its own schedule and processes records in small batches. It does not interfere with running automations.
How do I know if archival is working?
Check whether old execution records are being cleaned up from the main database on schedule. Your administrator can also review archival logs for confirmation.
Related Pages
- Monitoring Automations — how to watch your automations run and check results
- Logging — how to read detailed step messages and troubleshoot failures