Skip to content

Backup & Recovery

Protect your AgentOps data with regular PostgreSQL backups, Redis persistence, and documented recovery procedures.

What to Back Up

Component Data Priority
PostgreSQL Users, tenants, agents, policies, audit logs, anomalies, HITL Critical
Redis Pub/sub state, cached sessions Medium (rebuildable)
OPA policies Rego templates in policies/templates/ Low (in Git)
Storage Async export files in storage/ Medium
Environment .env, secrets Critical

Audit logs are compliance-critical

Audit logs may be subject to regulatory retention requirements (GDPR, HIPAA, FZ-152). Plan retention and backup accordingly.

PostgreSQL Backup

Use the in-repo scripts from the project root (or set AGENTOPS_DIR):

# One-shot (default BACKUP_DIR=~/agentops-backups)
./scripts/backup_postgres.sh

# Optional offsite
BACKUP_S3_URI=s3://my-bucket/agentops ./scripts/backup_postgres.sh
# or
BACKUP_RSYNC_TARGET=backup@host:/var/backups/agentops ./scripts/backup_postgres.sh

Cron (daily at 02:15 UTC):

15 2 * * * AGENTOPS_DIR=/home/smdg/agentops BACKUP_DIR=/home/smdg/agentops-backups /home/smdg/agentops/scripts/backup_postgres.sh >> /home/smdg/agentops/backup.log 2>&1

Or install via helper:

AGENTOPS_DIR=/home/smdg/agentops BACKUP_DIR=/home/smdg/agentops-backups ./scripts/install_backup_cron.sh

Non-destructive restore drill (checksums + pg_restore --list, no downtime):

./scripts/verify_backup.sh ~/agentops-backups/<stamp>

The script dumps PostgreSQL (pg_dump -Fc), archives models/ + storage/, writes SHA256SUMS, retains 30 days, and optionally uploads to S3 or rsync. Demo DB is included when the demo stack is running.

Manual Backup (legacy one-liner)

# Docker
docker compose exec postgres pg_dump -U aegisai -Fc aegisai > aegisai_backup_$(date +%Y%m%d).dump

# Direct connection
pg_dump -h localhost -U aegisai -Fc aegisai > aegisai_backup.dump

Point-in-Time Recovery (PITR)

For production, enable WAL archiving in PostgreSQL:

# postgresql.conf
wal_level = replica
archive_mode = on
archive_command = 'cp %p /var/lib/postgresql/wal_archive/%f'

Restore Procedure

Full Restore (script)

# Point at a dump file or a backup directory created by backup_postgres.sh
./scripts/restore_postgres.sh ~/agentops-backups/20260727_021500
# or
./scripts/restore_postgres.sh ~/agentops-backups/20260727_021500/agentops_20260727_021500.dump

The script stops app/frontend, recreates the DB, restores the dump, optionally restores models_storage.tgz, runs Alembic, and brings services back up.

Manual restore

docker compose -f docker-compose.prod.yml --env-file .env stop app frontend
docker compose -f docker-compose.prod.yml --env-file .env exec -T postgres \
  psql -U aegisai -d postgres -c "DROP DATABASE IF EXISTS aegisai;"
docker compose -f docker-compose.prod.yml --env-file .env exec -T postgres \
  psql -U aegisai -d postgres -c "CREATE DATABASE aegisai;"
docker compose -f docker-compose.prod.yml --env-file .env exec -T postgres \
  pg_restore -U aegisai -d aegisai --no-owner < agentops_backup.dump
docker compose -f docker-compose.prod.yml --env-file .env up -d

Verify Restore

# Health check
curl -sf https://agentops.fun/api/v1/health/ready

# Verify data
curl https://agentops.fun/api/v1/stats -H "X-API-Key: <key>"
curl https://agentops.fun/api/v1/agents -H "X-API-Key: <key>"

Redis Backup

Redis data is primarily ephemeral (pub/sub, event bus). Enable persistence for session resilience:

# docker-compose.yml
redis:
  command: redis-server --appendonly yes
  volumes:
    - redis_data:/data

Manual Redis Backup

docker compose exec redis redis-cli BGSAVE
docker compose exec redis cp /data/dump.rdb /data/backup_$(date +%Y%m%d).rdb

Audit Log Export Backup

Export audit logs regularly for long-term compliance archives:

# Async export for large datasets
curl -X POST "http://localhost:8001/api/v1/audit/export/async?limit=100000" \
  -H "X-API-Key: <key>"

# Download and store offsite
curl "http://localhost:8001/api/v1/audit/export/download/<job_id>" \
  -H "X-API-Key: <key>" \
  -o audit_archive_$(date +%Y%m%d).jsonl

Store exports in S3, GCS, or encrypted local storage with appropriate retention policies.

Storage Directory

Async export files are stored in storage/:

# Backup storage directory
tar czf storage_backup_$(date +%Y%m%d).tar.gz storage/

Disaster Recovery Plan

Scenario RTO Target Procedure
Database corruption 1 hour Restore from latest pg_dump
Full server loss 4 hours Provision new VPS, restore DB, redeploy
Accidental data deletion 30 minutes Restore from backup, verify integrity
Redis failure 15 minutes Restart Redis (data rebuilds from events)
OPA failure 5 minutes Restart OPA (policies reload from templates)

Recovery Checklist

  1. Provision infrastructure (see VPS Deployment)
  2. Restore PostgreSQL from latest backup
  3. Run alembic upgrade head if needed
  4. Restore environment variables and secrets
  5. Start services: docker compose up -d
  6. Verify health: /api/v1/health/ready
  7. Verify data integrity: check agent count, recent audit logs
  8. Re-create super-admin if user table was empty
  9. Notify users of recovery completion

Backup Testing

Untested backups are not backups

Test restore procedures quarterly:

  1. Restore to a staging environment
  2. Verify all tables have expected row counts
  3. Confirm API endpoints return correct data
  4. Test login with restored user credentials
  5. Document any issues and update procedures

Retention Policies

Data Type Recommended Retention Storage
PostgreSQL daily backups 30 days Local + offsite
Audit log exports Per compliance requirement (1–7 years) Encrypted object storage
Redis snapshots 7 days Local
Application logs 90 days Log aggregation service