Backup & Recovery

This page describes the backup strategies and disaster recovery procedures available for AutoMem deployments. It covers the three recovery layers, scheduled backup automation, configuration options, backup formats, and four recovery paths for different failure scenarios. For monitoring backup health and detecting data drift, see Health Monitoring.

Backup Architecture Overview

AutoMem implements a defense-in-depth backup strategy with three recovery layers plus scheduled automation:

graph TB
    subgraph "Layer 1: Persistent Volumes"
        RailwayVol["Railway Volume Snapshots<br/>Automatic every 24h<br/>Recovery: 5 minutes"]
        VolumeFeatures["- One-click restore<br/>- Railway Dashboard access<br/>- Included with Railway Pro<br/>- FalkorDB only"]
    end

    subgraph "Layer 2: Dual Storage Redundancy"
        FalkorDB["FalkorDB<br/>Canonical record<br/>Graph + metadata"]
        Qdrant["Qdrant<br/>Backup record<br/>Vectors + payloads"]
        RecoverQdrant["recover_from_qdrant.py<br/>Recovery: 10 minutes<br/>Success rate: 99.7%"]

        FalkorDB <-->|"Redundancy"| Qdrant
        Qdrant --> RecoverQdrant
        RecoverQdrant --> FalkorDB
    end

    subgraph "Layer 3: Automated Backups"
        GitHubWorkflow[".github/workflows/backup.yml<br/>Every 6 hours<br/>Free tier: 2000 min/month"]
        BackupScript["backup_automem.py<br/>Compressed JSON exports"]
        LocalBackups["./backups/<br/>Last 7-14 backups"]
        S3Backups["S3 Storage<br/>Cross-region replication<br/>Recovery: 30 minutes"]

        GitHubWorkflow --> BackupScript
        BackupScript --> LocalBackups
        BackupScript --> S3Backups
    end

    Failure["Data Loss Event"] --> Layer1Check{"Layer 1<br/>Available?"}
    Layer1Check -->|Yes| RailwayVol
    Layer1Check -->|No| Layer2Check{"Layer 2<br/>Available?"}
    Layer2Check -->|Yes| RecoverQdrant
    Layer2Check -->|No| Layer3Check{"Layer 3<br/>Available?"}
    Layer3Check -->|Yes| S3Backups
    Layer3Check -->|No| DataLoss["Complete data loss<br/>Start fresh"]

Layer Responsibilities

Area	Mechanism	Recovery Speed	Scope	Platform Lock
Infrastructure	Railway volume snapshots	Instant	FalkorDB only	Yes (Railway)
Dual Storage	Real-time dual writes	Immediate	Both databases	No
Application	Script exports	Minutes	Both databases	No
Automation	Scheduled execution	N/A (prevention)	Both databases	No

Infrastructure Backups (Layer 1)

Railway Volume Snapshots

Railway provides automatic volume backups for the FalkorDB persistent volume configured in the deployment template.

The FalkorDB service uses a persistent volume mounted at /var/lib/falkordb/data. The Redis persistence settings ensure data durability with the following configuration:

RDB snapshots every 60 seconds if at least 1 write
AOF (Append-Only File) for write-ahead logging
fsync every second for durability

Accessing and restoring snapshots:

Railway Dashboard → FalkorDB service
“Backups” tab shows snapshot history
One-click restore from any snapshot

Limitations:

Only covers FalkorDB (Qdrant not included)
Cannot export or download backups
Platform-locked to Railway
Best for quick recovery from recent failures

Application-Level Backups (Layer 3)

backup_automem.py Script

The core backup script at scripts/backup_automem.py exports data from both FalkorDB and Qdrant to compressed JSON files.

Backup Format

FalkorDB Export Structure:

The FalkorDB export is a Cypher-driven graph dump, not a dump of the Redis keyspace. It is a JSON object with timestamp, graph_name, nodes, relationships, and stats keys, containing:

Nodes, each with its id, labels, and properties
Relationships, each with source_id, type, target_id, and properties
stats with node_count and relationship_count

Indexes and other Redis-level structures are not part of the export.

Qdrant Export Structure:

The Qdrant export is a JSON object with timestamp, collection_name, points, and stats keys, containing:

points, one entry per Qdrant point, each with id (the memory ID), vector, and payload
Vector embeddings in vector (dimensions depend on the configured VECTOR_SIZE, embedding provider, and any adopted existing Qdrant collection dimension)
Payload data in payload (memory content, metadata, tags)
stats with points_count and vector_size

Both exports are compressed using gzip with .json.gz extension, typically achieving 70-80% compression ratio.

Command-Line Usage

# Basic backup - creates timestamped backups in ./backups/falkordb/ and ./backups/qdrant/
python scripts/backup_automem.py

# With retention policy - after backing up, keeps the 7 most recent backup files per store, deleting the rest
python scripts/backup_automem.py --cleanup --keep 7

# Custom directory
python scripts/backup_automem.py --backup-dir /path/to/backups

# With S3 upload - requires boto3 and AWS credentials set via environment variables.
# The bucket is configurable; object keys are written under automem-backups/.
python scripts/backup_automem.py --s3-bucket my-automem-backups

Local Filesystem Storage

Backups are written to timestamped subdirectories:

backups/
├── falkordb/
│   ├── falkordb_20251020_143000.json.gz
│   ├── falkordb_20251020_203000.json.gz
│   └── ...
└── qdrant/
    ├── qdrant_20251020_143000.json.gz
    ├── qdrant_20251020_203000.json.gz
    └── ...

The --cleanup --keep N flag is a file count, not an age: it keeps the N most recent .json.gz files in each of backups/falkordb/ and backups/qdrant/ (ranked by file modification time) and deletes the rest. N defaults to 7.

S3 Cloud Storage

s3://my-automem-backups/
└── automem-backups/
    ├── falkordb/
    │   ├── falkordb_20251020_143000.json.gz
    │   └── ...
    └── qdrant/
        ├── qdrant_20251020_143000.json.gz
        └── ...

S3 Cost Estimation:

Component	Formula	Example
Storage	$0.023/GB/month	100MB backup = $0.0023/month
PUT requests	$0.005/1000 requests	4 backups/day = $0.60/year
GET requests (restore)	$0.0004/1000 requests	Negligible
Total (100MB, every 6h)	-	~$0.30/month

Automated Backup Methods

GitHub Actions Workflow

The recommended automation method uses GitHub Actions to run backups on a schedule without consuming Railway resources.

sequenceDiagram
    participant GHA as "GitHub Actions<br/>backup.yml"
    participant TCP as "Railway TCP Proxy<br/>monorail.proxy.rlwy.net"
    participant FDB as "FalkorDB<br/>:6379"
    participant QDR as "Qdrant Cloud<br/>HTTPS API"
    participant Script as "backup_automem.py"
    participant S3 as "S3 Bucket<br/>my-automem-backups"

    Note over GHA: Triggered every 6 hours<br/>or manually via workflow_dispatch

    GHA->>GHA: Checkout code
    GHA->>GHA: Install dependencies<br/>requirements.txt + boto3

    Note over GHA,TCP: Connectivity check (critical)
    GHA->>GHA: Validate FALKORDB_HOST<br/>Must NOT be *.railway.internal
    GHA->>TCP: Test TCP connection<br/>timeout 10s
    TCP->>FDB: Forward connection
    FDB-->>TCP: Connection OK
    TCP-->>GHA: ✅ Connectivity verified

    Note over GHA,S3: Backup execution
    GHA->>Script: Execute with env vars<br/>FALKORDB_*, QDRANT_*, AWS_*
    Script->>TCP: Connect via TCP Proxy
    TCP->>FDB: Redis protocol commands
    FDB-->>TCP: Export graph data
    TCP-->>Script: Graph JSON

    Script->>QDR: Scroll points (POST /collections/memories/points/scroll)
    QDR-->>Script: Vector data with payloads

    Script->>Script: Compress to .json.gz<br/>backups/falkordb/<br/>backups/qdrant/

    alt S3 Upload Enabled
        Script->>S3: Upload via boto3<br/>s3://my-automem-backups/automem-backups/
        S3-->>Script: Upload complete
    end

    Script-->>GHA: Exit 0 (success)
    GHA->>GHA: Log backup summary<br/>File sizes and timestamps

The workflow is defined in .github/workflows/backup.yml and triggers every 6 hours or manually via workflow_dispatch.

Required GitHub Secrets

Secret	Purpose	Example	Used By
`FALKORDB_HOST`	Railway TCP proxy domain	`monorail.proxy.rlwy.net`	`FalkorDB()` connection
`FALKORDB_PORT`	Railway TCP proxy port	`12345`	`FalkorDB()` connection
`FALKORDB_PASSWORD`	FalkorDB authentication	Generated by Railway	`FalkorDB(password=)`
`QDRANT_URL`	Qdrant endpoint	`https://xyz.qdrant.io`	`QdrantClient(url=)`
`QDRANT_API_KEY`	Qdrant authentication	API key from Qdrant Cloud	`QdrantClient(api_key=)`
`AWS_ACCESS_KEY_ID`	S3 upload (optional)	AWS credentials	`boto3.client('s3')`
`AWS_SECRET_ACCESS_KEY`	S3 upload (optional)	AWS credentials	`boto3.client('s3')`
`AWS_DEFAULT_REGION`	S3 region (optional)	`us-east-1`	AWS SDK environment resolution

The TCP proxy endpoint is found in Railway Dashboard → FalkorDB service → Settings → Networking → TCP Proxy.

Backup Configuration Reference

Environment Variables

Variable	Required	Default	Description
`AUTOMEM_BACKUP_DIR`	No	`./backups`	Backup directory used by `backup_automem.py` (output) and `restore_from_backup.py` (input); overridden by `--backup-dir`
`FALKORDB_HOST`	No	`localhost`	FalkorDB hostname or IP
`FALKORDB_PORT`	No	`6379`	FalkorDB Redis port
`FALKORDB_PASSWORD`	No	-	FalkorDB authentication password (needed only for password-protected instances)
`FALKORDB_GRAPH`	No	`memories`	Graph database name
`QDRANT_URL`	No	`http://localhost:6333`	Qdrant endpoint URL
`QDRANT_API_KEY`	No	-	Qdrant API authentication (leave unset for a Qdrant instance without authentication)
`QDRANT_COLLECTION`	No	`memories`	Qdrant collection name
`AWS_ACCESS_KEY_ID`	No	-	For S3 upload
`AWS_SECRET_ACCESS_KEY`	No	-	For S3 upload
`AWS_DEFAULT_REGION`	No	-	Optional S3 region consumed by boto3 / AWS SDK environment resolution

backup_automem.py always exports Qdrant as well as FalkorDB and has no option to skip it. If Qdrant is unreachable, the run fails after the FalkorDB file is written, so point QDRANT_URL at a running Qdrant. restore_from_backup.py can skip Qdrant with --falkordb-only.

Retention Policy Recommendations

Use Case	Backup Frequency	Retention Target	`--keep` Value	Storage Location	Estimated Cost
Personal/Development	Every 24 hours	7 days	`--keep 7`	Local only	$0
Team/Small Production	Every 6 hours	14 days	`--keep 56`	Local + S3	~$0.50/month
Production	Every 1-6 hours	30 days	`--keep 120` to `--keep 720`	S3 with versioning	~$2-5/month
Enterprise	Every 1 hour	90 days + archive	`--keep 2160`	S3 + cross-region	~$10-20/month

--keep takes a number of files per store, not a number of days, so convert a retention target into a count: days multiplied by backups per day. --cleanup prunes only the local backup directory. backup_automem.py does not delete S3 objects, so set S3 retention with a bucket lifecycle rule. The repository’s GitHub Actions workflow runs every 6 hours with --keep 14, which keeps the 14 most recent files per store in the runner’s local backups/ directory.

Backup Method Comparison

Method	Scope	Speed	Automation	Platform Lock	Cost	Best For
Railway Volumes	FalkorDB only	Instant	Automatic	Yes	Included	Quick recovery
GitHub Actions	Both databases	5-10 min	Scheduled	No	Free	Most users
Manual Script	Both databases	5-10 min	Manual	No	Free	Development

Disaster Recovery

Recovery Decision Matrix

Failure Scenario	Data Available	Recovery Method	Primary Tool	RTO
FalkorDB data loss	Qdrant intact	Qdrant-based rebuild	`recover_from_qdrant.py`	2-5 min
FalkorDB persistence disabled	Qdrant intact	Qdrant-based rebuild	`recover_from_qdrant.py`	2-5 min
Qdrant data loss	FalkorDB intact	Background re-embedding	Enrichment queue	30-60 min
Both databases corrupted	Backup files (S3/local)	File restoration	`restore_from_backup.py` + recovery	10-20 min
Railway volume failure	Railway snapshots	Volume restore	Railway dashboard	5-10 min
Drift detected (5-50%)	Both databases available	Selective sync	`health_monitor.py --auto-recover`	1-2 min

Recovery Decision Tree

flowchart TD
    Start["Data Loss Detected"] --> Assess["Assess Damage"]

    Assess --> FalkorLost{"FalkorDB<br/>lost?"}
    Assess --> QdrantLost{"Qdrant<br/>lost?"}

    FalkorLost -->|Yes| QdrantIntact{"Qdrant<br/>intact?"}
    FalkorLost -->|No| NoAction1["No action needed"]

    QdrantIntact -->|Yes| RecoverQdrant["Use recover_from_qdrant.py<br/>⚡ 10 min / 99.7% success"]
    QdrantIntact -->|No| BothLost["Both databases lost"]

    QdrantLost -->|Yes| FalkorIntact{"FalkorDB<br/>intact?"}
    QdrantLost -->|No| NoAction2["No action needed"]

    FalkorIntact -->|Yes| RebuildQdrant["Rebuild Qdrant from FalkorDB<br/>POST /admin/reembed<br/>⚡ 15-20 min"]
    FalkorIntact -->|No| BothLost

    BothLost --> RailwayBackup{"Railway volume<br/>backups?"}
    RailwayBackup -->|Yes| RailwayRestore["Railway Dashboard restore<br/>⚡ 5 min / FalkorDB only<br/>Then rebuild Qdrant"]
    RailwayBackup -->|No| S3Backup{"S3 or local<br/>backups?"}

    S3Backup -->|Yes| S3Restore["1. Restore Qdrant from S3<br/>2. recover_from_qdrant.py<br/>⏱️ 30 min total"]
    S3Backup -->|No| CompleteFailure["Complete data loss<br/>Start fresh"]

    RecoverQdrant --> Verify["Verify Integrity<br/>Compare counts<br/>Check sample memories"]
    RebuildQdrant --> Verify
    RailwayRestore --> Verify
    S3Restore --> Verify

Method 1: Qdrant-Based Recovery

The fastest and most reliable recovery method. Clears the FalkorDB graph and re-creates Memory nodes from Qdrant’s vector payloads. Relationships are not restored.

When to use:

FalkorDB lost all data (container restart, persistence misconfiguration)
FalkorDB corrupted but Qdrant intact
Health monitor detects >50% drift with Qdrant having more data

Recovery process details:

Function	Purpose	Code Location
`qdrant_client.scroll()`	Fetch all vectors with payloads	Qdrant SDK call
`restore_memory_to_graph_only()`	Filters `RESERVED_FIELDS` from metadata inline (no separate filter function)	`scripts/recover_from_qdrant.py`
`CREATE (m:Memory {...})`	Rebuild memory nodes	Cypher query in recovery loop

Execution steps:

# Step 1: Verify Qdrant availability
curl https://your-qdrant-url/collections/memories

# Step 2: Run recovery script with database environment variables
FALKORDB_HOST=falkordb.railway.internal \
FALKORDB_PORT=6379 \
FALKORDB_PASSWORD=your-password \
QDRANT_URL=https://your-qdrant-url \
QDRANT_API_KEY=your-key \
python scripts/recover_from_qdrant.py

# Step 3: Monitor recovery progress in stdout

Known limitations:

99.7% recovery rate: In testing, 2/780 memories failed due to malformed data
Relationship loss: The script re-creates Memory nodes only, so relationship edges are not recovered
Recent writes: Memories written in the last 2 seconds (before embedding completes) may be missing from Qdrant

Method 2: Backup File Restoration

Used when both databases are lost or corrupted. Restores from compressed JSON backups stored locally or in S3.

When to use:

Both FalkorDB and Qdrant lost
Recovery from specific point in time needed
Testing disaster recovery procedures

Backup file structure:

backups/
├── falkordb/
│   ├── falkordb_20251020_143000.json.gz
│   └── falkordb_20251020_083000.json.gz
└── qdrant/
    ├── qdrant_20251020_143000.json.gz
    └── qdrant_20251020_083000.json.gz

Each Qdrant backup is a JSON object with timestamp, collection_name, points, and stats. stats holds points_count and vector_size. Each entry in points has id (the memory ID), vector (float array, dimensions match VECTOR_SIZE config), and payload (the memory’s stored fields, such as content, tags, tag_prefixes, importance, timestamp, type, confidence, updated_at, last_accessed, and metadata).

The exact payload keys depend on which code path wrote the point, and the backup stores them as found. Every AutoMem code path that builds a Qdrant memory payload writes content, tags, importance, timestamp, type, confidence, updated_at, last_accessed, and metadata. Beyond that core set:

tag_prefixes is written by every path except the /admin/reembed and /admin/sync endpoints and scripts/reembed_embeddings.py.
t_valid and t_invalid are written only when a client supplies the embedding on create or when a memory is updated.
The embedding worker also writes relevance_score, and consolidation can later set relevance_score and archived on existing points.

Restoration Flow

sequenceDiagram
    participant Operator
    participant S3 as S3 Bucket / Local
    participant Restore as restore_from_backup.py
    participant Qdrant
    participant Recovery as recover_from_qdrant.py
    participant FalkorDB

    Operator->>S3: Download backup<br/>aws s3 cp or local copy
    Operator->>Restore: Execute with backup path

    Restore->>Restore: Decompress JSON.gz<br/>gzip.open()
    Restore->>Restore: Parse JSON object<br/>json.load()

    loop For each point
        Restore->>Qdrant: Upsert point<br/>qdrant_client.upsert()
    end

    Restore-->>Operator: Qdrant restoration complete<br/>780 points restored

    Operator->>Recovery: python recover_from_qdrant.py
    Recovery->>Qdrant: Scroll all points
    Recovery->>FalkorDB: Rebuild graph
    Recovery-->>Operator: Full recovery complete

Execution steps:

# Step 1: Download backup file into the directory structure restore_from_backup.py
# expects — <backup-dir>/qdrant/qdrant_<timestamp>.json.gz — and leave it compressed.
# backup_automem.py uploads objects under automem-backups/<store>/<file>;
# the restore script decompresses the local file itself via gzip.open().
mkdir -p ./backups/qdrant
aws s3 cp s3://my-automem-backups/automem-backups/qdrant/qdrant_20251020_143000.json.gz ./backups/qdrant/

# Step 2: Restore to Qdrant (--backup-dir defaults to ./backups; --qdrant-only
# restores just Qdrant here)
python scripts/restore_from_backup.py --backup-timestamp 20251020_143000 --qdrant-only

# Step 3: Rebuild FalkorDB from Qdrant
python scripts/recover_from_qdrant.py

Recovery Time Estimate

Database Size	Decompress	Qdrant Upload	FalkorDB Rebuild	Total
100 memories	<1 sec	5 sec	10 sec	~15 sec
1,000 memories	2 sec	30 sec	60 sec	~2 min
10,000 memories	10 sec	5 min	10 min	~15 min
100,000 memories	30 sec	30 min	60 min	~90 min

Method 3: Railway Volume Restore

Uses Railway’s built-in volume snapshots for instant recovery of FalkorDB data.

When to use:

FalkorDB volume corruption
Need to rollback to specific snapshot
Quick recovery without external dependencies

Restoration steps:

Log in to Railway dashboard, navigate to FalkorDB service, click “Volumes” tab
View available snapshots (sorted by date), note timestamp and size
Click “Restore” next to chosen snapshot and confirm (irreversible action)
Railway stops the FalkorDB service, replaces volume with snapshot data, restarts the service
Verify service health with curl https://your-automem-url/health

Limitations:

Railway-locked: Cannot export snapshots outside Railway platform
FalkorDB only: Does not restore Qdrant data
Snapshot frequency: Default 24-hour intervals (may lose up to 24 hours of data)
Retention policy: Depends on Railway plan (Pro plan: 30 days)

Method 4: Drift-Based Selective Recovery

Uses the health monitor to detect and automatically fix inconsistencies between FalkorDB and Qdrant without full recovery.

When to use:

Health monitor detects 5-50% drift
Both databases online but inconsistent
Partial write failures suspected
Preventive maintenance

# Auto-recovery is a CLI flag, not an environment variable — there is no
# persistent env-var equivalent. Run the health-monitor process/cron with
# the flag baked into its invocation:
python scripts/health_monitor.py --auto-recover

Recovery Validation

After any recovery procedure, verify system integrity before returning to production.

graph TB
    Start[Recovery Complete]

    Start --> V1{1. Database<br/>Connectivity}
    V1 -->|Pass| V2{2. Memory<br/>Count Match}
    V1 -->|Fail| Fix1[Check connection strings<br/>Verify credentials]

    V2 -->|Pass| V3{3. Relationship<br/>Integrity}
    V2 -->|Fail| Fix2[Re-run recovery<br/>Check for errors]

    V3 -->|Pass| V4{4. Embedding<br/>Coverage}
    V3 -->|Fail| Fix3[Investigate missing edges<br/>Check entity extraction]

    V4 -->|Pass| V5{5. Search<br/>Functionality}
    V4 -->|Fail| Fix4[Re-run embedding worker<br/>Verify Qdrant indexes]

    V5 -->|Pass| Complete[✓ Validation Complete<br/>Resume Operations]
    V5 -->|Fail| Fix5[Test vector search<br/>Rebuild Qdrant collection]

    Fix1 --> V1
    Fix2 --> V2
    Fix3 --> V3
    Fix4 --> V4
    Fix5 --> V5

Validation Procedures

1. Database Connectivity:

curl https://your-automem-url/health

2. Memory Count Verification:

# Check FalkorDB count matches Qdrant count
curl https://your-automem-url/health | jq '{memory_count, vector_count, sync_status}'

3. Relationship Integrity:

# Check aggregate graph edge counts and relationship distribution
curl -H "Authorization: Bearer your-api-token" \
  https://your-automem-url/graph/stats \
  | jq '{edges: .totals.edges, by_relationship}'

4. Embedding Coverage:

# /analyze has no embedding_coverage field. Compare vector_count against
# memory_count from /health instead:
curl https://your-automem-url/health | jq '{memory_count, vector_count}'

5. Search Functionality:

# Test recall with a known query
curl -X GET "https://your-automem-url/recall?query=test" \
  -H "Authorization: Bearer your-api-token"

Post-Recovery Monitoring

Monitor the system for 24 hours after recovery:

Metric	Check Frequency	Threshold	Action if Exceeded
Drift percentage	Every 5 min	>5%	Investigate write failures
Enrichment queue depth	Every 15 min	>100	Check worker health
Embedding queue depth	Every 15 min	>500	Verify OpenAI API key
API error rate	Every 5 min	>1%	Check logs for errors
Response time (p95)	Every 15 min	>2s	Investigate slow queries

Recovery Time Objectives (RTO)

Scenario	Method	Detection	Execution	Validation	Total RTO
FalkorDB lost, Qdrant intact	Qdrant recovery	Immediate	2-3 min	1 min	3-4 min
FalkorDB persistence disabled	Qdrant recovery	Immediate	2-3 min	1 min	3-4 min
Both databases corrupted	Backup restore	Varies	5-10 min	2 min	7-12 min
Railway volume failure	Volume restore	Immediate	3-5 min	1 min	4-6 min
5-20% drift detected	Selective sync	Auto (5 min)	1-2 min	1 min	7-8 min
Qdrant lost, FalkorDB intact	Re-embedding	Immediate	30-60 min	5 min	35-65 min

Recovery Point Objective (RPO):

Best case: 0 seconds (Qdrant recovery, dual storage)
Typical case: 2 seconds (embedding queue latency)
Worst case: 6 hours (last automated backup)

Troubleshooting Recovery Failures

Recovery Script Fails with Connection Error

Problem: recover_from_qdrant.py exits with “Failed to connect to FalkorDB”

Solution: Verify FALKORDB_HOST, FALKORDB_PORT, and FALKORDB_PASSWORD environment variables match the FalkorDB service configuration. For Railway deployments, use the internal hostname falkordb.railway.internal (or TCP proxy endpoint for external access).

Qdrant Recovery Shows 0% Success Rate

Problem: Script reports “Recovered 0/780 memories”

Solutions:

If collection doesn’t exist: Use backup restoration method (Method 2)
If payloads are missing: Qdrant was used for vectors only; recovery not possible without payloads

Memory Types Corrupted After Recovery

Problem: /analyze shows invalid memory types like “str”, “int”, “boolean”

Cause: Using old version of recover_from_qdrant.py without RESERVED_FIELDS filtering

Solution: Update to v0.5.0+ of AutoMem. If already on old version, run scripts/cleanup_memory_types.py to fix corrupted types.

Drift Persists After Recovery

Problem: Health monitor still reports >5% drift after running recovery

Solutions:

If FalkorDB has more: Recent Qdrant writes may have failed; check Qdrant API key
If Qdrant has more: FalkorDB may be read-only; check REDIS_ARGS includes --save and --appendonly
If inconsistent: May need bidirectional sync; open a GitHub issue

Testing Recovery Procedures

Regular recovery testing ensures procedures work when needed.

Recommended Testing Schedule

Environment	Test Frequency	Test Type	Data Source
Development	Weekly	Full recovery	Test data
Staging	Monthly	Full recovery	Production replica
Production	Quarterly	Validation only	Verify backup integrity

Verifying Backup Integrity

# Check backup files exist and have reasonable sizes
ls -lh backups/falkordb/ backups/qdrant/

# Validate memory counts from backup (the file is an object, so count its points)
zcat backups/qdrant/qdrant_latest.json.gz | jq '.stats.points_count, (.points | length)'

# Test decompression
gunzip -t backups/falkordb/falkordb_latest.json.gz && echo "OK"