ElevenLabs

ElevenLabs Agents connect to AutoMem through the same remote MCP sidecar used by other cloud platforms (ChatGPT, Claude.ai). ElevenLabs supports custom HTTP headers, so it can use the more secure header-based authentication. Claude.ai (Web) can use a header or a URL token too, while ChatGPT is limited to URL-based tokens. The sidecar itself accepts a token from either place.

Note

ElevenLabs has a 30-second idle timeout. The sidecar’s SSE transport sends heartbeats every 20 seconds to keep the connection alive. Streamable HTTP is also supported.

---

Architecture

The MCP sidecar bridges ElevenLabs voice agents to AutoMem’s HTTP API:

graph TB
    subgraph "Remote MCP (HTTP/SSE Transport)"
        EL["ElevenLabs Agents Platform"]
        SIDECAR["MCP Sidecar — POST /mcp / GET /mcp/sse"]
        AUTOMEM["AutoMem API :8001"]

        EL -->|"HTTPS + Bearer token"| SIDECAR
        SIDECAR -->|"HTTP"| AUTOMEM
    end

---

Prerequisites

1. A deployed AutoMem service
2. The MCP sidecar deployed and accessible over HTTPS
3. An ElevenLabs account with Agents access

---

Deploy the MCP Sidecar

The sidecar is included in the AutoMem Railway one-click template. See Claude.ai (Web) for full deployment instructions.

Required environment variables:

Variable	Purpose
AUTOMEM_API_URL	AutoMem service URL
AUTOMEM_API_TOKEN	Default token for API authentication
PORT	Listen port (default: 8080)

ElevenLabs-specific note: The SSE transport heartbeat (every 20 seconds) is specifically designed to keep connections alive past ElevenLabs’ 30-second idle timeout. Ensure the sidecar deployment does not have its own idle timeout shorter than 30 seconds.

---

Connect ElevenLabs Agent

Configure the MCP server in your ElevenLabs Agent settings:

Option 1: Header-Based Authentication (Recommended)

ElevenLabs supports custom HTTP headers, so you can use the more secure Authorization header:

• Server URL: https://your-mcp-bridge.up.railway.app/mcp/sse
• Custom Header Name: Authorization
• Custom Header Value: Bearer YOUR_AUTOMEM_TOKEN

Or using Streamable HTTP:

• Server URL: https://your-mcp-bridge.up.railway.app/mcp
• Custom Header Name: Authorization
• Custom Header Value: Bearer YOUR_AUTOMEM_TOKEN

Option 2: URL-Based Authentication

SSE transport:

https://your-mcp-bridge.up.railway.app/mcp/sse?api_token=YOUR_AUTOMEM_TOKEN

Streamable HTTP:

https://your-mcp-bridge.up.railway.app/mcp?api_token=YOUR_AUTOMEM_TOKEN

Tip

Use Option 1 (header-based) when possible. URL-based tokens appear in server logs, reverse proxy logs, and potentially in ElevenLabs’ own analytics.

---

Voice Agent Memory Patterns

ElevenLabs voice agents benefit from memory in several ways:

User preference recall — Remember communication preferences across sessions:

recall_memory(
  tags: ["preference", "elevenlabs"],
  limit: 5
)

Context continuity — Resume conversations with context from previous sessions:

recall_memory(
  query: "last conversation with this user",
  time_query: "last 7 days"
)

Knowledge persistence — Store facts the user shares:

store_memory(
  content: "User's name is Alex. Prefers brief responses.",
  type: "Preference",
  importance: 0.9,
  tags: ["elevenlabs", "user-preferences"]
)

---

Available Memory Tools

All six AutoMem tools are accessible to ElevenLabs agents:

Tool	Description
store_memory	Store voice-captured information
recall_memory	Find relevant memories for current conversation
associate_memories	Create relationships between related memories
update_memory	Update existing memory content or metadata
delete_memory	Remove a specific memory
check_database_health	Verify AutoMem service is running

---

Connection Stability

ElevenLabs has a 30-second idle timeout on SSE connections. The sidecar handles this automatically:

• SSE heartbeats sent every 20 seconds (within the 30s timeout window)
• Heartbeat format: : ping\n\n (SSE comment, not a data event)
• Anti-buffering headers set: X-Accel-Buffering: no, Cache-Control: no-cache, no-transform

If you observe connection drops:

1. Check that no intermediate proxy is buffering the SSE stream
2. Verify the proxy passes through SSE headers
3. Consider using Streamable HTTP (/mcp) instead of SSE (/mcp/sse)

---

Verification

Test the connection through your ElevenLabs agent:

"Check the health of the AutoMem service"

Test memory:

"Remember that I prefer concise responses."
"What are my preferences?"

---

Troubleshooting

Agent cannot connect

1. Verify sidecar health: curl https://your-mcp-bridge.up.railway.app/health. It returns 200 whenever the sidecar process is up, even if AutoMem is down, so read the body: upstream reports the sidecar’s last check of AutoMem (reachable, unreachable, unconfigured when AUTOMEM_API_TOKEN is not set on the sidecar, or unknown before the first check) and upstream_error gives the reason when AutoMem cannot be reached, while upstream_details carries AutoMem’s own health report. That check runs at startup and every 30 seconds by default. GET /ready runs the check on each request and returns 503 unless AutoMem reports healthy.
2. Check TLS certificate is valid
3. Ensure port 443 is reachable from ElevenLabs’ servers

401 Unauthorized

1. For header auth: verify the header name is exactly Authorization and value is Bearer YOUR_TOKEN
2. For URL auth: verify token is URL-encoded if it contains special characters
3. Test the token against AutoMem directly, not through the sidecar: curl -i -H "Authorization: Bearer $TOKEN" "https://your-automem.up.railway.app/recall?query=test&limit=1". A 401 means AutoMem rejected the token; a 200 means the token was accepted. Do not use /health for this: neither the sidecar’s /health nor AutoMem’s own /health checks the token.

The sidecar itself returns 401 only when a request carries no token and AUTOMEM_API_TOKEN is not set on the sidecar. If AUTOMEM_API_TOKEN is set, a request with no token uses that default, and a token in the request is used in preference to it. A token that is present but wrong is forwarded to AutoMem. Connecting and listing tools still succeed, and a tool call that reads or writes memories fails with an MCP tool error (for example AutoMem error: Unauthorized) instead of an HTTP 401 from the sidecar. check_database_health can still succeed with a wrong token, because AutoMem’s /health does not check it.

Connection drops during agent conversation

The SSE heartbeat (every 20 seconds) should prevent ElevenLabs’ 30-second timeout. If drops persist:

• Switch to Streamable HTTP (/mcp) — lower latency, no idle timeout issues
• Check sidecar logs for session expiration (sessions expire after 1 hour of inactivity)

Memory service unreachable

When AutoMem is unreachable or returns an error, the sidecar does not turn it into an HTTP 502. The tool call comes back as an MCP tool error (isError: true) whose text describes the failure, for example AutoMem error: fetch failed, AutoMem request timed out, or AutoMem service is temporarily unavailable.

Verify AUTOMEM_API_URL in sidecar environment variables points to the correct AutoMem service. On Railway, use the internal hostname: http://memory-service.railway.internal:8001. If AUTOMEM_API_URL is not set, the sidecar falls back to the legacy AUTOMEM_ENDPOINT, then to http://127.0.0.1:8001, instead of reporting a configuration error, so a missing variable usually shows up as this same tool error.

---

Related Platforms

Other cloud platforms using the same MCP sidecar:

• ChatGPT — URL auth only
• Claude.ai (Web) — header or URL auth