20 min read Updated Aug 2026 Memory

How to Build a Memory System for AI Agents

Agents without memory are amnesiacs — every session starts from zero. I built GrayMatter, a hybrid memory system that gives my 3-agent fleet shared recall across 22K+ entities and 147K relations.

Memory isn't storage — it's retrieval. A million facts mean nothing if you can't find the right one in context-window time.

The memory problem

Three failure modes in agent memory:

Context bloat — feeding entire history burns 70% of context window on irrelevant data
Cross-session amnesia — agent forgets what it did yesterday
Cross-agent silos — Discus doesn't know what Hermit learned

Solve all three with: tiered retrieval + shared backend + structured schema.

GrayMatter: hybrid vector+keyword+graph

Architecture:

Agent query
    ↓
[FTS5 keyword match] → [Vector similarity (bge-m3 1024d)] → [Graph traversal]
    ↓
Ranked results (hybrid score)
    ↓
Injected into agent context

Why hybrid? Keyword for exact names/dates, vector for semantic similarity, graph for relationship context. Each catches what the others miss.

Stats: 22K entities, 6.8K typed entities, 147K relations, sub-100ms retrieval at 1024 dimensions.

FTS5 for instant recall

SQLite FTS5 handles exact-match and prefix queries. Critical for agent memory — when an agent asks "what did we decide about X?", it needs exact recall, not semantic similarity.

CREATE VIRTUAL TABLE memory_fts USING fts5( content, entity_id UNINDEXED, source UNINDEXED, timestamp UNINDEXED, tokenize = 'porter unicode61' ); -- Query with ranking SELECT entity_id, content, bm25(memory_fts) AS score FROM memory_fts WHERE content MATCH :query ORDER BY score LIMIT 10;

FTS5 bm25 ranking is deterministic and fast — no GPU needed, runs on any VPS.

Tiered retention (hot/warm/cold)

Not all memories are equal. Three tiers:

Tier	Retention	Storage	Use case
Hot	7 days	RAM (Redis)	Active tasks, recent decisions
Warm	90 days	SQLite + vectors	Project history, patterns
Cold	Forever	Compressed JSON	Archival facts, old sessions

Hot tier is queried first. If miss, fall back to warm. Cold is only for explicit historical lookups.

Schema: entities, relations, facts

Structured memory beats free-text:

{
  "entity": {
    "id": "discus-asus",
    "type": "agent",
    "name": "Discus",
    "properties": {
      "runtime": "hermes",
      "host": "asus-um3406ha",
      "role": "primary"
    }
  },
  "relation": {
    "source": "discus-asus",
    "type": "manages",
    "target": "graymatter"
  },
  "fact": {
    "entity": "discus-asus",
    "predicate": "last_heartbeat",
    "value": "2026-08-29T04:00:00Z",
    "source": "cron-log"
  }
}

Agents query by entity type, relation path, or fact predicate. Graph traversal finds indirect connections.

Integration with agents

Hermes agents talk to GrayMatter via REST at http://localhost:8768:

# Store a fact
POST /entities
{
  "name": "openclaw-gateway-status",
  "type": "system_state",
  "properties": {"status": "running", "uptime_h": 142}
}

# Retrieve context for a query
GET /search?q=openclaw+gateway+status&limit=5

# Traverse graph from entity
GET /entities/discus-asus/relations?depth=2

Never expose GrayMatter to the public internet. Agents access via localhost or Tailscale mesh only.

Next: The Complete Guide to Self-Hosting AI Agents — VPS hardening, Tailscale mesh, 24/7 uptime.