Why we built our own engine for agent memory

I want to explain the least fashionable decision we made building Aether: instead of putting a memory layer on top of an existing storage engine, we wrote our own engine, in Rust, and built the memory layer on it. It cost us time we will never get back. I'd make the same call again. Here's the reasoning, including the parts that didn't go our way.

TL;DR

• Agent memory is a workload, not a feature: tiny constant writes, and recall that mixes semantic similarity with a time window and a per-user filter in one query. General-purpose stores fight you on every part of that.
• We wrote our own storage and retrieval engine in Rust so those constraints are design inputs, not workarounds — which is why retrieval is sub-millisecond, memory ships as a single self-hostable binary, and the pricing is flat instead of metered.
• It cost us real time to market and made every bug ours. The trade we took in exchange: first-party SDK breadth (Python, TypeScript, Go, and .NET) and one bill instead of three.

Agents forget, and the fix is harder than it looks

If you've shipped an agent or a copilot, you've hit the wall: a user tells your agent something on Tuesday — their deploy preferences, their kid's name, the fact that they already tried restarting the cluster — and on Thursday the agent asks again. The context window is not memory. It's a scratchpad that gets wiped.

The obvious fix looks simple. Store what the agent learns, embed it, retrieve it later. A weekend project.

Then you discover that memory is not search. A memory query is almost never "find the most similar text." It's "what did this user tell me about deploys, in the last month, in this project, weighting their recent correction over the stale preference it replaced."

That's semantic similarity and a time window and an entity filter and recency ranking, in one query. Bolt that onto a generic store and you end up hand-rolling all of it: an extraction pipeline that turns conversation into facts, per-entity namespacing so user A's memories never surface in user B's recall, session-thread bookkeeping, timestamp-aware ranking that decays old facts. Plus an embedding API. Plus object storage for the source documents. Three bills, and a pile of glue code you maintain forever.

The obvious move was to wrap something

When we started, the well-trodden path was clear: be a layer. Put a memory abstraction on top of a store the customer brings or rents, ship fast, let someone else worry about storage.

That approach can work, and I won't pretend otherwise. Mem0 built real traction on it — AWS picked them as the memory provider for its Agent SDK, which is not a small endorsement. Zep's Graphiti temporal knowledge graph is genuinely interesting work on time-aware reasoning. Letta carries real research pedigree from the MemGPT line and goes deeper on the agent runtime than we do. These are serious teams, and "be a layer" got them to market faster than we got there.

We rejected it anyway, for three reasons that all reduce to the same observation: memory workloads have a specific shape, and a general-purpose store fights you on every part of it.

Writes are tiny and constant. Memory isn't bulk ingest. It's a handful of small facts per conversation turn, on every turn, all day. Engines tuned for batch loading and occasional queries have the exact opposite shape.

Recall mixes similarity with time and entity filters. In the stores you'd wrap, filters live around the similarity search — you over-fetch candidates and post-filter, or pre-filter and hope the index cooperates. For memory, the time window and the entity scope aren't decorations on the query. They are the query. We wanted them in the retrieval path natively, not simulated on top.

One store means one meter, and one more service to run. If your backend is a metered vector DB plus object storage plus an embedding API, your architecture is three services and your bill is three meters. We wanted the whole thing to be one process — so it self-hosts as a single binary and prices as one flat rate — which is only possible if you own the storage format, the index, and the embedding path yourself.

When you don't own the engine, each of these is a workaround you write and rewrite. When you do, they're design inputs. That's the whole argument.

What owning the engine bought us

A short hot path. Memory recall sits inside the agent loop, between the user's message and the model call. Every millisecond of retrieval is a millisecond added to time-to-first-token, on every turn. Because retrieval runs in our own engine — no translation layer, no second hop to someone else's store — we get sub-millisecond retrieval, and we can keep it that way, because nobody upstream can regress it.

A boundary you can prove. Every memory is scoped to an entity in the engine itself, so one user's memories can't surface in another user's recall. The isolation is enforced in the retrieval path, not bolted on as a filter you have to remember to apply.

Flat pricing, because we control the cost floor. If your backend is someone else's metered API, your pricing inherits the meter. We know what a write and a recall cost us, end to end, so we charge flat monthly rates — Free at $0, Pro at $49, up through Business at $499 — with no per-API-call metering. Your bill doesn't grow because your agent had a busy week.

One platform, one bill. Document storage, semantic search, and embeddings are included. The DIY alternative — pgvector plus a hand-rolled extraction pipeline plus an embedding API plus object storage — is three bills and the glue code in between.

What it cost us

This post would be dishonest without this section.

We were slower to market. Writing a storage and retrieval engine in Rust takes longer than composing managed services. Competitors shipped, raised, and built mindshare while we were still debugging engine internals. That gap was real, and we felt it.

We maintain everything. When recall misranks or a write path misbehaves, there's no upstream project to file an issue against. From the storage format to the ranking logic, every bug is ours. That's the deal, and it never expires.

There's no plugin ecosystem yet. Mem0 has roughly 58K GitHub stars and a large community of integrators. We don't have that gravity, and pretending otherwise would be silly. What we chose to build instead is first-party SDK breadth: Python, TypeScript, Go, and .NET at full parity. No dedicated memory layer (Mem0, Letta, Zep, Cognee) ships a first-party .NET SDK, and none ships all four of Python/TypeScript/Go/.NET — Aether does. If you're a .NET team building agents, you've probably noticed the silence.

Where the engine shows up

Everything above surfaces in the Memory API. The headline is two calls — remember and recall — with entity scoping, timestamps and time-window recall, fact extraction, recency ranking, and session threads handled by the engine instead of by you.

The shape of it, across sessions:

from aether import Memory
 
mem = Memory("user_4127", api_key="aether_...")
 
# Tuesday, in a support session: the agent learns something.
mem.remember(
    "Runs self-hosted CI; deploy advice must not assume GitHub Actions",
)
 
# The following week — new session, same user. The agent recalls it.
hits = mem.recall("what should I know before giving deploy advice?")
# hits[0].text -> "Runs self-hosted CI; deploy advice must not assume GitHub Actions"

The point is the second half: a different session, days later, and the agent still knows. Memory across sessions is the product. The engine is what makes it fast and scoped.

Want the hands-on version? Give your AI agent persistent memory across sessions walks the same idea end to end in TypeScript and Python.

The bottom line

We built our own engine because agent memory is a workload, not a feature — and workloads deserve engines shaped for them. Owning the engine made us slower and made every bug ours. It's also why retrieval is sub-millisecond, memory self-hosts as one binary, and the bill is flat.

Your agents need memory. Not three bills.

Docs and SDKs are at aetherdb.ai. If you read this far and think we made the wrong trade, I'd genuinely like to hear the argument: support@aetherdb.ai.