agent-engine-sdk-memory

The problem it solves

Memory gives an agent a persistent store of the facts, conversations, procedures and vocabulary that matter — and fills that store itself, by extracting durable learned facts from the conversations your agent has with an LLM. The extraction itself is done by an LLM you configure.

For a platform-deployed agent, none of this needs wiring. The runtime records each conversation turn to short-term memory as the agent workflow runs, and promotion and extraction then proceed asynchronously in the background, so nothing blocks a response while a fact is distilled. An agent that already knows something durable does not have to wait for extraction to find it — it can write the fact directly with save_semantic and the other typed writes.

Reading it back works either way round. An agent can go after one kind of fact at a time — search_semantic for what it has learned, search_episodes for what happened before, search_taxonomic for what a term means, discover_procedures for how something is done — and handle the results itself.

Or it can hand the whole job over. build_context_from_sources takes one spec per source, each declaring its own retrieval mode, filter and candidate count, then retrieves from every source on its own terms, ranks and de-duplicates the results together, trims them to a token budget, and returns a single block of context to drop into the next prompt. build_context does the same with one mode and one filter for every source; reach for the per-source form when they need to differ.

Two layers

Memory is split by how long things live and how they are shaped.

Short-term memory (STM) is the raw conversation: one record per turn, scoped to a session, written as it happens. It is cheap to write and complete — everything said, in order.

Long-term memory is what survives the conversation, distilled into four shapes because they answer different questions:

Projects can also declare custom types for domain records the four shapes do not fit.

How writes become memory

You do not write long-term memory by hand, though you can. The normal path is:

record_turn(...)          you write turns as the conversation happens
      │
      ▼   turns accumulate into a session
   snapshot                a contiguous run of turns, summarised
      │
      ▼   an LLM reads the snapshot and extracts what is durable
  semantic · episodic · procedural · taxonomic

Extraction is asynchronous and runs in the background, so record_turn stays a fast write. A fact learned in one conversation is therefore available to the next one, not to the next turn.

The consequence worth designing around: recent turns are visible immediately through STM, extracted knowledge appears a little later. Ask for stm when you need what was just said.

When extraction fails

Background extraction depends on an embedding service, an extraction LLM, and the database, and any of them can fail. Your writes are insulated from that: record_turn succeeded when it returned, whatever happens downstream.

Behind the scenes, a failed step is classified by one question — can the condition change without the request changing? Environmental failures (network errors, provider outages, rate limits, an API key mid-rotation) are retried automatically with backoff, paced to the failure: a network blip retries within seconds, a credential problem more slowly, since keys are rotated by humans. Content-determined failures are not retried, because a retry cannot change the outcome; they are recorded server-side where operators can see them, rather than looping forever.

This applies to the whole extraction path, not only its outer edges. A step that fails no longer quietly produces an empty result: it either retries, or is recorded where operators can act on it.

One unusable item does not discard the rest. If a single memory cannot be embedded — its content is too long for the model, or the provider rejects it — that memory is still stored, and the others in the same batch are unaffected. What the stored memory lacks is a vector. In practice:

• It is still found by text search, and by hybrid — hybrid combines both rankings, so the text side still surfaces it.

• It is not found by semantic search, which compares vectors only.

This is a good reason to prefer hybrid as your default retrieval mode. It is already the better choice for exact identifiers and rare words, and it also means a memory that could not be embedded still reaches you.

What this means for an agent:

• A provider outage delays extracted knowledge; it does not lose your turns. STM is written synchronously and is unaffected — keep asking for stm when you need what was just said.

• Retries are bounded: a dependency outage that outlasts them stops retrying rather than looping indefinitely, and the failure is recorded rather than discarded.

• None of this surfaces through the SDK as an error. Extraction failures are a server-side concern; the SDK-visible signal is extracted memories not (yet) appearing in retrieval, or appearing through text and hybrid but not semantic search.

How reading works

Assembled prose, or the records themselves — two different questions.

``build_context_from_sources(...)`` is the one to reach for by default. You give it a query and a token budget; it retrieves across the memory types you name, ranks the results together, drops near-duplicates, trims to the budget, and returns something you can put straight into a prompt. It is the whole retrieval pipeline behind one call, and it takes one spec per source, so modes and filters are set per source rather than once for all of them.

Retrieval can match three ways. semantic compares embeddings and finds things that mean the same; text matches words and finds things that say the same; hybrid runs both and fuses the rankings. Hybrid is usually the right default — vector search alone misses exact identifiers and rare words, text search alone misses paraphrases.

``build_context(...)`` is the simpler builder: same pipeline, but one mode and one filter apply to every source it reads.

``search(...)`` and the per-type searches (search_semantic, search_episodes and so on) return ranked records instead of assembled prose, for when you want to inspect or post-process them yourself.

What scopes a memory

Every record carries the identity it was written under, and every read is filtered by that identity in the database query rather than afterwards. The fields are: organization, user, project, session, and optionally the agent, plus a visibility that decides whether a record is private to its user or readable more widely.

This matters for a practical reason: a search or build_context scoped to a user will not surface another user’s private memory, because the constraint is part of the query rather than a filter applied to its results. So bind(...) is not a convenience — it is how you declare the scope that subsequent reads and writes operate within, and getting it wrong writes one user’s memory under another’s identity.

user_id and visibility answer different questions

user_id is whose memory a record is. visibility is how far it reaches:

No visibility value reaches outside the project the record was written in. Memory is stored per project, so a project boundary is not something visibility can cross — org is a historical name, and it means “not restricted to one user”, not “visible to other projects”.

The two are independent, and on a read they combine as and, never as or. Each field you supply adds one more equality condition to the query; each field you omit leaves that dimension unconstrained:

The third row is the one that surprises people. search_semantic(query, user_id="user_1", visibility="org") does not mean “user_1’s memories plus the org’s”. It means “the org-visible memories that user_1 owns”, which is smaller than either constraint alone. There is no union: to read shared knowledge and a user’s own private memory, make two calls and merge the results yourself.

The hosted platform

The managed service. Pass a service-account access token and your project id.

Mint the token with the agentengine CLI. Create a service account once — the client secret is shown only once, so save it immediately:

agentengine service-account create my-agent --project-id <your-project-id> --role AGENT_DEVELOPER

Then exchange the client ID and secret for a short-lived (1-hour) access token (curl prompts for the client secret so it stays out of shell history):

ACCESS_TOKEN=$(curl --fail-with-body --silent --show-error      --user <client-id>      --data grant_type=client_credentials      https://agentengine.mongodb.com/api/v1/oauth/token | jq -er .access_token)

memory = Memory(service_account_token="<your-access-token>", project_id="<your-project-id>")

The platform pins your credential to its project, so a project_id that does not match is rejected. A blank service_account_token raises ValueError.

api_key (and AGENTIC_MEMORY_API_KEY) remain accepted as a deprecated alias and emit a DeprecationWarning; passing both the new and the legacy input raises ValueError. Project API keys can no longer be minted over HTTP, so new integrations must use a service-account token.

Local development

Point at a backend you run yourself, such as the stack agentengine dev up starts.

memory = Memory(base_url="http://localhost:8080")

Leave project_id empty for local development.

Inside a platform agent (app-bound)

When your agent runs on Atlas Agent Engine, you don’t construct or connect Memory at all — the platform injects a ready, pre-wired instance, with identity, tenancy, and transport already bound. Application code passes none of the inputs above; it uses the handle the runtime hands it. Identity (user, session, org, project) resolves from the ambient runtime context, so operations can be called directly:

# `memory` is supplied by the platform runtime — do not construct it.
memory.record_turn(role="user", content="I'm allergic to penicillin.")
context = memory.build_context(query="What medications should I avoid?")
# bind(...) is still available to scope a call chain to a specific identity.

The app-bound path has a few capability gaps (tool-call / model turn metadata and some list-style reads); see the App-bound column in What each connection supports and `docs/capability-matrix.md <docs/capability-matrix.md>`__.

Env vars are a fallback

Each input also falls back to an AGENTIC_MEMORY_* environment variable when you omit the argument. An explicit argument always wins. Memory() with no arguments reads all three from the environment.

How routing works

project_id decides where calls go. Auth never does.

• Set project_id, and the SDK calls your project’s routes, /api/v1/projects/{project_id}/memory/*.

• Leave it empty, and the SDK calls the backend directly, /api/v1/memory/*.

If project_id is set but the backend has no matching route, such as a local backend, the call raises `MemoryRouteNotFoundError <#errors>`__ with a hint to unset it. The reverse also holds. Target the hosted service with no project_id and the call 404s with a hint to set it.

Advanced: authenticated direct backend

Auth and routing are independent, so you can pass a service_account_token with project_id empty. The SDK then sends authenticated calls straight to the backend at base_url, on the direct routes, skipping the project path. This suits a hosted Orchestration Engine (OE) reached directly.

What each connection supports

Capabilities follow where the call is routed, not auth.

An unsupported call raises `MemoryNotSupportedError <#errors>`__ — before any network request for known per-backend gaps, or after the response for capability gaps only the platform can report (see Errors) — never a silent failure or a raw HTTP error. The full per-backend reference, including app-bound gaps on type-specific CRUD, is in `docs/capability-matrix.md <docs/capability-matrix.md>`__.

App-bound agents are the exception to all of the above. Inside a deployed platform agent, the platform injects a ready runtime, so application code passes none of these inputs.

Settings you cannot change later

Some parts of the config are effectively write-once, so decide them before you start writing memories:

• Metadata partition keys — the declared filterable attributes. A key cannot be removed or have its type changed once declared.

• Custom memory type declarations — once a type is accepted, its collection and tag set cannot be edited or removed through the config API; declare a new type name instead.

• Search indexes — provisioned only when they do not already exist. Adding a partition key after the server has started does not rebuild the existing index; the drift is logged, and a filter on the new key then fails at the database rather than being silently ignored. Recreating the index is a manual operation.

Log level, extraction LLM and the enabled extraction types can be changed and re-applied. Changing the embedding model or dimension requires migrating and re-embedding existing memories; dimension changes also require rebuilding the vector search indexes.

Setting visibility on a write

Every write takes a visibility. The default differs by memory type, because the types are used differently:

Taxonomic memory defaults to org because a domain vocabulary is shared by definition — a term and its meaning are rarely one user’s private business. Everything else defaults to private, so a fact you save is scoped to the user it was learned from unless you say otherwise.

record_turn is the exception: it takes no visibility, and no user_id either. A conversation turn is always written under the bound user and session, so bind(...) before recording turns — there is no per-call way to set the user on a turn.

# Private to user_1 — the default.
session.save_semantic(text="Prefers window seats.", label="seat-preference")
# Readable across the whole organization.
session.save_semantic(
    text="Refunds over $500 need manager approval.",
    label="refund-policy",
    visibility="org",
)

Scoping a read

Reads leave visibility unset, so they are constrained only by the identity that resolves — usually the bound user_id. That is the right default for personalization: you get everything that user owns, at any visibility.

To reach shared knowledge you pass a visibility, and here the and rule bites. A handle bound to user_id="user_1" still contributes that user_id, so this reads only the org-visible records user_1 owns:

session = memory.bind(MemoryRequestContext(user_id="user_1", session_id="thread_123"))
session.search_semantic("refund policy", visibility="org")   # user_1 AND org

Passing user_id=None does not widen it. A None or blank argument counts as “not supplied” at every tier, so it falls through to the bound value. Read across all users with a handle that has no user_id bound:

# The original handle is unbound, so it carries no user_id.
memory.search_semantic("refund policy", visibility="org")
# Or bind only the parts you want.
thread = memory.bind(MemoryRequestContext(session_id="thread_123"))
thread.search_semantic("refund policy", visibility="org")

So an assistant that needs both the user’s own history and the team’s shared knowledge issues two reads and merges them:

personal = session.search_semantic(query, top_k=5)                       # user_1, any visibility
shared = memory.search_semantic(query, visibility="org", top_k=5)        # org-wide, any owner

Reading back your own metadata

record_turn takes an optional metadata dict, and short-term retrieval returns it in its own slot on the chunk: chunk.metadata["metadata"]. Your keys are kept apart from the platform’s turn fields (session_id, role, turn_seq, …) rather than mixed in with them, so a key of yours never collides with one of theirs — you can name a key role or session_id and read your own value back.

Two limits that collision-safety does not cover. Values have to survive the round trip, so they must be JSON-serializable and of a sane size. And to be filterable a key additionally has to be declarable as a metadata partition key: each dot-separated segment must match ^[a-z][a-z0-9_]{0,63}$, at most one dot is allowed, and a few names are reserved (org_id, project_id, user_id, agent_id, content, embedding, embedding_model_id, created_at). A key outside that shape — Tier, $foo, a.b.c — is still stored and returned, it just cannot be filtered on.

An empty dict is treated as no metadata: metadata={} records the turn without a metadata slot on the chunk, so read it with chunk.metadata.get("metadata", {}) if the caller might not have supplied any. Semantic memory behaves identically, which is what lets one read path cover both. Episodic, taxonomic and procedural behave the same way: the dict you write comes back under chunk.metadata["metadata"], and an empty one means no slot at all. Episodic is the one exception: a compatibility filter drops the whole dict, unrelated keys included, if it happens to contain citation_score, llm_confidence_score and combined_score together, since that shape also matches a pre-migration internal blob. Retrieval may also carry chunk.metadata["contextual_metadata"] — that is the platform’s own extraction artifact, not your data, and it is not part of any contract you should depend on.

Short-term memory is scoped to a session, so the write and the read have to name the same one — bind it once and pass it through:

from agent_engine_sdk_memory import (
    Memory,
    MemoryRequestContext,
    MemorySource,
    RetrievalMode,
    SourceSpec,
)
session_id = "session-42"
memory = Memory(base_url="http://localhost:8080").bind(
    MemoryRequestContext(user_id="user-1", session_id=session_id)
)
memory.record_turn(
    role="user",
    content="The engagement is green and the rollout continues.",
    metadata={"engagement_id": "eng-alpha", "tier": 2},
)
context = memory.build_context_from_sources(
    query="what is the engagement status?",
    sources=[
        SourceSpec(
            source=MemorySource.STM,
            mode=RetrievalMode.HYBRID,
            metadata_filter={"metadata.engagement_id": "eng-alpha"},
        )
    ],
    session_id=session_id,
    include_memories=True,
)
for chunk in context.selected_memories or []:
    print(chunk.metadata["metadata"]["engagement_id"])

Atlas Search indexes the write asynchronously, so a read issued immediately after may not see the turn yet.

One spelling difference to know: you read the bare name inside the nested dict, chunk.metadata["metadata"]["engagement_id"], but you filter on the qualified index path, {"metadata.engagement_id": ...}. Both acknowledge the nesting — the filter addresses the stored document, where your dict really does live under metadata.

Filtering on the bare name is rejected with an error listing the accepted paths, so that mistake tells you the answer.

Filterable metadata has to be declared for the project first: engagement_id above has to appear under metadata_partition_key, with short_term listed in metadata_partition_index. See Declaring filterable metadata under Use it.

Package layout

agent_engine_sdk_memory/
├── memory.py        # Memory — the public, transport-free facade
├── protocol.py      # MemoryRequestContext, MemoryRuntime + MemoryCrudClient seams
├── identity.py      # resolve_identity — call arg > bind ctx > runtime ctx
├── errors.py        # MemoryIdentityError + the typed transport-error family
├── models.py        # Pydantic models for memory requests/responses
├── validation.py    # require_positive_max_tokens — public build_context guard
├── _transport.py    # _HttpTransport — shared retry / error-mapping / lifecycle
├── _http_runtime.py # _HttpMemoryRuntime — the workflow ops + per-backend profiles
├── _direct_crud.py  # empty-tenancy MemoryCrudClient over the shared transport
├── _wire.py         # custom-type request-body builders shared by the CRUD clients
├── _tag_syntax.py   # client-side custom-type name + tag-syntax checks
├── _client.py       # MemoryClient — internal HTTP client for the Memory Server API
└── _denylist.py     # do-not-add dependency denylist (see below)

Memory is transport-free: it delegates the workflow operations (record_turn, build_context, the searches, discover_procedures) to an injected MemoryRuntime, and the type-specific CRUD to an injected MemoryCrudClient. The public surface is frozen by a snapshot test in tests/test_package_contract.py; MemoryClient stays internal. Per-call tenancy (org_id, body-level project_id) never appears in a public signature; the only project_id on the public surface is the optional constructor route-shape selector, which goes into the URL, not a request body.

Dependency denylist

The pydantic + httpx-only constraint is enforced, not aspirational. _denylist.py lists known-heavy distributions and import names (langchain, fastapi, pymongo, …), and tests/test_package_contract.py fails if importing the package pulls any of them into sys.modules.

Who uses it

agent-engine-runner-shared declares this package as a workspace dependency and constructs the internal MemoryClient in agent_engine_runner_shared/memory.py, passing its execution-context lookup as an execution_id_provider callable so per-request execution-id headers work without this package importing any platform code.