Integrate MongoDB with Feast

Online and Offline Stores

• The online store is a key-value store backed by a single MongoDB collection, optimized for low-latency retrieval of the latest features per entity during online inference.

• The offline store is a compute and translation layer that queries rows of historical feature data stored in a MongoDB collection (typically named feature_history) for training datasets, scoring and materialization (promoting data to the online store).

Common Workflow Patterns

A typical end-to-end workflow looks like this:

1. Define entities, feature views, and data sources that point to MongoDB-backed collections.

2. Ingest feature data into the offline store via offline_write_batch, which accepts a PyArrow table as input and inserts the data into the feature_history MongoDB collection following the offline store schema.

3. Generate training data using get_historical_features, which runs an efficient point-in-time join over historical feature rows stored in MongoDB.

4. Materialize the latest feature values from the offline store into the online store using pull_latest_from_table_or_query and online_write_batch.

5. Serve features online via Feast's online APIs, which read from a single MongoDB collection keyed by a serialized entity key.

Concept Mapping

Data Model

The MongoDB offline store uses a single shared collection (by default feature_history) that stores append-only historical feature rows for all feature views.

Each document represents one observation of one entity for one FeatureView at a specific event timestamp:

{
  "entity_id": "Binary(...)",
  "feature_view": "driver_stats",
  "event_timestamp": "ISODate(2024-01-15T12:00:00Z)",
  "created_at": "ISODate(2024-01-15T12:01:00Z)",
  "features": {
    "conv_rate": 0.72,
    "acc_rate": 0.91,
    "avg_daily_trips": 14
  }
}

Key properties:

• Append-only: historical data is treated as immutable; corrections are written as new rows with newer created_at timestamps rather than in-place updates.

• Time-series friendly: event_timestamp represents when the feature value was observed; created_at is used as a tie-breaker when multiple observations share the same event timestamp.

• Feature grouping by FeatureView: feature_view identifies which FeatureView the row belongs to, so a single collection can host multiple FVs.

A single compound index supports all major query patterns:

(entity_id ASC, feature_view ASC, event_timestamp DESC, created_at DESC)

This index enables efficient range scans over entities and feature views, while ensuring that the most recent observation per (entity_id, feature_view) is seen first during aggregation.

Core Offline Operations

The MongoDB offline store implements the standard Feast offline store interface:

• offline_write_batch - Writes a pyarrow.Table of feature data into the underlying MongoDB collection, using the configured MongoDBSource metadata to determine connection_string, database, and collection.

• get_historical_features - Given an entity_df of entities and event timestamps plus a set of FeatureViews, returns a widened table where each row includes point-in-time correct feature values: for each (entity_id, event_timestamp) pair, the most recent feature value whose event_timestamp <= entity_event_timestamp and within TTL is selected.

• pull_latest_from_table_or_query - Returns one row per entity containing the latest feature values in a time window, used by Feast's materialization engine to seed the online store.

• pull_all_from_table_or_query - Retrieves all rows from a data source in a specified date range for export or inspection, backed by the same feature_history schema and index.

• persist (via RetrievalJob.persist) - Writes the result of a historical feature query to a separate collection or external sink via SavedDatasetStorage, distinct from feature_history.

FeatureStore.write_to_offline_store(feature_view_name, df)
  → provider.ingest_df_to_offline_store(feature_view, arrow_table)
    → OfflineStore.offline_write_batch(config, feature_view, table, progress)

$match { feature_view, event_timestamp: {$gte, $lte} }
→ $sort { entity_id, event_timestamp DESC, created_at DESC }
→ $group $first by entity_id
→ $project { entity_id, event_timestamp, features.* }

Aggregation Implementation

The recommended offline implementation is the aggregation-based MongoDB offline store, named MongoDBOfflineStore.

Key characteristics:

• Uses a single feature_history collection shared by all FeatureViews, distinguished by feature_view.

• Relies on the compound index (entity_id, feature_view, event_timestamp, created_at) for all queries, avoiding full collection scans.

• Uses server-side $group $first for "scoring" workloads (one row per entity), and pd.merge_asof for "training" workloads with repeated entity IDs, balancing correctness and performance.

• Bounded memory usage via chunking, so large entity_df values can be processed without exhausting RAM.

Benchmarks show this implementation provides the best combination of throughput and memory efficiency compared to alternative MongoDB offline approaches.

scoring_path = (
    entity_df[all_entity_id_cols].drop_duplicates().shape[0]
    == len(entity_df)
)

$match  →  $sort  →  $group $first  →  $project

# Merge on entity_id (left = entity_df rows, right = server results)
merged = result[["_fv_entity_id", event_timestamp_col]].merge(
    fv_join, on="_fv_entity_id", how="left"
)
# Null out rows where the server doc is in the future or outside TTL
future_mask = merged["_fv_ts"] > merged[event_timestamp_col]
if fv.ttl:
    ttl_mask = merged["_fv_ts"] < (
        merged[event_timestamp_col] - fv.ttl
    )
    bad_mask = future_mask | ttl_mask
else:
    bad_mask = future_mask
for feat in features:
    vals = merged[feat].copy()
    vals[bad_mask | merged["_fv_ts"].isna()] = None
    result[col] = vals.values

$match  →  (no $group)

result = pd.merge_asof(
    result.sort_values(event_timestamp_col),
    fv_df_subset.sort_values("_fv_ts"),
    left_on=event_timestamp_col,
    right_on="_fv_ts",
    by="_fv_entity_id",
    direction="backward",
)

if len(working_df) <= CHUNK_SIZE:
    result_df = _run_single(working_df, coll)
else:
    chunks = [
        _run_single(chunk, coll)
        for chunk in _chunk_dataframe(working_df, CHUNK_SIZE)
    ]
    result_df = pd.concat(chunks, ignore_index=True)

if "features" in fv_df.columns:
    for feat in features:
        src_col = reverse_fm.get(feat, feat)
        fv_df[feat] = fv_df["features"].apply(
            lambda d, _s=src_col: (
                d.get(_s) if isinstance(d, dict) else None
            )
        )
    fv_df = fv_df.drop(columns=["features"])

Offline Store Capabilities

Additional conveniences like exporting directly to data lakes or warehouses depend on the specific RetrievalJob implementation and are expected to follow Feast's standard patterns for offline stores.

Data Model

The MongoDB online store uses a single collection for all FeatureViews, keyed by the serialized entity key.

• _id: serialized_entity_key(entity_key), produced by Feast's stable encoding function that sorts entity names and values and encodes them into bytes.

• features: nested subdocument where each FeatureView maintains its own feature namespace.

• event_timestamps: per-FeatureView timestamps indicating when the latest value for that FeatureView was written.

• created_timestamp or updated_at: bookkeeping fields useful for TTL indexing and diagnostics.

Example (simplified):

{
  "_id": "b\"<serialized_entity_key>\"",
  "features": {
    "driver_stats": {
      "rating": 4.91,
      "trips_last_7d": 132
    },
    "pricing": {
      "surge_multiplier": 1.2
    }
  },
  "event_timestamps": {
    "driver_stats": "ISODate(2026-01-01T12:00:00Z)",
    "pricing": "ISODate(2026-01-21T12:00:00Z)"
  },
  "created_timestamp": "ISODate(2026-01-21T12:00:00Z)"
}

Design rationale:

• A single collection keeps each entity's state in one document, which matches Feast's expectation of key-based lookups and avoids fragmenting state across per-FeatureView collections.

• Using the serialized entity key as _id reuses Feast's deterministic encoding, avoids duplicate primary keys across collections, and keeps retrieval to a single key lookup per entity.

Core Online Operations

The MongoDB online store implements Feast's standard online store API:

• online_write_batch - During materialization, Feast writes the latest feature values for each entity into MongoDB documents. Each batch upsert updates only the relevant nested features.<feature_view> subdocument and its corresponding entry in event_timestamps, keeping entity documents atomic and consistent.

• online_read and get_online_features - Online serving resolves entity keys into _id values using the same serialization logic as offline, then performs key lookups. Each lookup returns all requested features for the entity in a single round trip, leveraging the nested features structure.

• TTL and freshness - Feature TTL is configured on the FeatureView and used primarily in offline PIT joins; online TTL can be implemented with an index on updated_at or similar timestamp, consistent with Feast's notion that offline stores are append-only while online stores hold the latest state.

Offline Store Configuration

The offline store is configured using MongoDBOfflineStoreConfig:

class MongoDBOfflineStoreConfig(FeastConfigBaseModel):
    type: str = "...MongoDBOfflineStore"
    connection_string: str = "mongodb://localhost:27017"
    database: str = "feast"
    collection: str = "feature_history"

Example feature_store.yaml:

offline_store:
  type: feast.infra.offline_stores.contrib.mongodb_offline_store.mongodb.MongoDBOfflineStore
  connection_string: "mongodb+srv://user:pass@cluster.mongodb.net"
  database: feast
  collection: feature_history

MongoDBSource is the corresponding DataSource. Its name field becomes the feature_view discriminator stored in every document. For full configuration options, see the MongoDB Data Source reference in the Feast docs.

source = MongoDBSource(
    name="driver_stats",
    timestamp_field="event_timestamp",
    created_timestamp_column="created_at",
)