Docs Menu

Optimize Sync Atlas Usage

On this page

Atlas Device Sync uses space in your app's synced Atlas cluster to store utility data for sync. This includes a history of changes to each realm. Atlas App Services minimizes that space usage via backend compaction, which runs automatically. You can optimize further by specifying a client maximum offline time, which enables advanced backend compaction.

The App Services backend keeps a history of changes to underlying data for each realm, similar to the MongoDB oplog. App Services uses this history to synchronize data between the backend and clients. App Services stores history in your synced Atlas cluster.

Backend compaction is a maintenance process that automatically runs for all Apps. During compaction, the backend optimizes a realm's history by removing unneeded changesets. The process removes any instruction whose effect is later overwritten by a newer instruction.

Note
Using Backend Compaction With Partition-Based Sync.

Backend Compaction is supported by Partition-Based Sync. App Services does not currently support using backend compaction with Flexible Sync.

Example

Consider the following realm history:

Original History
CREATE table1.object1
UPDATE table1.object1.x = 4
UPDATE table1.object1.x = 10
UPDATE table1.object1.x = 42

The following history would also converge to the same state, but without unneeded interim changes:

Compacted History
CREATE table1.object1
UPDATE table1.object1.x = 42

Backend compaction:

  • reduces space usage in your Atlas cluster, since your history contains fewer instructions.
  • reduces the time and data usage of sync, since clients can converge on the same result with fewer instructions.

Backend compaction runs regularly on all synced clusters as part of Atlas Device Sync.

Advanced backend compaction uses client sync states to further optimize a changeset. However, this optimization can prevent convergence for clients that have only seen a subset of history. As a result, advanced backend compaction only occurs for history older than the client maximum offline time.

Example

Consider the following realm history:

Original History
CREATE table2.object2
CREATE table1.object1
DELETE table2.object2

Because there is a gap in time between the CREATE and DELETE operations for table2.object2, a client could have synchronized the CREATE, but not the DELETE. As a result, compaction can remove the CREATE, but not the DELETE:

Compacted History
CREATE table1.object1
DELETE table2.object2

This guarantees client convergence. If the history didn't include the DELETE, clients that synced the CREATE would never delete table2.object2, but other clients would never create that object in the first place. Instead, all clients get a command to delete that object, and any clients who never created table2.object2 in the first place ignore the DELETE.

The following history uses advanced backend compaction to converge on the same state with even fewer steps:

Advanced Compacted History
CREATE table1.obj1

Because table2.object2 was deleted, it has no impact on the end state of the realm. As a result, advanced backend compaction can completely remove both the CREATE and the DELETE operations for table2.object2 from history. However, any clients that synchronized the CREATE operation but not the DELETE operation will not be able to converge to a common end state.

Advanced backend compaction only runs on Apps that specify a client maximum offline time. Only history older than the client maximum offline time experiences advanced backend compaction.

The client maximum offline time controls the age limit of history optimized with advanced backend compaction. This indirectly changes how long a client can remain offline between synchronization sessions with the backend. Clients that do not synchronize for more than the specified number of days may experience a client reset the next time they connect with the backend.

The lower the client maximum offline time, the sooner compaction can apply advanced backend compaction to history. The resulting optimizations usually result in less storage usage in the synced Atlas cluster.

Note
Using Client Maximum Offline Time With Partition-Based Sync.

Client Maximum Offline Time is supported by Partition-Based Sync. App Services does not currently support setting client maximum offline time with Flexible Sync.

Warning
Client Maximum Offline Time Causes Permanent Changes to History

Client maximum offline time enables advanced backend compaction for older history. This kind of compaction permanently changes affected history, and can cause client resets in the future even after disabling the feature. Use caution before enabling.

Sync should always converge at the same end state on all clients. In order to converge during a sync, clients require the full history of changes beginning immediately after their last sync. When a client does not sync for a long period of time, advanced backend compaction can alter the history in ways that prevent the client from converging. Since synchronization relies on all clients converging on a common result, such a client cannot synchronize.

Example

Consider the following example that contains a CREATE and a DELETE for table2.object2:

Original History
CREATE table2.object2
CREATE table1.object1
DELETE table2.object2

When this history becomes older than the client maximum offline time, advanced backend compaction optimizes the backend history into the following:

Advanced Compacted History
CREATE table1.obj1

However, a client contains this subset of history, containing only the CREATE operation for table2.object2:

Partial Client History
CREATE table2.object2
CREATE table1.object1

Since compaction pruned the DELETE operation for table2.object2, the client will never delete that object. The client cannot converge to the same state as the backend and all other clients.

As a result, the client must complete a client reset before it can resume synchronization. In a client reset scenario, the client deletes the client-local copy of a realm and downloads the current state of that realm from the backend. Synchronization then resumes using the new copy of the realm.

The client maximum offline time controls how long your backend waits before applying advanced backend compaction. After the specified number of days without syncing, clients may experience a client reset the next time they connect with the backend.

Applications that do not specify the client maximum offline time never apply advanced backend compaction. This means that clients can connect after any period of time offline -- weeks, months, or even years -- and synchronize changes. As time passes, frequently-edited realms accumulate many changes. With a large changeset, synchronization requires more time and data usage.

Advanced backend compaction causes permanent, irreversible changes to history. As a result, increasing the client maximum offline time does not immediately change the length of time before clients experience a client reset. Existing history has already been changed by advanced compaction, requiring a client reset. New history needs time to accumulate up to the new client maximum offline time.

Disabling the client maximum offline time feature stops additional advanced backend compaction, but history that has already been changed by advanced compaction will permanently cause client resets in clients.

Decreasing client maximum offline time also does not immediately change the length of time before clients experience a client reset. Client resets begin taking place earlier once the regularly scheduled compaction job applies advanced backend compaction to the newly eligible history.

For Flexible Sync configuration, the amount of Atlas storage space used is directly proportional to the number of queryable fields you have set up. Queryable fields use storage on the backing Atlas cluster. The more queryable fields you configure, the more storage you use on the backing cluster.

You can configure up to 10 queryable fields per application for Flexible Sync. If you have a large number of collections in an App, you may need to use the same queryable field across multiple collections. Combine this with rules and permissions for more granular control over who can access which collections.

Example

Your App may contain 20 or 30 collections, but you can only configure 10 queryable fields. You must re-use queryable fields across collections in order to sync objects from every collection.

For best performance, open a synced realm with a broad query. Then, add more refined queries to expose targeted sets of data in the client application. Slicing off working sets from a broad query provides better performance than opening multiple synced realms using more granular queries.

When you configure queryable fields, consider the broad queries you use for Sync, and select fewer fields that support those broad queries.

Example

In a task tracker app, prefer broad queries such as assignee == currentUser or projectName == selectedProject for a Sync query. This gives you a couple of broad fields against which to Sync documents. In the client, you can further refine your query for things like tasks of a certain priority or completion status to slice off a working set.

  • Atlas Device Sync uses space in your synced Atlas cluster to store change history.
  • Backend compaction reduces the space usage of this history.
  • Advanced backend compaction reduces the space usage of this history even more, but can cause client resets for clients who have not connected to the backend in more than the client maximum offline time (in days).
  • Apps that do not specify a client maximum offline time do not apply advanced backend compaction, so clients of any age can sync without experiencing a client reset.
  • Adding additional queryable fields to a flexible sync configuration will increase storage consumed on an Atlas Cluster. Using broad queries and selecting fewer fields that support broad queries decreases storage consumed.
←  Pause or Terminate SyncError Handling →
Give Feedback
© 2022 MongoDB, Inc.

About

  • Careers
  • Investor Relations
  • Legal Notices
  • Privacy Notices
  • Security Information
  • Trust Center
© 2022 MongoDB, Inc.