Atlas Device Sync Protocol

On this page

Overview

Key Concepts
Changeset
Operational Transformation
Client File Identifier
Network Security
Sync Session Process
Client Connects to the App Server
Client Initiates a Sync Session
App Services Allocates a New Client File Identifier
Client Sends a Client Identifier
Client Uploads & Downloads Sync Changesets
Client Terminates the Sync Session
Request Types
Client -> Server Messages
Server -> Client Messages

Realm is now Atlas Device SDK – Learn More

Overview

Atlas Device Sync uses a protocol to correctly and efficiently sync data changes in real time across multiple clients that each maintain their own local Realm files. The protocol defines a set of pre-defined request types as well as a process by which a client, like a Realm SDK, can connect to an Atlas App Services application server and sync data.

Note

The Realm SDKs internally implement and manage the sync protocol, so for most applications you don't need to understand the sync protocol to use Device Sync. This page covers the protocol at a high level and is not an implementation spec.

Key Concepts

Changeset

A changeset is a list of instructions that describe granular modifications made to a known object state or version by one or more write operations. Changesets are the base unit of the sync protocol. Synced realm clients send changesets to the Device Sync server whenever they perform a write operation. The server sends each connected client the changesets for write operations executed by other clients.

The Device Sync server accepts changesets from any connected sync client (including changes in a synced MongoDB cluster) at any time and uses an operational transformation algorithm to serialize changes into a linear order and resolve conflicting changesets before sending them to connected clients.

Note

Delta Sync

When you make a change to a synced object, App Services does not re-upload the entire object. Instead, App Services sends only the difference ("delta") between before and after. The service compresses the deltas with zlib compression. This reduces network load, which is especially useful in mobile network conditions.

Operational Transformation

An operational transformation is a function that, given two changesets, produces a third changeset that represents logically applying one of the given changesets after the other. Device Sync uses operational transformation to resolve conflicts between changesets from different sync clients that apply to the same base state.

Realm is an offline-first local database even when sync is enabled, which means that any device may perform offline writes and upload the corresponding changesets later when network connectivity is re-established. The operational transformation algorithm is designed to gracefully handle changesets that arrive "out of order" with respect to the logical server clock such that every synced Realm file converges to the same version of each changed object.

Tip

An operational transformation on Realm changesets is analogous to a rebase operation in Git.

Client File Identifier

A client file identifier is a value that uniquely identifies a synced client Realm file and its corresponding server file. The server generates a client file identifier whenever an SDK requests one during its initial sync of a Realm file. Each identifier is a 64-bit, non-zero, positive signed integer strictly less than 2^63.

Note

The server guarantees that all identifiers generated on behalf of a particular server file are unique with respect to each other. The server is free to generate identical identifiers for two client files if they are associated with different server files.

Network Security

The SDK synchronizes with the application server over a WebSocket connection secured by HTTPS using TLS 1.3.

Sync Session Process

To initiate, execute, and terminate a Device Sync session, a Realm SDK and application server send and receive a set of protocol-specific requests.

The SDK negotiates a WebSocket connection over HTTP and then establishes a sync session by sending BIND and IDENT requests to the server over the WebSocket connection. Once the session is established, the SDK and server send synced changesets for a given Realm file to each other via UPLOAD and DOWNLOAD messages. To end the session, the SDK sends an UNBIND request.

Realm SDK                      App Server
    |                                  |
    |  <---- 1. HTTP Handshake ----->  |
    |                                  |
    |  --------- 2. BIND ----------->  |
    |                                  |
    |  <-- 3. IDENT (first time) ----  |
    |                                  |
    |  --------- 4. IDENT ---------->  |
    |                                  |
    |  <---- 5. UPLOAD/DOWNLOAD ---->  |
    |                                  |
    |  --------- 6. UNBIND --------->  |

Client Connects to the App Server

The sync protocol is primarily handled over a WebSocket connection between the SDK and the server. To establish a connection, the SDK sends a handshake HTTP request that includes the following:

a protocol version
a WebSocket key
a valid access token for an authenticated App Services application user

The server sends an HTTP 101 Switching Protocols response that specifies a WebSocket connection for the SDK. The rest of the sync protocol occurs over this connection.

Client Initiates a Sync Session

To begin a sync session, a Realm SDK sends a BIND request to a Device Sync server. The request identifies a specific local Realm Database file to sync and includes a WebSocket connection key that the server will use to open a bidirectional connection to the SDK.

If the SDK is attempting to sync a particular Realm Database file for the first time, it does not yet possess a server-generated client identifier for the file. In this case, the BIND request also indicates that the Device Sync server should allocate one.

App Services Allocates a New Client File Identifier

If a BIND request indicates that the SDK needs a client file identifier, the Device Sync server generates a unique value for the specified Realm Database file and sends it to the SDK in an IDENT response. When the SDK receives the IDENT, it stores the new client identifier persistently in the local Realm Database file.

An SDK only needs to request a client file identifier the first time it syncs each Realm Database file. For subsequent sync sessions, the SDK can use the persisted identifier.

Client Sends a Client Identifier

Once an SDK has initiated a sync session with a BIND request, it must identify the local Realm Database file that it wants to sync. To do this, the SDK sends the application server an IDENT message that contains the client file identifier. If the SDK has previously synced the realm with the server, it can specify the most recently synced server version to optimize the sync process.

When it receives the IDENT message, the server establishes the session. The SDK and server can can now freely send upload and download sync changesets at any time.

Client Uploads & Downloads Sync Changesets

Once a sync session is established, the SDK and server can freely send and receive UPLOAD and DOWNLOAD messages to sync changes whenever they occur.

The SDK sends an UPLOAD message for every changeset it applies except for those that it received in a DOWNLOAD message from the server.

When the server receives an UPLOAD message, it applies operational transformations to resolve any conflicts with other changesets and then applies the transformed changeset to the server version of the realm. This triggers the server to send DOWNLOAD messages to other connected clients, including the synced Atlas cluster which mirrors the server realm. A DOWNLOAD message groups one or more transformed changesets in chronological order from oldest to most recent according to the server's history. The SDK applies the changesets in the same order.

Client Terminates the Sync Session

Once a sync session is established, Device Sync servers will continue to accept UPLOAD messages and send DOWNLOAD messages until the SDK terminates the session. To terminate a sync session, an SDK sends an UNBIND request to the Device Sync server.

Request Types

Client -> Server Messages

The following table describes the request types that a sync client can send to a Device Sync server:

Request	Description
`BIND`	Starts a new sync session on the server and provides a signed authorization token for the current application user. If the client does not yet possess a client file identifier for the Realm file it wants to sync, this also indicates that the server should generate one and send it back to the client. A client must send a BIND before it can send any other requests.
`IDENT`	Provides the client file identifier that indicates the following: the Realm file to sync the client realm's current version the client realm's most recently synced server version This request is related to but distinct from the `IDENT` message sent by the server when a client requests a client file identifier.
`UPLOAD`	Specifies one or more changesets for operations that occurred on the client. The changesets are listed by client version in increasing order.
`TRANSACT`	Specifies a changeset that describes a serialized transaction that occurred on the client. The client may not upload any other changesets until the server confirms or rejects the transaction.
`UNBIND`	Ends a running sync session. A client may not send any other requests for an `UNBOUND` session.
`MARK`	Requests that the server notify the client when it has synced the latest changeset in the server history (at the time of the request).
`REFRESH`	Re-authorizes a current sync session with a new user token.
`STATE_REQUEST`	Requests that the server send one or more `STATE` messages, which the client uses to download the current server version of the Realm file. Clients issue state requests when they asynchronously open a synced realm.
`CLIENT_VERSION_REQUEST`	Requests that the server send the client version of the latest changeset that was sent by the client and processed by the server. This is most commonly used when an SDK executes a client reset.
`PING`	Indicates that the client is still connected and that the server should maintain the sync session. A client must send at least one PING to the server every 10 minutes. The server acknowledges to each PING with a `PONG`. If the server has not received a PING from a client in more than 10 minutes, it considers the client to be disconnected and may automatically end the session.

Server -> Client Messages

The following table describes the request types that the Device Sync server can send to a sync client:

Request	Description
`IDENT`	Provides a client file identifier that the server generated in response to a `BIND` that requested the identifier.
`DOWNLOAD`	Specifies one or more changesets (up to 16MB total) for operations that occurred on other clients. The changesets are listed by server version in increasing order. The changesets in a DOWNLOAD may not be the exact changesets uploaded by other clients. Instead, they may be equivalent changesets output by Device Sync's operational transformation algorithm.
`UNBOUND`	Specifies that the server ended a sync session in response to an `UNBIND`.
`TRANSACT`	Indicates whether or not the server successfully processed a changeset specified in a `TRANSACT` from the client.
`MARK`	Indicates that the server has sent the client the latest changeset that was in the server history when the server received a `MARK` from the client.
`STATE`	Contains one or more segments of encoded data that the client can concatenate to construct the latest server version of the realm. Sent in response to a `STATE_REQUEST`.
`CLIENT_VERSION`	Specifies the client version of the latest changeset that was sent by the client and processed by the server. Sent in response to a `CLIENT_VERSION_REQUEST`.
`ERROR`	Indicates that the server encountered an issue that appears to have been caused by the connected client. For details, see Sync Client Errors.
`PONG`	Acknowledges a `PING`. If a client does not receive a PONG acknowledgement, it indicates that the client cannot currently communicate with the server over the network and that the server may not have received the corresponding `PING`.

← Conflict Resolution

Migrate Device Sync Modes →