When MongoDB acquired Realm in 2019, we knew we wanted to give developers the easiest and fastest way to synchronize data on-device with a backend in the cloud.
In an offline-first environment, edge-to-cloud data sync typically requires thousands of lines of complex conflict resolution and networking code, and leaves developers with code bloat that slows the development of new features in the long-term.MongoDB Realm’s sync simplifies moving data between the Realm Mobile Database and MongoDB Atlas. With huge amounts of boilerplate code eliminated, teams are able to focus on the features that drive 5-star app reviews and happy users.
Since bringing Realm Sync GA in February 2021, we’ve seen it transform the way developers are building data synchronization into their mobile applications. But we’ve also seen developers creating workarounds for complex sync use cases. With that in mind, we’ve been hard at work building the next iteration of Realm Sync, which we’re calling Realm Flexible Sync.
Realm Flexible Sync takes into account a year’s worth of user feedback on partition-based sync, and aims to make syncing data to MongoDB Atlas a simple and idiomatic process by using a client-defined query to define the data synced to user applications.
#How Flexible Sync Works
Flexible Sync lets developers start writing code that syncs data more quickly – allowing you to choose which data is synced via a language-native query and to change the queries that define your syncing data at any time.
With Flexible Sync, developers can enable devices to define a query on the client side using the Realm SDK’s query-language, which will execute on MongoDB Atlas to identify the set of data to Sync. Any documents that match the query will be translated to Realm Objects and saved to the client device’s local disk. The query will be maintained on the server, which will check in real-time to identify if new document insertions, updates, or deletions on Atlas change the query results. Relevant changes on the server-side will be replicated down to the client in real-time, and any changes from the client will be similarly replicated to Atlas.
Flexible Sync is distinctly different from the partition-based sync used by Realm Sync today.
With partition-based sync, developers must configure a partition field for their Atlas database. This partition field lives on each document within the Atlas database that the operator wants to sync. Clients can then request access to different partitions of the Atlas database, using the different values of the partition key field. When a client opens a synchronized Realm they pass in the partition key value as a parameter. The sync server receives the value from the client, and sends any documents down to the client that match the partition key value. These documents are automatically translated as Realm Objects and stored on the client’s disk for offline access.
Partition-based sync works well for applications where data is static and compartmentalized, and where permissions models rarely need to change. With Flexible Sync, we’re making fine-grained and flexible permissioning possible, and opening up new app use cases through simplifying the syncing of data that requires ranged or dynamic queries.
Unlike with partition-based sync, Flexible Sync makes it seamless to implement the document-level permission model when syncing data - meaning synced fields can be limited based on a user’s role. We expect this to be available at beta, and with field-level permissions coming after that.
Consider a healthcare app, with different field-level permissions for Patients, Doctors, and Administrative staff using the application. A patient collection contains user data about the patient, their health history, procedures undergone, and prognosis. The patient accessing the app would only be able to see their full healthcare history, along with their own personal information. Meanwhile, a doctor using the app would be able to see any patients assigned to their care, along with healthcare history and prognosis. But doctors viewing patient data would be unable to view certain personal identifying information, like social security numbers. Administrative staff who handle billing would have another set of field-level permissions, seeing only the data required to successfully bill the patient.
Under the hood, this is made possible when Flexible Sync runs the query sent by the client, obtains the result set, and then subtracts any data from the result set sent down to the client based on the permissions. The server guards against clients receiving data they aren’t allowed to see, and developers can trust that the server will enforce compliance, even if a query is written with mistakes. In this way, Flexible Sync simplifies sharing subsets of data across groups of users and makes it easier for your application's permissions to mirror complex organizations and business requirements.
Flexible Sync also allows clients to share some documents but not others, based on the ResultSet of their query. Consider a company where teams typically share all the data within their respective teams, but not across teams. When a new project requires teams to collaborate, Flexible Sync makes this easy. The shared project documents could have a field called allowedTeams: [ marketing, sales]. Each member of the team would have a client-side query, searching for all documents on allowedTeams matching marketing or sales using an $in operator, depending on what team that user was a member of.
#Ranged & Dynamic Queries
One of Flexible Sync's primary benefits is that it allows for simple synchronization of data that falls into a range – such as a time window – and automatically adds and removes documents as they fall in and out of range.
Consider an app used by a company’s workforce, where the users only need to see the last seven days of work orders. With partition-based sync, a time-based trigger needed to fire daily to move work orders in and out of the relevant partition. With Flexible Sync, a developer can write a ranged query that automatically includes and removes data as time passes and the 7-day window changes. By adding a time based range component to the query, code is streamlined. The sync resultset gets a built-in TTL, which previously had to be implemented by the operator on the server-side.
Flexible Sync also enables much more dynamic queries, based on user inputs. Consider a shopping app with millions of products in its Inventory collection. As users apply filters in the app – viewing only pants that are under $30 dollars and size large – the query parameters can be combined with logical ANDs and ORs to produce increasingly complex queries, and narrow down the search result even further. All of these query results are combined into a single realm file on the client’s device, which significantly simplifies code required on the client-side.
Ultimately, our decision to build Flexible Sync is driven by the Realm team’s desire to eliminate every possible piece of boilerplate code for developers. We’re motivated by delivering a sync service that can fit any use case or schema design pattern you can imagine, so that you can spend your time building features rather than implementing workarounds.
The Flexible Sync project represents the next evolution of Realm Sync. We’re working hard to get to a public beta by the end of 2021, and believe this query-based sync has the potential to become the standard for Realm Sync enabled applications. We won’t have every feature available on day one, but iterative releases over the course of 2022 will continuously bring you more query operators and permissions integrations.
Interested in joining the early access program? Sign-up here and we’ll let you know when Realm Flexible Sync is available in beta.