I’m trying to future-proof my app by migrating from Partition-based Sync to Flexible Sync and I’ve run into an issue.
My app’s model has been continuously extended in the last two years. Many new fields were added, some of which are marked as required. This has not been an issue, despite many documents in Atlas not containing these fields.
However, when converting an existing App Service from Partition-based Sync to Flexible Sync or creating a new App Service with the app’s schema, documents in Atlas that are missing these fields are excluded from sync. The App Service’s Log contains the following error message:
Failed to convert MongoDB document with configured schema during initial sync
"Detailed Error: document is missing required field(s): [someFieldName]"
Questions:
Does Flexible Sync have a different handling of required fields that are missing in Atlas? Or is the Partition-based Sync only still working because it has been running continuously while the new fields were added?
Is there a way to set a default value for a field in the Schema, so that the App Service handles this gracefully?
“When you add a new required property, you must update existing documents with the new property or they do not sync to the client.”
This doesn’t seem to be entirely correct, as a running Sync appears to continue to work fine, even for objects that are missing the required field. I suspect it would stop syncing these after terminating and restarting sync.
Generally speaking, schema changes have been by far the most painful issue for me with Atlas Device Sync. I’m never really sure what data is (successfully) committed to Atlas and what might break sync.
Especially the fact that many errors are silent is a huge problem from my point of view. I’ve just discovered the (hidden) collection __realm_sync.unsynced_documents and it’s not pretty.
Hello! Sync actually does have a way to gracefully handle required fields that are missing in Atlas. The way to do this is: when adding the new required field, also populate that field’s default property. Sync uses this to fill in any missing required fields. Using this value will also make it so that Atlas documents don’t become unsyncable after adding the required field. You can read more about default here: https://www.mongodb.com/docs/atlas/app-services/sync/data-model/update-schema/#add-a-default-value-on-a-property
Sync actually does have a way to gracefully handle required fields that are missing in Atlas. The way to do this is: when adding the new required field, also populate that field’s default property.
That’s exactly what I have been doing, but this is not sufficient to ensure that sync can be terminated/restarted without issue.
The additional required field isn’t added to existing documents in Atlas. An existing, already-running Device Sync continues to work fine, but a new App Service/Device Sync with the same schema would fail to sync these documents – which is precisely my issue.
That’s helpful; thanks.
Do you happen to know why the Metrics indicate 122 Unsyncable Documents, while __realm_sync.unsynced_documents has 82 000 entries?
If you terminate and re-enable Device Sync with these default values for required fields added, then the Atlas documents missing those required fields will no longer be unsyncable. What were the issues you encountered when doing this?
Do you happen to know why the Metrics indicate 122 Unsyncable Documents, while __realm_sync.unsynced_documents has 82 000 entries?
In Partition-based Sync, the unsynced_documents collection is shared among all Partition-based Sync Apps. If you have multiple apps, that could the cause for the difference here.
When I tried to create a new App Service/Device Sync with the same schema that an existing Sync used, many documents weren’t synced to the client. Errors were logged in the App Service Logs (see first post of this thread).
By default values, you mean the ones in the client app that uses the SDK, correct? Or is there a way to specify default values in the App Service schema?
I do have multiple apps, but I’ve looked up a few ObjectIDs and they were all from the main app (which only shows 122 Unsyncable Documents on the Metrics page).
Is there a way to show a list of the unsyncable documents that are counted on the Metrics page?
Update: I’ve deleted all other App Service apps and the number of entries in unsynced_documents hasn’t changed.
When doing this, ensure that the type of the default value matches the bsonType
Is there a way to show a list of the unsyncable documents that are counted on the Metrics page?
Update: I’ve deleted all other App Service apps and the number of entries in unsynced_documents hasn’t changed.
Unfortunately there’s not a way to distinguish between these. The unsynced documents collection is internal, so it doesn’t always update immediately when doing things like deleting the app. I’d recommend referring to the value in the Metrics page as the source of truth for this.
Interesting; I had failed to find any documentation of this ability and the UI doesn’t seem to provide a way to set a default value either. I suspect this will solve my issues. Thanks!
Hope it helps! The doc that I linked is referring to this, but I realize that it’s not very clear on the fact that this in the the App Services UI. I’ll follow up on updating that page
Also, regarding the unsynced_documents collection, you could filter them by your app’s id, which is found in the App Services URL (i.e. https://services.cloud.mongodb.com/groups/<groupID>/apps/<appID>/dashboard)
I can confirm that adding a default value to the App Service Schema fields solved my issue and initial sync ran successfully.
A few follow-up questions:
Unsyncable Documents in Device Sync’s Metrics only show objects that failed to sync from the App Service to Atlas, right?
Documents in Atlas that failed to sync to Device Sync aren’t shown anywhere except in the logs, correct?
Schema validation will fail for required fields that are missing in Atlas, even if they have a default value in the App Service’s Schema. Is that as it should be?
Changing an App Service Schema field’s default value doesn’t sync the new value to the client SDK. Should this change perhaps trigger the prompt to Reinitialize Sync?
As for the unsynced_documents, all the 80K entries are definitely for the same app that only shows 122 Unsyncable Documents in App Service’s Metrics, and they reference items that I have updated with the missing fields (and these updates are visible in Atlas, so they were definitely synced). I assume the unsynced_documents collection will be updated eventually.
Unsyncable documents are documents that live in your Atlas collection that cannot be used with Device Sync due to an issue with the schema validation. This is only for documents coming from Atlas to Device Sync. Documents that come from devices are not tracked as unsyncable documents. You can read more about this here https://www.mongodb.com/docs/atlas/app-services/mongodb/internal-database-usage/#unsynced-documents
I’m not totally clear what do you mean about schema validation failing for required fields w/ a default value that are missing in Atlas . Where are you seeing this failure & what did you do to trigger it?
Changing the default value in the schema currently does not cause Sync to process that change immediately - this is known issue and not probably something that we’ll change in the short term.
Regarding the unsynced_documents collection, I’d be interested to poke around the app a bit to see what’s going on, as the behavior you’re describing sounds a bit strange. If you’re interested, feel free to DM me your app ID and I can look into it
