I am having problems to sync documents which contain timeseries data. Using the bucket pattern, I store the data for one hour in one document respectively. One document contains about 1MB of data.
This data is parsed from files, so it is not rare that a new document containing the data of one full hour is added and not filled timestamp by timestamp over time.
When syncing one such document, the cluster seems to be overloaded by the process of translating the Realm objects into Atlas documents. After adding one document to a synced Realm, the primary node starts to insert the data, but restarts after a while. The primary node is reassigned multiple times. During this time I cannot access the “Browse Collections” UI.
The cluster continues to work for hours, but I cannot find the document in the collection. The number of documents and size of the collection in the “Browse Collections” UI grows, but the documents are not shown. They are also not retrievable via the mongodb shell or Compass.
The realm logs don’t show any errors when syncing the data, but afterwards occasionally a MaxIntegrationsAttempsError is logged.
I am using the Free Tier M0 cluster.
Are 1MB documents too big to be added to a synced Realm at once? Should I split the document into smaller buckets or add the bucket first and fill it over time? Am I missing something else?
Thank you in advance!