Running out of memory on initial sync

I have a database set up that uses Realm Sync, which is roughly 256Mb in size. When I try to perform an initial sync, I’m getting crashes on lower spec devices, as they are running out of memory. This is happening mostly on Android during testing, but possibly only because the iOS devices we have to test have more memory available.

Due to historical reasons, I am storing images an videos in blobs in the database itself, these are not huge (in the range of 300kb each), but there are quite a few of them. The client has a requirement that everything must work offline after the initial sync (the app is used in a medical context, and cannot rely on having internet access).

I have the app set up so that it gets a single Realm instance, which it is storing in the application, and then does all access through that instance. The instance is lazy loaded, and is closed and deallocated when the app goes into the background. The data is stored in a menu hierarchy, with 2 levels, and content items of various types attached to the second menu item. These content items are where 95% of the total data are held. While the initial sync is running, I have a download listener attached to the sync session so I can report the progress of the sync.

How can I get around this memory issue with a minimum of rewriting? Am I going to have to bite the bullet and move my content data out of blobs into some kind of storage, or is there a workaround that allows me to process these in smaller chunks?

Thanks!

@Ada_Lovelace : Have you figured out a solution to your problem?

I wonder whether partitioning might be an option here? If a particular view could open a realm for a single partition then it could avoid trying to sync the entire database in one go.

I wrote a script that ran through the database and compressed all of the images. This has brought the reported database size in Atlas/Database/Collections down to 85Mb, which may be small enough to work. However, when performing a sync, it still reports the download size as in excess of 300Mb. Do I need to compact the db? I’ve tried doing so through mongosh, but I keep getting unauthorized errors, even after adding access rights to the admin user for everything.

Btw, this is a new database - I do not care about retaining any history at this point.

Hi @Ada_Lovelace

Yes, you get the point: what’s still around is the history of the previous content of the DB: you can force a fresh start by Terminating and re-Enabling Sync (after a suitable delay - better leave a few minutes for the various processes to wind down before restarting).

On the app side, you should handle the Client Reset, but at this point a simple app re-install would do. Additionally, it’s probably a good idea to implement the shouldCompactAtLaunch callback, to clear up the history on the local DB as well from time to time.

For iOS, there’s an example of both in our repository

Thank you, I’ll try that.

Hello again,

@Andrew_Morgan I’ve implemented the history reset and compact, and the db is now much smaller. I’m still running into issues on low spec Android devices though, so I was hoping you could explain the partitioning idea? I’ve changed the partition for all of my content items, but now when I sync the database, the content array is empty, presumably because the ObjectId is now pointing to an object outside its partition. Would I need to change the ObjectId into a string in the database, and then use that to fetch the objects from my content partition, or is there a way to tell Realm to not try to resolve these dangling ObjectIds?

Many thanks