Migrating the data from offline Realm to Realm Cloud Sync

[Gagik_Kyurkchyan]

We have an application, that up until now was totally offline and was using Realm as a database. We are currently trying to implement a cloud-synced version of the app. Obviously, we need to somehow migrate the offline data for the existing users to the Realm cloud.

In the offline version of the app, all of our models had the Id as a primary key

[PrimaryKey]
public string Id { get; set; } = Guid.NewGuid().ToString();

Because we didn’t have any explicit MapTo attributes, internally, the primary key is stored as Id as well.

Inside Realm sync, there’s a requirement that the primary key should be named as _id internally. Also, we should define a new property for the partition key. So, in the synced version of the Realm, we should have at least the following:

[PrimaryKey]
[MapTo("_id")]
public string Id { get; set; } = Guid.NewGuid().ToString();

[Required]
[MapTo("_partitionKey")]
public string PartitionKey { get; set; }

The approach we’ve decided to take is comprised of 2 main phases

1. First of all, we will need to update the schema of the offline Realm to the point, where the schema would be final and acceptable to be used by the Realm cloud sync. This involves both the necessary changes to migrate somehow the primary key to _id and introduce _partitionKey. Also some changes in general schema improvements and cleanup internal to our company to make the database future proof. This is done to safeguard us as much as possible from schema changes after we’ve migrated fully to Realm sync. At the completion of this phase, we will have a schema, that’s simply ready to be hooked to cloud sync without any further changes. But we won’t physically connect to the Realm sync and users will still be using the offline version of the app.
2. After this, we will implement the Realm cloud sync. On the app launch, we will detect that the users are on the previous version of the app and will suggest them migrate their data to the synced version. We’ve already implemented code that deep copies data from offline Realm to synced Realm that has the same schema. We will simply copy over their existing data to their user-specific partition inside the Realm cloud. Job done.

So, now the questions and the challenges that we are facing:

1. I am facing a problem with phase 1. I can’t find a way of changing the name of the primary key. Is that possible in the C# client? I really don’t want to have 2 separate schemas for the offline and the synced versions. As that would mean duplicating a tone of code to support both schema versions, which essentially are going to the same classes duplicated into 2 different namespaces with slightly different schema.
2. During the initial rounds of hooking Realm sync into the existing schema the app would crash with exceptions like this one

   The program ‘XXX.exe’ has exited with code -1073740791 (0xc0000409).
   

   It’s was extremely hard to understand what was causing those issues. Sometimes, I would have a missing partition key, sometimes, user auth was incorrectly configured. Are there any tools/logs that would help when investigating such issues? Cause the only investigation we can do now is a mere “needle in a haystack” search or guessing.
3. Is the strategy that I have described above a good one? Perhaps, there’s a better approach?

Thanks in advance for the help.

[papafe (Ferdinando Papale)]

Hi @Gagik_Kyurkchyan, thanks for your message.

Regarding your questions:

1- It should be possible to change the primary key during a migration, but at the moment it’s possible to do it only using the dynamic API (this is something we’re going to fix though). You’ll need to have a migration block similar to the following one:

        var config = new RealmConfiguration
        {
            SchemaVersion = 1,
            MigrationCallback = (migration, oldSchemaVersion) =>
            {
                var newPeople = migration.NewRealm.All<Person>();
                var oldPeople = (IQueryable<RealmObject>)migration.OldRealm.DynamicApi.All("Person");

                for (var i = 0; i < newPeople.Count(); i++)
                {
                    var oldPerson = oldPeople.ElementAt(i);
                    var newPerson = newPeople.ElementAt(i);

                    var oldPersonId = oldPerson.DynamicApi.Get<string>("Id");
                    newPerson.DynamicApi.Set("_id", oldPersonId);
                }
            }
        };

2 - I definitely agree that the error message is not helpful in this cases. If it happens again please feel free to create an issue in Github and we can take a look.
3 - The approach you want to use makes sense, even though you could skip the intermediary step and just copy the content of the current offline DB to the sync one, so you won’t need to worry about migrations and problems with primary keys.

[Gagik_Kyurkchyan]

@papafe thanks a lot for the reply! I was thinking about using DynamicApi but I couldn’t find that property on the RealmObject subclasses. Apparently, it was added recently, and after I updated the package to the latest 10.2.* from 10.1.* I had, and used the code you suggested, I was able to do the primary key migration.

I did one small improvement over what you suggested. There’s no need to use oldPerson.DynamicApi.Get<string>("Id") instead because we are already in the “dynamic” realm, we can simply write oldPerson.Id which will refer to the old Id property value.

As for the other points, thanks for sharing your thoughts on those. Now we are one step closer to the cloud realm which is very exciting!

[papafe (Ferdinando Papale)]

I’m glad I was of help!

And yes, you’re correct, in the older versions of Realm the dynamic API was accessed differently. You’re also correct that in this case you don’t need to to cast migration.OldRealm.DynamicApi.All("Person") to (IQueryable<RealmObject>) and use directly the dynamic api. The solution I have sent is slightly more generic because it works also on platforms that use AOT such as Unity with IL2CPP.