Schema Changes - Additive

Raymond_Brack · October 13, 2020, 5:48am

I currently have a Realm class as follows;

public class JobResponse : RealmObject
{
	[Required]
	[PrimaryKey]
	[MapTo("id")]
	public string Id { get; set; }
	[Required]
	[MapTo("jobId")]
	public string JobId { get; set; }
	[Required]
	[MapTo("runId")]
	public string RunId { get; set; }
	[Required]
	[MapTo("driverId")]
	public string DriverId { get; set; }
	[MapTo("sequence")]
	public int Sequence { get; set; }
	[MapTo("completed")]
	public int Completed { get; set; }
	[MapTo("reasonId")]
	public int? ReasonId { get; set; }
	[MapTo("reason")]
	public string Reason { get; set; }
	[MapTo("driverNote")]
	public string DriverNote { get; set; }
	[MapTo("clientSignature")]
	public string ClientSignature { get; set; }
	[MapTo("clientSignatureImage")]
	public byte[] ClientSignatureImage { get; set; }
	[MapTo("driverSignature")]
	public string DriverSignature { get; set; }
	[MapTo("driverSignatureImage")]
	public byte[] DriverSignatureImage { get; set; }
	[MapTo("signedBy")]
	public string SignedBy { get; set; }
	[MapTo("latitude")]
	public float Latitude { get; set; }
	[MapTo("longitude")]
	public float Longitude { get; set; }
	[MapTo("image")]
	public byte[] Image { get; set; }
	[MapTo("responseDtTm")]
	public DateTimeOffset ResponseDtTm { get; set; }
	[MapTo("processedDtTm")]
	public DateTimeOffset? ProcessedDtTm { get; set; }
}

The ClientSignature and DriverSignature properties were used early in the project to store a JSON string of points from the signature pad but these were later replaced by the ClientSignatureImage and DriverSignatureImage properties.

Recently we added two nullable fileds, JobStart and JobFinish, and decided to remove the the original ClientSignature and DriverSignature fields as follows;

public class JobResponse : RealmObject
{
	[Required]
	[PrimaryKey]
	[MapTo("id")]
	public string Id { get; set; }
	[Required]
	[MapTo("jobId")]
	public string JobId { get; set; }
	[Required]
	[MapTo("runId")]
	public string RunId { get; set; }
	[Required]
	[MapTo("driverId")]
	public string DriverId { get; set; }
	[MapTo("sequence")]
	public int Sequence { get; set; }
	[MapTo("completed")]
	public int Completed { get; set; }
	[MapTo("reasonId")]
	public int? ReasonId { get; set; }
	[MapTo("reason")]
	public string Reason { get; set; }
	[MapTo("driverNote")]
	public string DriverNote { get; set; }
	[MapTo("clientSignatureImage")]
	public byte[] ClientSignature { get; set; }
	[MapTo("driverSignatureImage")]
	public byte[] DriverSignature { get; set; }
	[MapTo("signedBy")]
	public string SignedBy { get; set; }
	[MapTo("latitude")]
	public float Latitude { get; set; }
	[MapTo("longitude")]
	public float Longitude { get; set; }
	[MapTo("image")]
	public byte[] Image { get; set; }
	[MapTo("responseDtTm")]
	public DateTimeOffset ResponseDtTm { get; set; }
	[MapTo("processedDtTm")]
	public DateTimeOffset? ProcessedDtTm { get; set; }
	[MapTo("jobStart")]
	public DateTimeOffset? JobStart { get; set; }
	[MapTo("jobFinish")]
	public DateTimeOffset? JobFinish { get; set; }
}

From what I understand from the documentation this is an additive change however the application now throws the following exception;

Realms.Exceptions.RealmMigrationNeededException: Migration is required due to the following errors: - Property ‘JobResponse.clientSignature’ has been removed. - Property ‘JobResponse.driverSignature’ has been removed. - Property ‘JobResponse.jobStart’ has been added. - Property ‘JobResponse.jobFinish’ has been added.

Since this is using a Full Sync Configuration I can’t use a Migration callback to update the model and there would be a large amount of work to code the changes required to update the Xamarin and Server code to migrate the users to new Realm files as not all users will be updated at the same time.

Can someone explain why this is not an additive change and what I could possibly do change the model so it is.

nirinchev · October 13, 2020, 1:46pm

You’re right - this is an additive change and the exception should not be thrown. My only guess would be that you’re accidentally opening a Realm without a FullSyncConfiguration. Can you post the section of the code where the exception is thrown along with the version of the SDK you’re using?

Raymond_Brack · October 13, 2020, 10:13pm

Following is the code to open the Realm;

private readonly string _realmName = AppConfig.RealmName;

private Realm OpenRealm()
{
	Realm realm = null;
	...
	_realmFile = $"~/{_realmName}";
	...
	_config = ConnectionServices.GetRealmConfiguration(_realmFile, _user);
	...
	realm = ConnectionServices.ConnectToSyncServer(_config);
	return realm;
}

public static FullSyncConfiguration GetRealmConfiguration(string realmName, User user)
{
	FullSyncConfiguration config;
	var serverUrl = new Uri(realmName, UriKind.Relative);
	config = new FullSyncConfiguration(serverUrl, user)
	{
		SchemaVersion = 1,
		ObjectClasses = new[] { typeof(Driver), typeof(Run), typeof(Job), typeof(Service), typeof(JobResponse), typeof(ServiceResponse), typeof(Option), typeof(AppLog) }
	};
	return config;
}

public static Realm ConnectToSyncServer(FullSyncConfiguration config)
{
	Realm realm;
	...
	realm = Realm.GetInstance(config);
	...
	return realm;
}

The schema version has not been changed in the sync config nor have any of the object classes other than the JobResponse class.

I recently upgraded Realm to v5.1.1. I also forgot to mention the same code is used by two services used to process the data from Realm to the mobile devices and to process the data from the devices. Neither of these services threw the error.

This is the stack trace from App Center;

NativeException.ThrowIfNecessary (System.Func`2[T,TResult] overrider)
SharedRealmHandle.Open (Realms.Native.Configuration configuration, Realms.Schema.RealmSchema schema, System.Byte[] encryptionKey)
RealmConfiguration.CreateRealm (Realms.Schema.RealmSchema schema)
Realm.GetInstance (Realms.RealmConfigurationBase config, Realms.Schema.RealmSchema schema)
Realm.GetInstance (Realms.RealmConfigurationBase config)
ConnectionServices.ConnectToSyncServer (Realms.Sync.FullSyncConfiguration config)

ConnectionServices.ConnectToSyncServer (Realms.Sync.FullSyncConfiguration config)
AppConfig.LogEntry (Realms.Sync.FullSyncConfiguration config, RealmSx.Shared.Enums+LoggingLevels loggingLevel, RealmSx.Shared.Enums+LoggingLevels logLevel, System.String logEntry, System.String stackTrace)
SrxRunSheet.Views.ServicePage..ctor (Realms.Sync.FullSyncConfiguration config, RealmSx.Shared.Enums+LoggingLevels loggingLevel, SrxRunSheet.ViewModels.ServiceVm service) [0x00027] in <9a65769ae7ba4c0fbecda6e40a12d3c1>:0
JobPage.OnServiceSelected (System.Object sender, Xamarin.Forms.ItemTappedEventArgs e)
AsyncMethodBuilderCore+<>c.<ThrowAsync>b__7_0 (System.Object state)
SyncContext+<>c__DisplayClass2_0.<Post>b__0 ()
Thread+RunnableImplementor.Run ()
IRunnableInvoker.n_Run (System.IntPtr jnienv, System.IntPtr native__this)
(wrapper dynamic-method) Android.Runtime.DynamicMethodNameCounter.50(intptr,intptr)

nirinchev · October 13, 2020, 10:49pm

That’s suuuper interesting. FullSyncConfiguration will call its base CreateRealm method:

github.com

realm/realm-dotnet/blob/8d38bc61cef166130e27ea1a4920184101c589e4/Realm/Realm/Configurations/SyncConfigurationBase.cs#L219


      
          /// </remarks>
          public static void EnableSessionMultiplexing()
          {
              SharedRealmHandleExtensions.EnableSessionMultiplexing();
          }
          
          
internal override Realm CreateRealm(RealmSchema schema)
          {
              var configuration = CreateConfiguration();
          
          
    var srHandle = SharedRealmHandleExtensions.OpenWithSync(configuration, ToNative(), schema, EncryptionKey);
              if (IsDynamic && !schema.Any())
              {
                  srHandle.GetSchema(nativeSchema => schema = RealmSchema.CreateFromObjectStoreSchema(nativeSchema));
              }
          
          
    return new Realm(srHandle, this, schema);
          }
          
          
[SuppressMessage("Reliability", "CA2000:Dispose objects before losing scope", Justification = "The Realm instance will own its handle")]
          internal override async Task<Realm> CreateRealmAsync(RealmSchema schema, CancellationToken cancellationToken)

As you can see, this one calls SharedRealmHandleExtensions.OpenWithSync. Instead, your stacktrace points to RealmConfiguration.CreateRealm:

github.com

realm/realm-dotnet/blob/8d38bc61cef166130e27ea1a4920184101c589e4/Realm/Realm/Configurations/RealmConfiguration.cs#L149


      
          if (ShouldCompactOnLaunch != null)
          {
              var handle = GCHandle.Alloc(ShouldCompactOnLaunch);
              configuration.should_compact_callback = ShouldCompactOnLaunchCallback;
              configuration.managed_should_compact_delegate = GCHandle.ToIntPtr(handle);
          }
          
          
var srPtr = IntPtr.Zero;
          try
          {
              srPtr = SharedRealmHandle.Open(configuration, schema, EncryptionKey);
          }
          catch (ManagedExceptionDuringMigrationException)
          {
              throw new AggregateException("Exception occurred in a Realm migration callback. See inner exception for more details.", migration?.MigrationException);
          }
          
          
var srHandle = new SharedRealmHandle(srPtr);
          if (IsDynamic && !schema.Any())
          {
              srHandle.GetSchema(nativeSchema => schema = RealmSchema.CreateFromObjectStoreSchema(nativeSchema));

RealmConfiguration is the class for opening a local Realm, which implies that somewhere in ConnectionServices.ConnectToSyncServer you’re either doing Realm.GetInstance() - without passing in the config parameter - or explicitly getting a RealmConfiguration instance and passing it to Realm.GetInstance.

Note that this may also be a subtle bug where ConnectToSyncServer is called with config: null. My guess would be that JobPage.OnServiceSelected is invoking an event that constructs ServicePage with null config, which then flows all the way to Realm.GetInstance.

Raymond_Brack · October 14, 2020, 4:25am

Your prognosis was spot on. I had moved a line of code in the page constructor above the point where the configuration was instantiated. Have placed a guard statement in my code to ensure this doesn’t happen again.

From Realm’s perspective would it make sense to raise an exception if ConnectToSyncServer is called with config: null? The current exception sends you looking in totally the wrong direction

nirinchev · October 14, 2020, 9:44am

ConnectToSyncServer is your method, so we can’t raise an exception there The first method we could throw one in is Realm.GetInstance. Unfortunately, due to the way we chose to implement it - with optional arguments instead of overloads, we can’t distinguish between Realm.GetInstance() and Realm.GetInstance(null) - those are identical from the compiler perspective, so we assume you were trying to open the default local Realm and go for it.