Sync Issues Copying Object Across Realms - Partition values not copying on deep relations

I have been hitting an issue when copying managed objects to a new Realm. I have a legacy Realm app being converted to MongoDb Realm. I know to use ‘.create()’ for copying to a new Realm. In legacy Realm there were no partition values for Realms, so when a copy was made the nested referenced Objects or List<>s (aka: relations via the ‘foreign_key’) were not a problem. I am finding that unless I update the partition values (deep) the local data will not sync to the cloud. Of course, the data writes fine locally but does not sync to the cloud.

My issue is regarding Objects that are nested with Objects or List<>s - that would be related (by use of the foreign_key relationship). Also, this is for Object type not EmbeddedObjects. Since each related Object or Array of Objects would have a partition key and value. Upon the copy (i.e., the target) such related objects do not have the new partition value, rather it holds the old (source) value.

To solve this, I have created some services and an Extension to Realm’s Object in order to unmanage them from Realm before writing the copy. This seems to work so far, though I have not completed all thorough testing yet. I want to make sure that I am not missing something (i.e., a better way to handle this) or do I need build more for this idea to work -please advise. If I am on the right path, then this code can help others.

I am including the code for my Swift Extension. The code is somewhat re-created for this post (so not exact) and I am not including the related services I built to complete what I need (since it doesn’t matter for the issue). FYI - I have a design to “further partition” (due to the data denormalization) when I need to create duplicate objects across Realms that would cause a primary-key duplication issue (within same Collection) for MongoDB.


/// NOTE: Code shown is not actual code, rather recreated for this post.
	
//
//  Object+Ext(DeepUnManageRealm).swift
//  SAMPLE CODE
//
//  Created by Reveel on 5/12/22.
//  Copyright © 2022 Reveel® All rights reserved.
//


import Foundation
import RealmSwift


/// A Protocol for Unmanaging Realm Objects (Deep)
protocol UnmanageRealmObject: AnyObject {
 func unmanageDeep(_ newPartitionValue: String?, _ furtherPartitioning: Bool) -> Self
// END
// END
} // END of Protocol for 'UnmanageRealmObject'

// NOTE: Extension for Unmanaging Realm Objects (Deep) - For 'Object'
extension Object: UnmanageRealmObject {
 
	/// This method will unmanage a Realm object deep and update partitioning value WITHOUT forcing to further partitioning.
	/// - Parameters:
	/// 	- newPartitionValue: pass 'nil' to unmanage deep without changing partition value.
	internal func unmanageAndUpdatePartitionsDeep(_ newPartitionValue: String?) -> Self {
		if let haveSourcePartition = self.realm?.configuration.syncConfiguration?.partitionValue {
			if haveSourcePartition.stringValue != newPartitionValue {
				return unmanageDeep(newPartitionValue, false)
			}
			else {
				return unmanageDeep(haveSourcePartition.stringValue, false)
			}
		}
		else {
			fatalError("During DEV - crashing bc NO Realm or Partition on source object. Check method: '\(#function)'", file: #file, line: #line)
		}
	} // End of 'unmanageAndUpdatePartitionsDeep' method

	/// This method will unmanage a Realm object deep and update partitioning value -AND- will force apply Further Partition practice.
	/// - Parameters:
	/// 	- newPartitionValue: pass 'nil' to unmanage deep without changing partition value.
	internal func unmanageAndUpdateFurtherPartitioningDeep(_ newPartitionValue: String?) -> Self {
		
		// Custom Code for needs to further partition and do it auto/conditionally
		
		return unmanageDeep(newPartitionValue, true)
	} // End of 'unmanageAndUpdateFurtherPartitioningDeep' method


 
	/// This method will unmanage a Realm object deep
	/// - Parameters:
	/// 	- newPartitionValue: pass 'nil' to unmanage deep without changing partition value.
	/// 	- furtherPartitioning: pass 'true' to force apply Further Partition practice.
	internal func unmanageDeep(_ newPartitionValue: String?, _ furtherPartitioning: Bool = false) -> Self {
	 
		let unmanaged = type(of: self).init()
		let partitioningKey = "partitioningKey"
		let furtherPartitionPropertyName = "YOUR-NAME"

		for property in objectSchema.properties {
		 
		 guard var propertyValue = value(forKey: property.name) else { continue; }
			
			var processedFurtherPartitioning: Bool = false
			
			if property.isArray {
				// NOTE: For Realm's List<>.
				let doUnmanage = propertyValue as? UnmanageRealmObject
				unmanaged.setValue(doUnmanage?.unmanageDeep(newPartitionValue, furtherPartitioning), forKey: property.name)
			}
			else if property.isMap || property.isSet {
				// NOTE: For Realm's Map (aka: Dictionary) -OR- Set.
				unmanaged.setValue(propertyValue, forKey: property.name)
			}
			else if property.type == .object {
				 // TODO: Test if this handles EmbeddedObjects by Reference (i.e. 'related' by 'foreign_key') when not fully embedded.
				 // NOTE: For Realm's Object and assuming it handles EmbeddedObject (when such is a reference via foreign_key in Schema).
				let doUnmanage = propertyValue as? UnmanageRealmObject
				unmanaged.setValue(doUnmanage?.unmanageDeep(newPartitionValue, furtherPartitioning), forKey: property.name)
			}
			else {
			 
				 if property.name == partitioningKey, let haveNewPartition = newPartitionValue, partitioningKey != haveNewPartition {
					 let oldPartition = propertyValue
					 propertyValue = haveNewPartition

					 processedFurtherPartitioning = RealmPartitionServices.shared.checkToApplyFurtherPartition(property: property)
				 } // End of IF-"have New Partition Value"

				 
				 if furtherPartitioning, property.name == furtherPartitionPropertyName {
					 let oldValue = propertyValue
					 
					 if oldValue is String {
						 if !processedFurtherPartitioning {
							 propertyValue = RealmPartitionServices.shared.createFurtherPartitioning(value: propertyValue as! String)
						 }
					 }
					 else {
						 // means non-standard type for further-partitioning for Reveel, which for DEV we crash but for PROD we Error-Log
						 fatalError("During DEV - crashing bc UN-KNOWN OLD-Type for Further-Partitioning. Check method: '\(#function)'", file: #file, line: #line)
					 }
				 } // End of IF-"Update for Further Partitioning"
			 
				unmanaged.setValue(propertyValue, forKey: property.name)
			}
		} // End of for-in-LOOP

		return unmanaged
 } // End of internal-method 'unmanageDeep', For: overrided method
	
// END
// END
} // END of 'Object' EXTENSION', For: 'UnmanageRealmObject'


// NOTE: Extension for Unmanaging Realm Objects (Deep) - For 'List'
extension List: UnmanageRealmObject {
	
	/// This method will unmanage a Realm object deep
	/// - Parameters:
	/// 	- newPartitionValue: pass 'nil' to unmanage deep without changing partition value.
	/// 	- furtherPartitioning: pass 'true' to apply Further Partition practice.
	internal func unmanageDeep(_ newPartitionValue: String?, _ furtherPartitioning: Bool = false) -> List<Element> {
	 
		let unmanaged = List<Element>()
		forEach {
			if let haveUnmanage = $0 as? UnmanageRealmObject {
				let unmanage = haveUnmanage.unmanageDeep(newPartitionValue, furtherPartitioning) as! Element
			 unmanaged.append(unmanage)
		 }
		 else {
			 // NOTE: Handled for when NOT an 'Element' type since Realm passes primtives as values
			 unmanaged.append($0)
		 }
	 } // End of For-EACH

		return unmanaged
	} // End of internal-method 'unmanageDeep'


	internal func getUnmanagedArray(_ newPartitionValue: String?, _ furtherPartitioning: Bool = false) -> [Element] {
	 
		return Array(self.unmanageDeep(newPartitionValue, furtherPartitioning))
	} // End of internal-method 'getUnmanagedArray'

	internal func getUnmanagedList(_ newPartitionValue: String?, _ furtherPartitioning: Bool = false) -> List<Element> {
	 
		return self.unmanageDeep(newPartitionValue, furtherPartitioning)
	} // End of internal-method 'getUnmanagedList'
 
// END
// END
} // END of 'List' EXTENSION', For: 'unmanageDeep', 'getUnmanagedArray' and 'getUnmanagedList'


// NOTE: Extension for Unmanaging Realm Objects (Deep) - For 'AnyRealmCollection'
extension AnyRealmCollection {
 
	internal func getUnmanagedList(_ newPartitionValue: String?, _ furtherPartitioning: Bool = false) -> List<Element> {
	 
		let unmanaged = List<Element>()
	 
		forEach {
			unmanaged.append($0)
		} // End of For-EACH

		return unmanaged.unmanageDeep(newPartitionValue, furtherPartitioning)
	} // End of internal-method 'getUnmanagedList'

	internal func getUnmanagedArray(_ newPartitionValue: String?, _ furtherPartitioning: Bool = false) -> [Element] {
	 
		return Array(self.getUnmanagedList(newPartitionValue, furtherPartitioning))
	} // End of internal-method 'getUnmanagedArray'

	internal func getUnmanagedAnyRealmCollection(_ newPartitionValue: String?, _ furtherPartitioning: Bool = false) -> AnyRealmCollection<Element> {
	 
		return AnyRealmCollection(self.getUnmanagedList(newPartitionValue, furtherPartitioning))
	} // End of internal-method 'getUnmanagedAnyRealmCollection'

// END
// END
} // END of 'AnyRealmCollection' EXTENSION' for 'getUnmanagedList', 'getUnmanagedArray' and 'getUnmanagedAnyRealmCollection'


// NOTE: Extension for Unmanaging Realm Objects (Deep) - For 'Result'
extension Results {
 
	internal func getUnmanagedList(_ newPartitionValue: String?, _ furtherPartitioning: Bool = false) -> List<Element> {
	 
		let unmanaged = List<Element>()
		forEach {
			unmanaged.append($0)
		} // End of For-EACH

		return unmanaged.unmanageDeep(newPartitionValue, furtherPartitioning)
	} // End of internal-method 'getUnmanagedList'

	internal func getUnmanagedArray(_ newPartitionValue: String?, _ furtherPartitioning: Bool = false) -> [Element] {
	 
		return Array(self.getUnmanagedList(newPartitionValue, furtherPartitioning))
	} // End of internal-method 'getUnmanagedArray'

	internal func getUnmanagedAnyRealmCollection(_ newPartitionValue: String?, _ furtherPartitioning: Bool = false) -> AnyRealmCollection<Element> {
	 
		return AnyRealmCollection(self.getUnmanagedList(newPartitionValue, furtherPartitioning))
	} // End of internal-method 'getUnmanagedAnyRealmCollection'

// END
// END
} // END of 'Results' EXTENSION', For: 'getUnmanagedList', 'getUnmanagedArray' and 'getUnmanagedAnyRealmCollection'

I kinda get the question but some of it is unclear.

Those aren’t really nested. Lists (in the case) are references to other managed objects - all of those objects will need to have objectId properties

@Persisted(primaryKey: true) var _id: ObjectId

such related objects do not have the new partition value, rather it holds the old (source) value

If the object doesn’t have the new partition value, it won’t be part of that partition. Can you clarify? Do you want it to be in that partition or perhaps you want those objects in their own discreet partition?

From your code:

// TODO: Test if this handles EmbeddedObjects by Reference (i.e. ‘related’ by ‘foreign_key’) when not fully embedded.

An embedded object is either embedded or it doesn’t exist; there are no other options there. Also, embedded objects can’t be referenced per se - they only exist within their parent object, so you access them through their parent object with dot notation

let someEmbddedObjectProperty = parentObject.embeddedObject.property

As you know, an object can only be managed by one Realm - so making an unmanaged copy of an object is one path you can go down but a copy of an object will still be that object.

It sounds more like you want to migrate your objects from an old-style object to a new MongoDB Realm Managed object. Perhaps that’s the point to your code?

Since there is no migration in Sync, what we did was pass the old object (via init) to a different new synced object and populate it based on the old objects properties.

class MyOldSchoolPersonClass: Object {
   @Persisted var name = ""
}

the old object above is passed to the new managed object

class MyCoolSyncedPersonClass: Object {
   @Persisted(primaryKey: true) var _id: ObjectId
   @Persisted var _partitionKey = "" //showing this for clarity
   @Persisted var name = ""

   convenience init(oldPerson: MyCoolSyncedPersonClass, partition: String) {
      self.init()
      self._partitionKey = partition
      self.name = oldPerson.name
}

No idea if any of that will help but I think we need more clarity on what the issue is (and that’s a lot of code in your question for us to parse through - does it work for your use case?). It might be good to add your models to the question so we understand what you’re looking at as well. (if you add them, please keep them small and only include relevant properties)

Hi there Jay - thanks for offering to help!

Let me start off by clarifying some things and then I will answer some of the questions you posed.

Clarifications:
• This is not for a data migration - rather normal usage by users.
• I had mentioned these nested are through relations (aka references) not EmbeddedObjects, so yes related, which is through the primary-key (foreign_key). Additionally as an FYI, those related Objects & Array of Objects can have any allowed type of primary-key and does not have to only be an ObjectId.
• Data needs to be denormalized for this use case in-hand. This is because it is not possible to use Sync Permissions at an object/document level via partition-based. Additionally, this case will not work with current (or even coming soon) version of Flexible-Sync either.
• The issue is that Realm (on MongoDB) does not offer a way (that I’m aware of) to copy an Object to a new Realm that is managed by another Realm - when there are nested relations to Objects or Array of Objects. Since using ‘.create()’ only updates the top level and not deep. That’s the problem.
The goal is to be able copy across Realms and update the partition values for all the relationships (deep), so that it will sync to the cloud properly. Therefore, I am asking to know if there is an existing process (that I don’t know about) -OR- is this process (i.e, my Extension) to unmanage first in-order to update partition values and then copy to new Realm the only way for this to work? -This was not an issue in legacy Realm, hence thought it may already be solved [:crossed_fingers:].

Responses:

My “TODO” in supplied code

  1. This was added for testing how a referenced relationship from within an EmbeddedObject is handled (as per this link). Even though it was a personal note, I should have written it better. I do not currently have this case but would test later since it is a possibility.

If the object doesn’t have the new partition value, it won’t be part of that partition. Can you clarify?

  1. I think the above clarification I provided answers this. Basically, for a copy to work, the partition-values need to be updated deep too for the (new) target Realm’s partition - not some other partition. In any case the partition values need to be updated deep and not just at the top-level.

RE code posted: does it work for your use case?

  1. Yes it works as planned, though I have not tested all possibilities, rather so far most of my use case issues.

As you know, an object can only be managed by one Realm - so making an unmanaged copy of an object is one path you can go down but a copy of an object will still be that object.`

  1. That is not precise. Yes a copy could just be a copy. Though for this purpose Realm docs say to use ‘.create()’ to copy across Realms, but that is not updating related partition values. Since I could not find another process I decided to unmanage the object in question, in-order to update the partition values deep. Mind you that updating within a write transaction is not ideal for this. Hence I wanted to do it outside of a write block (since writes are blocking transactions), that is why I chose to unmanage first.

… so I am hoping you or anyone (e.g., Realm team) can please advise (quickly) on what to do - Thank you in advance!

That added a lot of clarity - a question remains though because we don’t know what the objects look.

For simplicity, I will refer to Person and Dog where the Person has some kind of List relationship to Dogs

does that mean you are creating “manual” relationships via a primary key (ObjectId, String, etc)?

class Person: Object {
   var dogList = List<String> //you store the dogs primary key here?
}

instead of

class Person: Object {
  var dogList = List<Dog> //or do you store the Dog object here?
}

I ask as it can impact how the data is copied.

If it’s the prior, the manual relationship maintains by just copying the Person object to a new person object. e.g. no deep copy needed.

Whereas if it’s the latter, using .create is not a deep copy so that has be addressed separately.

The other question is; for simplicity, are there existing “dog” objects that have no relationship back to a “person”? Or are all the dogs spoken for (e.g. no stray dogs; lol)

Hello Jay,

To you use your example it would be the second one with ‘List<Dog>’, since ‘String’ is not Object type. I have been pretty specific on my posts that these are the type Object (cap on the ‘O’ for Realm type), or an Array of Objects.
– Of course the models can be deeper - e.g., Dog could have an Object of ‘Breed’ within it - etc., etc.

does that mean you are creating “manual” relationships via a primary key (ObjectId, String, etc)?

– All Objects must have a primary-key and a partition value (if using partitions). The primary-key is the relationship for the references that are nested (Object or List). There is no “manual” relationship, these relationships are setup in the Schema. Basically whenever you are not using an EmbeddedObject this is how you have relationships for nested Objects or Array of Objects.

Before we get too lost on tangents, I’d like to stay on focus here on the issue for a solution. Since ‘.create()’ will NOT update partition values deep on related Objects…
• is there another Realm supported way to handle copying across Realms on MongoDB?
• am I missing something?
• shall I proceed as I first posted?

Thank you

Thanks for the clarification. I was asking due to this

because this

var dogList = List<Dog>

is not a foreign key relationship, it’s an object relationship.

Hopefully someone will be able to provide an answer.

Jay - Yeah let’s see if someone from Realm Team can chime in!

I’d like to correct your recent comment. From an object view point it is a “To-Many” type relationship and which for the Schema (as mentioned) it is through the the ‘foreign_key’:

var dogList = List<Dog>
is not a foreign key relationship, it’s an object relationship.

– in that example you provided ‘List<Dog>’ is a relationship in the Schema. It is for an Array of Objects (aka: “To-Many”) as you can see more on this Realm docs link. For an Object (aka: “To-One”) you can see this link. Both are setup in Realm’s Schema where the primary-key for an Object (i.e., ‘_id’) makes the relationship connection; this is done via the ‘foreign_key’ which makes that connection for such relations. Hope that helps.

As you know Realm objects are not required to have keys in all cases. e.g. local only realm objects do not need keys to have forward or even inverse relationships. Therefore that relationship is to the object, not the key (it’s an object database after all)

We have no idea how your objects were structured before and after since they were not included in the question: Since you’re going from local to sync, the questions I presented are trying to get details on what those deep objects look like (and how the relationship was/is set up) instead of trying to decipher it from the code.

I will close with this from the Realm Swift documentation which is what prompted the question about how the objects relationships were/are implemented:

A relationship is when an object property references another object

Is there foreign_key in your actual Realm Swift object or is that the primary_key property? Perhaps including your models would clarify it for someone on the Realm Team.

@Jay - This whole post is about syncing; please look at the links I provided in my last post (“To-Many” & “To-One”). The ‘foreign_key’ is part of the Schema and is used with the primary-key to connect to the related Objects. Therefore, the models by way of an SDK (i.e., iOS in this case) would not have a property of ‘foreign_key’, rather depend on ‘_id’ (aka: primary-key). This is needed for Realm to work with MongoDB and sync.

Jay let’s please leave it here - as these posts will make it hard for someone to follow what is going here since this is all off topic. Thank you for trying though - I do appreciate that.