Partition Strategy improvements needed

MongoDB Atlas App Services & Realm Atlas Device Sync
,
1

Hey there,

Currently I am struggeling with my partitioning strategy and maybe you can give me a hint how to improve it.

Basically my app is meant for bands to plan their gigs. So you as a User can log in and you are assigned to one or more Bands. Every Band also can have one or more Events. To simplify things I have made a litte schema to visualize my data-structure.

3240×2186 413 KB

As you can see, I use a partition-key for my current user and this user has a list of bands, where I can append and remove them. (This pattern is from the documentation in the Relations: Many-To-Many section).

From the bands Object I can take the bandID and pass it as a partition-key for all of the events each band has. This does somehow conform to the way team realms should be partitioned (based on the documentation).

  • The big advantage is, that I only need the partition key for having all events loaded at once, for every band and member.
  • But the disadvantage is, that I only can display events from one band at once, due to the limitation of only be able to use one partition key (or bandID) at once.

But I somehow need that behaviour in the app. When you are in for example two bands at once, the overview should show all events they are assigned to.

Things I tried/thought of so far to solve this:
Trying to use multiple partition-keys at once — I think this is not a very elegant solution for the problem, but I tried, haha

Appending the Events also to the Band as a List — this was the most valueable solution up until now, but I think there will be syncinc issues bc the events no longer will be a top-level object. Instead they’ll become a children of band and grandchildren of user.

Somehow use Realm Triggers — but there I have no clue in wich way they could help me, I just have seen them in different tutorials and articles like RChat.

2

Let me first compliment you on a clear question and the graphic was very helpful. Good job.

The only thing is needed is a bit more about the actual structure of the partition keys you’re using - e.g. is the Band object literally using a single unique string as it’s paritition key?

class Bank: Object {
    @objc dynamic var id = ...
    @objc dynamic var _partitionKey = "" // is this "band_0" and "band_1"?
}

Onto to the question - while partitioning helps to organize a large amount data into smaller ‘chunks’ for each individual client - but it should not be used as a live filter for dynamic data, which seems like that’s kinda what you’re doing here and why you’re running into that limitation.

Can you give us a bit more info about how your partition keys are currently structured?

2 Likes
3

Yes — you are right. The band collection is accessed with a static and public partition-key. Every Band has a bandID which is a UUID as String This is then used to:

  • Append the Band to the user’s band-list.
  • Recieve the corresponding Event data, because every event also has the partition-key from the band, so that when you are in a band every event gets loaded.

I also updated the graphic to get you a deeper context of my current approach.

2696×2236 429 KB

If the first visual explanation is not sufficient enough, here is another one which is displaying the data structure I have at the moment.

2696×2006 370 KB

Due to my usage of my partition-key I am limited to only get data from one band at once, like shown here. GIF

Possible approach to this problem:

2782×2226 425 KB

I would like to append the events to the bands for each and every user. After that the user only will need their UserID as a partition-key. This seems to be straight forward. But — if User#1 is editing Event dsj34 and commits it’s changes to the database, with his unique partition-key, will User#2 recieve the changes?

4

Just thinking through this, it seems that perhaps the way the partition key is being used is the issue


A partitionKey can be used to limit the amount of data downloaded to the client and can also be used as a type of security to isolate data but using it as a filter for, in this case, dynamic data may be the issue.

Here’s an example;

Suppose we have users (that have bands) and each user has it’s own partition key (akin to each band having it’s own partition key).

The app will need to share data amongst users but that sharing is undefined and may change over time (dynamic).

class UserClass: Object {
     @objc dynamic var partitionKey  = ""
}

Then we have users, user_0 and user_1 and user_2 etc and thats the partition key for each.

The first issue is that if we have 10,000 users, you would end up with 10,000 files on your drive (each partition, a Realm, has a corresponding file on your drive). While that may be ok, it’s not very efficient - it would be way better to have one Realm with all of the users.

The second issue is sharing of data - remember that the users need to know about each other; user_0 may have a band that also has user_1 in it for example.

Lastly all users need to dynamically have access to all bands. So this loops back to the first case whereas every band has a unique partitionKey, so if there are 10,000 bands, you have 10,000 files on your drive.

The followup issue is the code to support that. For each Realm you’re app is accessing, it need to ‘open a connection’ so 10,000 users and 10,000 bands; ugh, what a coding nightmare!

So drawing your attention to Partition Data Into Realms, look at the first pic shown, circles, stars and triangles and say your users are circles, and the bands are stars. You can then see there are blue and red realms; but that’s not what you have - your data is more dynamic. Sometimes the star (the band) may be blue, but other times it bay be red, other times it could be both.

My knee-jerk reaction is the abandon the current partition strategy and just simplify it, users have a users partition, bands a have a bands partition etc. Then, thinking through it, your data is so dynamics, why isolate the partitions at all? Just use a generic partition _partitionKey = “My_Band_App” and have user objects and band objects all within one tidy realm. 10,000 users and 10,000 bands would easily be handled by a single Realm.

Need to think through this a bit as there may be an option for creating ‘teams’ but need to explore that.

1 Like
5

Kudos for your deep investigation into this, really appreciate it and it helps a lot!

Just use a generic partition _partitionKey = “My_Band_App” and have user objects and band objects all within one tidy realm. 10,000 users and 10,000 bands would easily be handled by a single Realm.

I think this is the thing that is surprising to me right now — that this approach is somehow feasible. Maybe I misinterpreted the whole concept of partition-keys and overestimated the performance usage I will get, if I just have single large collections for bands and users.

But yeah, I will check it out and try a cleaner approach and will provide feedback here, once I’m ready.