How to add DynamicRealmObject?

MongoDB Atlas App Services & Realm Realm
,
1

I setup realm (v10.3.0) in c# (VS) and it worked well with classic static objects (inheriting RealmObject). Now I need to create a dynamic object at runtime and thought about sth. like this :

RealmConfiguration config = new RealmConfiguration(realm_db_name);
config.SchemaVersion = 2;
config.ShouldDeleteIfMigrationNeeded = true;
config.IsDynamic = true;
                
var realm = Realm.GetInstance(config);

realm.Write(() =>
{

                    DynamicRealmObject tmp_obj = realm.DynamicApi.CreateObject("test", "time");

                    for (int min_idx = 0; min_idx < values.Length; min_idx++)
                    {
                        for (int col_idx = 0; col_idx < headers.Length - 1; col_idx++)
                        {
                            tmp_obj.DynamicApi.Set(headers[col_idx], double_values[col_idx,min_idx]);
                        } 

                        realm.Add(tmp_obj);

                    }
});

(headers is a string[] with the header names, double_values is a 2D double array with the values)

However, I get the following error :

The class test is not in the limited set of classes for this realm

I also tried to set a different schema version (from 1 to 2), but nothing changed. If I catch the current schema it still holds the old static class I used previously.

Any ideas?

2

If I understand your use case correctly, then this is not something that is supported by Realm, unfortunately. DynamicRealmObject is intended to be used to access existing properties dynamically, but it cannot create new ones. What you seem to need is an API to create the schema programmatically which is not yet exposed by the .NET SDK.

That being said, since you want to store simple data (i.e. not multi-level), you can probably use a dictionary and a strongly typed object instead:

public class HeaderContainer : RealmObject
{
    public IDictionary<string, double> Headers { get; }
}

// Later...

realm.Write(() =>
{

    var container = realm.Add(new HeaderContainer());

    for (int min_idx = 0; min_idx < values.Length; min_idx++)
    {
        for (int col_idx = 0; col_idx < headers.Length - 1; col_idx++)
        {
            container.Headers[headers[col_idx]] = double_values[col_idx,min_idx];
        } 
    }
});
1 Like
3

Oh thank you that sounds like a pretty good workaround! It work’s!
But unfortunately it has huge performance downsides…to insert 129600 containers I need 85000ms now, with classic static object it was 8000ms. :confused:

4

Hm, it is expected that dictionaries are slower than static objects, but that difference is a bit surprising. I can try and benchmark things, but I’ll need a bit more info - can you share the code that you’re using to insert the containers as well as a rough estimate on how many items there are in each dictionary as well as the size of the string keys.

6

Well yeah…I’m benchmarking too atm …would welcome help from an expert of course :slight_smile:
I just tried async-parallel adding of containers and got ~20000ms (which is a bit surprising, because most times I tried async in other approaches and db’s the overall-time didn’t really improve). Anyways - This is way too high, with SQlite it got ~8000ms to insert these data (with a giant insert into command which was pre-read which is not a good solution…).

I got 71 columns with 90(days)*24(hours)*60(minutes) values, so it’s 129600 containers with 9201600 values overall. The first col is time which is measured in added-seconds, so it’s 60 ; 120; 180 and so on. The others are all doubles, you may create them with random numbers or static values…

I’ve made a code snipped for you

//realm_config

            RealmConfiguration config = new RealmConfiguration(realm_db_name);
            config.SchemaVersion = 3;
            config.ShouldDeleteIfMigrationNeeded = true;
            config.IsDynamic = false;

            bool dynamic_objects_creation = true;
            
            //config.MigrationCallback = ;
            
            var realm = Realm.GetInstance(config);
            //var schema = realm.Schema;

//data_preparation

var headers = new string[71];

for (int i = 0; i < 71; i++)
{
headers[i] = “col”+i;
}

            double[,] double_vals = new double[headers.Length, 129600];
  var timestamp = 0;
  Random rnd = new Random();
            for (int min_idx = 0; min_idx < 129600; min_idx++)
            {
  timestamp += 60;

                for (int col_idx = 0; col_idx < headers.Length -1; col_idx++)
                {
  	if(col_idx == 0) double_vals[col_idx, min_idx] = timestamp;
                    double_vals[col_idx, min_idx] = rnd.NextDouble();
                }
            }

//write data
var stopWatch = new stopWatch();
stopWatch.Start();

realm.Write(() =>
{
if (dynamic_objects_creation)
{
// old approach : not supported by realm yet : DynamicRealmObject tmp_obj = realm.DynamicApi.CreateObject(“pv_plant”, “time”);

                    for (int min_idx = 0; min_idx < values.Length; min_idx++)
                    {
                        var container = realm.Add(new HeaderContainer());

                        for (int col_idx = 0; col_idx < headers.Length - 1; col_idx++)
                        {
                            container.Headers[headers[col_idx]] = double_vals[col_idx, min_idx];

                        }
                    }
                }

});

stopWatch.Stop();
Console.Write(stopWatch.Elapsed);

7

Okay, so I did a little benchmarking and adding the data accounts for about half the time (~14s on my machine) and then committing the transaction and waiting for the write to disk accounts for the other half (~16s). Unfortunately, even if we did something magical about bulk inserting all the data in a way that happens instantaneously, the commit would still take 16s. I haven’t profiled the internals of this yet, but I am fairly convinced it’s just a lot of data that needs to be stored. Unfortunately, we don’t do any optimizations around reusing dictionary keys, so we’ll store 71 identical strings 130k times.

The reason why a statically typed object works a lot faster is that we don’t store the property names and instead only write the double values.

I think our best shot is to expose an API to programmatically construct the schema, but that will only work if you know that these headers are always the same - i.e. they may change from time to time, but they’ll be identical for all objects in the Realm and you can either migrate from the previous schema when the headers change or you can just use ShouldDeleteIfMigrationNeeded as you’re currently doing.

1 Like
8

As a follow-up, I did an incredibly hacky modification using reflection to access some of the internal Realm API and was able to get the times down to 3s for insertion and 4s for the commit, thus bringing the total to just under 7s, roughly in line with the results you were seeing with SQLite.

You can find the project here, but the general idea is that I’m using reflection to build the schema dynamically, as the method along with the facilities to build the schema are internal. I’m creating a new object type with property names equal to the header values we’ve created. Then, I’m using the dynamic API to create the objects and add them to the Realm.

This is obviously super hacky and we’d like to eventually expose a proper API for dynamic schema generation, but should hopefully unblock you. I don’t envision the API being fundamentally different from what is currently internal, so when a new version is released that exposes them, it should be a fairly straightforward migration on your end to stop using reflection.

2 Likes