Preserve lazy-loading reactivity of realm collections after transforming them to another object

We have a pretty complex application that uses MongoDB App Services and Realm.

Realm database has a fantastic feature of lazy-loading data when the data is actually accessed. This has pretty exciting effects. Imagine you have a screen that shows a list of items. Those items can be tens of thousands. In the traditional database or API approach, one would implement some sort of data pagination - either automatic, when the user reaches the end of the list, the next chunk of data is loaded, or manual - the user manually selects the page that they should load. With realm, you can simply fetch all of the items of an entity and directly bind those items to a ListView control in Xamarin. For example

public class Product : RealmObject
{
    public string Name { get; set; }
    public decimal Price { get; set; }
}

public class ProductsViewModel
{
    private readonly Realm _realm;

    public IEnumerable<Product> Products { get; set; }
    public ProductsViewModel(Realm realm)
    {
        _realm = realm;
    }

    public void LoadProducts()
    {
        Products = _realm.All<Product>();
    }
}

//The xaml page
<ListView
    ItemsSource="{Binding Products}"
    CachingStrategy="RecycleElement">
    <ListView.ItemTemplate>
        <DataTemplate>
            <TextCell Text="{Binding Name}"/>    
        </DataTemplate>
    </ListView.ItemTemplate>
</ListView>

In the above example, even if there are tens of thousands of products, the ListView will load instantly and will scroll smoothly because of the lazy-loading magic.

The issue arises when we need to transform the native realm object Product into a different class. Imagine, we want to create a new view model out of the Product class, called ProductViewModel :

public class ProductViewModel
{
    public ProductViewModel(Product product)
    {
        Name = product.Name;
        FormattedPrice = $"{product.Price:C}";
    }
    public string Name { get; set; }
    public string FormattedPrice { get; set; }
}

This is a trivial example where we simply format the currency into a string. In our real-world application, the scenario is much more complicated, but it’s good to show the issue we are facing. Once would guess that changing the ProductsViewModel to something like this would do the job:

public class ProductsViewModel
{
    private readonly Realms.Realm _realm;

    public IEnumerable<ProductViewModel> Products { get; set; }
    public ProductsViewModel(Realms.Realm realm)
    {
        _realm = realm;
    }

    public void LoadProducts()
    {
        Products = _realm.All<Product>().Select(p => new ProductViewModel(p));
    }
}

However, this approach is naive and will essentially break the lazy-loading aspect of the list. What it will do is iterate over every item in the list; for every item, it will fetch the name and the price of the product, will create a new ProductViewModel, and only after all of these operations are completed, the list will show any data. If the product collection is extensive, this will take a significant amount of time to complete.

How can we approach solving this problem the right way? Essentially, we need some reactive collection transform operator which does the following.

  1. It preserves the reactive notifications of the original collection. If the original Realm collection changes, the transformed collection should also omit collection change events
  2. Items of the collection should be evaluated lazily whenever the list shows

If we do not find a solution, we will have to add artificial pagination so that the transformed objects are created only for a subset of items, which won’t take a lot of time. But this is a super undesired approach, as it breaks all of the benefits that the Realm database has.

P.S.
If we go further, there’s also another issue we do have, which probably is outside of this specific topic and is a more complicated variant of this issue. Imagine you have a list that should show data from 2 different Realm collections. How would we manage this? We need a mechanism to merge 2 Realm collections reactively.

Hi @Gagik_Kyurkchyan :slight_smile:

Unfortunately what you are trying to do is not easily achievable with realm. It obviously depends on how complex your use case is, but I can think of two general solution:

  • Define wrapper properties around the realm properties, and then bind the UI to your wrapper properties. You could even define ProductViewModel inside Product, so that’s easier to manage. This starts to become more complex if you need to forward notifications from the base object.
  • Use value converters to transform the properties of your classes in order to be shown in the UI. This is a little more cumbersome, but at least you would not need to worry about notifications from the property to the UI.

Finally, both these approaches starts to get complex if you need two-way data binding.

If you have more complicated use cases that you can show we can give a look and maybe give you some suggestions about what you could do with it.

@Gagik_Kyurkchyan still regarding this, I’ve opened an issue in our repo to track something that could be useful in your case.

Hey @papafe

Nice to see you again :slight_smile: And thanks for the quick reply as always.

I wouldn’t actually assume that Realm would support something like this out of the box. But rather, there might be some open-source libraries that are meant to solve this problem.

I am investigating Dynamic Data currently, which looks promising. I will try digging deeper into it and see if I can create a fully reactive collection pipeline with it.

Anyway, thanks for the reply one more time, I will keep digging and update this thread with the outcomes.

1 Like

Hey @papafe and anybody who would face this issue, we’ve solved the issue by using a combination of DynamicData and Lazy<>.

We have the same Product and ProductViewModel. Let’s modify the ProductsViewModel logic to ensure everything is reactive and lazy. Here’s how the ProductsViewModel would look like:

public class ProductsViewModel : IDisposable
{
    private readonly Realm _realm;

    private readonly CompositeDisposable _bindings = new();

    //--1--
    private readonly ObservableCollectionExtended<Lazy<ProductViewModel>> _products;

    //--2--
    public ReadOnlyObservableCollection<Lazy<ProductViewModel>> Products { get; }

    public ProductsViewModel(Realm realm)
    {
        //--3--
        _products = new();
        Products = new(_products);
        _realm = realm;
    }

    public void LoadProducts()
    {
        //--4--
        var results = (IRealmCollection<Product>)_realm.All<Product>().OrderBy(p => p.Name);

        _bindings.Add(results
                      //--5--
                      .ToObservableChangeSet<IRealmCollection<Product>, Product>()
                      //--6--
                      .Transform(product => new Lazy<ProductViewModel>(() => new ProductViewModel(product)))
                      //--7--
                      .Bind(_products)
                      .Subscribe());
    }

    public void Dispose()
    {
        //--8--
        _realm?.Dispose();
        _bindings.Dispose();
    }
}

And inside the page xaml, instead of binding to Name we now bind to Value.Name, as our items are of type Lazy<ProductViewModel

<ListView
    ItemsSource="{Binding Products}">
    <ListView.ItemTemplate>
        <DataTemplate>
            <TextCell Text="{Binding Value.Name}" />
        </DataTemplate>
    </ListView.ItemTemplate>
</ListView>

Let’s break down the changes.

  1. Set up a private variable ObservableCollectionExtended that will host our collection of ProductViewModel. This is a special extension of ObservableCollection by DynamicData that enables the reactive chained operations that DynamicData allows. Another emphasis here is the usage of Lazy<ProductViewModel>. This will guarantee that the ProductViewModel won’t be created until it’s accessed. Thus, the realm data that the ProductViewModel is based on will be accessed lazily when it’s needed
  2. Because we do not want to allow modification of the ObservableCollectionExtended for the view model consumers, the good design pattern is wrapping the ObservableCollection with ReadOnlyObservableCollection and exposing the read-only version to the public
  3. We initialize the observable collection and the read-only observable collection to wrap it
  4. We fetch the products from the realm, and we order the data as we want it.
    a) It’s crucial to set the ordering at the level of the realm query. Even though DynamicData supports ordering of collection, if we use the DynamicData version, it will cause all of the items to be loaded into memory, which we were trying to avoid in the first place
    b) We need to cast the IQueryable result that the realm database returns to IRealmCollection. Under the hood, it is an IRealmCollection so the cast is safe. The reason for this is to get a collection that implements INotifyCollectionChanged. Otherwise, we won’t be able to use the DynamicData extension called ToObservableChangeSet later on.
  5. We convert the collection to IObservable<IChangeSet<>>, which is the bases of the reactive transformation that DynamicData has. To learn more about this, read the DynamicData documentation
  6. To transform the Product realm object to Lazy<ProductViewModel>, we use the transform operator
  7. Finally, we bind the result to the _products observable collection, and we Subscribe to initiate the reactive chain
  8. Do not forget to dispose of all of the disposables to avoid memory leaks

This approach guarantees end-to-end reactiveness and lazy loading. Whenever a new product is added to the database, the DynamicData magic transformation will ensure that the target collection is updated as well. And as we create a Lazy object, the data will actually be loaded whenever it’s accessed. And you won’t have to implement any pagination or lazy loading at the UI level. It’s as simple as binding to a collection. ENJOY :slight_smile: