Efficient realm query pagination and transformation while preserving lazy loading and change notifications

This is a topic that I’ve been struggling with since I started using the Realm database.

I’ve raised this question earlier in this thread and I thought that I have found a solution. However, after doing a very deep performance analysis that you can find in this article - Creating memory efficient reactive IObservable<IChangeset<T>> based on Realm database query · reactivemarbles/DynamicData · Discussion #894 (github.com), I’ve understood that the solution I thought I found has major memory and performance issues.

Let me restate the problem for visibility purposes.

We have a pretty complex application that uses MongoDB App Services and Realm.

The realm database has a fantastic feature that allows data to be lazy-loaded when accessed. This has pretty exciting effects. Imagine you have a screen that shows a list of items. Those items can be tens of thousands. In the traditional database or API approach, one would implement some sort of data pagination - either automatic, when the user reaches the end of the list, the next chunk of data is loaded, or manual - the user manually selects the page they should load. With realm, you can fetch all of the items of an entity and directly bind those items to a ListView control in Xamarin. For example:

public class Product : RealmObject
    public string Name { get; set; }
    public decimal Price { get; set; }

public class ProductsViewModel
    private readonly Realm _realm;

    public IEnumerable<Product> Products { get; set; }
    public ProductsViewModel(Realm realm)
        _realm = realm;

    public void LoadProducts()
        Products = _realm.All<Product>();

//The xaml page
    ItemsSource="{Binding Products}"
            <TextCell Text="{Binding Name}"/>    

In the above example, even if there are tens of thousands of products, the ListView will load instantly and will scroll smoothly because of the lazy-loading magic.

The issue arises when we need to transform the native realm object Product into a different class. Imagine we want to create a new view model out of the Product class, called ProductViewModel :

public class ProductViewModel
    public ProductViewModel(Product product)
        Name = product.Name;
        FormattedPrice = $"{product.Price:C}";
    public string Name { get; set; }
    public string FormattedPrice { get; set; }

This is a trivial example of simply formatting the currency into a string. The scenario is much more complicated in our real-world application, but it’s good to show the issue we are facing. One would guess that changing the ProductsViewModel to something like this would do the job:

public class ProductsViewModel
    private readonly Realms.Realm _realm;

    public IEnumerable<ProductViewModel> Products { get; set; }
    public ProductsViewModel(Realms.Realm realm)
        _realm = realm;

    public void LoadProducts()
        Products = _realm.All<Product>().Select(p => new ProductViewModel(p));

However, this approach is naive and will essentially break the list’s lazy-loading aspect. It will iterate over every item in the list, fetching the product’s name and price and creating a new ProductViewModel. Only after all of these operations are completed will the list show any data. If the product collection is extensive, this will take significant time to complete.

How can we approach solving this problem the right way? Essentially, we need some reactive collection transform operator that does the following.

  1. It preserves the reactive notifications of the original collection. If the original Realm collection changes, the transformed collection should also omit collection change events
  2. Items of the collection should be evaluated lazily whenever the list shows

The solution that I thought I’ve found was based on using DynamicData and hooking it to the Realm query results`

                                       .Filter(item => item.IsValid)
                                       .Transform(p => new ProductViewModel(p));

However, as I mentioned above, I’ve explained in detail in this article why this is not working and essentially materializes the realm collection under the hood.

In that dynamic data discussion thread, we’ve concluded that the only real solution to the problem that will preserve all benefits of the realm query (reactiveness and lazy loading) is the one that would be provided by the Realm SDK itself - a real pagination and transformation support. Something like this

realm.All<Product>().Skip(10).Take(50).Select(p => new ProductViewModel(p));

The above LINQ statement will not work now as the Skip, Take, Select LINQ operators are not supported.

As a temporary solution, we’ve implemented an artificial pagination so that the transformed objects are created only for a subset of items, which won’t take much time. But this is a super undesired approach, as it breaks all of the Realm database’s benefits. This solution is described in the Approach 5 of the article I mentioned a couple of times.

If we go further, there’s another issue we do have, which probably is outside of this specific topic and is a more complicated variant. Imagine you have a list that should show data from 2 different Realm collections. How would we manage this? We need a mechanism to merge 2 Realm collections reactively.

Hi @Gagik_Kyurkchyan, thanks a lot for your question.
It seems that the crux of your question is about how to create a wrapper class (like your ProductViewModel ) around a Realm object that preserves the reactivity, and that can also be used in an observable collection.

In this case what we suggest is to try to include the property that you would need on your wrapper/viewModel directly on the RealmClass, and then use an approach similar to what is shown in this article, especially in the part where it talks about Vector3 and how to model types we do not support.
The main idea in your case would be to:

  • Add the properties you need directly on the wrapper/view model class
  • Change their getters (and eventually setters) so that they do the appropriate conversions to/from the relative realm properties
  • Override OnPropertyChanged to raise notifications for these additional properties

For the example that you posted this would be something like:

public class Product : RealmObject
    public string Name { get; set; }
    public decimal Price { get; set; }

    public string FormattedPrice => $"{Price:C}";

    protected override void OnPropertyChanged(string propertyName)

        if (propertyName == nameof(Price))

Would this be something suitable for your use case? I understand this could get complex with more properties, but we could eventually simplify the handling of raising notifications on our end.

Regarding your questions about merging two Realm collections, we do not have any particular recommendations and it depends on your case. If these two collections are queries on the same table then you can probably write a single query to have both in the same collection. Otherwise it gets more complicated, but it depends on your specific use case.

Dear @papafe

Thanks a lot for getting back to me. Really appreciate your answer.

Indeed, utilizing property change notifications will solve some of the problems in some scenarios. However, I have intentionally simplified the example than in real life. Unfortunately, this will not solve my problem for several reasons.

The first reason is more stylistic/architectural. Essentially, different view models will require different transformations of the same model. Putting all this “stress” on the model will bloat its size and break the separation of Model and ViewModel in MVVM architecture. As the application grows, the model will acquire more concerns of different view models. Anyways, I could live with it if it was the only problem.

The second problem is much more complicated. The model transforms into a ViewModel that’s not just a bag of properties or transformed properties. ViewModel can have injected services, compose multiple objects into a single succinct interface that’s consumable for the view. I will bring only this single example from a real project (and this is not the most complicated example BTW), I think you would be able to extrapolate this to more examples that can and will happen in real-world applications.

public class VideoMentionViewModel : BaseDisposableViewModel
    public static readonly UpvoteMissingFeatureRequest UpvoteAmenRequest = new(() => CommonStrings.UpvoteAmenDialogTitle, Feature.SayAmen);

    public static readonly UpvoteMissingFeatureRequest UpvoteCommentRequest = new(() => CommonStrings.UpvoteCommentDialogTitle, Feature.CommentOnMentions);

    public static readonly UpvoteMissingFeatureRequest UpvoteShareRequest = new(() => CommonStrings.UpvoteShareDialogTitle, Feature.ShareMentions);

    public static readonly UpvoteMissingFeatureRequest UpvoteSaveRequest = new(() => CommonStrings.UpvoteSaveDialogTitle, Feature.SaveMentions);

    public static readonly UpvoteMissingFeatureRequest UpvoteFollowRequest = new(() => CommonStrings.UpvoteFollowDialogTitle, Feature.FollowAuthors);

    public VideoMentionViewModel(VideoMention videoMention,
                                 IAsyncCommand upvoteMissingFeatureCommand,
                                 ICultureService cultureService,
                                 ICurrentBibleService currentBibleService,
                                 CompositeDisposable disposables) : base(disposables)
        VideoMention = videoMention;
        UpvoteMissingFeatureCommand = upvoteMissingFeatureCommand;

        VideoMentionSummary = cultureService.Culture.Select(culture =>
            var components = new List<string>
            return string.Join(" • ", components);
        MentionCountSummary = Strings.ResourceManager.GetFormattedCountLocalizableString(videoMention.Video!.Mentions.Count(),
        //TODO: Implement amens
        AmenCountSummary = "2";
        MentionLocation = $"{videoMention.StartSeconds.FormatSeconds()} / {videoMention.Video!.DurationInSeconds.FormatSeconds()}";
        BibleLocation = currentBibleService.GetBibleLocation(videoMention.VerseId)
                                           .Select(location => location.ToString());

    public VideoMention VideoMention { get; }
    public IAsyncCommand UpvoteMissingFeatureCommand { get; }
    public IObservable<string> VideoMentionSummary { get; }
    public IObservable<string> BibleLocation { get; }
    public LocalizableString MentionCountSummary { get; }
    public string AmenCountSummary { get; }
    public string MentionLocation { get; }

    private static LocalizableString GetTimeAgo(DateTimeOffset creationDate)
        TimeSpan ts = DateTimeOffset.UtcNow.Subtract(creationDate);
        double delta = ts.TotalSeconds;

        string key;
        object? parameter = null;
        switch (delta)
            // less than a minute
            case < 60:
                key = nameof(Strings.VideoMentionJustNow);
            // less than an hour
            case < 3600:
                key = ts.Minutes == 1 ? nameof(Strings.VideoMentionOneMinuteAgo) : nameof(Strings.VideoMentionMinutesAgo);
                parameter = ts.Minutes;
            // less than a day
            case < 86400:
                key = ts.Hours == 1 ? nameof(Strings.VideoMentionOneHourAgo) : nameof(Strings.VideoMentionHoursAgo);
                parameter = ts.Hours;
            // less than a week
            case < 604800:
                key = ts.Days == 1 ? nameof(Strings.VideoMentionOneDayAgo) : nameof(Strings.VideoMentionDaysAgo);
                parameter = ts.Days;
            // less than a month
            case < 2592000:
                int weeks = Convert.ToInt32(Math.Floor((double)ts.Days / 7));
                key = weeks <= 1 ? nameof(Strings.VideoMentionOneWeekAgo) : nameof(Strings.VideoMentionWeeksAgo);
                parameter = weeks;
            // less than a year
            case < 31536000:
                int months = Convert.ToInt32(Math.Floor((double)ts.Days / 30));
                key = months <= 1 ? nameof(Strings.VideoMentionOneMonthAgo) : nameof(Strings.VideoMentionMonthsAgo);
                parameter = months;
                int years = Convert.ToInt32(Math.Floor((double)ts.Days / 365));
                key = years <= 1 ? nameof(Strings.VideoMentionOneYearAgo) : nameof(Strings.VideoMentionYearsAgo);

        object[]? parameters = parameter != null ? new[] { parameter } : Array.Empty<object>();
        return Strings.ResourceManager.GetLocalizableString(key, parameters);

Let me explain what’s happening here. First, we have VideoMention realm entity which has a related Video entity. We have a screen where we need to show the list of VideoMention-s. However, we can’t show the video mention by itself. We need to transform and aggregate the data from the realm object. Particularly

  • Our application is localizable. Every textual content that we display should be dynamically updated when the culture changes. Thus, the video model references ICultureService to generate reactive localizable text
  • We need to show formatted information about the location of the video mention within the mention. (this part could actually be solved by property changes inside the realm entity, though undesirable due to breakage in separation of concerns)
  • The video mention is essentially a bible verse tag inside a vide. Users may have multiple Bibles. We need to show the video mention summary based on the bible that the user has selected. We should reactively update the video mention when the user selects a different bible.
  • The video mention has different actions that the user can perform. In MVVM those are called commands. I inject the commands into the VideoMentionViewModel from its parent view model that hosts the collection of VideoMentionViewModels. This way, all items in the list share the same commands and we can bind the commands directly from the view.
  • Further, some view models must have subscriptions to other services. We need to dispose the view models when the screen is closed, otherwise, we will have a tone of memory leaks. I don’t think it will be safe to dispose the realm objects and I envision that this will cause a tone of other issues.

The only way to achieve the above is by transforming the realm entity into a view model. And this brings me to the original problem that I’ve described - there should be a way, built into the realm to do this efficiently. Either through native pagination, or a native support for lazy transformation (aka, objects are transformed when they are requested).

For the time being, as I’ve mentioned earlier, I’ve implemented my own version of pagination which does the following


This, of course, will break the reactive collection change notifications.

I hope this makes sense, and I’ve seen in other community posts that people request this kind of functionality. If needed, I can bring up more examples like this.

@Gagik_Kyurkchyan I understand your point, and I can definitely see why my previous suggestion didn’t exactly cut it, as it was more appropriate for simpler cases.

In this case I would suggest something similar to have I’ve done in this PR. I thought it would be easier to read in a PR than copy-pasting the whole code here.

Here the main idea would be to create a WrapperCollection for realm queries, that essentially emits the same NotifyCollectionChanged events as the collection query, and also creates the ViewModel class when requested.
This ViewModel class is a wrapper around the realm object, that takes care of emitting notifications for all its properties that depends on the realm object properties, like Summary in the example.

I understand this is not super clean, but let me know if this would make sense in your case.

Hey @papafe

Thanks for sharing the code. I will be off for several days and will dig into the code once I am back and get back to you.


1 Like

Dear @papafe

Finally, I had a chance to play with the suggested solution. Unfortunately, it has a major issue - it is not lazy.

As soon as you bind the collection to the list these lines will be invoked

public IEnumerator<TViewModel> GetEnumerator()
    foreach (var item in collection)
        yield return viewModelFactory(item);

This means, that the view model will be created for all of the items inside the collection. I’ve pushed this suggestion to my own sample project. Trying pulling the main branch of the kyurkchyan/RxRealm project. I utilized the WrapperCollection inside the RxRealm.Core.ViewModels.ProductsViewModel/ RxRealm.Pages.ProductsPage. If you try running the application you will see 3 tabs -

  • Paginated - this is the tab that contains the best solution at the moment - a manual implementation of pagination that I’ve mentioned earlier
  • Virtualized - this is an attempt of using dynamic data virtualization to solve the problem. But it doesn’t work due to the fact that the full collection is materialized causing a significant UI thread blockage
  • Wrapper Collection - This is the tab you should select to see the application’s behaviour.

The application has a built-in database of 1 million products. When you open the app for the first time, the database will be installed.

As soon as you navigate to the “Wrapper Collection” page, you will see a significant delay until the UI becomes responsive. If you put a breakpoint on the yeald return... line, you will see that it gets invoked for each and every product (not just the products that are currently visible in the UI).

I tried hacking around with ChatGPT to find a solution that does lazy evaluation, but no luck. Whatever I do with IEnumerable, everything materializes the Realm collection and doesn’t support lazy loading of items.

Hi @Gagik_Kyurkchyan ,

Unfortunately I don’t have access to DevExpress, so I can’t run your project as it is, but let me do some investigation and I’ll get back to you.

Ok, I’ve made a small test that you can find in the same PR. I have added 200 objects to the realm and I’ve added Console.WriteLine both when the enumerator is called and when the indexer is called. You can see the changes I’ve done in the last commit.

If you run the test you’ll see that the enumerator is not called at all, and that the indexer is called only for the first 16 elements, that are the ones that are shown on screen. If you scroll, you’ll see the indexer being called for the elements as they appear.
(To be honest, I can see the indexer is called 3 times for each element, but I can imagine this is due some internal inefficiency of MAUI).

So, I think the problem here is understanding what exactly is the DevExpress control doing. With the default MAUI controls, when a collection is bound to the UI, the framework decides how to populate the list on the screen depending on the runtime type of what is bound.
So, for instance, if the actual type of the bound collection is IReadOnlyList, then the framework knows that it has an indexer and just uses that.
If the actual type is “only” IEnumerable instead, then it know it can only use the enumerator (and so it’ll be less efficient). You can verify this by switching IReadOnlyList for IEnumerable in the wrapper collection (here). If you do so and run the project, you’ll see that the enumerator is called for all the elements of the list (200) and not only for the first 16 that are shown on screen.

All of this to say that probably we need to understand how does the DevExpress control that you’re using make its decision internally on which method to calls when populating the UI. Once that is understood, probably you can modify the wrapper collection to work with what the control expects for a “lazy approach”.

It’s a lot of text but I hope I’ve been clear enough :wink:

1 Like


Very interesting indeed! It’s a very deep insight, and it gives me some directions to explore. I will play with this a bit tomorrow and get back to you.

BTW, I owe you a meal :slight_smile: I hope we will meet someday so I can return the favor, haha!


Glad to be of help :wink:

Btw, I’ve given a look at the documentation from DevExpress, and it seems that the collection you’re binding to needs to implement IList or IList<T> to have lazy loading. It’s slightly more annoying because IList has many more methods than IReadOnlyList that I’ve used in my example, but I think if you set IList.IsReadonly to true, probably you don’t need to implement any of the collection modification methods.

Hello @papafe

I’ve been playing around implementing the more advanced version of the WrapperCollection and there is some progress that I’ve made. Sorry, this is going to be a long post.

First of all, I have been able to achieve laziness by implementing the IList interface, more specifically, the methods that are required for list read-only access. This is already an amazing achievement.

As you remember, my goal was 2 things when trying to achieve efficient implementation

  1. Laziness (which I can already say we’ve achieved)
  2. Reactiveness - aka, updating the collection when the database changes

And the second point was the subject of my focus for the past couple of days. Unfortunately, the basic implementation of the INotficyCollectionChanged interface inside teh wrapper collection is not working properly for several reasons

When we simply forward the event to the underlying realm collection, we use Model entity inside the events. For instance, when an object is added the INotifyCollectionChanged will contain the Product realm object inside the NewItems. However, as our collection is of ProductViewModel type, we actually need to translate/map the INotifyCollectionChanged event’s internal object collection of Product realm entities to ProductViewModel.

There are various challenges associated with implementing such mapping efficiently. First, we need to store a mapping between the model and the view model - a map of Product to ProductViewModel - we need this to cache the view models that we’ve already created. Otherwise, view model objects will be created every time their are accessed. There’s a really nice class that I’ve utilized from ReactiveUI called MemoizingMRUCache to do jus this. Because our products collection can be huge, we don’t want to pre-create all view models. We rather need to create view models when they are needed. Also, we don’t want to cache too many view models - we only need the ones that are accessed the latest. That’s exactly what MemoizingMRUCache will do.

Then, we need to subscribe to the INotifyCollectionChanged event and handle all events.


Deletion is very problematic. When realm collection raises the deletion event, it has 2 main properties

  • OldItems - containing the deleted objects.
  • OldStartIndex
    Now, we need to create a new INotifyCollectionChanged event that will have the same OldStartIndex and OldItems with the old view models. The challenge is that the deleted objects are invalid realm object. If you try to use them to get the view model from the mapping, you will get an exception. Thus, we will need to find another way

I have solved this issue, but in a very nasty way. Pull my code on the main branch to see what I’ve done. The explanation is going to be very long, I won’t put it here. TBH, I would love to have a call with you to solve this problem once and for all, and I believe it will be a big win for the community in general if we solve this.

Handling INotifyCollectionChanged

I have done some tests with my current implementation on the main branch of the repo, and the results are not satisfactory yet. Everything is very slow. I really need to dive deep to understand how to improve the performance.

However, there’s something that I would really need a help. How exactly does realm raize INotifyCollectionChanged events? There are 5 actions

  • Add
  • Remove
  • Replace
  • Move
  • Reset

Does realm collection raise them all? If yes, can you please explain how can simulate replace, move or reset. (add and remove are more or less clear). Also, I would love to see if the events can also be raised with collection inside the OldItems or NewItems instead of a single value. Generally, to be able to validate my implementation, I would need to simulate all possible realm collection change event use cases.

Action items

For the time being, here are the 2 main things

  • If possible, please describe how realm utilizes INotifyCollectionChanged so I can implement the necessary testing strategy
  • If you would like to dive deeper into the code I’ve come up with so far and brainstorm together to solve this issue once and for all, I would love to have a pair programming session with you @papafe

Hey @Gagik_Kyurkchyan,

I see that the problem is only partially solved. For sure we can have a call together to have a faster iteration about the issues that you’re seeing. Unfortunately next week I’ll not be at work, and I’ll have a shoddy connection, so we can only meet from May 28th onwards.

For now a couple of things.


You’re correct, you would need to “translate” the CollectionChanged from the realm collection to the wrapper collection. To be honest I was sure that I tested adding/removing elements when I sent my code, but it seems I didn’t, so I’m sorry for that. I just tested that adding the translation layer solves the issue. I’ll push it shortly to my PR in case you want to take a look. This of course doesn’t completely solve your problem because, as you said, you could be recreating the view model multiple times.
I still need to take a look at the MemoizingMRUCache class, but I suppose we can talk about it during our meeting too.


Here the bad news is that you’re correct, unfortunately you cannot access delete items because they’ll be invalid. So we actually use a special “sentinel” value when raising CollectionChanged after a deletion.
Good news is that, at least from my tests on MAUI, it seems that oldItems are not used by the UI framework, so you can just fill it with default(TViewModel)

How do we utilitze INotifyCollectionChanged

Internally we raise CollectionChanged by translating the “pure” realm notifications obtained with results.SubscribeForNotifications. This is the method where we do the translation.

Going forward

I did not have the time to go over your code, so I am sorry if some of the things I have posted are not super appropriate for your use case, but I wanted to give you some pointers until we have our call, where we can talk and (hopefully) solve your issues :smiley:

Hey @papafe

I think I’ve almost solved the issue in an ideal way that fits into the ideology of our application and what I was trying to achieve.

I’ve decided to take a slightly different strategy than I initially had. Essentially, initially I wanted to achieve a super-generalized implementation. However, that kind of implementation comes with a cost - it’s hard to implement and you would have to deal with cache management and proper complex abstractions if you wanted to make it right. Instead, I decided to address this issue with more specific requirements of our application. And in our case those requirements come to these 2 points

  1. Pagination
  2. Reactive change notifications


Pagination solves several problems.

  1. If we need to make transforms, we can do so only for a very limited set of items - the ones that are currently loaded in the list and ignore anything else
  2. It is lazy and memory efficient by it’s nature - you load only the data that the user wants/needs to see

Further, to make our first-time user experience top-notch, we are going to implement 2 versions of each DB request

  • One that talks to the APIs
  • One that talks to the synchronized Realm database

Essentially, we will display the data that comes first. We need this so that we return the data to the user as soon as possible instead of waiting for all data to sync to the device. So, we will need to implement paginated API requests anyway. If we have a proper abstraction in place, we can use the same type of data access for both the database and the APIs through pagination, and the UX will also be consistent.


So, I’ve taken the PaginatedResults<T> class that I already had as one of the best solutions to this problem, and made it even better through a subclass of it called[ PaginatedRealmResults<T>]( RxRealm/RxRealm.Core/PaginatedRealmResults.cs at main · kyurkchyan/RxRealm (github.com)). This new class does two things

  1. Translates the pagination requests to realm query
  2. Handles the INotifyCollectionChanged events

The interesting part about handling the INotifyCollectionChanged events, is that we only need to handle the CRUD operations only for the indices that are currently already loaded inside the collection. Let’s say our page size is 50, and we’ve loaded 2 pages - 100 items. If a change happens with an index of 105, we can safely ignore that, as the user hasn’t reached that page yet and the data will be loaded once they reach that point.

Benefits of this hybrid implementation

There are multiple benefits of this implementation.

First of is that I can later add PaginatedApiResults<T> subclass, which will translate the pagination requests to REST API requests and the consumers will use the same interface without knowing whether the results are coming from API or the database.

The second is that I am using DynamicData APIs to load the collections. This means that I can use its reach APIs to transform, sort, dispose of, and cache the collection the way I want it without worrying about performance issues as we only work with a small subset of data.

I’ve integrated this functionality into the sample app inside a page called “Paginated”. Everything works as expected - all CRUD operations update the UI reactively. So, as far as I can tell, I’ve reached the ideal implementation that covers all use-cases for our application. There are 2 remaining questions though.

Remaining questions

I did take a look at the source code where you map the realm change notifications to INotifyCollectionChanged events. However, it didn’t give me the information that I need to be able to simulate all possible scenarios of change notifications. Particularly

  • While I was doing unit tests of the new PaginatedRealmResults<T> class, I found that if I insert 10 items in the middle of the query, it will generate a Reset event instead of add. I didn’t understand why.
  • I don’t know how to simulate Replace/Move events.

I tried to generate Move event like this

  • loaded a list of products ordered by price. In my test setup each product has a unique price and I generate 100 of them with prices 1, 2,…, 100
    Pick the first product and set its price to 50.5, which will put it in a new position—instead of the first, it will have a position of 50.

When I do this, instead of a Move event, I receive a Reset event.

So, to avoid strange behaviors and be able to test all possible notification types, I need to understand clearly when and how each change event will happen. Once we have this, I would say that my issue will be fully resolved. I will be happy to discuss this during a call once you are available after May 28th as well if we do not resolve this asynchronously through this forum.