Do relationship properties get populated during .toList()?

Do relationship properties get populated during .toList()?

Hi,

We’ve recently discovered Realm for Flutter and are finding it really awesome!

We are currently migrating to Realm and during the first stage we are injecting Realm db into the Data Access layer instead of http services. It means that we cannot currently use the live data feature (but plan to migrate to it during the subsequent stages) and we have to query and map all data before returning it to presentation level.

Maybe you can help us understand how to query data from Realm in more optimal way.

Example models:

@RealmModel()
class $ProductEntity {
  @PrimaryKey()
  late int id;
  String name = '';

  late List<$ProductCommentEntity> comments;
}

@RealmModel()
class $ProductCommentEntity {
  @PrimaryKey()
  late String id;
  int productId = 0;
  int userId = 0;
  String comment = '';


  @Backlink(#comments)
  late Iterable<$ProductEntity> product;
}
  1. Will the comment list be also be hydrated during the .toList() call or will it stay “live” and be populated only during actual enumeration:
var products = realm
	.query<ProductEntity>('comments.@count > 0 SORT(name ASC)')
	.skip(index)
	.take(size)
	.toList();
  1. What would be better solution performance wise?
var realm
	.query<ProductEntity>('comments.@count > 0 SORT(name ASC)')
	.skip(index)
	.take(size)
	.toList()
	.map((x) {
	  final product = x.toModel();
	  return ProductExperienceFull(
		product: product,
		commentedByMe: x.comments.any((y) => y.userId == myUserId),
	  );
	})
	.toList();

or


var realm
	.query<ProductEntity>('comments.@count > 0 SORT(name ASC)')
	.skip(index)
	.take(size)
	.toList()
	.map((x) {
	  final product = x.toModel();
	  return ProductExperienceFull(
		product: product,
		commentedByMe: realm.query<ProductCommentEntity>('productId = \$0 && userId = \$1', [x.id, myUserId]).isNotEmpty,
	  );
	})
	.toList();

1

The .toList will create native handles for all elements in the results, and fix which object is at what index. It is one of the things I typically warn against. It won’t actually hydrate the object though. Whenever you access a property on realm object, it will read lazily from the database, so the objects are still live… that is until you map them later, which brings me to your second question

2

I’m a bit worried by the skip(index) in both examples. As is, it will create a handle for the first index elements, and while that is not super expensive, it is a total waste in this case. We should probably improve on that.

Disregarding that…

Well, it depends :slight_smile: … on the length of the comment list fx.

In the first your reading the userId into dart for each element in the list, but the list is already available, and the overhead of retrieving an int is very low.

In the second you avoid the traversal on the dart side, but on the other hand since you have no index on productId the query engine is forced to iterate over all products.

However in your example I would expect the first to be fastest, but it never hurts to meassure.

That said, creating a window abstraction, that avoids skip will be a good idea… At least for now.

3 Likes

I have a PR in review to make skip on RealmResults efficient.

1 Like

Hi @Kasper_Nielsen1,

Wow! Thanks for the detailed response and for the .skip() fix!

One more question regarding the hydration.
I thought that objects’ non-relationship/non-ListResult properties are hydrated fully when you access it by index:

final firstComment = product.comments[0]; // <-- all the fields of firstComment are fully fetched from db

or if you access at least one prop:

final firstComment = product.comments[0]; // <-- firstComment not hydrated
var userId = firstComment.userId; // <-- firstComment is fully hydrated now

but after reading you answer I’m getting the feeling that the properties values are fetched one by one only when accessed:

product.comments
  .toList()
  .map(
	(x) => ProductCommentEntity(
		id: x.id, 				// <-- fetch id
		productId: x.productId, // <-- fetch productId
		userId: x.userId, 		// <-- fetch userId
		comment: x.comment, 	// <-- fetch comment
	),
  )
  .toList()

Can you please tell which case is true?
I mean it does not really matter if it works fast, but it is good to understand how the technology you work with works :slight_smile:

P.S. It would be helpful to have some kind of logs when there is an actual fetch request to db backend - then it would be easier to debug and optimize such things.

They are fetched property by property. That is one reason mapping from realm objects to another is not the best architecture for realm. Not only do you loose the liveness, but you also pay for the hydration of properties you may never need.

The only thing that is actually cached is the exact position of the object in the database in a given version. Hence once the native handle is created, we know exactly what memory addresses to fetch all props from.

Realm uses mmap, so it is really as efficient as following a native pointer + the overhead of moving the value into Dart memory. In general Dart FFI is very efficient, but for Strings there is slightly larger overhead since they are converted from UTF8 to Dart’s UTF16 based format.

But for anything sane we are talking single-digit nanoseconds for property access.

Wrt. logging you can increase the verbosity with:

Realm.logging.logLevel = RealmLogLevel.trace; // or what you prefer

You can also set your own logger:

Realm.logger = Logger.detached("custom logger")..level = RealmLogLevel.detail;

You can enable very detailed logging.

I can also recommend this issue comment from github:

and

However, note that you no longer needs to do these tricks, since RealmResults.elementAt has since been implemented efficiently. Still the discussion is illuminating I think.

There is also this issue, that I think does a good job explaining why you don’t need to do pagination with realm, and why you should be aware of ìtemExtent in ListView.builder when working with large lists.

I would also like to point out that Iterable.elementAt is implemented efficeintly for RealmResults. Think about this before calling toList. Even if you need to use indexing later, you can rest assured that elementAt is O(1) instead of O(n).

Hi @Kasper_Nielsen1,

Thank you for the explanation, it’s clearer now.

I also appreciate the tip on log levels - it’s quite helpful!

Regarding the attached issues - I actually went through them before starting the implementation. They, along with your Youtube streams on Realm, were instrumental in helping me grasp the proper use of Realm for Flutter. This was the main reason I chose Realm over other NoSQL database implementations. Live data Iterables are truly an amazing and superior concept!

Our initial goal is to quickly transition our app to use offline storage by replacing the http DAL with a local database. If all goes well in production, and we find that Realm is stable enough for such a strong coupling, we can gradually rewrite the app to use Realm the right way.

However, it would be immensely helpful to have more elaborate documentation on best practices, given that the core concept differs from the typical “query and map everything” or “data streams” approach that everyone is used to. It’s also not immediately clear when and why one should or shouldn’t use .query() as opposed to .where(), those .toList() gotchas, etc.

In any case, thank you very much for your dedication!

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.