How to partition an endless feed? (for example reddit, twitter, facebook, youtube comments...)

My problem is that in an “endless” feed, there can be thousands upon thousands of elements. The client obviously shouldn’t (and doesn’t in apps like Twitter) download them all at once. Only small chunks get downloaded, like 30 elements, and then, IF the user scrolls far enough down, the next chunk loads.

How could I achieve this functionality in realm? It is also very important to keep in mind the steady growing of new elements that get added to the server, so just making like a partition-1 for the first 30 elements, partition-2, partition-3 and so one will not work (or does it?).

This is such a common theme I think there clearly has to be some kind of solution. I also haven’t really looked into flexible syncing, maybe that’s the solution? Excited to read your comments.

I was wondering the same. I am currently developing (with flexible sync) an iOS app which contains user-generated content. From what I understand, for flexible sync, each subscription will locally download all the data based on its query, so having one subscription with a broad query won’t be scalable.
I haven’t been able to come up with an elegant solution yet, it would be great if someone could chime in…

1 Like

My approach, for now, is just doing it with functions.

I have a function ‘loadNextDataChunk(int offset, queryParameters, startTime)’ which will return the next x amounts of data every time the user scrolled far enough down.

Hello, I’m trying to achieve this exact functionality. Would you mind elaborating how you used the function to pass the results back to the realm app? (i.e the App service you used and the architecture) Thank you!

1 Like

Hello, my Realm function code looks like this:

exports = function(query, skipAmount, limitAmount){
  let col = context.services.get("mongodb-atlas").db("Database name").collection("collection name");
    
  let result = await col.find({'title': {'$regex': query, '$options': 'i'}}).skip(skipAmount).limit(limitAmount);
  if(result.modifiedCount == 0){
    console.log("Could not get results");
    return false;
  }
  
  return JSON.stringify(result);
};

This starts at the newest data and goes further down to older posts as the user scrolls. The parameters skipAmount has to be maintained from client app and is just a a whole number of how many posts already got loaded, limitAmount is the amount to load each time the user reaches bottom.

Hello, Thank you very much. This helped a lot!

1 Like