Is there any reason cross-session cursors are not allowed?

Hello!

We have an interesting challenge. Doing exports for our customers at scale in a microservice environment. Right now, when performing an export, we perform. find using a limit + skip query, and then iterate over the DB by limiting the backpressure. Each find chunk is spread over 10 different workers allowing us to perform live updates in our core infrastructure.

A better way to solve this problem would have been to retrieve the cursorID and then perform getMore commands using cursorID across all our workers.

The issue is when doing this, we are getting an error Cannot run getMore on cursor <CURSOR_ID>, which was created in session <SESSION_ID>, in session <SESSION_ID>.

OK. So it is not possible doing cross session cursors. But what if I hack the MongoDB driver? It’s what I have been working on with success by crafting getMore commands having a lsid payload. Using this trick, I have been able to perform cross workers cursors.

So my question is simple. Why not allow developers to pass their own sessionID parameter? Let’s say I specifically want to create a long-running job, right now, my only option will be using skip/limit. Most Databases provide a clean way to continue a cursor job programmatically. It is technically possible on MongoDB using a lsid + cursorid, why not offering this on official drivers?

Hi @Baptiste_Jamin,

OK. So it is not possible doing cross session cursors. But what if I hack the MongoDB driver? It’s what I have been working on with success by crafting getMore commands having a lsid payload. Using this trick, I have been able to perform cross workers cursors.

Since MongoDB 3.6, compatible drivers would create implicit sessions for CRUD operations. getMore operations (per the documentation) must be called from within a session, however as the Driver abstracts the implicit session management away passing the lsid value isn’t publicly exposed.

We have an interesting challenge. Doing exports for our customers at scale in a microservice environment. Right now, when performing an export, we perform. find using a limit + skip query, and then iterate over the DB by limiting the backpressure. Each find chunk is spread over 10 different workers allowing us to perform live updates in our core infrastructure.

Note that these types of high offset queries (aka “paging via skip/limit”) are typically not performant (regardless of whether you use MongoDB or an RDBMS [1][2]). To better distribute this workload across workers an alternate strategy would be to segment your data into ranges.

This could look something like:

// OLD
db.foo.find({ status: "draft" }).skip(1000).limit(100);
db.foo.find({ status: "draft" }).skip(1100).limit(100);
db.foo.find({ status: "draft" }).skip(1200).limit(100);

// NEW
db.foo.find({ status: "draft", created_at: { $gte:  new Date("2021-01-01"), $lt: new Date("2021-02-01") } })
db.foo.find({ status: "draft", created_at: { $gte:  new Date("2021-02-01"), $lt: new Date("2021-03-01") } })
db.foo.find({ status: "draft", created_at: { $gte:  new Date("2021-03-01"), $lt: new Date("2021-04-01") } })

This would allow you to bypass using a single cursor for the entire operation and instead leverage multiple cursors, which could be distributed across the cluster (assuming a non-default read preference is used).

Additionally, assuming an appropriate index exists it could be used to improve the overall performance of each “batch”. Obviously this isn’t a 1:1 replacement for skip/limit as the batch quantities may vary but using knowledge of your data ingestion and distribution an appropriate filter could be identified.

So my question is simple. Why not allow developers to pass their own sessionID parameter? Let’s say I specifically want to create a long-running job, right now, my only option will be using skip/limit. Most Databases provide a clean way to continue a cursor job programmatically. It is technically possible on MongoDB using a lsid + cursorid, why not offering this on official drivers?

There are internal scenarios (such as working with Serverless instances) that would require additional tooling aside form just passing an lsid to the getMore to ensure it would function properly.

To ensure your microservice exports can scale appropriately a better strategy may be to look at range-based queries for distributing the workload.

2 Likes

We just made a NodeJS package allowing to perform cross-worker cursors: GitHub - crisp-oss/node-mongodb-native-cross-cursor: 📡 A MongoDB driver extension allowing to consume MongoDB cursors accross multiple instances.