Need information about OP_MSG's more_to_come flag bit

Kartik_Pattaswamy · September 18, 2023, 5:48pm

Hello,

I read the OP_MSG specifications in the wire protocol documentation and saw that there is a flag bit called more_to_come. I am trying to learn more about how this bit is toggled and how it would affect the requestID or responseTo fields for request/response messages. Is this bit commonly toggled? An example of a query to run to reproduce the scenario on wireshark would be great.

Thanks so much!!

John_Sewell · September 18, 2023, 7:58pm

From the server source code looks like this is a trigger for that flag:

github.com

mongodb/mongo/blob/1e1c5733b96ea87f10d4a75fcd99c560fa179ccb/src/mongo/client/dbclient_cursor.cpp#L145


      
              auto msg = assembleCommandRequest(_client, _ns.dbName(), getMoreRequest.toBSON({}), _readPref);
          
          
    // Set the exhaust flag if needed.
              if (_isExhaust) {
                  OpMsg::setFlag(&msg, OpMsg::kExhaustSupported);
              }
              return msg;
          }
          
          
bool DBClientCursor::init() {
              invariant(!_connectionHasPendingReplies);
              Message toSend = assembleInit();
              MONGO_verify(_client);
              Message reply;
              try {
                  reply = _client->call(toSend, &_originalHost);
              } catch (const DBException&) {
                  // log msg temp?
                  LOGV2(20127, "DBClientCursor::init call() failed");
                  // We always want to throw on network exceptions.
                  throw;

github.com

mongodb/mongo/blob/1e1c5733b96ea87f10d4a75fcd99c560fa179ccb/src/mongo/client/dbclient_cursor.h#L107


      
                         boost::optional<BSONObj> postBatchResumeToken = boost::none);
          
          
virtual ~DBClientCursor();
          
          
/**
           * If true, safe to call next(). Requests more from server if necessary.
           */
          virtual bool more();
          
          
bool hasMoreToCome() const {
              invariant(_isInitialized);
              return _connectionHasPendingReplies;
          }
          
          
/**
           * If true, there is more in our local buffers to be fetched via next(). Returns false when a
           * getMore request back to server would be required. You can use this if you want to exhaust
           * whatever data has been fetched to the client already but then perhaps stop.
           */
          int objsLeftInBatch() const {
              invariant(_isInitialized);

github.com

mongodb/mongo-python-driver/blob/db4406b95198654066b72faaa8827492daedbb13/pymongo/message.py#L1553


      
              docs = self.unpack_response(codec_options=codec_options)
              assert self.number_returned == 1
              return docs[0]
          
          
def raw_command_response(self) -> NoReturn:
              """Return the bytes of the command response."""
              # This should never be called on _OpReply.
              raise NotImplementedError
          
          
@property
          def more_to_come(self) -> bool:
              """Is the moreToCome bit set on this response?"""
              return False
          
          
@classmethod
          def unpack(cls, msg: bytes) -> _OpReply:
              """Construct an _OpReply from raw bytes."""
              # PYTHON-945: ignore starting_from field.
              flags, cursor_id, _, number_returned = cls.UNPACK_FROM(msg)
          
          
    documents = msg[20:]

James_Kovacs · September 18, 2023, 10:39pm

Hi, @Kartik_Pattaswamy,

A good source of information is the moreToCome section of the OP_MSG spec. For a high-level overview of OP_MSG, I would recommend this blog post by one of our engineers.

Sincerely,
James

Kartik_Pattaswamy · September 18, 2023, 10:49pm

Hello,
Thanks a lot for the response. I realize that the more to come bit is set internally for bulk writes/find operations but what actually makes a bulk operation and causes the bit to be set by the client or server? I am trying to create a query that sets the bit and capture the traffic through wireshark.

James_Kovacs · September 19, 2023, 12:32am

The server will set the OP_MSG moreToCome flag on responses to exhaust cursors. Exhaust cursors are only valid for handshake messages (e.g. hello and legacy hello) and getMore. The easiest scenario to reproduce is to monitor the initial handshake between a 4.4-compatible driver and a 4.4 or later cluster. Streamable monitoring was introduced in 4.4. The initial handshake will start with OP_QUERY since the driver doesn’t know whether the cluster speaks OP_MSG. It will then upgrade to OP_MSG.

Another option is to opt into Stable API, which was introduced in MongoDB 5.0. Because we know the driver is talking to a MongoDB 5.0 cluster, the initial handshake is performed over OP_MSG.

I hope that this helps in your repro efforts.

Kartik_Pattaswamy · September 20, 2023, 4:57pm

Thanks a lot for the response. I was able to repro the server sending a message with moreToCome flag set. Is there also a way to repro the client also setting that flag on a request? I have tried inserting a document larger than the max BSON size as well sending a find query to receive many documents. Would I be right in assuming that this bit is usually set on server response?

James_Kovacs · September 20, 2023, 9:35pm

A single document cannot exceed the maximum BSON size of 16MB. According to the spec, drivers can set moreToCome on a request for unacknowledged writes, but these are generally discouraged. Reviewing the .NET/C# Driver code, we do not set moreToCome for unacknowledged writes. Typically moreToCome is only set by the server.

system · September 25, 2023, 9:36pm

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.