MongoDB change streams not working with one specific cluster

Working with Data Java Frameworks
, , , ,
1

I have a spring boot microservice that communicates with the MongoDB cluster deployed on MongoDB Atlas. The mongodb driver version is 4.6.1 In this microservice, we actually trigger some functions if data is inserted, updated, deleted in some specific collections. For catching these events, we are using mongodb-driver-reactivestreams library. I am facing a srange issue. I have 2 clusters in MongoDB Atlas that can be seen below in the following image:

image
image2560×1528 249 KB

The microservice starts without any errors if i try to connect it with the first cluster Cooperants-DE but when i connect it with the second cluster ScopeSet I see this error in the logs when microservice starts

ocean-docker-server-elastic-search-1  | 2023-05-04 20:03:16,697 ERROR r.c.p.Operators@[annel-group-0-handler-executor] - Operator called default onErrorDropped
ocean-docker-server-elastic-search-1  | reactor.core.Exceptions$ErrorCallbackNotImplemented: com.mongodb.MongoCommandException: Command failed with error 8000 (AtlasError): 'Error modifying $match value for change streams / $currentOp. err=Oplog ns RegEx queries must begin with ^' on server ac-3htpqvx-shard-00-02.hor3ln3.mongodb.net:27017. The full response is {"ok": 0, "errmsg": "Error modifying $match value for change streams / $currentOp. err=Oplog ns RegEx queries must begin with ^", "code": 8000, "codeName": "AtlasError"}
ocean-docker-server-elastic-search-1  | Caused by: com.mongodb.MongoCommandException: Command failed with error 8000 (AtlasError): 'Error modifying $match value for change streams / $currentOp. err=Oplog ns RegEx queries must begin with ^' on server ac-3htpqvx-shard-00-02.hor3ln3.mongodb.net:27017. The full response is {"ok": 0, "errmsg": "Error modifying $match value for change streams / $currentOp. err=Oplog ns RegEx queries must begin with ^", "code": 8000, "codeName": "AtlasError"}
ocean-docker-server-elastic-search-1  |         at com.mongodb.internal.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:198)
ocean-docker-server-elastic-search-1  |         at com.mongodb.internal.connection.InternalStreamConnection$2$1.onResult(InternalStreamConnection.java:512)
ocean-docker-server-elastic-search-1  |         at com.mongodb.internal.connection.InternalStreamConnection$2$1.onResult(InternalStreamConnection.java:498)
ocean-docker-server-elastic-search-1  |         at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback$MessageCallback.onResult(InternalStreamConnection.java:821)
ocean-docker-server-elastic-search-1  |         at com.mongodb.internal.connection.InternalStreamConnection$MessageHeaderCallback$MessageCallback.onResult(InternalStreamConnection.java:785)
ocean-docker-server-elastic-search-1  |         at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:645)
ocean-docker-server-elastic-search-1  |         at com.mongodb.internal.connection.InternalStreamConnection$5.completed(InternalStreamConnection.java:642)
ocean-docker-server-elastic-search-1  |         at com.mongodb.internal.connection.AsynchronousChannelStream$BasicCompletionHandler.completed(AsynchronousChannelStream.java:250)
ocean-docker-server-elastic-search-1  |         at com.mongodb.internal.connection.AsynchronousChannelStream$BasicCompletionHandler.completed(AsynchronousChannelStream.java:233)
ocean-docker-server-elastic-search-1  |         at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannel.lambda$read$4(AsynchronousTlsChannel.java:122)
ocean-docker-server-elastic-search-1  |         at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
ocean-docker-server-elastic-search-1  |         at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
ocean-docker-server-elastic-search-1  |         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
ocean-docker-server-elastic-search-1  |         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
ocean-docker-server-elastic-search-1  |         at java.base/java.lang.Thread.run(Unknown Source)

Can someone guide me what I am doing wrong? I have also tried to run the microservice with the MongoDB version 6.0.5 that I deployed locally on my machine and microservice starts successfully without any errors but only when I try to connect it with the ScopeSet cluster the above error appears in the logs. Can some one guide me in this regard what i am doing wrong? Thanks

2

Hello :wave: @Usman_Sajid,

Apologies for the late response.

Could you please share if you have any Atlas database triggers associated with this and clarify what the functions are doing?

I see you have two different MongoDB Cluster tiers i.e., Cooperants-DE with M30 (dedicated tier) and ScopeSet with M0 (shared tier).

Based on the error message, it appears that there is an issue with your aggregation pipeline. Could you please share the aggregation pipeline with us so that we can reproduce it in our environment?

Also, please specify the specific database and collection you are querying. Based on the error, it seems that you are trying to access the oplog from the local database of MongoDB Atlas.

Shared tiers (M0/M2/M5) have certain restrictions in place that make them slightly different from a dedicated clusters. Please refer to the Manage Clusters and Atlas M0 (Free Cluster), M2, and M5 Limitations documentation for more details. Although we strive to make them functionally equivalent, some features are exclusively available on dedicated clusters. In this particular case, one restriction is that when performing a regex query on an oplog in a shared tier deployment, it must be preceded by an anchor (^).

Could you modify your query so that it includes this anchor and see if it works? For example:

Atlas atlas-gbccyn-shard-0 [primary] local> db.oplog.rs.findOne({ns:{$regex:"sample"}})
MongoServerError: Oplog ns RegEx queries must begin with ^
Atlas atlas-gbccyn-shard-0 [primary] local> db.oplog.rs.findOne({ns:{$regex:"^sample"}})
{
  op: 'c',
  ns: 'sample_analytics.$cmd',
  ui: new UUID("9e2aa592-eb09-4b4d-9f89-e03036b45540"),
  o: {
    startIndexBuild: 'transactions',
    indexBuildUUID: new UUID("487cd2d1-ca36-400e-b654-a10754eb9d57"),
    indexes: [
      {
        v: 2,
        key: { 'transactions.amount': 1 },
        name: 'transactions.amount_1',
        sparse: false
      }
    ]
  },
  ts: Timestamp({ t: 1664519274, i: 2 }),
  t: Long("96"),
  v: Long("2"),
  wall: ISODate("2022-09-30T06:27:54.871Z")
}

Best,
Kushagra

1 Like