Watch for changes in documents that match a specific criteria

Working with Data
,
1

I wish to monitor a collection for changes using changeStreams. I only want to watch the documents in my collection which have a filed which matches a specific value. I am not looking for changes in that field but in the other.
For example.
I have a mongoose model called Trainer.
I wish to montior all trainers with fields called “available” set to “true” and “language” set to “english”.
Availability stays unchanged but antother field, say, “location” can change.
Can I watch for the change in location on all available trainers who speak English?
I cannot seem to do this without seeing a change on all trainers. (I have some who speak other languages but are also available.

Thanks for your help team.

3

Hi @John_Parsons and welcome to MongoDB community forums!!

Based on my understanding, you wish to view the updates in the change stream for only the documents that satisfy the match condition fields called “available” set to “true” and “language” set to “english”.
I tried the following code the:

const mongoose = require('mongoose');

// Connect to MongoDB
mongoose.connect('mongodb+srv://findThief:findThief@cluster0.sqm88.mongodb.net/?retryWrites=true&w=majority/test', {
  useNewUrlParser: true,
  useUnifiedTopology: true,
});

const db = mongoose.connection;
db.on('error', console.error.bind(console, 'MongoDB connection error:'));

// Define Trainer schema
const trainerSchema = new mongoose.Schema({
  available: Boolean,
  language: String,
});
const Trainer = mongoose.model('Trainer', trainerSchema);
const pipeline = [
  {
    $match: {
      available: true,
      language: "English",
    },
  },
];

const changeStream = Trainer.watch(pipeline, { fullDocument: 'updateLookup' });
console.log(changeStream);

console.log('Change stream started.');
changeStream.on('change', (change) => {
  console.log(change);
});

The output for the above when an update operation is performed is:

{'_id': {'_data': '8264B625FF000000172B022C0100296E5A1004B72A49ECB85648E49A08B734CBEA93AF46645F6964006464B5240E413B98DBF731CF880004'}, 'operationType': 'update', 'clusterTime': Timestamp(1689658879, 23), 'wallTime': datetime.datetime(2023, 7, 18, 5, 41, 19, 410000), 'fullDocument': {'_id': ObjectId('64b5240e413b98dbf731cf88'), 'name': 'XYZ', 'available': True, 'language': 'English', 'location': 'Changed Location'}, 'ns': {'db': 'test', 'coll': 'trainers'}, 'documentKey': {'_id': ObjectId('64b5240e413b98dbf731cf88')}, 'updateDescription': {'updatedFields': {'location': 'updated Location'}, 'removedFields': [], 'truncatedArrays': []}}
{'_id': {'_data': '8264B625FF000000182B022C0100296E5A1004B72A49ECB85648E49A08B734CBEA93AF46645F6964006464B5244F413B98DBF731CF890004'}, 'operationType': 'update', 'clusterTime': Timestamp(1689658879, 24), 'wallTime': datetime.datetime(2023, 7, 18, 5, 41, 19, 410000), 'fullDocument': {'_id': ObjectId('64b5244f413b98dbf731cf89'), 'name': 'ABC', 'available': True, 'language': 'English', 'location': 'Changed Location'}, 'ns': {'db': 'test', 'coll': 'trainers'}, 'documentKey': {'_id': ObjectId('64b5244f413b98dbf731cf89')}, 'updateDescription': {'updatedFields': {'location': 'updated Location'}, 'removedFields': [], 'truncatedArrays': []}}

The pipeline in the above code would match the requirements in the change stream and would should the results as fullDocument in the change stream output.

Could you please try the above code and let us know if it works for you.

Regards
Aasawari

1 Like
4

Hi @John_Parsons

arrgh, sniped by @Aasawari (happily with a code example) my pipeline is slightly different.

A pipeline would need to be added to the watch.

Something like:

[ 
  { "$match": 
    { "operationType": "update", 
     "fullDocument.available": true,
     "fullDocument.language": "english",
     "updateDescription.updatedFields.location": { "$exists": true } 
    } 
  }
]

This would strictly match updates where the field location is updated along with the matching criteria for available and language.

If you want to also catch other scenarios like a new matching document being inserted the the pipeline would need further modification.

5

Thanks for the quick reply, sorry I have taken so long to get back to you. Things to do.

It still doesn’t quite work. I get the console.log that the change stream has started, that indicates it is “watching” for changes. When I make a change tot the document (using MongoDB Compass on my desktop) I get no reaction.

6

Hi @John_Parsons

Would you mind sharing the code snippet that you are using to get the updates in the console screen ?

Thanks
Aasawari

7

Thanks Asawari for your reply:

The following code is in place:

const pipeline =[
    {$match: {available: true}}
  ];

  const changeStream = Trainer.watch(pipeline, { fullDocument: 'updateLookup' });

  changeStream.on('change', data => console.log(data));

If I leave pipline empty:
const pipeline = [ ];

It works, as soon as I put anything in the pipeline it doesn’t respond, just hangs waiting for a change (although one has been made).
Thanks for any help. I believe I am not composing the pipeline correctly but have run out of ideas to try. I have left only the one criteria to test if it works, if I can make it work with one I can add the others later.

John

8

Did you try the pipeline I suggested ?

9

Yes, no reaction, as if there was no change registered.
John

10

Sorry, I left a bit out:

const pipeline =[
  {$match: {available: true}}
  ];
  const changeStream = Instructor.watch(pipeline, { fullDocument: 'updateLookup' });

  if (changeStream) {console.log('Change stream started.');}

  changeStream.on('change', data => console.log(data));

The if statement tells me that a changeStream object has been created. “'Change stream started.” gets logged to the console. But on changing a document in the database I get no response.

Thanks
John

11

With a blank pipeline have you logged the document raised in the event to the console to validate the paths in your query?

12

Thanks John,
Yes I have, the expected document was logged and the result for the updated field was correct. All is working fine. I just don’t seem to be able to do what I want which is only check for change in documents which match certain criteria.
All suggestions appreciated.
John

13

Ok, I suggested that as we tested change stream pipelines and when we swapped out to full document and did a few other changes the shape of the event changed so our pipeline no longer matched so filtered out all documents!

Your pipeline is:

const pipeline =[
    {$match: {available: true}}
  ];

From one of your above posts? Should this not be:

const pipeline =[
    {$match: {'fullDocument.available': true}}
  ];

as @chris suggested above?

14

Thanks fo ryour time that did the trick. The next stumbling block is to match a language as well. The required language is an element of an array (where the subject speaks mor than one language.
language:[“english”, “german”, “french”]
I am not sure how to formulate the request.

const pipeline =[
    {
      $match: {
        'fullDocument.available': true,
        'fullDocument.language': {$in:['english']}
    }
    }
  ];

Doesnt work.
What is the correct formulation please? The documentation is quaite vague.
On another note, where can I find documentation on the fullDocument notation and the allowed options in the watch(pipeline, options) function.
Thanks again for helping me learn.
Regards
John

15 
db.getCollection("Test").find({'fullDocument.languages':'English'})

This will search for a document with an element in the array matching the filter condition.

You could use the $in operator the other way around, so if you had a list of documents with courses or something, you could find any that are in one of a number of values, i.e.

db.getCollection("Test").find({'fullDocument.courseLanguage':$in:['Esperanto':'English']})

See here:

Under the array section.

16

Thank you John, things are coming along. I have now hit the next hurdle. I don’t seem to be able to access the data retrieved.
In Visual Studio code I have been able to access the changed fields and the following has been logged to the console.

updateDescription: {
    updatedFields: { 'location.coordinates.1': '2' },
    removedFields: [],
    truncatedArrays: []
  }

But I cannot get at the data in the updatedFields field.

I have tried this:

const changedData = (data.updateDescription.updatedFields);
    console.log(changedData);

which results in this:
{ 'location.coordinates.1': '2' }

but:
console.log(changedData.location);

is undefined, I can’t get at the data.
I want to be able to access the array “location” and extract the two values, location.coordinates[0] and location.coordinates.[1]

The data object behaves like a normal object allowing me to get at its values through dot notation but only down to updatedFields. Then I am blocked.

Again, any light you can shine on this will be greatly appreciated. there is some fundamental logic that I don’t understand.

John

17

Sorry all,
Looking for the complex solution instead of the simple one. The values are accessed through the fullDocument object and not through the updateDescription object.
John