Strange Behavior of MongoDB queries in numeric string data. find() and aggregation pipeline $match

Working with Data Node.js Frameworks
, ,
1

I am working with Data.
I have applied the find() query and aggregation pipeline for the same data.

Now in find() query, it is loosely checking the data while in the aggregation pipeline, in $match it is checking strictly. Here I am giving the example.

For example in MongoDB data is stored like status: ‘2’. That means the value of status is in a String(To be a precise numeric string). Now when I apply find() query on status like find(status: 2), it will return all the data with status 2. Please note that in find() query I am passing the Number, not the string, and still it is working fine.

Now the same thing I have applied in the aggregation pipeline in $match. Example. $match : {status : 2}. then it is not returning anything !!!

This is such weird behavior.
You can try these two queries in any kind of numeric string data. Stored the value in a Numeric string. In the query parameter pass the numeric value in both queries, find() and aggregation pipeline $match.

Ping me if you need further explanation.
Also, correct me if I am wrong.

2

While your title reads Strange behaviour of MongoDB, I would be surprised if that would be case. As far as I know, MongoDB does not do automatic type conversion.

What I suspect is that you are using an abstraction layer, something like mongoose with a schema that defines status as being of type string and when you use the something like mongoose to query it silently does type conversion for you. And when you do not use the something like mongoose there is not such conversion, so you do not $match anything.

If my hypothesis of something like mongoose is wrong please share sample documents and your code.

3

Your suspect is right. I am using a mongoose.

But still, In mongoose also there should not be automatic type conversion. It will create chaos.
While applying the query in the mongo shell, we know that there will NOT be automatic type conversion. So we will expect that it will be the same for all types of abstraction layers.

Personally, I experienced lots of trouble due to this behavior.

Now in mongoose, I don’t know what is behind the scene, while I apply the mongoose query.
So according to you, is it Bug or is it expected?? Can you please say your views with justification?

4

I beg to differ. I do not know much about mongoose but I know it uses schema. In your schema you define the type of a field. When you define a type you then have 2 choices:
1 - you generate an error when you use the wrong type
2 - you convert the value to the correct type if such a conversion is possible

In your first post you do not mentioned that you get an error. You mention about the string “2” to be interchangeable in one case and not in the other. To me it really looks like there is type conversion happening in one case and not the other.

If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck

Expected, but I do not much about mongoose.

Share your schema, share sample data and share your code. Others, more savvy with mongoose will be able to jump. My views are clear: if you pass a number and you get a string then there is type conversion and it is not the mongodb native driver that does it.

5

It looks like I was right about type conversion in mongoose.

6

See I am talking about retrieving data. Not about inserting data.

The schema is perfect. The data storing process is perfect.

The problem is when I apply the find() query and $match, then it is showing this behavior.

In data storing, it is not doing any type of conversion. When I apply a parameter in the query then while searching in the database, it is doing type conversion in the find() query only. In $match is not doing type conversion while searching.

7

We are making progress.

I understand but if one part of mongoose (data insertion using the schema) does type conversion it would make sense that the other part (data retrieval using the schema) does the same data conversion.

const testSchema = new mongoose.Schema(
{
    field : { type: String }
}

test = { field : 5 } /* 5 the number, not the string */

Test.insertOne( test )

/* Document is stored as { "field" : "5" } 5 the string because of type
   conversion */

/* In the following findOne you really want result to hold the Document
   inserted.  Do you? I think you do and it would not make sense if you
   did not. */

result = Test.findOne( test ) 

/* If result is a document with field:"5", the string,
   then type conversion is done. */

I did not imply that either was wrong when I asked you to share your schema, documents and code. I think it could help comprehension because we could use your real schema and document rather than the one I made up above with approximate syntax.

The SO post I shared seems to indicate the contrary.

We are making progress because we went from

to

It makes sense that mongoose would not do type conversion in $match. In a lot of $match, there is no schema related to the $match-ed documents. In a pipeline, documents are altered or even generated (by $group). It would take a lot of complicated code to determine which schema to use, if any, to do type conversion. And it will be very confusing. For 1 $match you have conversion because we have a schema and for another one there is none because it is generated document.

My conclusion is

1 - type conversion occurs (may be not in your case) when documents are inserted because of the schema

Personally, a warning or error would be better.

2 - type conversion occurs in simple queries where it is clear which schema is used

It makes sense because conversion occurs at insertion.

3 - type conversion does not occurs in pipeline for $match because it would be hard to do and it will be very confusion because it cannot be done on some of the $match

4 - storing numbers as string is a bad idea

A string with a couple of digits will take more space than the number and it is slower to compare. Numbers as strings are not in the natural sort order “2” > “111” but 2 < 111.

5 - I am sure about the following.

Hopefully, mongoose has a way for you to communicate and ease your frustration.