Why does Mongoose remove the key when specified in the schema?

Randall_Barker · September 25, 2022, 5:20am

I have spent the past hour or two puzzled over why my data doesn’t seem to be coming from the database. I’ve narrowed it down to being the result of the schema, but I’m stumped as to how I get around that.

Here’s what happens:
When I specify the schema for an array of objects (may be doing this wrong), that particular key and all the data with it get stripped from any query result. When I remove this from the schema, it suddenly works again.

Here’s why I can’t just keep it out: I need an ObjectId generated for all objects in the array, and only way I can do that without manually supplying one is by using a subdocument schema. I’ve tried reading into documentation, looking this up various ways, looking at examples, and none of it seems to be giving me the solution I’m looking for. I can insert data just fine and validation passes, but it just gets removed when trying to read it.

Schema:

const Form = new Schema({
    schema: { type: Object, required: true }, // This is a form schema unrelated to mongo
    title: { type: String, required: false },
    subtitle: { type: String, required: false },
    acceptingResponses: { type: Boolean, required: false },
    userSubmissions:  [ // Ends up being an array with objects each containing a key and following this schema
        new Schema({
            status: { type: String, required: true },
            response: { type: Object, required: true },
            respondant: { type: Object, required: true },
        },{
            _id: true,
            required: true
        })
    ]
},{
    strict: false
});

I’ve also tried setting userSubmissions to { type: Array, required: true } with the same issue.

SourabhBagrecha · September 29, 2022, 4:37am

Hi @Randall_Barker,
I tried to reproduce your issue, but the validation got failed while inserting can you help me by sharing the following details in order to better diagnose the issue?

The version of mongoose you are using in your application server
The version of MongoDB instance that you are connecting your app with
The version of Node.js running your application server

I am assuming you mean the array of userSubmissions by "it" here, is that correct?
Can you also share the commands you use to create and query the Form documents or a self-contained code example that reproduces this issue?

If you have any doubts, please feel free to reach out to us.

Thanks and Regards.
Sourabh Bagrecha,
MongoDB

Randall_Barker · September 29, 2022, 4:52am

I may have left this detail out that may have been very important. Insertion is done on a push of data that was manually entered, not entered through mongoose. Data is pushed into the userSubmissions array which may be skipped on validation, unsure of this behavior.

models.Forms.updateOne({_id: formId},{$push: {userSubmissions: responseData}}).then(resolve).catch(reject);

After one submission, the array looks as follows:

"userSubmissions": [{
        "respondant": {
            "id": "62436ac8a19e9b08e18dab88",
            "userId": 12345,
            "username": "######"
        },
        "status": "pending",
        "response": {
            "question1": ["answer3","answer1","answer7"],
            "question2": "answer4",
            "question3": "being translucent"
        }
    }],

Randall_Barker · September 29, 2022, 4:54am

For whatever reason, I cannot edit my reply.
My wording may have been slightly confusing. The document was created through manual entry into MongoDB Compass. Responses are pushed into the array. If this isn’t validated when being pushed in, that may very well explain why I’m not receiving any error.

SourabhBagrecha · October 7, 2022, 3:50am

Hi @Randall_Barker,
I have reproduced the issue and it seems like mongoose indeed doesn’t return the array when a key named "schema" exists in the document.
I have raised a similar issue a few days back about the "schema" key causing failed validations even with valid documents. And this issue has already been added to v6.6.6 milestones, check out the following to learn more:

github.com/Automattic/mongoose

Array validation getting failed because of the "schema" key inside the Schema

opened 06:36AM - 28 Sep 22 UTC

closed 05:32PM - 20 Oct 22 UTC

sourabhbagrecha

confirmed-bug

### Prerequisites - [X] I have written a descriptive issue title - [X] I hav…e searched existing issues to ensure the bug has not already been reported ### Mongoose version 6.2.2 ### Node.js version 16.14.2 ### MongoDB server version 4.4.6 ### Description Hi Team, I have observed a bug in the schema class/validator provided by mongoose. When a schema definition contains a key named: `schema` and another key containing an array of documents, the validation gets failed even with completely valid documents. ### Steps to Reproduce #### Step 1. Defining a schema that contains a key named `schema` and another key that embeds an array. ```javascript const AuthorSchema = new Schema({ fullName: { type: "String", required: true }, }); const BookSchema = new Schema({ schema: { type: "String", required: true }, title: { type: "String", required: true }, authors: [AuthorSchema], }); const Book = model("book", BookSchema); ``` #### Step 2. Insert a document in the collection ```javascript const book = await Book.create({ schema: "design", authors: [{ fullName: "Sourabh Bagrecha" }], title: "The power of JavaScript", }); ``` #### Step 3. Which throws the following error: ```bash mongoose/node_modules/mongoose/lib/document.js:3055 this.$__.validationError = new ValidationError(this); ^ ValidationError: book validation failed: authors: Cast to Array failed for value "[ { fullName: 'Sourabh Bagrecha' } ]" (type Array) at path "authors" because of "TypeError" at model.Document.invalidate (/Users/sourabh/Work/repro/mongoose/node_modules/mongoose/lib/document.js:3055:32) ...call stack clipped... { errors: { authors: CastError: Cast to Array failed for value "[ { fullName: 'Sourabh Bagrecha' } ]" (type Array) at path "authors" because of "TypeError" at model.$set (/Users/sourabh/Work/repro/mongoose/node_modules/mongoose/lib/document.js:1417:9) ...call stack clipped... { stringValue: `"[ { fullName: 'Sourabh Bagrecha' } ]"`, messageFormat: undefined, kind: 'Array', value: [ { fullName: 'Sourabh Bagrecha' } ], path: 'authors', reason: TypeError: doc.schema.path is not a function at new MongooseDocumentArray (/Users/sourabh/Work/repro/mongoose/node_modules/mongoose/lib/types/DocumentArray/index.js:60:47) ...call stack clipped... valueType: 'Array' } }, _message: 'book validation failed' } ``` Note that the validation error does not occur when: - The field schema is removed - The field schema is renamed to Schema (uppercase S) - The array field is removed The validation error occurs only when both the schema field AND the array field exist in the schema #### Step 4. Now if we update the schema by uppercasing the key `Schema` from lowercased `schema` ```javascript const BookSchema = new Schema({ Schema: { type: "String", required: true }, // ^^^^^^^ title: { type: "String", required: true }, authors: [AuthorSchema], }); const book = await Book.create({ Schema: "design", // ^^^^^^^ authors: [{ fullName: "Sourabh Bagrecha" }], title: "The power of JavaScript", }); ``` #### Step 5. It will work completely fine if schema was spelled differently (e.g. with an uppercase S, i.e.: `Schema`) ```javascript { "Schema": "design", "title": "The power of JavaScript", "authors": [ { "fullName": "Sourabh Bagrecha", "_id": new ObjectId("633343a2396c4534d5a70e19") } ], "_id": new ObjectId("633343a2396c4534d5a70e18"), "__v": 0 } ``` ### Expected Behavior I understand that `schema` is a reserved keyword that can be used to fetch the document's schema. <img width="200" alt="image" src="https://user-images.githubusercontent.com/27056663/192612592-e80a56b5-4a42-4a1b-8d72-db6ca2d67432.png"> But in that case, it should throw a warning(compile-time) while we are specifying the `schema` key in our schema definition. Also, the validation is only throwing the error, when we are specifying an embedded array of documents in our definition. In all the other cases, having the `schema` key in our definition doesn't throw any validation error at the time of insert/update. The expected behavior IMO would be a consistent error irrespective of the embedded array.

However, when I change the key name: "schema" to something else, it starts working perfectly fine.
Therefore, as a quick fix if you could change the key name "schema" to "formSchema" or uppercased "Schema", it will work as expected.

Also, please avoid skipping mongoose’s schema validation process, and don’t insert documents manually unless that’s really needed.

If you have any doubts, please feel free to reach out to us.

Thanks and Regards.
Sourabh Bagrecha,
MongoDB

system · October 12, 2022, 3:51am

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.