2.19 breaks projections?

Working with Data Drivers
,
1

Just upgraded to 2.19 and any query using .Project is failing. Originally I thought it may have been due to the AllowedTypes feature, but forcing LinqV2 seems to fix the issue.

Is there a new syntax or requirement in 2.19 for Project queries? If I remove the .Project, the query returns the entire document as expected. If I .Project, then I just get the default back. Projecting to anonymous types also fails.

For example:

var query = Builders<MongoMessage>.Filter.In(m => m.ObjectId, messageIds);
      var toDelete = await cs.MessageCollection.Find(query).Project(m => new DeleteInfoWrapper(m.ObjectId, m.GridFsObjectId, m.AttachmentGridFsObjectId)).ToListAsync().ConfigureAwait(false);
2

Hi, @Nick_Judson,

Thank you for reaching out to us about this issue.

Project should continue to work with LINQ3 and I was able to get your code snippet working with anonymous types:

using System;
using MongoDB.Bson;
using MongoDB.Driver;
using MongoDB.Driver.Linq;

var client = new MongoClient();
var db = client.GetDatabase("test");
var coll = db.GetCollection<MongoMessage>("messages");

var messageIds = new[] { ObjectId.Empty };
var filter = Builders<MongoMessage>.Filter.In(m => m.ObjectId, messageIds);
var query = coll.Find(filter).Project(m => new {m.ObjectId, m.GridFsObjectId, m.AttachmentGridFsObjectId});

Console.WriteLine(query);

public class MongoMessage
{
    public ObjectId ObjectId { get; set; }
    public ObjectId GridFsObjectId { get; set; }
    public ObjectId AttachmentGridFsObjectId { get; set; }
}

I was also able to get it working with the wrapper class:

var query = coll.Find(filter).Project(m => new DeleteInfoWrapper(m.ObjectId, m.GridFsObjectId, m.AttachmentGridFsObjectId));

with the wrapper defined as:

public class DeleteInfoWrapper
{
    public ObjectId ObjectId { get; }
    public ObjectId GridFsObjectId { get; }
    public ObjectId AttachmentGridFsObjectId { get; }

    public DeleteInfoWrapper(ObjectId objectId, ObjectId gridFsObjectId, ObjectId attachmentGridFsObjectId)
    {
        ObjectId = objectId;
        GridFsObjectId = gridFsObjectId;
        AttachmentGridFsObjectId = attachmentGridFsObjectId;
    }
}

One difference between LINQ2 and LINQ3 is that LINQ3 now requires a matching property name for each constructor parameter. If you remove lines 3 to 5 of the wrapper above, LINQ3 will fail with the following exception:

MongoDB.Driver.Linq.ExpressionNotSupportedException: Expression not supported: new DeleteInfoWrapper(m.ObjectId, m.GridFsObjectId, m.AttachmentGridFsObjectId) because constructor parameter objectId does not match any property.

Please ensure that DeleteInfoWrapper has properties matching the constructor parameters (differing only by case). If this is not the root cause of the problem, please provided a small, self-contained repro (including class definitions) so that we can investigate further. The full exception message including stack trace would also help.

Sincerely,
James

1 Like
3

In one case, no exception is thrown, but a Default object is returned. In two cases it appears the following exception is thrown:

System.InvalidOperationException : ReadName can only be called when State is Name, not when State is EndOfDocument.
   at MongoDB.Bson.IO.BsonReader.ThrowInvalidState(String methodName, BsonReaderState[] validStates)
   at MongoDB.Bson.IO.BsonBinaryReader.ReadName(INameDecoder nameDecoder)
   at MongoDB.Bson.IO.IBsonReaderExtensions.ReadName(IBsonReader reader)
   at MongoDB.Bson.IO.IBsonReaderExtensions.VerifyName(IBsonReader reader, String expectedName)
   at MongoDB.Bson.IO.IBsonReaderExtensions.ReadName(IBsonReader reader, String name)
   at MongoDB.Driver.Linq.Linq3Implementation.Serializers.WrappedValueSerializer`1.Deserialize(BsonDeserializationContext context, BsonDeserializationArgs args)
   at MongoDB.Bson.Serialization.IBsonSerializerExtensions.Deserialize[TValue](IBsonSerializer`1 serializer, BsonDeserializationContext context)
   at MongoDB.Driver.Core.Operations.CursorBatchDeserializationHelper.DeserializeBatch[TDocument](RawBsonArray batch, IBsonSerializer`1 documentSerializer, MessageEncoderSettings messageEncoderSettings)
   at MongoDB.Driver.Core.Operations.FindOperation`1.CreateFirstCursorBatch(BsonDocument cursorDocument)
   at MongoDB.Driver.Core.Operations.FindOperation`1.CreateCursor(IChannelSourceHandle channelSource, IChannelHandle channel, BsonDocument commandResult)
   at MongoDB.Driver.Core.Operations.FindOperation`1.<ExecuteAsync>d__129.MoveNext()

In the projection using the wrapper object, the constructor parameters do match the properties.

I will have to investigate further…

4

Hi, Nick,

Thank you for providing the exception and stack trace. The WrappedValueSerializer indicates that you are trying to deserialize a primitive value wrapped in a _v. For example if the driver wanted the server to return the value 42, the driver would issue a query such that the resulting int was wrapped as { "_v": 42 }. This is because BSON requires a top-level BSON document and doesn’t allow top-level primitives.

The InvalidOperationException indicates that the driver attempted to deserialize the _v field, but found the BSON EndOfDocument marker. Something went wrong in the query generation or execution, but we can’t tell exactly what. A self-contained repro would really help in understanding the root cause of the problem.

Thanks in advance,
James

5

@James_Kovacs - I have reproduced the issue. My document objects all make use of [BsonElement("xxx")] in order to shrink the document size.

It appears that if you use this attribute, projections are broken. If I remove this attribute, my projections start to work again (although [BsonId] still doesn’t seem right).

Try defining MongoMessage as and re-run your test

    [BsonId]
    [DataMember]
    public ObjectId ObjectId { get; set; }

    [BsonElement("grid")]
    [DataMember]
    [BsonIgnoreIfDefault]
    public ObjectId GridFsObjectId { get; set; }

    [BsonElement("agrd")]
    [DataMember]
    [BsonIgnoreIfDefault]
    public ObjectId AttachmentGridFsObjectId { get; set; }
6

With respect to the stack trace above, it appears to be related to projecting the ObjectId property.

 var upperLimitObjectId = await cs.MessageCollection.Find(filter).Sort(Builders<MongoMessage>.Sort.Descending(m => m.ObjectId)).Limit(1).Project(m => m.ObjectId).FirstOrDefaultAsync().ConfigureAwait(false);

The non-throwing issue appears related to projecting any property with a POCO class name different from the document property name, where the BsonElement attribute is used to set the document property name.

7

Hi, Nick,

I tried adding [BsonElement] with an abbreviated field name, but could not reproduce the issue. Here is my complete repro:

using System;
using System.Runtime.Serialization;
using MongoDB.Bson;
using MongoDB.Bson.Serialization.Attributes;
using MongoDB.Driver;

var client = new MongoClient();
var db = client.GetDatabase("test");
var coll = db.GetCollection<MongoMessage>("messages");

var messageIds = new[] { ObjectId.Empty };
var filter = Builders<MongoMessage>.Filter.In(m => m.ObjectId, messageIds);
var query = coll.Find(filter).Project(m => new DeleteInfoWrapper(m.ObjectId, m.GridFsObjectId, m.AttachmentGridFsObjectId));

Console.WriteLine(query);

public class MongoMessage
{
    [BsonId]
    [DataMember]
    public ObjectId ObjectId { get; set; }

    [BsonElement("grid")]
    [DataMember]
    [BsonIgnoreIfDefault]
    public ObjectId GridFsObjectId { get; set; }

    [BsonElement("agrd")]
    [DataMember]
    [BsonIgnoreIfDefault]
    public ObjectId AttachmentGridFsObjectId { get; set; }
}

public class DeleteInfoWrapper
{
    public ObjectId ObjectId { get; }
    public ObjectId GridFsObjectId { get; }
    public ObjectId AttachmentGridFsObjectId { get; }

    public DeleteInfoWrapper(ObjectId objectId, ObjectId gridFsObjectId, ObjectId attachmentGridFsObjectId)
    {
        ObjectId = objectId;
        GridFsObjectId = gridFsObjectId;
        AttachmentGridFsObjectId = attachmentGridFsObjectId;
    }
}

The resulting output is:

find({ "_id" : { "$in" : [ObjectId("000000000000000000000000")] } }, { "ObjectId" : "$_id", "GridFsObjectId" : "$grid", "AttachmentGridFsObjectId" : "$agrd", "_id" : 0 })

[BsonId] and [BsonElement] attributes are being respected and generating the correct MQL.

Please take my repro as a starting point and modify it to reproduce your issue. Once we have been able to reproduce the issue in the standalone repro, we can file a CSHARP ticket to find the root cause and resolve it. Thank you for your collaboration in this investigation.

Sincerely,
James

8

Looks like the issue only shows up with server versions 4.2 and below. 4.4 and 6.0 seem to work as expected. 4.2 isn’t EOL yet is it?

Here is the repro:

using MongoDB.Driver.Core.Configuration;
using MongoDB.Driver.Linq;
using MongoDB.Driver;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using DnsClient;
using MongoDB.Bson;
using MongoDB.Bson.Serialization.Attributes;
using System.Runtime.Serialization;

namespace Mongo219Repro
{
    internal class Program
    {
        static async Task Main(string[] args)
        {
            var database = GetClient().GetDatabase("Test");
            var messageCollection = database.GetCollection<MongoMessage>($"Test_Message");

            var ckIndex = new CreateIndexModel<MongoMessage>(Builders<MongoMessage>.IndexKeys.Ascending(m => m.CorrelationKey), new CreateIndexOptions { Name = "IX_CorrelationKey", Sparse = true });
            await messageCollection.Indexes.CreateOneAsync(ckIndex).ConfigureAwait(false);

            var mongoMessage = new MongoMessage
            {
                AttachmentDataBytes = null,
                AttachmentSize = 1234,
                QueueType = 4,
                Data = "This is test data",
                DataType = typeof(string).FullName,
                DataSize = 55,
                SourceUri = "manual",
                CorrelationKey = Guid.NewGuid(),
                AccountNumber = "12345",
                ModifiedDateTime = DateTime.UtcNow,
                ExamId = "1234",
                HasProcessingHistory = true,
                Keywords = new List<string>(),
                PatientId = "1234",
                MessageType = "1234",
                MessageDateTime = DateTime.UtcNow,
                MessageDateTimeUtcOffset = 2,
                MessageIdentifier = "1234",
                SendingFacility = "1234",
                MessageExtension = "1234",
                PurgedOnProcessed = false,
                AttachmentGridFsObjectId = ObjectId.GenerateNewId(),
                GridFsObjectId = ObjectId.GenerateNewId(),
                ObjectId = ObjectId.GenerateNewId(),
            };

            await messageCollection.InsertOneAsync(mongoMessage).ConfigureAwait(false);

            var query = Builders<MongoMessage>.Filter.In(m => m.ObjectId, new[] { mongoMessage.ObjectId });
            var noProjection = await messageCollection.Find(query).FirstOrDefaultAsync().ConfigureAwait(false);

            Console.WriteLine($"Without projection: Expecting {mongoMessage.ObjectId}, got {noProjection.ObjectId}");
            Console.WriteLine($"Without projection: Expecting {mongoMessage.AttachmentGridFsObjectId}, got {noProjection.AttachmentGridFsObjectId}");
            Console.WriteLine($"Without projection: Expecting {mongoMessage.GridFsObjectId}, got {noProjection.GridFsObjectId}");

            var withProjection = await messageCollection.Find(query).Project(m => new { m.ObjectId, m.GridFsObjectId, m.AttachmentGridFsObjectId }).FirstOrDefaultAsync().ConfigureAwait(false);

            Console.WriteLine($"With projection: Expecting {mongoMessage.ObjectId}, got {withProjection.ObjectId}");
            Console.WriteLine($"With projection: Expecting {mongoMessage.AttachmentGridFsObjectId}, got {withProjection.AttachmentGridFsObjectId}");
            Console.WriteLine($"With projection: Expecting {mongoMessage.GridFsObjectId}, got {withProjection.GridFsObjectId}");

            var filter = Builders<MongoMessage>.Filter.Empty;
            try
            {
                var upperLimitObjectId = await messageCollection.Find(filter).Sort(Builders<MongoMessage>.Sort.Descending(m => m.ObjectId)).Limit(1).Project(m => m.ObjectId).FirstOrDefaultAsync().ConfigureAwait(false);
            }
            catch (Exception ex) 
            {
                Console.WriteLine($"Exception thrown in projection: {ex.Message}");
            }

            Console.ReadLine();
        }

        private static MongoClient GetClient()
        {
            var settings = MongoClientSettings.FromConnectionString("mongodb://localhost:27017");

            // default settings
            settings.ApplicationName = "Test";
            settings.ConnectTimeout = TimeSpan.FromSeconds(10);
            settings.SocketTimeout = TimeSpan.FromSeconds(60);  // query timeout

            // settings.UseSsl = false;
            settings.SocketTimeout = TimeSpan.FromSeconds(60);
            settings.ServerSelectionTimeout = TimeSpan.FromSeconds(10);
            //settings.LinqProvider = LinqProvider.V2;

            if (!string.IsNullOrWhiteSpace(settings.ReplicaSetName))
                settings.WriteConcern = WriteConcern.WMajority;

            return new MongoClient(settings);
        }

        [DataContract(IsReference = true)]
        [BsonIgnoreExtraElements]
        public class MongoMessage
        {
            [BsonId]
            [DataMember]
            public ObjectId ObjectId { get; set; }

            [BsonElement("mdt")]
            [BsonIgnoreIfNull]
            [DataMember]
            public DateTime? ModifiedDateTime { get; set; }

            [BsonElement("kw")]
            [DataMember]
            public List<string> Keywords { get; set; }

            [BsonElement("grid")]
            [DataMember]
            [BsonIgnoreIfDefault]
            public ObjectId GridFsObjectId { get; set; }

            [BsonElement("agrd")]
            [DataMember]
            [BsonIgnoreIfDefault]
            public ObjectId AttachmentGridFsObjectId { get; set; }

            [BsonElement("data")]
            [DataMember]
            [BsonIgnoreIfNull]
            public string Data { get; set; }

            [BsonElement("sz")]
            [DataMember]
            public int DataSize { get; set; }

            [BsonElement("tp")]
            [DataMember]
            [BsonIgnoreIfNull]
            public string DataType { get; set; }

            [BsonElement("qt")]
            [DataMember]
            public int QueueType { get; set; }

            [BsonElement("ck")]
            [DataMember]
            [BsonIgnoreIfNull]
            public Guid? CorrelationKey { get; set; }

            [BsonElement("dk")]
            [DataMember]
            [BsonIgnoreIfNull]
            public Guid? SourceDeviceKey { get; set; }

            [BsonElement("ph")]
            [DataMember]
            public bool HasProcessingHistory { get; set; }

            [BsonElement("src")]
            [DataMember]
            [BsonIgnoreIfNull]
            public string SourceUri { get; set; }

            [BsonElement("asz")]
            [DataMember]
            [BsonIgnoreIfDefault]
            public int AttachmentSize { get; set; }

            [BsonElement("abt")]
            [DataMember]
            [BsonIgnoreIfNull]
            public byte[] AttachmentDataBytes { get; set; }

            [BsonElement("me")]
            [DataMember]
            [BsonIgnoreIfNull]
            public string MessageExtension { get; set; }

            [BsonElement("__1")]
            [DataMember]
            [BsonIgnoreIfNull]
            public string MessageIdentifier { get; set; }

            [BsonElement("__2")]
            [DataMember]
            [BsonIgnoreIfNull]
            public string MessageType { get; set; }

            [BsonElement("__3")]
            [DataMember]
            [BsonIgnoreIfNull]
            public string PatientId { get; set; }

            [BsonElement("__4")]
            [DataMember]
            [BsonIgnoreIfNull]
            public string AccountNumber { get; set; }

            [BsonElement("__5")]
            [DataMember]
            [BsonIgnoreIfNull]
            public string ExamId { get; set; }

            [BsonElement("__6")]
            [DataMember]
            [BsonIgnoreIfNull]
            // [BsonDateTimeOptions(Kind = DateTimeKind.Local)] <-- we ToLocal this value as mongoDB by default converts datetimes to UTC on the way in (but not the way out)
            public DateTime? MessageDateTime { get; set; }

            [BsonElement("__61")]
            [DataMember]
            [BsonIgnoreIfNull]
            public short? MessageDateTimeUtcOffset { get; set; }

            [BsonElement("__7")]
            [DataMember]
            [BsonIgnoreIfNull]
            public string SendingFacility { get; set; }

            // null out message payloads on processed/filtered etc.
            [BsonElement("pop")]
            [DataMember]
            public bool PurgedOnProcessed { get; set; }

            #region Deserialized Object

            public void ClearMessageObject()
            {
                Data = null;
            }

            #endregion
        }
    }
}

This is the output when run against 4.2 or lower:

image
image1114×246 19.9 KB

This is the output when run against 4.4 and above:

image
image977×240 17.4 KB

9

@James_Kovacs - are you able to reproduce this?

10

Hi, Nick,

Thank you for your patience and thank you for filing CSHARP-4507. I was able to reproduce the issue with the provided code along with the additional detail that it only fails on MongoDB 4.2 and earlier. I will explain the problem and then some potential workarounds while we work on a fix.

To answer your first question, MongoDB 4.2 is still a supported server version though it will reach end-of-life in April 2023. See our support policy for full details.

Now let’s discuss the root cause of this issue. The problem stems from the renaming of the fields in your LINQ query. The following code will display the MQL sent to the server. The same MQL is sent to MongoDB regardless of the server version.

var query = coll.Find(filter).Project(m => new DeleteInfoWrapper(m.ObjectId, m.GridFsObjectId, m.AttachmentGridFsObjectId));
Console.WriteLine(query);

The resulting MQL is:

find({ "_id" : { "$in" : [ObjectId("63d856bb965b1cf474a00a6b"), ObjectId("63d856ba965b1cf474a00a68")] } }, { "ObjectId" : "$_id", "GridFsObjectId" : "$grid", "AttachmentGridFsObjectId" : "$agrd", "_id" : 0 })

The second argument to the find command is the projection. Note the syntax "CsharpFieldName": "$databaseFieldName". For example "GridFsObjectId": "$grid". This is where the problem lies. In MongoDB 4.4 and later, you could use this $fieldName syntax to rename fields in projections - whether those projections were part of a find operation or an aggregation pipeline.

However in MongoDB 4.2 and earlier, you could only use this syntax in aggregation pipelines, but not in find projections. Find projections only allowed the older, simpler syntax of fieldName: 1 to include a field. This is further complicated by the fact that for backwards compatibility, fieldName: VALUE where VALUE was truthy in the JavaScript sense. This meant that 0, false, null, and undefined are interpreted as false and pretty much everything else was interpreted as true. Thus MongoDB 4.2 (using find) sees "GridFsObjectId": "$grid" as "GridFsObjectId": true. Since there is no field in the document named GridFsObjectId, the field is omitted leading to the observed behaviour.

In MongoDB 4.4, we enhanced the find projection to use the same syntax as the aggregation pipeline. Thus MongoDB 4.4 sees "GridFsObjectId": "$grid" as "GridFsObjectId": "$grid" and correctly renames the field grid (in the database) to GridFsObjectId in the returned document.

Possible workarounds for this issue (in no particular order):

  • Upgrade to MongoDB 4.4 or later.
  • Continue using LINQ2.
  • Refactor your Fluent Find queries to Fluent Aggregate.

The first two should be self-explanatory. I will note that LINQ3 has greatly enhanced capabilities including support for new aggregation features. LINQ2 will be deprecated in an upcoming version and removed in the 3.0.0 driver. We have not announced a public timeline for the 3.0.0 driver.

Refactoring to use Fluent Aggregation is probably the most straightforward. You can use the same FilterDefinition<> and ProjectionDefinition<,> as you are currently using with Find/Project. Rather than coll.Find(filter).Project(projection) you would instead use coll.Aggregate().Match(filter).Project(projection).

var query = coll.Aggregate().Match(filter).Project(m => new DeleteInfoWrapper(m.ObjectId, m.GridFsObjectId, m.AttachmentGridFsObjectId));
Console.WriteLine(query);

The resulting MQL produces an aggregation pipeline rather than a find command, but the query results are the same:

aggregate([{ "$match" : { "_id" : { "$in" : [ObjectId("63d856bb965b1cf474a00a6b"), ObjectId("63d856ba965b1cf474a00a68")] } } }, { "$project" : { "ObjectId" : "$_id", "GridFsObjectId" : "$grid", "AttachmentGridFsObjectId" : "$agrd", "_id" : 0 } }])

This aggregation pipeline - including the projection - will work on even very old server versions. I tested it on MongoDB 3.6 and it produced the correct results.

Please follow CSHARP-4507 for the fix. Thank you again for reporting it. Let us know if you have any additional questions.

Sincerely,
James

11

Just a FYI for anyone finding this thread. This also affects using MongoDB.Driver 2.19.0 with Azure Cosmos for MongoDB API and using Find().Project() queries. (same error, same stack trace)

12

Hi Nick,

I am also experiencing a breaking change going from .Net Driver 2.18 → 2.19.1 (the most recent one available on NuGet).

I have the following simple code in a method to retrieve a smaller version of a bigger class object from the database. This exact code runs perfect when i force Linq to use V2 instead of V3.
Since V2 will be deprecated soon, I don’t know why Linq V3 (which is in 2.19) is causing this error:

MongoDB.Driver.Linq.ExpressionNotSupportedException: ‘Expression not supported:
f.Categories.Where(c => (c.Name == “Functions”)).FirstOrDefault().’

This is the method that works perfect in V2 but throws aforementioned error in V3.

public async Task<List<ShortFOW>> GetAllFOWOnlyFunctions(){
 ProjectionDefinition<BigFOW, ShortFOW> project = Builders<BigFOW>.Projection.Expression( f =>
  new ShortFOW()
  {
   Id = f.Id,
   Name = f.Name,
   // This next line: causes the error with the where clause:
   Functions = f.Categories.Where(c => c.Name == "SomeName").FirstOrDefault() 
 });

var results = await FOWCollection
                    .Find(FilterBuilder.Empty)
                    .SortBy(f => f.Name)
                    .Project(project)
                    .ToListAsync();

return results;
}

Please help me out or point me in the correct direction if you can!
Thanks in advance!

13

I believe that in prior versions, a projection like this would actually be handled by the driver (client-side), whereas in the latest version it’s the DB itself that is handling the projection.

I’m not sure you can have a .Where in your projection like that. Does it work if it’s just f.Categories?

14

Hi Nick,
Thank you for your quick reply. I have just tested the projection without the where clause: i.e. f.Categories
This works just fine, so the problem is when i try to only get 1 type of category back instead of all the List categories.

Also, 2.19 seems to work just fine when forced to use Linq V2.
Do you have any alternative way to achieve my goal? Which is exactly what the 2.18 driver did.

public class BigFOW
{
    [BsonId]
    [BsonRepresentation(BsonType.ObjectId)]
    public string Id { get; set; }
    public string Name                  { get; set; } = string.Empty;
    public int NumberOfRequests { get; set; } = 0;
    public List<Category> Categories    { get; set; } = new();
}

public class ShortFOW
{
    public string Id { get; set; }
    public string Name { get; set; }
    public Category Functions { get; set; } = new ();
}

Thanks again Nick!

15

I suspect your previous query was partly happening on the client-side within the driver. I’m not spending much time in Mongo these days but your query looks like it could probably be done with an aggregation pipeline?

Maybe check out How to retrieve only a subset of an array of embedded documents in MongoDB? - Stack Overflow

16

I see, so it seems i need to take a different route.
Thanks for pointing me in the right direction!
Kind regards,

Am

17

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.