Ok ! I found my issue, I need to remove the bson.A wrapped around my bson.D object. The bson.A cause the $set to be treated as an aggregation command, which does not support empty objects.
See: MongoServerError: Invalid $set :: caused by :: an empty object is not a valid value - #4 by Jason_Tran
2 Likes