Error decoding bson objectid in json payload in golang

Peter_Geerts · November 15, 2023, 11:08am

Hi I am pretty new to mongodb and I am trying to get example code that comes with a book to work against mongodb. I did a local installation on windows and it seems that the server comes up fine. The default port is there as expected. My golang executable which does a db connect at the outset works fine and says connecting to the database. However when I offer it a json payload derived from the code in the book it gives me a json parsing error on the location key “3” in the following json:

{
 //   "id": 4,
   "name":"opera aida",
   "startdate":768346784368,
   "enddate":43988943,
   "duration":120,
   "location":{
      "id":3,
      "name":"West street Opera House",
      "address":"11 west street , AZ 73646",
      "country":"U.S.A",
      "opentime":7,
      "closetime":20,
      "Hall":{
         "name":"Cesar Hall",
         "location":"second floor, room 2210",
         "capacity":10
      }
   }
}

The golang main struct is event which is as follows:

type Event struct {
	ID        bson.ObjectId `bson:"_id"`
	Name      string
	Duration  int
	StartDate int64
	EndDate   int64
	Location  Location
}

And the included Location is as follows:

type Location struct {
	ID        bson.ObjectId `bson:"_id"`
	Name      string
	Address   string
	Country   string
	OpenTime  int
	CloseTime int
	Halls     []Hall
}

The parsing looks as follows:

        event := persistence.Event{}
	err := json.NewDecoder(r.Body).Decode(&event)
	if nil != err {
		w.WriteHeader(500)
		fmt.Fprintf(w, `{"error": "error occured while decoding event data %s"}`, err)
		return
	}

And postman gives me the following error message:

{“error”: “error occured while decoding event data invalid ObjectId in JSON: 3”}

If I decomment the key (4) in event in the json above I get the same error message type

Anyone any ideas on how to tackle this ?

Thanks

Peter Geerts

Preston_Vasquez · November 15, 2023, 6:15pm

Hi @Peter_Geerts , thank you for the question. Based on the code you’ve submitted, it appears you are using mgo , is this correct? The ObjectID type in the official MongoDB Go Driver is located in the primitive package. Using the official Go Driver, I would expect the types to look something like this:

type Event struct {
	ID        primitive.ObjectID `bson:"_id"`
	Name      string
	Duration  int
	StartDate int64
	EndDate   int64
	Location  Location
}

Also note that you are using a JSON decoder but are applying a bson tag to the ID field. While there is nothing wrong with doing this, the JSON decoder will attempt to decode the key “id” into the ID field. The case you provide will attempt to use the primitive.ObjectID’s decode logic to decode an int, which will invariably result in the following error:

not an extended JSON ObjectID

Object IDs are always 12 bytes that represent a timestamp, random value, and incrementation. You can read more about them here. I would expect your JSON “id” fields to be a hexadecimal string. For example,

{"id": "542c2b97bac0595474108b48"}

Peter_Geerts · November 16, 2023, 11:13am

Hi Preston,

thanks for the extensive reply which touches in more detail upon things which I sort of “guessed”.

First of - the code I presented is as-is (except for modifying some package name references) from a book dating back to 2017. And yes it uses mgo and already discovered it is not supported anymore and you should use the official mongodb go packages. Aside from this parsing issue I also notice that mgo behaves rather strangely because I get messages about “unable to locate cluster” while I am certain that my local mongodb instance runs perfectly. As an exercise I have used the same json payload to create an entry in the database with mongodb shell and this works without problems (and yes I then get a generated 12 byte object id).

Yesterday I did a small test of converting my code from mgo to the official MongoDB driver by substituting the relevant package names. This results in quite some rework since I get a lot of errors obviously and I have to figure out how things need to be changed whilst knowing neither package very well.
Some things that come to mind:

I get the impression that offering this json payload with id values of “3” or “4” will never work either in the past (older mongodb releases) or present. Is this correct?
Would it work against my current code (and mgo bson tag) if I specify a 12byte id in json? Because if it could I would like to try this first. Should I get passed this parsing stage the code wil try to do a db insert and we will see what happens next.
If mongodb always works with the 12 byte ID fields it seems to me these are not really “usable” from a end-user perspective. I understand the notion that the user wants to create an “event” with id “4” or a location with id “3”. You could then offer search capability based on these simple id which can be memorized but this will never happen with 12 byte values. Could you make the go code such that the ID’s are self generated (as mongodb shell seems to do). Basically leave the ID field out of the struct (json parsing will then work) and make it irrelevant for end-user purposes. If you would like an user related id in the data that field would have to be incorporated in the “user” part of the struct definition as e.g. an int.
It can then be used as a searchable field like "name " etc.

Preston_Vasquez · November 16, 2023, 4:46pm

@Peter_Geerts the _id field can be any non-array type. From the documentation:

The field name _id is reserved for use as a primary key; its value must be unique in the collection, is immutable, and may be of any type other than an array. If the _id contains subfields, the subfield names cannot begin with a ($) symbol.

For instance, if you wanted it to be an integer you would type the struct like this:

type Event struct {
	ID        int `bson:"_id"`
	Name      string
	Duration  int
	StartDate int64
	EndDate   int64
	Location  Location
}

If you wanted to leave it as a primitive.ObjectID, you can programmatically generate this data using the current time with primitive.NewObjectID.

Peter_Geerts · November 16, 2023, 6:48pm

@Preston_Vasquez so if I understand you correctly there are basically two options:

Redefine the ID as an int as part of the struct you listed in which case giving it a value of e.g. “3” in the json for adding a record in the collection might work (including the json parsing)? If the key in mongodb
actually then has this value then key values become user responsibility because the next insert with the same key will probably fail (uniqueness constraint)?
Leave the key definition as it is (primitive.objectid or bson.objectid). I understand your suggestion of somehow generating a key in the go code (don’t know whether mgo supports this) and including this key in the struct before inserting into the database. I fear however a sort chicken and egg situation because I still need to json parse the data as part of the http request before I can assign the generated key to the struct.
I have tried ommiting the id from the json payload altogether but also then other terrible things happen and it also fails. So I am not really sure about the way forward in this case

thanks Peter

Preston_Vasquez · November 16, 2023, 8:53pm

@Peter_Geerts The server will automatically generate _id as an object id if it is excluded from the document. You do not need to programmatically generate this value, I was simply pointing out you can do that if you need / want. If you decide to use your own ID type, then yes it is your responsibility to maintain logic for a duplication event. From the documentation:

In MongoDB, each document stored in a collection requires a unique _id field that acts as a primary key. If an inserted document omits the _id field, the MongoDB driver automatically generates an ObjectId for the _id field.

For example, consider you have the JSON payload {"id":2,"name":"Preston"} but you want the server to auto-assign and object ID. Then you can decode this JSON into the following struct:

type Developers struct {
    Name string
}

And it will insert something like this:

{ _id: ObjectId("655680d19596fedd21075b2e"), name: 'Preston' }

Peter_Geerts · November 17, 2023, 8:22pm

@Preston_Vasquez I did some tests today with basically omitting all key values (“id”) in the json I listed earlier. This allows the json parsing to proceed succesfully and when I list the struct contents as delivered to the server both keys are empty (“”) but the rest is all there. What I see in the code (mgo version) that before a db insert is done the objectid’s are checked for validity. There is a (mgo) IsValid function for checking objectid’s but I am not sure whether an empty key qualifies as being invalid but in case it is invalid they generate an objectid before insertion.
The problem I also have is that mgo keeps having problems with connecting to the database and I am sure the db insert won’t work once I get to that point.
Therefore I started with refactoring this mgo code out and replacing it with the official golang packages as you indicated. I did some modifications already and at this point I am mostly left with the actual db crud statements but I will figure that out as best I can.

I understand what you are saying in your latest post. It looks like that actual record keys (objectid’s) are best treated as opaque since they will never be used for search purposes by users.
The problem with the example you gave is I think that the id value of 2 is basically ignored and only the name field remains which is confusing to users because searching for an id :2 will give no results.
Suppose I change my event struct into the following:

type Event struct {
	ID        int 
	Name      string
	Duration  int
	StartDate int64
	EndDate   int64
	Location  Location
}

will the ID field then treated as user data and stored as such and subsequently mongodb be smart enough
to see there is no objectid and generate one? Or is ID/id some sort of default name for the objectid and treated as such (and using it will fail) and do I therefore need the think of another name to avoid a clash?

thanks Peter