IntDecodeValue can only truncate float64 to an integer type when truncation is enabled

Zhihong_GUO · March 7, 2020, 12:05pm

Hello, when I search the database by pipeline and aggregation, I will get this error when I execute cursor.Decode to my struct. But if I search the db by collection.Find (ctx, filter), then no such error when executing the cursor Decode.

The data stored in db is something like:

{
	"_id" : "AAX",
	"user_id" : ObjectId("5cc6ebf58a158c00010b3d74"),
	"name" : "mon super Bureau",
	"status" : "active",
	"location" : {
		"type" : "Point",
		"coordinates" : [
			48.1259880065918,
			-1.6275769472122192
		],
		"accuracy" : 14
	},
	"creation_date" : ISODate("2019-05-28T14:26:03.579Z")
}

The result returned looks like:

{
      "id": "AAX",
      "user_id": "5cc6ebf58a158c00010b3d74",
      "name" : "mon super Bureau", 
      "status": "active",
      "location": {
        "type": "Point",
        "coordinates": [
          48.125988,
          -1.627576
        ],
        "accuracy": 14
      },
  "creation_date": {
    "seconds": 1559053563,
    "nanos": 579000000
  },
}

It looks the lng and lat in location has been truncated and cause such error. But why if I use Find, such error will not happen? any parameters should I set for the pipeline and aggregation?

thanks,

James

Divjot_Arora · March 9, 2020, 1:18pm

Hi,

The error you’re seeing in the aggregation is expected because decoding would lose precision and therefore could result in data loss. I’m not sure why you’re not seeing the error when using Find because the Cursor returned by Find and Aggregate is the same, so the Decode code path should be the same. Is it possible that the filter you’re providing to Find is filtering out the document so it’s never returned? If not, can you share the code you’re using for Find as well as the definition of the struct you’re decoding into?

If you want the driver to ignore the precision loss and truncate each float into an integer, you can use the trunacte struct tag:

type Foo struct {
    Coordinates []int `bson:"coordinates,truncate"`
}

This will tell the driver that you’re opting into truncating any floats into integers, even if that will cause loss of precision.

– Divjot

Zhihong_GUO · March 9, 2020, 3:01pm

Hello Divjot,

Thank you again for your help

the struct for decoding is

type Place struct {

ID string `json:"id,omitempty" bson:"_id,omitempty"`

UserID primitive.ObjectID `json:"user_id,omitempty" bson:"user_id,omitempty"`

Name string `json:"name,omitempty" bson:"name,omitempty"`

Status Status `json:"status,omitempty" bson:"status,omitempty"`

Location geo.GeoJSON `json:"location,omitempty" bson:"location,omitempty"`

CreationDate time.Time `json:"creation_date,omitempty" bson:"creation_date,omitempty"`

}
the GeoJSON is:
type GeoJSON struct {
Type string json:"type,omitempty" bson:"type,omitempty"
Coordinates float32 json:"coordinates,omitempty" bson:"coordinates,omitempty"
Accuracy int json:"accuracy,omitempty" bson:"accuracy,omitempty"
}

I check the filter used by Find, which is search place by text:
jsonFilter = {"$and" : [{"$text":{"$search" : %q}}, {"status" : "active"}]}

The other difference is there is no skip and limit on Find. The code of Find looks like:
cur, err := collection.Find(context.TODO(), filter)
defer cur.Close(context.TODO())
places := place.Place{}
err = cur.All(context.TODO(), &places)

if err != nil {
	return errors.InternalServerError("place_internal_error", err.Error())
}

after the cur.All, the places get all of the result and no error happens, so it is not returned from “place_internal_error”

Divjot_Arora · March 9, 2020, 3:12pm

Can you share the code for your call to Collection.Aggregate as well as the struct the aggregation is decoding into? Apologies for asking you to type out so much code, but it is super helpful for us when we have code we can copy/paste and run to reproduce errors.

– Divjot

Zhihong_GUO · March 9, 2020, 3:21pm

geoStage := createGeoStage(in.Lng, in.Lat, in.Radius)
matchStage := createMatchStage(in.UserId)

countStage := `{ "$count":"number"}`

counting, err := BuildPipelineFromJsons( geoStage, matchStage, countStage)

omit the code to count the places as there are no error

searching, err := BuildPipelineFromJsons( geoStage, matchStage)
searching = appendToPipeline(searching, "$skip", 0)
searching = appendToPipeline(searching, "$limit", 20)
cur, err := collection.Aggregate(context.TODO(), searching, options.Aggregate())
defer cur.Close(context.TODO())
for cur.Next(context.TODO()) {
	place := place.Place{}
	err = cur.Decode(&place)

}

the other different of Find and Aggregation is in Find I call CountDocuments to get the total number of places

Zhihong_GUO · March 9, 2020, 3:24pm

func appendToPipeline(pipeline mongo.Pipeline, operator string, parameter int32) mongo.Pipeline {
stageRaw := fmt.Sprintf({%q: %v}, operator, parameter)
var stageCooked bson.D
err := bson.UnmarshalExtJSON(byte(stageRaw), false, &stageCooked)
if err == nil {
pipeline = append(pipeline, stageCooked)
}
return pipeline
}

func createGeoStage(lng float32, lat float32, radius int32) (jsonStage string) {

geoStage := `
{
	"$geoNear":{
		"includeLocs":"location",
		"distanceField":"distance",
		"near":{
			"type":"Point",
			"coordinates":[ %g, %g]
		},
		"maxDistance": %v,
		"spherical":true
	}
}`
geoStage = fmt.Sprintf(geoStage, lng, lat, radius)
return geoStage

}

func createMatchStage(userID string) (jsonStage string) {

if id, err := primitive.ObjectIDFromHex(userID); err == nil {
	matchStage := `
	{
		"$match":{
			"$and":[
				{
					"$or":[
						{
							"proprietary":false
						},
						{
							"user_id": {"$oid": %q}
						}
					]
				},
				{
					"status":"active"
				}
			]
		}
	}`
	matchStage = fmt.Sprintf(matchStage, id)
	return matchStage
} else {
	return `
	{
		"$match":{
			"$and":[
				{
					"proprietary":false
				},
				{
					"status":"active"
				}
			]
		}
	}`
}

}

Zhihong_GUO · March 9, 2020, 3:24pm

that’s all of the related code

Divjot_Arora · March 9, 2020, 5:30pm

Hmm I’m not sure what’s going on here. The error is from IntDecodeValue and the only integer field in your struct is GeoJSON.Accuracy so my guess is that the server is returning the accuracy field as a float instead of an integer.

One thing to try: instead of calling cur.Decode on the cursor from the Aggregate call, can you print out cur.Current (i.e fmt.Println(cur.Current))? This will print out the exact BSON document the driver received from the server and could help understand what field is malformed.

– Divjot

Zhihong_GUO · March 10, 2020, 1:54am

Hello Divjot,

Thanks for your guidance. It really helps to find the root of the issue; and forgive my careless yesterday: when I copy the Place struct I missed the last one, there is a “distance” at the end of the struct but I failed to copy it. I believe this field is the root of the error.

type Place struct {
	
	ID string `json:"id,omitempty" bson:"_id,omitempty"`
	
	UserID primitive.ObjectID `json:"user_id,omitempty" bson:"user_id,omitempty"`
	
	Name string `json:"name,omitempty" bson:"name,omitempty"`
	
	Status Status `json:"status,omitempty" bson:"status,omitempty"`

	Location geo.GeoJSON `json:"location,omitempty" bson:"location,omitempty"`

	CreationDate time.Time `json:"creation_date,omitempty" bson:"creation_date,omitempty"`
	
	Distance int32 `json:"distance,omitempty" bson:"distance,omitempty"`
}

When I Find the place in fact it is a kind of “search by text”, so no additional “distance” value generated by db. When I use $geoNear I think the db will sort the result by “distance” by default, and the db generated a double value for “distance”. When I send the result to the front end I remove the “distance”, making it more difficult to compare the difference of output from “Find” vs. “Aggregate”. . Everything becomes clear when Print the cur.Current.

Thank you again for your time and the great help!

Last question, is there a way to set the “distance” in float32 when db generating this value? or I have to change the “Distance” to be in64 in my Place struct?

Regards,

James

Divjot_Arora · March 10, 2020, 1:15pm

A BSON double value is always 64 bits, so I don’t think there’s a way to change the aggregation to output a float32. Also, doing so wouldn’t help because you’d be trying to decode a float32 into an int32, which could still cause precision loss. My advice would be to update the Distance field to be a float64 to guarantee that precision is never lost.

Zhihong_GUO · March 10, 2020, 1:17pm

Hello Divjot,

Got it, many thanks.

James