Storing deeply nested data

Working with Data Drivers
1

A long story about why, but we need to be able to store a freeform document a user provides.

I take a payload and I store it in a database using a Key/Value approach. e.g.

// GlobalKeypair represents a single entry in the globals DB
type GlobalKeypair struct {
	Key   string      `json:"key"  bson:"key"`
	Value interface{} `json:"value"  bson:"value"`
}
filter := bson.M{"key": key}
kp := GlobalKeypair{
		Key:   key,
		Value: value,
	}
info, err := coll.ReplaceOne(
		context.Background(), filter, kp, options.Replace().SetUpsert(true))

The issue is when it comes back out of the database, it’s getting garbled. I’m thinking due to the freeform nature of the interface.
So I store this:

GlobalKeypair{Key: "some-key",
                Value: map[string]interface{}{
			"lid-brain-test-1": map[string]interface{}{
				"brain": "StreamingBrain",
				"floor": 0.65,
			},
			"lid-brain-test-2": map[string]interface{}{
				"brain": "BulkBrain",
				"floor": 0.65,
			},
		}
}

And when I pull it out, I get this:

"some-key": [
        {
            "Key": "lid-brain-test-1",
            "Value": [
                {
                    "Key": "brain",
                    "Value": "StreamingBrain"
                },
                {
                    "Key": "floor",
                    "Value": 0.65
                }
            ]
        },
        {
            "Key": "lid-brain-test-2",
            "Value": [
                {
                    "Key": "brain",
                    "Value": "BulkBrain"
                },
                {
                    "Key": "floor",
                    "Value": 0.65
                }
            ]
        }
    ],

I think something is happening under the covers where it’s structuring my interface into a bson.E, but I’m at the end of my rope. I can’t figure out what’s going on.

2

I found a way around and I’m posting here for others.

Essentially, rather than using an interface{}, I use an bytes now, this bypasses whatever weirdness is going on when mongo unpacks an object.

// GlobalKeypair represents a single entry in the globals DB
type GlobalKeypair struct {
	Key   string      `json:"key"  bson:"key"`
	Value []byte     `json:"value"  bson:"value"`
}
// value is an interface{} - get it to an []bytes
valBytes, err := json.Marshal(value)
if err != nil {
	return fmt.Errorf("Could not Marshal global into JSON: %v", err)
}
filter := bson.M{"key": key}
kp := GlobalKeypair{
		Key:   key,
		Value: valBytes,
	}
info, err := coll.ReplaceOne(
		context.Background(), filter, kp, options.Replace().SetUpsert(true))

Then you can get it back out with:

	kp := GlobalKeypair{}
	if err != nil {
		return value, err
	}
	err = result.Decode(&kp)
	if len(kp.Value) == 0 {
		// No value - not found
		return value, db.ErrNotFound
	}
	if err = json.Unmarshal(kp.Value, &value); err != nil {
		return value, fmt.Errorf("Could not pack global into JSON: %v", err)
	}

I was still hoping someone could help me understand why interface{} isn’t preserved and is instead unpacked as a nested slice.

3

Hi @TopherGopher,

The reason this happens is because the driver unpacks BSON documents in to a bson.D object, which doesn’t support converting to/from JSON. The bson.D type is internally represented as a slice to structs with fields named Key and Value, so that explains why the JSON is structured that way. If you know that the user-provided value will always be a document, you can change your type to

type GlobalKeypair struct {
    Key string
    Value bson.M
}

Unlike bson.D, the bson.M type is simply map[string]interface{} so the standard library json functions can handle it. Can you let me know if this works for you? I’ve also opened https://jira.mongodb.org/browse/GODRIVER-1765 to make this actually work for bson.D as well.

– Divjot

4

Hey @Divjot_Arora -
Thanks for the response and thank you for making the ticket - much obliged. :smile:
Unfortunately, we can’t use a map for the value because it’s freeform.

Our use case:

  • Python developers access our application via golang REST - no direct access to the DB.
  • Developers needed a common location to store a handful of freeform globals. Specifically, 4 values that all could access and modify. One is a float, one is a string, one is an array, one is a dictionary in a dictionary.
  • We created endpoints that allowed them to interact with the table in a freeform way using interface{} in gin REST endpoints.

Unfortunately, the bson.M as a Value technique was only compatible with one of our values. The byte however has been working awesome. I get out exactly what I put in, which is what I would expect when using an interface{}.

This might be something to consider as part of ease-of-use improvements. For example, if you get a freeform interface{} as a type, don’t coerce it to a bson.D because we lose the original structure. Instead, store it as a byte. It seems to be reliable and works for all types.

5

The following code can convert everything back to go concrete types, but the real problem is that the benchmarks for bson.Unmarshal(myBytes, &myInterface) are eye-popping. Unmarshalling 376 bytes of bson took 14406ns/op in my benchmarks. By comparison, the same json can be unmarshaled to an interface{} in 1107ns/op on my box. The benchmark did not include the code below.

func convertToGoConcreteTypes(val interface{}) interface{} {
	if document, ok := val.(primitive.D); ok {
		result := make(map[string]interface{})
		for _, element := range document {
			result[element.Key] = convertToGoConcreteTypes(element.Value)
		}
		return result
	}
	if array, ok := val.(primitive.A); ok {
		result := make([]interface{}, 0)
		for _, i := range array {
			result = append(result, convertToGoConcreteTypes(i))
		}
		return result
	}
	if primitiveMap, ok := val.(primitive.M); ok {
		result := make(map[string]interface{}, 0)
		for k, v := range primitiveMap {
			result[k] = convertToGoConcreteTypes(v)
		}
		return result
	}
	return val
}
6

Also, if I add the conversion code to the bson.Unmarshal above I get 16650ns/op. This begs the question: Can I convert 376 bytes of bson to json in less than 15000ns? The answer has to be yes.