Dynamic collection creation with unique indexes

First, the context. Then, the question.

I have the need for a database to exist in a sort of “pre-provisioned” state, with certain - call them “original” - collections existing (with no documents) that have uniqueIndexes applied (prior to any writes, of course). Then, at runtime, I am dynamically creating collections - call them “novel” - to which I apply uniqueIndexes immediately.

The problem is that multiple applications will be starting independently, each with write access to the “original” collections. There’s no clear point at which unique-createIndexes should be applied, unlike the “novel” collections which can be uniquely-indexed by the application which creates them at runtime. I am considering 2 approaches to the “provisioning” of these “original” collections’ indexes:

  1. Create a separate application whose responsibility is only to uniquely index/provision these “original” collections. Upon successful termination of this application, the N independent applications which have write access to these “original” collections can safely assume that these collections have been uniquely indexed and don’t have to perform any findOne/findAll prior to document insertion.
  2. Have each independent application attempt to index the “original” collections independently, inevitable encountering an error like “ErrIndexAlreadyExists” (a fabricate error name). This is what I was doing, but it relies on the particular error string, which might change with a different version of the mongo-go-driver.

The 1st approach has the disadvantage of an extra and awkward step in the deployment procedure. Also, if someone forgets to run this application (i.e. ./buildIndexes) then there’s no failsafe. The disadvantage of the second case is two-pronged: 1) it relies on the error string (this can probably be avoided using an error value I’m not aware of); 2) it (kind of) requires each application to take the responsibility of indexing these collections, duplicating work.

My question is this: What is the best practice/recommended procedure for establishing a unique set of indexes on a set of collections that a developer knows will be written to independently by multiple applications?

1 Like

Hi John,

I tried running some code to create the same index multiple times and did not get back any errors from a 4.2 server. My understanding is that IndexView.CreateOne/IndexView.CreateMany should be a no-op if all of the specified indexes already exist. Note that specifying an index with the same name as one that already exists but a different key pattern will return an error because the index specifications don’t match.

Can you try running some code against your servers to call IndexView.CreateOne multiple times to see if you get back any errors? If you do, please post your code and the error in a reply to this conversation.

– Divjot

2 Likes

To be clear, I’m trying to replicate this behavior: https://docs.mongodb.com/manual/core/index-unique/#unique-constraint-across-separate-documents.

I’m calling *mongo.Collection.Indexes().CreateOne with a mongo.IndexModel argument. My code is below. It errors on the second call to indexCollection (indexcnt == 1) with:

➜  ./mongo_test
2020/03/03 12:50:24 indexed the collection 1 time(s).
2020/03/03 12:50:24 failed to index DB: (IndexOptionsConflict) Inde
x with name: param3_1_param1_1_param2_1 already exists with a diffe
rent name

Ahh, so let me give this index a common name across my applications (and in this test code) to see if it doesn’t break.

I’m using mongo-go-driver v1.3.1 with go 1.14. My mongod version is

➜ ./bin/mongod -version
db version v4.2.2
git version: a0bbbff6ada159e19298d37946ac8dc4b497eadf
allocator: system
modules: none
build environment:
    distarch: x86_64
    target_arch: x86_64

My code

package main

import (
	"context"
	"flag"
	"log"
	"time"

	"go.mongodb.org/mongo-driver/mongo"
	"go.mongodb.org/mongo-driver/mongo/options"
	"go.mongodb.org/mongo-driver/mongo/readpref"
)

func main() {
	flagURI := flag.String("uri", "mongodb://localhost:27017", "URI of the MongoDB host (e.g. mongodb://localhost:27017")
	flag.Parse()
	client, err := mongo.NewClient(options.Client().ApplyURI(*flagURI))
	if err != nil {
		log.Fatalf("failed to obtain a MongoDB client: %s", err)
	}
	ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
	defer cancel()
	err = client.Connect(ctx)
	if err != nil {
		log.Fatalf("failed to connect to mongod: %s", err)
	}
	if err := client.Ping(ctx, readpref.Primary()); err != nil {
		log.Fatalf("failed to ping mongod: %s", err)
	}
	db := client.Database("brand-new-db")
	coll := db.Collection("brand-new-collection")
	var indexcnt int
	if err := indexCollection(coll); err != nil {
		log.Printf("indexed the collection %d time(s).", indexcnt)
		log.Fatalf("failed to index DB: %s", err)
	}
	indexcnt++
	if err := indexCollection(coll); err != nil {
		log.Printf("indexed the collection %d time(s).", indexcnt)
		log.Fatalf("failed to index DB: %s", err)
	}
}

func indexCollection(coll *mongo.Collection) error {
	ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
	defer cancel()
	t := true
	_, err := coll.Indexes().CreateOne(ctx, mongo.IndexModel{
		Keys: map[string]int{
			"param1": 1,
			"param2": 1,
			"param3": 1,
		},
		Options: &options.IndexOptions{
			Unique: &t,
		},
	})
	return err
}

I have modified the indexCollection function to the below, to use a name. It still fails with the following error:

➜   go build && ./mongo_test
2020/03/03 13:14:56 indexed the collection 1 time(s).
2020/03/03 13:14:56 failed to index DB: (IndexKeySpecsConflict) Index must have unique name.The existing index: { v: 2, unique: true, key: { param1: 1, param2: 1, param3: 1 }, name: "3params", ns: "brand-new-db.brand-new-collection" } has the same name as the requested index: { v: 2, unique: true, key: { param3: 1, param1: 1, param2: 1 }, name: "3params", ns: "brand-new-db.brand-new-collection" }

I think the problem is that I’m not using an ordered mongo.D (document) type. I’ll run one more test and report back.

func indexCollection(coll *mongo.Collection) error {
	ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
	defer cancel()
	_, err := coll.Indexes().CreateOne(ctx, mongo.IndexModel{
		Keys: map[string]int{
			"param1": 1,
			"param2": 1,
			"param3": 1,
		},
		Options: options.Index().SetName("3params").SetUnique(true),
	})
	return err
}
1 Like

Got eeee. Thanks Divjot. No errors, now. Maybe the name isn’t necessary? But, I’ll keep it.

func indexCollection(coll *mongo.Collection) error {
	ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
	defer cancel()
	_, err := coll.Indexes().CreateOne(ctx, mongo.IndexModel{
		Keys: bson.D{
			{"param1", 1},
			{"param2", 1},
			{"param3", 1},
		},
		Options: options.Index().SetName("3params").SetUnique(true),
	})
	return err
}

Hi John,

Seems like you’ve figured it out. For completeness, the reason it was erroring before is because Go’s maps do not guarantee any ordering and the driver iterates over the Keys field to create the index specification document. This meant that different attempts would generate different documents, which actually represent different indexes on the server. Using bson.D is the way to go for these kinds of things as it guarantees ordering.

As for the name, you’re right that it isn’t necessary. If not specified, the driver will generate a name from the specification (e.g. for your Keys field, the name would be param1_1_param2_1_param3_1). The name option is there for you to override this behavior if you want to give the index a more meaningful name.

1 Like

You’re the man Divjot. Thanks for confirming everything.