Writing to databse blocked with golang v1.17

Working with Data Drivers
1

I tried to insert 1M documents to the collection with golang v1.17, the insertion of data is always blocked，when the counts of documents reach aboout 50,000, the counts didn’t change anymore. But it’s normal when I use golang v1.13.

MongoDB version: v4.0.11
mongo-driver version: go.mongodb.org/mongo-driver v1.8.4

2

Hello @zhou_yi, I tried inserting a million documents into a MongoDB database collection using the following and there was no issue. The document is a simple one with one string field, name.

MongoDB Server v4.2
MongoDB Golang Driver v1.8
Golang v1.17

I suggest you include the code you had tried, so that I or someone can suggest solution if there are any issues. Also, tell if there were any error messages after the inserts were blocked.

3

Hi @Prasad_Saya , thank you for your advice, I upgrade my MongoDB Server version from 4.0 to 4.4, but the block problem of Golangv1.17 is still remain, while the insertion with Golang1.13 is normal. The test code is as follows. And there was no error during the insertion, I also checked mongod.log and found no error.

type student struct {
	ID      int    `bson:"_id"`
	SId     int    `bson:"sId"`
	Subject string `bson:"subject"`
	Name    string
	Gender  string
	SNO     int
}

var (
	host   string
	port   string
	uname  string
	pwd    string
	dbname string
	authDB string
)

func parseStartParams() {
	flag.StringVar(&host, "h", "127.0.0.1", "server ip")
	flag.StringVar(&port, "P", "10019", "server port")
	flag.StringVar(&uname, "u", "tsmpadmin", "database user")
	flag.StringVar(&pwd, "p", "tsmpadmin2021_isv_com", "user's password")
	flag.StringVar(&dbname, "d", "temp", "database name")
	flag.StringVar(&authDB, "auth", "admin", "authDatabase")

	flag.Parse()
}

func main() {
	parseStartParams()

	client, err := connectMongo()
	if err != nil {
		return
	}
	defer client.Disconnect(context.TODO())

	db, err := getDatabase(client, dbname)
	if err != nil {
		return
	}

	colName := "students17"
	col := getCollection(db, colName)
	fmt.Println("go version：", runtime.Version())

	count := 0
	bt := time.Now()
	for i := 0; i < 1000000; i++ {
		s := student{
			ID:      1000000 + i,
			SId:     1000000 + i,
			Subject: "biology",
			Name:    fmt.Sprintf("stu%05d", i),
			SNO:     i,
		}
		if i%2 == 0 {
			s.Gender = "male"
		} else {
			s.Gender = "female"
		}

		_, err := col.InsertOne(context.TODO(), &s)
		if err != nil {
			fmt.Println("InsertOne error: ", err)
			break
		}

		count++
		if count%100000 == 0 {
			fmt.Printf("%d documents have been writen, cost %d ms\n", count, time.Since(bt)/time.Millisecond)
			bt = time.Now()
		}
	}
}

func getCollection(db *mongo.Database, colNam string) *mongo.Collection {
	return db.Collection(colNam)
}

func getDatabase(client *mongo.Client, dbnam string) (*mongo.Database, error) {
	dbs, err := client.ListDatabaseNames(context.TODO(), bson.D{})
	if err != nil {
		fmt.Println("ListDatabaseNames error:", err)
		return nil, err
	}

	var exist bool
	for _, db := range dbs {
		if db == dbnam {
			exist = true
			break
		}
	}
	if !exist {
		fmt.Printf("database %s not exist!\n", dbnam)
		return nil, errors.New("database not exist")
	}

	return client.Database(dbnam), nil
}

func connectMongo() (*mongo.Client, error) {
	host := fmt.Sprintf("mongodb://%s:%s@%s:%s/%s?retryWrites=false", uname, pwd, host, port, authDB)

	clientOptions := options.Client().ApplyURI(host)

	var err error
	client, err := mongo.Connect(context.TODO(), clientOptions)
	if err != nil {
		fmt.Println("mongo.Connect error:", err)
		return nil, err
	}

	err = client.Ping(context.TODO(), nil)
	if err != nil {
		fmt.Println("mongo client.ping error:", err)
		return nil, err
	}

	fmt.Printf("Connected to MongoDB!\n")

	return client, nil
}
4

Hello @zhou_yi, from your attached code I see one issue - that is using the _, err := col.InsertOne(context.TODO(), &s) within the for-loop of million iterations. For each InsertOne the program sends the insert to the database and get a result back. That is an inefficient operation. You can try something like this which allows make a bulk write - one call is made to the database and you get one result. For example:

coll := client.Database("test").Collection("large")
docs := []interface{}{}

for i := 0; i < 1000000; i++ {
	s:= strconv.Itoa(i)
	docs = append(docs, bson.D{{ "name", "document_"+s }})
}

opts := options.InsertMany().SetOrdered(false)

results, err := coll.InsertMany(context.TODO(), docs, opts)

if err != nil {
	panic(err)
}

fmt.Printf("Number of documents inserted: %d \n", len(results.InsertedIDs))

Also, note the insert options set as opts := options.InsertMany().SetOrdered(false). This can help insert the documents in a performant way (see Execution of Operations).

5

@Prasad_Saya Thanks a lot for your suggestion, the InserMany operation is ok, the insertion didn’t block again. But I still want to know why the InsertOne operation within the for-loop is normal with Golang1.13 but block with Golang1.17?

6

I don’t know what the reason is :innocent:

But, you can look into the details of Release history - Golang, you can see the improvements, bug fixes, new features, etc., for each version. You can browse the releases after 1.13 and see what issues are fixed which might have caused the problem in your case (its also possible you may not find the reason).

1 Like