Cannot insert all documents to the MongoDB cloud cluster: Prematurely reached end of stream

[invzbl3 (Maksym)]

I’m using JDK/JRE version like: 1.8.0_192
When I’m inserting documents to the Mongodb cloud cluster, I’m getting always the issue:

Caused by: com.mongodb.MongoSocketReadException: Prematurely reached end of stream
	at com.mongodb.internal.connection.SocketStream.read(SocketStream.java:112)
	at com.mongodb.internal.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:579)
	at com.mongodb.internal.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:444)
	at com.mongodb.internal.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:298)
	at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:258)
	at com.mongodb.internal.connection.CommandHelper.sendAndReceive(CommandHelper.java:83)
	at com.mongodb.internal.connection.CommandHelper.executeCommand(CommandHelper.java:33)
	at com.mongodb.internal.connection.InternalStreamConnectionInitializer.initializeConnectionDescription(InternalStreamConnectionInitializer.java:103)
	at com.mongodb.internal.connection.InternalStreamConnectionInitializer.initialize(InternalStreamConnectionInitializer.java:60)
	at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:128)
	at com.mongodb.internal.connection.UsageTrackingInternalConnection.open(UsageTrackingInternalConnection.java:50)
	at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.open(DefaultConnectionPool.java:435)
	at com.mongodb.internal.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:117)
	at com.mongodb.internal.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:102)
	at com.mongodb.internal.connection.DefaultServer.getConnection(DefaultServer.java:90)
	at com.mongodb.internal.binding.ClusterBinding$ClusterBindingConnectionSource.getConnection(ClusterBinding.java:112)
	at com.mongodb.client.internal.ClientSessionBinding$SessionBindingConnectionSource.getConnection(ClientSessionBinding.java:136)
	at com.mongodb.internal.operation.FindOperation$1.call(FindOperation.java:628)
	at com.mongodb.internal.operation.FindOperation$1.call(FindOperation.java:625)
	at com.mongodb.internal.operation.OperationHelper.withReadConnectionSource(OperationHelper.java:462)
	at com.mongodb.internal.operation.FindOperation.execute(FindOperation.java:625)
	at com.mongodb.internal.operation.FindOperation.execute(FindOperation.java:77)
	at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:190)
	at com.mongodb.client.internal.FindIterableImpl.first(FindIterableImpl.java:189)
	at gui.Controller.cloudData(Controller.java:120)
	at gui.Controller.lambda$pushAction$1(Controller.java:111)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at gui.Controller.pushAction(Controller.java:111)
	... 58 more

I’ve tried different connection strings for older drivers like:

mongodb://admin:password@cluster0-shard-00-00-ox90k.mongodb.net:27017,cluster0-shard-00-01-ox90k.mongodb.net:27017,cluster0-shard-00-02-ox90k.mongodb.net:27017/test?ssl=true&replicaSet=Cluster0-shard-0&authSource=admin&retryWrites=true&maxIdleTimeMS=40000

and without maxIdleTimeMS parameter:
mongodb://admin:password@cluster0-shard-00-00-ox90k.mongodb.net:27017,cluster0-shard-00-01-ox90k.mongodb.net:27017,cluster0-shard-00-02-ox90k.mongodb.net:27017/test?ssl=true&replicaSet=Cluster0-shard-0&authSource=admin&retryWrites=true

For newer drivers like:
mongodb+srv://admin:password@cluster0-ox90k.mongodb.net/test?retryWrites=true&w=majority

and like:

mongodb+srv://admin:password@cluster0-ox90k.mongodb.net/test?retryWrites=true&w=majority&readPreference=secondary&replicaSet=Cluster0-shard-0&ssl=true

It is worth noting that I can insert half of the documents, but then this error occurs.

And nothing helps here.

[coderkid (Kay Anar)]

@invzbl3

Caused by: com.mongodb.MongoSocketReadException: Prematurely reached end of stream

This looks like SSL connection issue… check whether or not JRE’s SSL keystore is set properly.

[invzbl3 (Maksym)]

Thanks for the answer, coderkid. can you a bit elaborate how to check it properly?
I’m not sure, but maybe these limits messages would be helpful here.

UPD: I suppose I’ve found, at least, place where is located trustore:

How do I find out what keystore my JVM is using?

And I have inside something like:

[invzbl3 (Maksym)]

To clarify a bit:

I’ve tried everything what I’ve already mentioned here: How to avoid an exception Prematurely reached end of stream using mongoDB Java driver 3.4+ or 3.6+? (during insertion) - Stack Overflow

as you can see my post above.

My Network Access looks like:

Also I can’t provide any logs, because:

“M0 Free Tier and M2/M5 shared clusters do not provide downloadable logs.”

Atlas M0 (Free Cluster), M2, and M5 Limitations — MongoDB Atlas

[wan (Wan Bachtiar)]

Hi @invzbl3,

The fact that you can connect and insert for a period of time, could be an indication that it’s not related to the keystore.

Looking at both of these log lines:

Caused by: com.mongodb.MongoSocketReadException: Prematurely reached end of stream
   ....
at com.mongodb.client.internal.FindIterableImpl.first(FindIterableImpl.java:189)

It seems that you are looping through an iterable from Find(), and within the loop you’re inserting documents. If so, it is possible that each of the loop process is taking too long, that either the find cursor has timeout or the server that the cursor is connected to was disconnected.

Could you provide more information about how you’re inserting the documents with a snippet code example ?

Regards,
Wan.

[invzbl3 (Maksym)]

Hi, @wan, thanks for the answer. My code looks like:

 public void data(MongoCollection<Document> collection, String title, String country) {

        Document found = collection.find(new Document("title", title)             
                .append("country", country)).first();

        if (found == null) {
            collection.insertOne(new Document("title", title)
                .append("country", country));
        } 
} 

The cause of error stacktrace shows me on line:

.append("country", country)).first();

By the same logic, I had this exactly error using code snippet like here:

How to avoid an exception Prematurely reached end of stream using mongoDB Java driver 3.4+ or 3.6+? (during insertion)

I have so far inserted about 50 documents instead of the 176 that I planned using this snippet.

[wan (Wan Bachtiar)]

Thanks for the snippet code.
This function is called within a loop right ? Also, are you running this in a concurrent/parallel operations ?

@invzbl3

at java.util.ArrayList.forEach(ArrayList.java:1257)

Could you check in Atlas cluster monitoring views the number of connections and operations displayed around the time you’re getting the error ? See also Monitor a Cluster for information how to view the metrics.

I’m suspecting that you could be hitting the shared clusters limitations.

Regards,
Wan.

[invzbl3 (Maksym)]

@invzbl3

at gui.Controller.cloudData(Controller.java:120) at gui.Controller.lambda$pushAction$1(Controller.java:111) at java.util.ArrayList.forEach(ArrayList.java:1257)

@wan, yes, you’re right about the loop. To correct a bit, my method of program is named like cloudData not data (as I mentioned earlier in snippet), so stacktrace is correct here.

Journal is my custom class of parameters that I’m adding inside list.

So I’m declaring firstly instance variable:
private List<Journal> journalList;

and then I’m using for each to call this method using getters like:

journalList.forEach(journal -> cloudData(journal.getTitle(), journal.getCountry()));

I don’t think it’s related to concurrent/parallel operations, because I simply using the same function until all documents from list will be added to MongoDb Tier cluster one by one.

My metrics looks like:

Also I’m getting alert like:

Can you tell me, please, can I somehow control/regulate limit of % connections from connection string or something? I suppose it can be solution here if I add something like connection limiter as parameter, but still can insert everything that I need to cluster.

P.s. I’ve tried to increase number of milliseconds for parameter maxIdleTimeMS up to 100000 or 1000000 instead of 40000, but it doesn’t affect the overall situation, the same error appears.

[wan (Wan Bachtiar)]

@invzbl3

Can you tell me, please, can I somehow control/regulate limit of % connections from connection string or something?

Depending on your use case, there are a number of ways that you could try.

You could try to reduce the connection pool limit using maxPoolSize parameter (default is 100). Alternatively could also change the connections pool limit via ConnectionPoolSettings.Builder().maxSize(int).
If you reduce the number of connection pool, you need to be mindful about the time limit a thread may wait for a connection to become available. The default is 2 minutes see also ConnectionPoolSettings.Builder().maxWaitTime(long, TimeUnit).

In addition to the above, try to restructure your code. Instead of inserting a document one by one you could utilise BulkWrites operations. Also, looking at your example method cloudData, where you try to find a document and insert if it doesn’t exist, try performing update upsert operation whenever possible.

Depending on your use case, you could also consider upgrading the cluster to a dedicated cluster. See also Free, Shared and Dedicated Cluster Comparison.

Regards,
Wan.

[invzbl3 (Maksym)]

Thanks, for valuable answer, @wan. I’ve changed my code to:

public void cloudData(MongoCollection<Document> collection, String title, String country) {
  List<WriteModel<Document>> updates = Collections.singletonList(
                new ReplaceOneModel<>(
                        new Document("title", title) // find part
                                .append("country", country),              
                        new Document("title", title) // update part
                                .append("country", country),            
                        new ReplaceOptions().upsert(true)             
                )
        );
}

It works also fine, but the same issue appears after half of inserted documents.

Also I’ve already tried to add maxPoolSize and reduce value to 80 and 50 instead of default 100 in format like:

mongodb+srv://admin:password@cluster0-ox90k.mongodb.net/test?retryWrites=true&w=majority&maxPoolSize=80

unfortunately to no avail, still here:
Caused by: com.mongodb.MongoSocketReadException: Prematurely reached end of stream

Probably yes, the most likely option as I see it’s really to pay for upgrade.

P.s. I’ll try to combine maxPoolSize and maxWaitTime, as you mentioned, but not sure if it possible solution for connection string.

[invzbl3 (Maksym)]

Finally solved. I should call mongodb.MongoClient.connect once, not each request. So I’ve restructured my code a bit, to call connection once instead of each time when I insert specific document.

mongoDB too many connections how to close them correctly?

Thanks for the help, everyone.