TLS inPlain buffer overflow error hangs indefinitely in scala/java driver 4.1.0

Working with Data Drivers & ODMs
#1

We are using the scala driver version 4.1.0 with TLS enabled. Our application runs smoothly for several hours before randomly erroring with the following log:

2020-08-19 04:08:50.628 | ERROR |  | 9818284 ms| c.m.i.c.t.a.AsynchronousTlsChannelGroup  | error in operation
java.lang.IllegalStateException: inPlain buffer insufficient despite having capacity of 32768
    at com.mongodb.internal.connection.tlschannel.impl.BufferHolder.enlarge(BufferHolder.java:101)
    at com.mongodb.internal.connection.tlschannel.impl.TlsChannelImpl.unwrapLoop(TlsChannelImpl.java:309)
    at com.mongodb.internal.connection.tlschannel.impl.TlsChannelImpl.readAndUnwrap(TlsChannelImpl.java:612)
    at com.mongodb.internal.connection.tlschannel.impl.TlsChannelImpl.read(TlsChannelImpl.java:235)
    at com.mongodb.internal.connection.tlschannel.ClientTlsChannel.read(ClientTlsChannel.java:168)
    at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.readHandlingTasks(AsynchronousTlsChannelGroup.java:594)
    at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.doRead(AsynchronousTlsChannelGroup.java:559)
    at com.mongodb.internal.connection.tlschannel.async.AsynchronousTlsChannelGroup.lambda$processRead$5(AsynchronousTlsChannelGroup.java:471)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

this error is thrown at https://github.com/mongodb/mongo-java-driver/blob/7cc4be219a15b08c69dfba8d02c3e014f808b90d/driver-core/src/main/com/mongodb/internal/connection/tlschannel/async/AsynchronousTlsChannelGroup.java#L594

and logged by https://github.com/mongodb/mongo-java-driver/blob/7cc4be219a15b08c69dfba8d02c3e014f808b90d/driver-core/src/main/com/mongodb/internal/connection/tlschannel/async/AsynchronousTlsChannelGroup.java#L473

The problem is, after this error is thrown, the Future never completes and the thread hangs indefinitely. No exception is thrown for us to recover from, and our application grinds to a halt.

The error occurs at random, we have not been able to reproduce it on demand.

We thought this was related to https://github.com/rozza/mongo-java-driver/pull/385 / https://jira.mongodb.org/browse/JAVA-3730 but upgrading the driver to 4.1.0 did not fix the problem. It sounds related because the pending read operation appears to hang indefinitely, and if it is related to unexpected network / socket issues then it would be difficult to reproduce.

#2

HI @Ryan_Foltz,

Thanks for posting, this is something we’d need to look into to understand more about what is happening here. Could you post a ticket to the JAVA jira project.

Looks like there is a bug in the tlschannel library.

Ross

#3

Thanks, I’ll open the ticket in a minute.

For the record, we worked around this by configuring a 10 second socket timeout. We still see the error occasionally, but it no longer brings everything to a halt.