I have a Mongo DB ETL application that will transfer data from one database to another database. The database can be in a different MongoDB server. The source and target table can be the same in the same database. So, basically any scenario is possible.
I have a timer that triggers every 5 seconds and does the migration for a particular date range. The timer will be disabled when ETL is going on. So there is no timer elapsed events in between ETL to prevent parallel ETL’s.
public class TimerClass
{
function Timer_Elapsed()
{
ETLClass etl = new ETLClass();
etl.StartETL();
{
}
public class ETLClass
{
public void CreateMongoDBSession(string extractMongoDBServerURL, string extractMongoDBDatabaseName,
string loadMongoDBServerURL, string loadMongoDBDatabaseName)
{
Uri extractURI = new Uri(extractMongoDBServerURL);
this.ExtractMongoDBMongoClient = new MongoClient(new MongoClientSettings()
{
Server = new MongoServerAddress(extractURI.Host, extractURI.Port),
ClusterConfigurator = clusterBuilder =>
{
clusterBuilder.Subscribe(e =>
{
this.EmitOnDebugInfoEvent($“{e.CommandName} - {e.Command.ToJson()}”);
});
}
});this.ExtractMongoDBDatabase = ExtractMongoDBMongoClient.GetDatabase(extractMongoDBDatabaseName); Uri loadURI = new Uri(loadMongoDBServerURL); this.LoadMongoDBMongoClient = new MongoClient(new MongoClientSettings() { Server = new MongoServerAddress(loadURI.Host, loadURI.Port), ClusterConfigurator = clusterBuilder => { clusterBuilder.Subscribe<CommandStartedEvent>(e => { this.EmitOnDebugInfoEvent($"{e.CommandName} - {e.Command.ToJson()}"); }); } });this.LoadMongoDBDatabase = LoadMongoDBMongoClient.GetDatabase(loadMongoDBDatabaseName);
}public void StartETL()
{
//Create Mongo DB session.
//Read data from DB.
//Translate data.
//Update data into DB.
}
}
Here there will be the creation of new IMongoClient and IMongoDatabase every ETL and use it till the end of ETL.
According to the doc’s it says that IMongoClient will be reused if the connection configuration is the same. So ideally it should create max 2 connections.
Secondly, there is no API to dispose of the connection. So, the driver should automatically do the disposing of the connections if in case there are creation on multiple connections(Which is not the case according to documents).
But I am getting error after every few hours continuously as shown below. This error will get ratified once the application is restarted. Interestingly because of this same error is throwing in other applications like Azure Service Bus client.
With some googling, it looks like applications are making connections multiple continuously would be the reason for this error.
Error in Robo 3t:
An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.
Error in C# application:
A timeout occured after 30000ms selecting a server using CompositeServerSelector{ Selectors = MongoDB.Driver.MongoClient+AreSessionsSupportedServerSelector, LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 } }
Q1. Why is the error different from DCS driver and Robo 3t. It seems Robo 3t error is accurate. Is there any configuration to get an exact error in c# application too.
Q2. Is there anything to with the subscription for logs which is making it to create separate connection and not disposing of connection due to un-subscription is not being done?
Q3. I think that when we create a new instance of MongoClient it will actually create a new connection every time because it is not a static function and the memory, stack allocation will be new. So, there is no concept of pooling here.
Q4. I am also thinking if the object creation of ETLClass is making a difference because MongoDB driver reference is in ETLClass. Basically this is similar to Q3.
Q5. It seems this pooling, connection sharing will work only when we use global instance. outside the ETLClass and have it only one-time initialized. Is this correct?
I think the issue is Q3 and Q4. This is a silly mistake.
With regards,
Nithin B.