Trying to use Relational Migrator with Kafka cluster

Hi there!

I’m trying to use Relational Migrator to migrate from SQL Server to MongoDB. I want to use Kafka just for academic purposes. I alredy have a Kafka cluster running locally. I know is working fine because i used the same Kafka server with Apache Flink, also for academic purposes.

The point is, i am following this guide:https://www.mongodb.com/docs/relational-migrator/installation/kafka-deployments/install-kafka-cluster/

I also checked with Kafka Connect that the new connectors are working.

My problem is, when i try to open Relational Migrator, there’s an error next to ‘Create Migration Job’ that says that Kafka has some kind of error. If a check the kafka server, i can see this:

WARN [SocketServer listenerType=ZK_BROKER, nodeId=0] Unexpected error from /127.0.0.1 (channelId=127.0.0.1:9092-127.0.0.1:50255-2); closing connection (org.apache.kafka.common.network.Selector)
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1195725856 larger than 904857600)

I suppose is related to the http connection. I tried to increment the socket.request.max.bytes in the server.properties of Kafka server, but doing that caused an out of memory error.

Here is the user.properties conf

spring.profiles.active: kafka
server.port: 8278
migrator.kafka.bootstrap.servers:localhost:9092
migrator.kafka.connect.url:localhost:9092

Any suggestions?

Ty very much for your time.

Thanks for trying out RM using Kafka!

A few follow up questions to help you diagnose the issue:

  1. The error shown near the ‘Create Migration Job’ button is a check which we do to attempt to validate the configuration provided. You mention that the new connectors are working, so are you able to start a job but you still see the error within the UI?

  2. Could you describe your Kafka setup detailing how many brokers are there and the version of Kafka?

  3. I notice the bootstrap servers and the connect url provided have the same port. Could you clarify how your Kafka Connect cluster is running?

Hello

First, thank you for your time.

Let’s answer the questions:

  1. No, i’m not able to start a job in Relational Migrator (i can actually with Apache Flink), but i can run the kafka connect command and i can see the statement in the logs, like ‘Added plugin…’.

  2. Since im running kafka standalone locally, i only have one broker.

  3. I think i answered your question in the previous point. I’m using kafka 3.7
    Maybe i have to change the connect url for localhost:8080?

I also tried to increment the socket.request.max.bytes in the server.properties of Kafka server. This causes and out of memory error. I tried to increment the memory of the JVM with export KAFKA_HEAP_OPTS=“-Xmx4G -Xms2G” and then i was able to start a job and the error disappeared. The problem was that, although Kafka didn’t show any errors, Relational Migrator showed this:

INFO org.apache.http.impl.execchain.RetryExec [pool-1-thread-1] I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://localhost:9092: The target server failed to respond
INFO org.apache.http.impl.execchain.RetryExec [pool-1-thread-1] Retrying request to {}->http://localhost:9092

And the running job never ends.

I hope this info helps.

Ty for your time again.

We have a reference docker-compose file which will guide you through an example setup running in Kafka mode. https://downloads.mongodb.org/migrator/1.6.0/docker-compose-migrator-kafka.yml

Note that the Kafka connect runs a seperate service so I expect that the connect url would be different from the bootstrap server. The default port for Kafka connect is usually 8083. Can you confirm you’re running Kafka connect as a seperate service?

Hope this reference docker file helps.