MongoDb-Kafka-Connector error

I am new to MongoDB-kafka - getting the following error —>

WARNING: Could not reach configured kafka connect system on http://localhost: 8083
Note: This script requires curl.

If using OSX please try reconfiguring Docker and increasing RAM and CPU. Then restart and try again.

Configuring MongoDB Connector —>

sleep 5

echo "\n\Configuring MongoDB Connector for Apache Kafka...\n\n"
curl --silent -X POST -H "Content-Type: application/json" -d @mongodb-sink.json  http://localhost:8083/connectors

My Sink-Connector →

{"name": "mongo-ts-sink",
    "config": {
      "connector.class":"com.mongodb.kafka.connect.MongoSinkConnector",
      "tasks.max":"1",
      "topics":"activity",
      "connection.uri"mongodb://mongo1",
      "database":"Stocks",
      "collection":"StockData",
      "key.converter":"org.apache.kafka.connect.storage.StringConverter",
      "value.converter":"org.apache.kafka.connect.json.JsonConverter",
      "value.converter.schemas.enable":"false",
      "timeseries.timefield":"tx_time",
      "timeseries.timefield.auto.convert":"true"
      
 }}

My connection —>

  connect:
    image: confluentinc/cp-kafka-connect-base:latest
    build:
      context: .
      dockerfile: Dockerfile-MongoConnect
    depends_on:
      - redpanda
    ports:
      - '8083:8083'

Thank you in advance.

How is your redpanda configured in your docker-compose file? Can you send the entire compose?

Hi Robert_Walters,

I npm installed curl - '“WARNING: Could not reach configured kafka connect system on http://localhost: 8083, Note: This script requires curl.” – > error is gone - see my → docker-compose.yml file below, but when i run ‘sh run.sh’, i am getting now

Kafka Connectors status:

run.sh: line 54: jq: command not found

Version of MongoDB Connector for Apache Kafka installed:

run.sh: line 58: jq: command not found

—> thats the last 2 echos on the run.sh

#!/bin/bash

set -e
(
if lsof -Pi :27017 -sTCP:LISTEN -t >/dev/null ; then
echo “Please terminate the local mongod on 27017, consider running ‘docker-compose down -v’”
exit 1
fi
)

echo “Starting docker .”
docker-compose up -d --build

sleep 5
echo “\n\nWaiting for the systems to be ready…”
function test_systems_available {
COUNTER=0
until $(curl --output /dev/null --silent --head --fail http://localhost:$1); do
printf ‘.’
sleep 2
let COUNTER+=1
if [[ $COUNTER -gt 30 ]]; then
MSG="\nWARNING: Could not reach configured kafka connect system on http://localhost:$1 \nNote: This script requires curl.\n"

      if [[ "$OSTYPE" == "darwin"* ]]; then
        MSG+="\nIf using OSX please try reconfiguring Docker and increasing RAM and CPU. Then restart and try again.\n\n"
      fi

    echo -e $MSG
    clean_up "$MSG"
    exit 1
  fi

done
}

test_systems_available 8083

#echo -e “\nConfiguring the MongoDB ReplicaSet of 1 node…\n”
#docker-compose exec mongo1 /usr/bin/mongo --eval ‘’‘rsconf = { _id : “rs0”, members: [ { _id : 0, host : “mongo1:27017”, priority: 1.0 }]};
#rs.initiate(rsconf);’’’

sleep 5

echo “\n\Configuring MongoDB Sink Connector for Apache Kafka…\n\n”
curl --silent -X POST -H “Content-Type: application/json” -d @mongodb-sink.json http://localhost:8083/connectors

echo “\n\Configuring MongoDB Source Connector for Apache Kafka…\n\n”
curl --silent -X POST -H “Content-Type: application/json” -d @mongodb-source.json http://localhost:8083/connectors

sleep 5

echo “\n\nKafka Connectors status:\n\n”
*curl -s “http://localhost:8083/connectors?expand=info&expand=status” | *

  •       jq '. | to_entries[] | [ .value.info.type, .key, .value.status.connector.state,.value.status.tasks[].state,.value.info.config."connector.class"]|join(":|:")' | \*
    
  •       column -s : -t| sed 's/\"//g'| sort*
    

echo “\n\nVersion of MongoDB Connector for Apache Kafka installed:\n”
curl --silent http://localhost:8083/connector-plugins | jq -c ‘. | select( .class == “com.mongodb.kafka.connect.MongoSourceConnector” or .class == “com.mongodb.kafka.connect.MongoSinkConnector” )’

echo ‘’’

==============================================================================================================

The following services are running:

MongoDB 1-node replica set on port 27017
Redpanda on 8082 (Redpanda proxy on 8083)
Kafka Connect on 8083
Node Server on 4000 is hosting the API and homepage

Status of kafka connectors:
sh status.sh - last 2 echos are not found

To tear down the environment and stop these serivces:
docker-compose down -v

==============================================================================================================
‘’’

sh status.sh

echo “Redpanda topics:\n”

curl --silent “http://localhost:8082/topics” | jq

echo “\nThe status of the connectors:\n”

curl -s “http://localhost:8083/connectors?expand=info&expand=status” |
jq ‘. | to_entries | [ .value.info.type, .key, .value.status.connector.state,.value.status.tasks.state,.value.info.config.“connector.class”]|join(":|:")’ |
column -s : -t| sed ‘s/"//g’| sort

echo “\nCurrently configured connectors\n”
curl --silent -X GET http://localhost:8083/connectors | jq

echo “\n\nVersion of MongoDB Connector for Apache Kafka installed:\n”
curl --silent http://localhost:8083/connector-plugins | jq -c ‘. | select( .class == “com.mongodb.kafka.connect.MongoSourceConnector” or .class == “com.mongodb.kafka.connect.MongoSinkConnector” )’

docker-compose.yml

version: ‘3.7’
services:

redpanda:
command:
- redpanda
- start
- --smp
- ‘1’
- --reserve-memory
- 0M
- --overprovisioned
- --node-id
- ‘0’
- --pandaproxy-addr
- 0.0.0.0:8082
- --advertise-pandaproxy-addr
- 127.0.0.1:8082
- --kafka-addr
- PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
- --advertise-kafka-addr
- PLAINTEXT://redpanda:29092,OUTSIDE://localhost:9092
image: docker.vectorized.io/vectorized/redpanda:latest
container_name: redpanda
hostname: redpanda
# networks:
# - localnet
ports:
- 9092:9092
- 29092:29092
- 8082:8082

connect:
image: confluentinc/cp-kafka-connect-base:latest
build:
context: .
dockerfile: Dockerfile-MongoConnect
hostname: connect
# container_name: connect
depends_on:
- redpanda
ports:
- “8083:8083”
# networks:
# - localnet
environment:
CONNECT_BOOTSTRAP_SERVERS: ‘redpanda:29092’
CONNECT_REST_ADVERTISED_HOST_NAME: connect
CONNECT_REST_PORT: 8083
CONNECT_GROUP_ID: connect-cluster-group
CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs
CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
CONNECT_OFFSET_FLUSH_INTERVAL_MS: 10000
CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets
CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status
CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
CONNECT_PLUGIN_PATH: “/usr/share/java,/usr/share/confluent-hub-components”
CONNECT_AUTO_CREATE_TOPICS_ENABLE: “true”
CONNECT_KEY_CONVERTER: “org.apache.kafka.connect.json.JsonConverter”
CONNECT_VALUE_CONVERTER: “org.apache.kafka.connect.json.JsonConverter”

mongo1:
image: “mongo:latest”
container_name: mongo1
command: --replSet rs0 --oplogSize 128
volumes:
- /data/db
# networks:
# - localnet
ports:
- “27017:27017”
restart: always

networks:

localnet:

attachable: true

volumes:

rs1:

nodesvr:
image: node:16
build:
context: .
dockerfile: Dockerfile-Nodesvr
depends_on:
- redpanda
- mongo1
volumes:
- ./backend/:/usr/app
# - /usr/app/node_modules
ports:
- “4000:4000”

  # env_file: ./server/.env # TODO - uncomment this to auto-load your .env file!
environment:
  - NODE_ENV=development
  # - CHOKIDAR_USEPOLLING=true

frontendsocket:
build:
context: ./frontendsocket/
command: npm start
volumes:
- ./frontendsocket/:/usr/app
- /usr/app/node_modules
depends_on:
- nodesvr
ports:
- “3000:3000”

Dockerfile-MongoConnect file

FROM confluentinc/cp-kafka-connect:latest

RUN confluent-hub install --no-prompt mongodb/kafka-connect-mongodb:latest

RUN confluent-hub install --no-prompt mongodb/kafka-connect-mongodb:latest

ENV CONNECT_PLUGIN_PATH="/usr/share/java,/usr/share/confluent-hub-components"

mongodb-sink.json

{“name”: “mongo-ts-sink”,
“config”: {
“connector.class”:“com.mongodb.kafka.connect.MongoSinkConnector”,
“tasks.max”:“1”,
“connection.uri”:“mongodb+srv://<username+password>@cluster0.ebt7p.mongodb.net/?retryWrites=true&w=majority”,
“topics”:“ChatData”,
“database”:“socketIo-MongoDb”,
“collection”:“chatfeed”,
“poll.max.batch.size”: “1000”,
“poll.await.time.ms”: “5000”,
“batch.size”: “1”,
“change.stream.full.document”: “updateLookup”,
“key.converter”: “org.apache.kafka.connect.storage.StringConverter”,
“value.converter”: “org.apache.kafka.connect.storage.StringConverter”,
“key.converter.schemas.enable”: “false”,
“value.converter.schemas.enable”: “false”,
“publish.full.document.only”: “true”,
“change.data.capture.handler”:“com.mongodb.kafka.connect.sink.cdc.mongodb.ChangeStreamHandler”

}}

mongodb-source.json

{
“name”: “mongo-source”,
“config”: {
“connector.class”: “com.mongodb.kafka.connect.MongoSourceConnector”,
“tasks.max”: “1”,
“connection.uri”: “mongodb+srv://<username+password>@cluster0.ebt7p.mongodb.net/?retryWrites=true&w=majority”,
“database”: “socketIo-MongoDb”,
“collection”:“chat”,
“copy.existing”: “true”,
“poll.max.batch.size”: “1000”,
“poll.await.time.ms”: “5000”,
“batch.size”: “1”,
“change.stream.full.document”: “updateLookup”,
“key.converter”: “org.apache.kafka.connect.storage.StringConverter”,
“value.converter”: “org.apache.kafka.connect.storage.StringConverter”,
“key.converter.schemas.enable”: “false”,
“value.converter.schemas.enable”: “false”,
“publish.full.document.only”: “true”
}
}

Even when i run.sh from → mongodb-redpanda-example-main repo in github, i am getting the same error in that repo

Thank you in advance

install jq

https://stedolan.github.io/jq/download/

then
do a docker ps to see if kafka connect and redpanda are running, if they are read the logs via docker logs xxxx where xxx is the container id, see if there are any errors

actually the issue is you are using the old name for the image, vectorized change it from

image: docker.vectorized.io/vectorized/redpanda:latest

to

‘image: docker.redpanda.com/vectorized/redpanda:latest

change that in the docker-compose file and retry

hi Robert_Walters,

Thank you, that error is gone but i am now unable to connect my react app to port 4000 in —> bundle.js:53895 GET http://localhost:4000/socket.io/?EIO=4&transport=polling&t=O8RKBri net::ERR_CONNECTION_REFUSED

docker-compose.yml

version: ‘3.7’
services:

redpanda:
command:
- redpanda
- start
- --smp
- ‘1’
- --reserve-memory
- 0M
- --overprovisioned
- --node-id
- ‘0’
- --pandaproxy-addr
- 0.0.0.0:8082
- --advertise-pandaproxy-addr
- 127.0.0.1:8082
- --kafka-addr
- PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
- --advertise-kafka-addr
- PLAINTEXT://redpanda:29092,OUTSIDE://localhost:9092
# image: docker.vectorized.io/vectorized/redpanda:latest
image: docker.redpanda.com/vectorized/redpanda:latest
container_name: redpanda
hostname: redpanda
# networks:
# - localnet
ports:
- 9092:9092
- 29092:29092
- 8082:8082

connect:
image: confluentinc/cp-kafka-connect-base:latest
build:
context: .
dockerfile: Dockerfile-MongoConnect
hostname: connect
# container_name: connect
depends_on:
- redpanda
ports:
- “8083:8083”
# networks:
# - localnet
environment:
CONNECT_BOOTSTRAP_SERVERS: ‘redpanda:29092’
CONNECT_REST_ADVERTISED_HOST_NAME: connect
CONNECT_REST_PORT: 8083
CONNECT_GROUP_ID: connect-cluster-group
CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs
CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
CONNECT_OFFSET_FLUSH_INTERVAL_MS: 10000
CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets
CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status
CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
CONNECT_PLUGIN_PATH: “/usr/share/java,/usr/share/confluent-hub-components”
CONNECT_AUTO_CREATE_TOPICS_ENABLE: “true”
CONNECT_KEY_CONVERTER: “org.apache.kafka.connect.json.JsonConverter”
CONNECT_VALUE_CONVERTER: “org.apache.kafka.connect.json.JsonConverter”

mongo1:
image: “mongo:latest”
# command: --replSet rs0 --oplogSize 128
volumes:
- /data/db
ports:
- “27017:27017”
restart: always

nodesvr:
image: node:16
build:
context: .
dockerfile: Dockerfile-Nodesvr
depends_on:
- redpanda
- mongo1

ports:
  - "4000:4000"

# backend:
#   build:
#     context: ./backend/
#   command: /usr/app/node_modules/.bin/nodemon server.js
#   volumes:
#     - ./backend/:/usr/app
#     - /usr/app/node_modules
#   depends_on:
#     - mongo1
#   ports:
#     - "4000:4000"

frontendsocket:
build:
context: ./frontendsocket/
command: npm start
volumes:
- ./frontendsocket/:/usr/app
- /usr/app/node_modules
depends_on:
- nodesvr
ports:
- “3000:3000”

Dockerfile-Nodesvr —>

FROM node:16

Create app directory /usr/src

WORKDIR /usr/src

Install app dependencies

A wildcard is used to ensure both package.json AND package-lock.json are copied

COPY ./package*.json ./

RUN npm install

RUN npm ci -qy

Bundle app source

COPY ./backend .

EXPOSE 4000

CMD [ “node”, “backend/server.js” ]

Thank you again

it looks like you do not have any networks defined for these, you commented out

# networks:
# - localnet

if you want all these components to be able to talk to each other they need to be on the same network so add networks: to each of the components then at the end of the docker compose file

networks:
  localnet:
    attachable: true

Thank you Robert_Walters, am still unable to send data to mongodb atlas db - bundle.js:53895 GET http://localhost:4000/socket.io/?EIO=4&transport=polling&t=O8Vho6a net::ERR_CONNECTION_REFUSED

const io = new Server(httpServer, {
cors: {
origin: “http://localhost:3000
}
});

my yml file —>

version: ‘3.7’
services:

redpanda:
command:
- redpanda
- start
- --smp
- ‘1’
- --reserve-memory
- 0M
- --overprovisioned
- --node-id
- ‘0’
- --pandaproxy-addr
- 0.0.0.0:8082
- --advertise-pandaproxy-addr
- 127.0.0.1:8082
- --kafka-addr
- PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
- --advertise-kafka-addr
- PLAINTEXT://redpanda:29092,OUTSIDE://localhost:9092
# image: docker.vectorized.io/vectorized/redpanda:latest
image: docker.redpanda.com/vectorized/redpanda:latest
container_name: redpanda
hostname: redpanda
networks:
- localnet
ports:
- 9092:9092
- 29092:29092
- 8082:8082

connect:
image: confluentinc/cp-kafka-connect-base:latest
build:
context: .
dockerfile: Dockerfile-MongoConnect
hostname: connect
# container_name: connect
depends_on:
- redpanda
ports:
- “8083:8083”
networks:
- localnet
environment:
CONNECT_BOOTSTRAP_SERVERS: ‘redpanda:29092’
CONNECT_REST_ADVERTISED_HOST_NAME: connect
CONNECT_REST_PORT: 8083
CONNECT_GROUP_ID: connect-cluster-group
CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs
CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
CONNECT_OFFSET_FLUSH_INTERVAL_MS: 10000
CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets
CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status
CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
CONNECT_PLUGIN_PATH: “/usr/share/java,/usr/share/confluent-hub-components”
CONNECT_AUTO_CREATE_TOPICS_ENABLE: “true”
CONNECT_KEY_CONVERTER: “org.apache.kafka.connect.json.JsonConverter”
CONNECT_VALUE_CONVERTER: “org.apache.kafka.connect.json.JsonConverter”

mongo1:
image: “mongo:latest”
# command: --replSet rs0 --oplogSize 128
volumes:
- /data/db
ports:
- “27017:27017”
networks:
- localnet
restart: always

nodesvr:
image: node:16
build:
context: .
dockerfile: Dockerfile-Nodesvr
depends_on:
- redpanda
- mongo1
networks:
- localnet

ports:
  - "4000:4000"

# backend:
#   build:
#     context: ./backend/
#   command: /usr/app/node_modules/.bin/nodemon server.js
#   volumes:
#     - ./backend/:/usr/app
#     - /usr/app/node_modules
#   depends_on:
#     - mongo1
#   ports:
#     - "4000:4000"

frontendsocket:
build:
context: ./frontendsocket/
command: npm start
volumes:
- ./frontendsocket/:/usr/app
- /usr/app/node_modules
depends_on:
- nodesvr
networks:
- localnet
ports:
- “3000:3000”

networks:
localnet:
attachable: true

Although below logs indicate i am connected to via port 4000

onesmusnyakotyo@onesmuss-iMac socketIo-MongoDb % docker compose logs nodesvr
nodesvr_1 | MongoDB connected…
nodesvr_1 | IO RUNNING on PORT 4000

Thank you again

you opened port 3000 in docker-compose but you are running on port 4000 within your code apparently so add the 4000 port to the docker-compose

ports:
 - “3000:3000”
 - "4000:4000"

Hi Robert_Walters,

Port 3000 is for my frontend(react app) ---->

frontendsocket:
build:
context: ./frontendsocket/
command: npm start
volumes:

  • ./frontendsocket/:/usr/app
  • /usr/app/node_modules
    depends_on:
  • nodesvr
    networks:
  • localnet
    ports:
  • “3000:3000”

Port 4000 for the nodeJs backend ---->

nodesvr:
image: node:16
build:
context: .
dockerfile: Dockerfile-Nodesvr
depends_on:
- redpanda
- mongo1
networks:
- localnet

ports:
  - "4000:4000"

Dockerfile-Nodesvr file ------>

FROM node:16

Create app directory /usr/src

WORKDIR /backend .

Install app dependencies

A wildcard is used to ensure both package.json AND package-lock.json are copied

COPY ./package*.json ./

RUN npm install

RUN npm ci -qy

Bundle app source

COPY . .

EXPOSE 4000

CMD [ “node”, “backend/server.js” ]

Thank you

is there anything the docker logs for mongodb ? are the containers still running? can you connect to mongodb from the host? Also you said MongoDB Atlas yet it looks like you are running an instnace of MongoDB locally, which one are you trying to connect to?

I want to connect to mongoDB atlas – uri: mongodb+srv://[username, password ]@cluster0.ebt7p.mongodb.net/?retryWrites=true&w=majority

docker compose logs mongo1 —>

mongo1_1 | {“t”:{"$date":“2022-07-21T20:35:58.956+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:23403, “ctx”:“initandlisten”,“msg”:“Build Info”,“attr”:{“buildInfo”:{“version”:“5.0.9”,“gitVersion”:“6f7dae919422dcd7f4892c10ff20cdc721ad00e6”,“openSSLVersion”:“OpenSSL 1.1.1f 31 Mar 2020”,“modules”:,“allocator”:“tcmalloc”,“environment”:{“distmod”:“ubuntu2004”,“distarch”:“x86_64”,“target_arch”:“x86_64”}}}}
mongo1_1 | {“t”:{"$date":“2022-07-21T20:35:58.956+00:00”},“s”:“I”, “c”:“CONTROL”, “id”:51765, “ctx”:“initandlisten”,“msg”:“Operating System”,“attr”:{“os”:{“name”:“Ubuntu”,“version”:“20.04”}}}

mongo1:
image: “mongo:latest”
# command: --replSet rs0 --oplogSize 128
volumes:
- /data/db
ports:
- “27017:27017”
networks:
- localnet
restart: always

nodesvr:
image: node:16
build:
context: .
dockerfile: Dockerfile-Nodesvr
depends_on:
- redpanda
- mongo1
networks:
- localnet

ports:
  - "4000:4000"

docker compose logs connect ---->

connect_1 | [2022-07-21 19:27:47,703] WARN [Producer clientId=connector-producer-mongo-source-0] Error while fetching metadata with correlation id 20237 : {socketIo-MongoDb.chat=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)
connect_1 | [2022-07-21 19:27:47,703] WARN [Consumer clientId=connector-consumer-mongo-ts-sink-0, groupId=connect-mongo-ts-sink] Error while fetching metadata with correlation id 21029 : {ChatData=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient)

you need to set auto topic create on in Kafka or create the topic beforehand look like

This is my connect environment variables as advised by redpanda team–>

environment:
CONNECT_BOOTSTRAP_SERVERS: ‘redpanda: 9092’
CONNECT_REST_ADVERTISED_HOST_NAME: connect
CONNECT_REST_PORT: 8083
CONNECT_GROUP_ID: connect-cluster-group
CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs
CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
CONNECT_CONFIG_STORAGE_PARTITIONS: 6
CONNECT_CONFIG_STORAGE_CLEANUP.POLICY: compact
CONNECT_OFFSET_FLUSH_INTERVAL_MS: 10000
CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets
CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status
CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
CONNECT_PLUGIN_PATH: “/usr/share/java,/usr/share/confluent-hub-components”
CONNECT_AUTO_CREATE_TOPICS_ENABLE: “true”
TOPIC_CREATION_ENABLE: “true”
TOPIC_CREATION_DEFAULT_REPLICATION_FACTOR: 3
TOPIC_CREATION_DEFAULT_PARTITIONS: 10
TOPIC_CREATION_DEFAULT__CLEANUP_POLICY: compact
TOPIC_CREATION_DEFAULT_COMPRESSION_TYPE: “lz4”
CONNECT_KEY_CONVERTER: “org.apache.kafka.connect.json.JsonConverter”
CONNECT_VALUE_CONVERTER: “org.apache.kafka.connect.json.JsonConverter”

Robert_Walters, can you advise on connecting to mongoDB Atlas in this setup, which is the local instance you mentioned above?, because i want my source connected to a mongoDB Atlas collection and then my sink connected to a different collection in mongoDB.

Did you enable the IP in MongoDB Atlas? See Accessing a MongoDB Atlas cluster in this blog MongoDB Atlas Tutorial | MongoDB.

Yes i did → 0.0.0.0/0 (includes your current IP address) and the database was working well before connecting to docker/kafka .

Thats the connection uri → mongodb+srv://[username, password ]@cluster0.ebt7p.mongodb.net/?retryWrites=true&w=majority

  1. For docker or redpanda kafka is there a specific IP that needs to be enabled?

  2. Please can you look at these mongo and node image is this correct config ---->

mongo1:
image: “mongo:latest”
container_name: mongodb
environment:
- MONGODB_CONNSTRING=mongodb+srv:// [username, password] @cluster0.ebt7p.mongodb.net/?retryWrites=true&w=majority
volumes:
- mongodb:/data/socketio-MongoDb
# - /data/socketio-MongoDb
networks:
- localnet

nodesvr:
image: node:16
build:
context: .
dockerfile: Dockerfile-Nodesvr
depends_on:
- redpanda
- mongo1
volumes:
- /app/node_modules
- ./nodesvr:/app
networks:
- localnet
ports:
- “4000:4000”

volumes:
mongodb:

if you are using Atlas you don’t need this at all, this is telling docker to install a MongoDB Cluster locally. (image:mongo:latest)

mongo1:
image: “mongo:latest”
container_name: mongodb
environment:
- MONGODB_CONNSTRING=mongodb+srv:// [username, password] @cluster0.ebt7p.mongodb.net/?retryWrites=true&w=majority
volumes:
- mongodb:/data/socketio-MongoDb
# - /data/socketio-MongoDb
networks:
- localnet

Thank you Robert_Walters for your advise, i removed the mongo1, now my connect logs (docker compose logs -f connect), is this a correct mongodb atlas connection —>

connect  | [2022-07-25 17:08:07,557] INFO Opened connection [connectionId{localValue:30, serverValue:112163}] to cluster0-shard-00-02.ebt7p.mongodb.net:27017 (org.mongodb.driver.connection)```

Looks like it works to me.

Thank you Robert_Walters, i am able to consume data from mongodb on cmd →

  "topic": "ChatData.socketIo-MongoDb.chat",
  "key": "{\"_id\": {\"_data\": \"8262DEF196000000122B022C0100296E5A1004FC2D06E10EF64CA2967AEFB29F6E510B46645F6964006462DEF1969F15802D8FA325BB0004\"}}",
  "value": "{\"_id\": {\"$oid\": \"62def1969f15802d8fa325bb\"}, \"name\": \"James Jamerson\", \"message\": \"Bonjour\"}",
  "timestamp": 1658778010584,
  "partition": 2,
  "offset": 2
}```

but sink is not sending the data to the collection on mongodb atlas ---> 

```{"name": "mongo-ts-sink",
    "config": {
      "connector.class":"com.mongodb.kafka.connect.MongoSinkConnector",
      "tasks.max":"1",
      "topics.regex": "chatData.*",
      "connection.uri":" ", 
      "database":"socketIo-MongoDb",
      "collection":"chatfeed",
      "auto.create": "true",
      "auto.evolve": "true",
      "insert.mode": "insert",
      "key.converter": "org.apache.kafka.connect.storage.StringConverter",
      "value.converter": "org.apache.kafka.connect.storage.StringConverter",
      "key.converter.schemas.enable": "false",
      "value.converter.schemas.enable": "false",
      "publish.full.document.only": "true"
            }
}```

what am i missing in this sink connector, thats stopping it from sending data to the 'chatfeed' collection

Thank you again Robert_Walters