Hi Robert_Walters,
I npm installed curl - '“WARNING: Could not reach configured kafka connect system on http://localhost: 8083, Note: This script requires curl.” – > error is gone - see my → docker-compose.yml file below, but when i run ‘sh run.sh’, i am getting now
Kafka Connectors status:
run.sh: line 54: jq: command not found
Version of MongoDB Connector for Apache Kafka installed:
run.sh: line 58: jq: command not found
—> thats the last 2 echos on the run.sh
#!/bin/bash
set -e
(
if lsof -Pi :27017 -sTCP:LISTEN -t >/dev/null ; then
echo “Please terminate the local mongod on 27017, consider running ‘docker-compose down -v’”
exit 1
fi
)
echo “Starting docker .”
docker-compose up -d --build
sleep 5
echo “\n\nWaiting for the systems to be ready…”
function test_systems_available {
COUNTER=0
until $(curl --output /dev/null --silent --head --fail http://localhost:$1); do
printf ‘.’
sleep 2
let COUNTER+=1
if [[ $COUNTER -gt 30 ]]; then
MSG=“\nWARNING: Could not reach configured kafka connect system on http://localhost:$1 \nNote: This script requires curl.\n”
if [[ "$OSTYPE" == "darwin"* ]]; then
MSG+="\nIf using OSX please try reconfiguring Docker and increasing RAM and CPU. Then restart and try again.\n\n"
fi
echo -e $MSG
clean_up "$MSG"
exit 1
fi
done
}
test_systems_available 8083
#echo -e “\nConfiguring the MongoDB ReplicaSet of 1 node…\n”
#docker-compose exec mongo1 /usr/bin/mongo --eval ‘’‘rsconf = { _id : “rs0”, members: [ { _id : 0, host : “mongo1:27017”, priority: 1.0 }]};
#rs.initiate(rsconf);’‘’
sleep 5
echo “\n\Configuring MongoDB Sink Connector for Apache Kafka…\n\n”
curl --silent -X POST -H “Content-Type: application/json” -d @mongodb-sink.json http://localhost:8083/connectors
echo “\n\Configuring MongoDB Source Connector for Apache Kafka…\n\n”
curl --silent -X POST -H “Content-Type: application/json” -d @mongodb-source.json http://localhost:8083/connectors
sleep 5
echo “\n\nKafka Connectors status:\n\n”
*curl -s “http://localhost:8083/connectors?expand=info&expand=status” | *
-
jq '. | to_entries[] | [ .value.info.type, .key, .value.status.connector.state,.value.status.tasks[].state,.value.info.config."connector.class"]|join(":|:")' | \*
-
column -s : -t| sed 's/\"//g'| sort*
echo “\n\nVersion of MongoDB Connector for Apache Kafka installed:\n”
curl --silent http://localhost:8083/connector-plugins | jq -c ‘. | select( .class == “com.mongodb.kafka.connect.MongoSourceConnector” or .class == “com.mongodb.kafka.connect.MongoSinkConnector” )’
echo ‘’’
==============================================================================================================
The following services are running:
MongoDB 1-node replica set on port 27017
Redpanda on 8082 (Redpanda proxy on 8083)
Kafka Connect on 8083
Node Server on 4000 is hosting the API and homepage
Status of kafka connectors:
sh status.sh - last 2 echos are not found
To tear down the environment and stop these serivces:
docker-compose down -v
==============================================================================================================
‘’’
sh status.sh
echo “Redpanda topics:\n”
curl --silent “http://localhost:8082/topics” | jq
echo “\nThe status of the connectors:\n”
curl -s “http://localhost:8083/connectors?expand=info&expand=status” |
jq ‘. | to_entries | [ .value.info.type, .key, .value.status.connector.state,.value.status.tasks.state,.value.info.config.“connector.class”]|join(“:|:”)’ |
column -s : -t| sed ‘s/"//g’| sort
echo “\nCurrently configured connectors\n”
curl --silent -X GET http://localhost:8083/connectors | jq
echo “\n\nVersion of MongoDB Connector for Apache Kafka installed:\n”
curl --silent http://localhost:8083/connector-plugins | jq -c ‘. | select( .class == “com.mongodb.kafka.connect.MongoSourceConnector” or .class == “com.mongodb.kafka.connect.MongoSinkConnector” )’
docker-compose.yml
version: ‘3.7’
services:
redpanda:
command:
- redpanda
- start
- --smp
- ‘1’
- --reserve-memory
- 0M
- --overprovisioned
- --node-id
- ‘0’
- --pandaproxy-addr
- 0.0.0.0:8082
- --advertise-pandaproxy-addr
- 127.0.0.1:8082
- --kafka-addr
- PLAINTEXT://0.0.0.0:29092,OUTSIDE://0.0.0.0:9092
- --advertise-kafka-addr
- PLAINTEXT://redpanda:29092,OUTSIDE://localhost:9092
image: docker.vectorized.io/vectorized/redpanda:latest
container_name: redpanda
hostname: redpanda
# networks:
# - localnet
ports:
- 9092:9092
- 29092:29092
- 8082:8082
connect:
image: confluentinc/cp-kafka-connect-base:latest
build:
context: .
dockerfile: Dockerfile-MongoConnect
hostname: connect
# container_name: connect
depends_on:
- redpanda
ports:
- “8083:8083”
# networks:
# - localnet
environment:
CONNECT_BOOTSTRAP_SERVERS: ‘redpanda:29092’
CONNECT_REST_ADVERTISED_HOST_NAME: connect
CONNECT_REST_PORT: 8083
CONNECT_GROUP_ID: connect-cluster-group
CONNECT_CONFIG_STORAGE_TOPIC: docker-connect-configs
CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR: 1
CONNECT_OFFSET_FLUSH_INTERVAL_MS: 10000
CONNECT_OFFSET_STORAGE_TOPIC: docker-connect-offsets
CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR: 1
CONNECT_STATUS_STORAGE_TOPIC: docker-connect-status
CONNECT_STATUS_STORAGE_REPLICATION_FACTOR: 1
CONNECT_PLUGIN_PATH: “/usr/share/java,/usr/share/confluent-hub-components”
CONNECT_AUTO_CREATE_TOPICS_ENABLE: “true”
CONNECT_KEY_CONVERTER: “org.apache.kafka.connect.json.JsonConverter”
CONNECT_VALUE_CONVERTER: “org.apache.kafka.connect.json.JsonConverter”
mongo1:
image: “mongo:latest”
container_name: mongo1
command: --replSet rs0 --oplogSize 128
volumes:
- /data/db
# networks:
# - localnet
ports:
- “27017:27017”
restart: always
networks:
localnet:
attachable: true
volumes:
rs1:
nodesvr:
image: node:16
build:
context: .
dockerfile: Dockerfile-Nodesvr
depends_on:
- redpanda
- mongo1
volumes:
- ./backend/:/usr/app
# - /usr/app/node_modules
ports:
- “4000:4000”
# env_file: ./server/.env # TODO - uncomment this to auto-load your .env file!
environment:
- NODE_ENV=development
# - CHOKIDAR_USEPOLLING=true
frontendsocket:
build:
context: ./frontendsocket/
command: npm start
volumes:
- ./frontendsocket/:/usr/app
- /usr/app/node_modules
depends_on:
- nodesvr
ports:
- “3000:3000”
Dockerfile-MongoConnect file
FROM confluentinc/cp-kafka-connect:latest
RUN confluent-hub install --no-prompt mongodb/kafka-connect-mongodb:latest
RUN confluent-hub install --no-prompt mongodb/kafka-connect-mongodb:latest
ENV CONNECT_PLUGIN_PATH=“/usr/share/java,/usr/share/confluent-hub-components”
mongodb-sink.json
{“name”: “mongo-ts-sink”,
“config”: {
“connector.class”:“com.mongodb.kafka.connect.MongoSinkConnector”,
“tasks.max”:“1”,
“connection.uri”:“mongodb+srv://<username+password>@cluster0.ebt7p.mongodb.net/?retryWrites=true&w=majority”,
“topics”:“ChatData”,
“database”:“socketIo-MongoDb”,
“collection”:“chatfeed”,
“poll.max.batch.size”: “1000”,
“poll.await.time.ms”: “5000”,
“batch.size”: “1”,
“change.stream.full.document”: “updateLookup”,
“key.converter”: “org.apache.kafka.connect.storage.StringConverter”,
“value.converter”: “org.apache.kafka.connect.storage.StringConverter”,
“key.converter.schemas.enable”: “false”,
“value.converter.schemas.enable”: “false”,
“publish.full.document.only”: “true”,
“change.data.capture.handler”:“com.mongodb.kafka.connect.sink.cdc.mongodb.ChangeStreamHandler”
}}
mongodb-source.json
{
“name”: “mongo-source”,
“config”: {
“connector.class”: “com.mongodb.kafka.connect.MongoSourceConnector”,
“tasks.max”: “1”,
“connection.uri”: “mongodb+srv://<username+password>@cluster0.ebt7p.mongodb.net/?retryWrites=true&w=majority”,
“database”: “socketIo-MongoDb”,
“collection”:“chat”,
“copy.existing”: “true”,
“poll.max.batch.size”: “1000”,
“poll.await.time.ms”: “5000”,
“batch.size”: “1”,
“change.stream.full.document”: “updateLookup”,
“key.converter”: “org.apache.kafka.connect.storage.StringConverter”,
“value.converter”: “org.apache.kafka.connect.storage.StringConverter”,
“key.converter.schemas.enable”: “false”,
“value.converter.schemas.enable”: “false”,
“publish.full.document.only”: “true”
}
}
Even when i run.sh from → mongodb-redpanda-example-main repo in github, i am getting the same error in that repo
Thank you in advance