we write this new version(10) for streaming from mongodb anything is good but it terminate
bin/spark-submit
the code is :
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.streaming.DataStreamWriter;
import org.apache.spark.sql.streaming.StreamingQuery;
import org.apache.spark.sql.streaming.Trigger;
import static org.apache.spark.sql.functions.*;
import java.util.concurrent.TimeoutException;
public final class MongoStructuredStreaming {
public static void main(final String[] args) {
/*
* Create the SparkSession.
* If config arguments are passed from the command line using --conf,
* parse args for the values to set.
*/
SparkSession spark = SparkSession.builder()
.master("local")
.appName("read_example")
.config("spark.mongodb.read.connection.uri", "mongodb://127.0.0.1/matching-engine.orders")
.config("spark.mongodb.write.connection.uri", "mongodb://127.0.0.1/matching-engine.orders")
.getOrCreate();
// define a streaming query
DataStreamWriter<Row> dataStreamWriter = spark.readStream()
.format("mongodb")
.load()
// manipulate your streaming data
.writeStream()
.format("console")
.trigger(Trigger.Continuous("1 second"))
.outputMode("append");
// run the query
try {
StreamingQuery query = dataStreamWriter.start();
} catch (TimeoutException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}
no errors found but application terminate it not shoould happen
the console is like this
23/03/09 08:23:52 INFO ContinuousExecution: Starting [id = 841c2f80-37f1-48ea-86f3-> 049166c84652, runId = d6a59ee6-44b4-4887-8c4a-b1d6f1b68019]. Use file:/C:/Users/joobin/AppData/Local/Temp/temporary-7336f1d4-9457-4c92-891b-f39ee54771b5 to store the query checkpoint.
23/03/09 08:23:52 INFO ContinuousExecution: Starting new streaming query.
23/03/09 08:23:52 INFO SparkContext: Invoking stop() from shutdown hook
23/03/09 08:23:52 INFO SparkUI: Stopped Spark web UI at http://DESKTOP-KK0D0F9:4041
23/03/09 08:23:52 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
23/03/09 08:23:52 INFO MemoryStore: MemoryStore cleared
23/03/09 08:23:52 INFO BlockManager: BlockManager stopped
23/03/09 08:23:52 INFO BlockManagerMaster: BlockManagerMaster stopped
23/03/09 08:23:52 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
23/03/09 08:23:52 INFO SparkContext: Successfully stopped SparkContext
23/03/09 08:23:52 INFO ShutdownHookManager: Shutdown hook called
the reading data from mongodb is ok but streaminng is not run in time
whats idea