How to print data to console in structured streaming using Spark 2.1.0 and Zeppelin 0.7.1?

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

How to print data to console in structured streaming using Spark 2.1.0 and Zeppelin 0.7.1?

kant kodali
Hi All,

I have the following code.

 val ds = sparkSession.readStream()
                .format("kafka")
                .option("kafka.bootstrap.servers",bootstrapServers))
                .option("subscribe", topicName)
                .option("checkpointLocation", hdfsCheckPointDir)
                .load();
 val ds1 = ds.select($"value")
 val query = ds1.writeStream.outputMode("append").format("console").start()
 query.awaitTermination()
There are no errors when I execute this code however I don't see any data being printed out to console? When I run my standalone test Kafka consumer jar I can see that it is receiving messages. so I am not sure what is going on with above code? any ideas?

Thanks!
Loading...