Zeppelin help: custom interpreter to create data, then using this data with existing interpreters

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Zeppelin help: custom interpreter to create data, then using this data with existing interpreters

Jack Armstrong
Hello,

I have created a custom interpreter that collects data from a service with a custom query language, and I would like to be able to use this data with existing interpreters in Zeppelin, like the spark interpreters. Basically the scenario I'm imagining is the custom interpreter runs, formats the data into a data frame/RDD, injects the data collected into the context, and then subsequent paragraphs have interpreters from the spark group that process this data further. This is similar to what happens in the "Zeppelin Tutorial/Basic Features (Spark)" notebook where scala code creates some data, uses "registerTempTable" to put the data into the spark context, and then this data can be used in SQL scripts in later paragraphs.

How can I accomplish this? Is there a simple solution involving calling something like "registerTempTable" in the custom interpreter and then run the other interpreters normally below as the tutorial does?

Thank you for any guidance.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Zeppelin help: custom interpreter to create data, then using this data with existing interpreters

Park Hoon
Hi, @Jack Armstrong

Since I don't have details for your custom interpreter and Its language, it's hard to describe the easy way. 

But generally speaking, spark supports custom data source. 
It means you can load data and create RDD, DataFrame from any source in your way. 

(See the `others` box) 



https://databricks.com/blog/2015/01/09/spark-sql-data-sources-api-unified-data-access-for-the-spark-platform.html

https://michalsenkyr.github.io/2017/02/spark-sql_datasource


Hope this would be helpful. 


Regard, 

On Wed, Aug 2, 2017 at 7:27 AM, Jack Armstrong <[hidden email]> wrote:
Hello,

I have created a custom interpreter that collects data from a service with a custom query language, and I would like to be able to use this data with existing interpreters in Zeppelin, like the spark interpreters. Basically the scenario I'm imagining is the custom interpreter runs, formats the data into a data frame/RDD, injects the data collected into the context, and then subsequent paragraphs have interpreters from the spark group that process this data further. This is similar to what happens in the "Zeppelin Tutorial/Basic Features (Spark)" notebook where scala code creates some data, uses "registerTempTable" to put the data into the spark context, and then this data can be used in SQL scripts in later paragraphs.

How can I accomplish this? Is there a simple solution involving calling something like "registerTempTable" in the custom interpreter and then run the other interpreters normally below as the tutorial does?

Thank you for any guidance.

Loading...