AWS ElasticSearch Service with Zeppelin

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

AWS ElasticSearch Service with Zeppelin

Saksham Srivastava

Hi,


Has anyone tried to connect AWS managed ElasticSearch cluster to Zeppelin? Our managed elastic-search domain end-point look something like this:

https://search-abc-xyz-prod-fdizuwhbc4663qlh2tqghghydajycyu.us-east-1.es.amazonaws.com

I have successfully added the elasticsearch interpreter in Zeppelin (0.7.2) 


What should be the values of the following configs:

elasticsearch.client.type
elasticsearch.cluster.name
elasticsearch.host
elasticsearch.port


I have tried few configs like port being 443 etc. but none of them worked. The error I get is :Problem with the Elasticsearch client, please check your configuration (host, port,...)

Any help would be greatly appreciated.


Thanks,

Saksham


Reply | Threaded
Open this post in threaded view
|

Re: AWS ElasticSearch Service with Zeppelin

Park Hoon
Hi, I think u need to use 9200 or 9300 depending on the client. 



https://stackoverflow.com/questions/42721259/elasticsearch-interpreter-can-not-connect-to-database/42731420#42731420 


Regard, 

On Fri, Sep 1, 2017 at 10:46 PM, Saksham Srivastava <[hidden email]> wrote:

Hi,


Has anyone tried to connect AWS managed ElasticSearch cluster to Zeppelin? Our managed elastic-search domain end-point look something like this:

https://search-abc-xyz-prod-fdizuwhbc4663qlh2tqghghydajycyu.us-east-1.es.amazonaws.com

I have successfully added the elasticsearch interpreter in Zeppelin (0.7.2) 


What should be the values of the following configs:

elasticsearch.client.type
elasticsearch.host
elasticsearch.port


I have tried few configs like port being 443 etc. but none of them worked. The error I get is :Problem with the Elasticsearch client, please check your configuration (host, port,...)

Any help would be greatly appreciated.


Thanks,

Saksham



Reply | Threaded
Open this post in threaded view
|

Re: AWS ElasticSearch Service with Zeppelin

Saksham Srivastava

Hi,


Thanks for the suggestion. Unfortunately seems like AWS ES does not support 'transport' client-type.

However, I got it working using http client-type. I will list down the steps, will be useful for anyone trying the same. This is specific to connecting Zeppelin to AWS managed ElasticSearch service.

1) Make sure the Zeppelin Server IP is added to the allowed list of IPs for AWS ElasticSearch.

2) Install the Zeppelin elasticsearch interpreter.

3) From the JSON response which you get in the browser for the ES endpoint (ES end point might look like this: https://search-abc-xyz-prod-fdizuwhbc4663qlh2tqghghydajycyu.us-east-1.es.amazonaws.com/ ) get the cluster name value.

"cluster_name" : "130397394056:abc"

4) Configure the interpreter with the following properties:

elasticsearch.client.type http
elasticsearch.cluster.name 130397394056:abc
elasticsearch.host search-abc-xyz-prod-fdizuwhbc4663qlh2tqghghydajycyu.us-east-1.es.amazonaws.com
elasticsearch.port 80

5) Test the note with the ES query. 


Thanks,

Saksham


From: Park Hoon <[hidden email]>
Sent: Monday, September 4, 2017 7:59:26 AM
To: [hidden email]
Subject: Re: AWS ElasticSearch Service with Zeppelin
 
Hi, I think u need to use 9200 or 9300 depending on the client. 



https://stackoverflow.com/questions/42721259/elasticsearch-interpreter-can-not-connect-to-database/42731420#42731420 


Regard, 

On Fri, Sep 1, 2017 at 10:46 PM, Saksham Srivastava <[hidden email]> wrote:

Hi,


Has anyone tried to connect AWS managed ElasticSearch cluster to Zeppelin? Our managed elastic-search domain end-point look something like this:

https://search-abc-xyz-prod-fdizuwhbc4663qlh2tqghghydajycyu.us-east-1.es.amazonaws.com

I have successfully added the elasticsearch interpreter in Zeppelin (0.7.2) 


What should be the values of the following configs:

elasticsearch.client.type
elasticsearch.host
elasticsearch.port


I have tried few configs like port being 443 etc. but none of them worked. The error I get is :Problem with the Elasticsearch client, please check your configuration (host, port,...)

Any help would be greatly appreciated.


Thanks,

Saksham



Reply | Threaded
Open this post in threaded view
|

Re: AWS ElasticSearch Service with Zeppelin

Patrick Maroney
Thanks Saksham and Park...

Anyone have tips/guidance/reference links for running Elastic Stack <==> Zeppelin with Elastic Stack v5.5, X-Pack, and SSL Certs?

Have not been able to get anything to work....

Patrick Maroney
Principal Engineer - Data Science & Analytics
Wapack Labs LLC


On Sep 4, 2017, at 5:42 AM, Saksham Srivastava <[hidden email]> wrote:

Hi,

Thanks for the suggestion. Unfortunately seems like AWS ES does not support 'transport' client-type.

However, I got it working using http client-type. I will list down the steps, will be useful for anyone trying the same. This is specific to connecting Zeppelin to AWS managed ElasticSearch service.

1) Make sure the Zeppelin Server IP is added to the allowed list of IPs for AWS ElasticSearch.
2) Install the Zeppelin elasticsearch interpreter.
3) From the JSON response which you get in the browser for the ES endpoint (ES end point might look like this: https://search-abc-xyz-prod-fdizuwhbc4663qlh2tqghghydajycyu.us-east-1.es.amazonaws.com/ ) get the cluster name value.
"cluster_name" : "130397394056:abc"
4) Configure the interpreter with the following properties:

elasticsearch.client.type http
elasticsearch.cluster.name 130397394056:abc
elasticsearch.port 80
5) Test the note with the ES query. 

Thanks,

Saksham

From: Park Hoon <[hidden email]>
Sent: Monday, September 4, 2017 7:59:26 AM
To: [hidden email]
Subject: Re: AWS ElasticSearch Service with Zeppelin
 
Hi, I think u need to use 9200 or 9300 depending on the client. 



https://stackoverflow.com/questions/42721259/elasticsearch-interpreter-can-not-connect-to-database/42731420#42731420 


Regard, 

On Fri, Sep 1, 2017 at 10:46 PM, Saksham Srivastava <[hidden email]> wrote:
Hi,

Has anyone tried to connect AWS managed ElasticSearch cluster to Zeppelin? Our managed elastic-search domain end-point look something like this:
I have successfully added the elasticsearch interpreter in Zeppelin (0.7.2) 

What should be the values of the following configs:

elasticsearch.client.type
elasticsearch.host
elasticsearch.port

I have tried few configs like port being 443 etc. but none of them worked. The error I get is :Problem with the Elasticsearch client, please check your configuration (host, port,...)

Any help would be greatly appreciated.

Thanks,
Saksham


signature.asc (859 bytes) Download Attachment