Accessing Spark UI from Zeppelin

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Accessing Spark UI from Zeppelin

ankit jain

Hi Zeppelin users,

We are following https://issues.apache.org/jira/browse/ZEPPELIN-2949 to launch spark ui.

Our Zeppelin instance is deployed on AWS EMR master node and setting zeppelin.spark.uiWebUrl to a url which elb maps to https://masternode:4040.

When user clicks on spark url within Zeppelin it redirects him to Yarn RM( something like http://masternode:20888/proxy/application_1511906080313_0023/) which fails to load.

Usually to access EMR Web interfaces requires to setup a SSH tunnel and change proxy settings in the browser - http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-history.html

Is there a way we can avoid users having to setup ssh tunnel and allow direct access to Spark UI?

Ideally, we will implement a filter which does Authentication on the user and then redirect to Spark UI – right now not sure what the redirect URL should be?


-- 
Thanks & Regards,
Anki
Reply | Threaded
Open this post in threaded view
|

Re: Accessing Spark UI from Zeppelin

Felix Cheung
You could set to replace http://masternode with your custom http hostname.

Perhaps you want that to be set to a known, public (and authenticated?) IP/url? If you do have it it can be set to the Zeppelin config before Zeppelin starts.


From: ankit jain <[hidden email]>
Sent: Thursday, December 14, 2017 2:35:32 PM
To: [hidden email]
Cc: Esteban de Jesus Hernandez; Jhon; Emmanuel; Oliver; Phil
Subject: Accessing Spark UI from Zeppelin
 

Hi Zeppelin users,

We are following https://issues.apache.org/jira/browse/ZEPPELIN-2949 to launch spark ui.

Our Zeppelin instance is deployed on AWS EMR master node and setting zeppelin.spark.uiWebUrl to a url which elb maps to https://masternode:4040.

When user clicks on spark url within Zeppelin it redirects him to Yarn RM( something like http://masternode:20888/proxy/application_1511906080313_0023/) which fails to load.

Usually to access EMR Web interfaces requires to setup a SSH tunnel and change proxy settings in the browser - http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-spark-history.html

Is there a way we can avoid users having to setup ssh tunnel and allow direct access to Spark UI?

Ideally, we will implement a filter which does Authentication on the user and then redirect to Spark UI – right now not sure what the redirect URL should be?


-- 
Thanks & Regards,
Anki