Unable to run Zeppelin Spark on YARN

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Unable to run Zeppelin Spark on YARN

Yeshwanth Jagini
Hi we are running cloudera CDH 5.9.1 .

while setting up zeppelin, i followed the documentation on website and specified following options

export ZEPPELIN_JAVA_OPTS="-Dhadoop.version=2.6.0-cdh5.9.1"                     # Additional jvm options. for example, export ZEPPELIN_JAVA_OPTS="-Dspark.executor.memory=8g -Dspark.cores.max=16"

export SPARK_HOME="/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/lib/spark"                            # (required) When it is defined, load it instead of Zeppelin embedded Spark libraries
export SPARK_SUBMIT_OPTIONS="--master yarn --deploy-mode client"                   # (optional) extra options to pass to spark submit. eg) "--driver-memory 512M --executor-memory 1G".
export SPARK_APP_NAME=Zeppelin                         # (optional) The name of spark application.

export HADOOP_CONF_DIR=/etc/hadoop/conf                         # yarn-site.xml is located in configuration directory in HADOOP_CONF_DIR.

export ZEPPELIN_IMPERSONATE_CMD='sudo -H -u ${ZEPPELIN_IMPERSONATE_USER} bash -c'       # Optional, when user want to run interpreter as end web user. eg) 'sudo -H -u ${ZEPPELIN_IMPERSONATE_USER} bash -c '

when running spark notebook, spark-submit is running in local mode and i cannot see the application in yarn resource manager.
is there any other configuration i am missing?

 
Thanks,
Yeshwanth Jagini
Reply | Threaded
Open this post in threaded view
|

Re: Unable to run Zeppelin Spark on YARN

Jianfeng (Jeff) Zhang

Could you try set yarn-client in interpreter setting page ?


Best Regard,
Jeff Zhang


From: Yeshwanth Jagini <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Friday, May 5, 2017 at 3:13 AM
To: "[hidden email]" <[hidden email]>
Subject: Unable to run Zeppelin Spark on YARN

Hi we are running cloudera CDH 5.9.1 .

while setting up zeppelin, i followed the documentation on website and specified following options

export ZEPPELIN_JAVA_OPTS="-Dhadoop.version=2.6.0-cdh5.9.1"                     # Additional jvm options. for example, export ZEPPELIN_JAVA_OPTS="-Dspark.executor.memory=8g -Dspark.cores.max=16"

export SPARK_HOME="/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/lib/spark"                            # (required) When it is defined, load it instead of Zeppelin embedded Spark libraries
export SPARK_SUBMIT_OPTIONS="--master yarn --deploy-mode client"                   # (optional) extra options to pass to spark submit. eg) "--driver-memory 512M --executor-memory 1G".
export SPARK_APP_NAME=Zeppelin                         # (optional) The name of spark application.

export HADOOP_CONF_DIR=/etc/hadoop/conf                         # yarn-site.xml is located in configuration directory in HADOOP_CONF_DIR.

export ZEPPELIN_IMPERSONATE_CMD='sudo -H -u ${ZEPPELIN_IMPERSONATE_USER} bash -c'       # Optional, when user want to run interpreter as end web user. eg) 'sudo -H -u ${ZEPPELIN_IMPERSONATE_USER} bash -c '

when running spark notebook, spark-submit is running in local mode and i cannot see the application in yarn resource manager.
is there any other configuration i am missing?

 
Thanks,
Yeshwanth Jagini
Reply | Threaded
Open this post in threaded view
|

Re: Unable to run Zeppelin Spark on YARN

Jongyoul Lee
Hi,

"--master yarn --deploy-mode client" will be overridden when create spark context by SparkInterpreter In zeppelin. You have to set those values in interpreter setting page

Regards,
Jongyoul

On Fri, May 5, 2017 at 8:33 AM, Jianfeng (Jeff) Zhang <[hidden email]> wrote:

Could you try set yarn-client in interpreter setting page ?


Best Regard,
Jeff Zhang


From: Yeshwanth Jagini <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Friday, May 5, 2017 at 3:13 AM
To: "[hidden email]" <[hidden email]>
Subject: Unable to run Zeppelin Spark on YARN

Hi we are running cloudera CDH 5.9.1 .

while setting up zeppelin, i followed the documentation on website and specified following options

export ZEPPELIN_JAVA_OPTS="-Dhadoop.version=2.6.0-cdh5.9.1"                     # Additional jvm options. for example, export ZEPPELIN_JAVA_OPTS="-Dspark.executor.memory=8g -Dspark.cores.max=16"

export SPARK_HOME="/opt/cloudera/parcels/CDH-5.9.1-1.cdh5.9.1.p0.4/lib/spark"                            # (required) When it is defined, load it instead of Zeppelin embedded Spark libraries
export SPARK_SUBMIT_OPTIONS="--master yarn --deploy-mode client"                   # (optional) extra options to pass to spark submit. eg) "--driver-memory 512M --executor-memory 1G".
export SPARK_APP_NAME=Zeppelin                         # (optional) The name of spark application.

export HADOOP_CONF_DIR=/etc/hadoop/conf                         # yarn-site.xml is located in configuration directory in HADOOP_CONF_DIR.

export ZEPPELIN_IMPERSONATE_CMD='sudo -H -u ${ZEPPELIN_IMPERSONATE_USER} bash -c'       # Optional, when user want to run interpreter as end web user. eg) 'sudo -H -u ${ZEPPELIN_IMPERSONATE_USER} bash -c '

when running spark notebook, spark-submit is running in local mode and i cannot see the application in yarn resource manager.
is there any other configuration i am missing?

 
Thanks,
Yeshwanth Jagini



--
이종열, Jongyoul Lee, 李宗烈