Multiple Spark Versions / Interpreter on CDH 5.9 Cluster

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Multiple Spark Versions / Interpreter on CDH 5.9 Cluster

Meier, Alexander
Hi there

We have a CDH 5.9 Cluster where zeppelin 0.7.1is running fine with spark 1.6, impala, hive, hbase and apache phoenix. As we have both, Spark 1.6 and Spark 2 distributed and activated on the cluster, we now also want to use a spark2 interpreter in zeppelin.

I created a new interpreter %spark2 in zeppelin belonging to the spark interpreter group. 
Also I added an interpreter property “SPARK_HOME” with value /opt/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2 
(I have also tried to set SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2 with the same result.)

When testing the interpreter:
%spark2
sc.version
spark.version 
we get
res0: String = 1.6.0
<console>:28: error: not found: value spark
              spark.version

Why is the interpreter still using Spark 1.6? I thought the interpreter property would override the SPARK_HOME setting in zeppelin-env.sh

Any ideas?

btw: we also use impersonation and run the interpreter instantiated per user in isolated mode, but that  as "just in case” info.

Best regards
Alex


Reply | Threaded
Open this post in threaded view
|

Re: Multiple Spark Versions / Interpreter on CDH 5.9 Cluster

Meier, Alexander
Hi there

Any ideas why the new interpreter is not using the specified SPARK_HOME?

Best regards
Alex Meier


On 18 Oct 2017, at 11:14, Meier, Alexander <[hidden email]> wrote:

Hi there

We have a CDH 5.9 Cluster where zeppelin 0.7.1is running fine with spark 1.6, impala, hive, hbase and apache phoenix. As we have both, Spark 1.6 and Spark 2 distributed and activated on the cluster, we now also want to use a spark2 interpreter in zeppelin.

I created a new interpreter %spark2 in zeppelin belonging to the spark interpreter group. 
Also I added an interpreter property “SPARK_HOME” with value /opt/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2 
(I have also tried to set SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2 with the same result.)

When testing the interpreter:
%spark2
sc.version
spark.version 
we get
res0: String = 1.6.0
<console>:28: error: not found: value spark
              spark.version

Why is the interpreter still using Spark 1.6? I thought the interpreter property would override the SPARK_HOME setting in zeppelin-env.sh

Any ideas?

btw: we also use impersonation and run the interpreter instantiated per user in isolated mode, but that  as "just in case” info.

Best regards
Alex



Reply | Threaded
Open this post in threaded view
|

Re: Multiple Spark Versions / Interpreter on CDH 5.9 Cluster

Jeff Zhang

Take a look at spark submit command

Refer this link for details



Meier, Alexander <[hidden email]>于2017年10月24日周二 上午2:54写道:
Hi there

Any ideas why the new interpreter is not using the specified SPARK_HOME?

Best regards
Alex Meier


On 18 Oct 2017, at 11:14, Meier, Alexander <[hidden email]> wrote:

Hi there

We have a CDH 5.9 Cluster where zeppelin 0.7.1is running fine with spark 1.6, impala, hive, hbase and apache phoenix. As we have both, Spark 1.6 and Spark 2 distributed and activated on the cluster, we now also want to use a spark2 interpreter in zeppelin.

I created a new interpreter %spark2 in zeppelin belonging to the spark interpreter group. 
Also I added an interpreter property “SPARK_HOME” with value /opt/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2 
(I have also tried to set SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2 with the same result.)

When testing the interpreter:
%spark2
sc.version
spark.version 
we get
res0: String = 1.6.0
<console>:28: error: not found: value spark
              spark.version

Why is the interpreter still using Spark 1.6? I thought the interpreter property would override the SPARK_HOME setting in zeppelin-env.sh

Any ideas?

btw: we also use impersonation and run the interpreter instantiated per user in isolated mode, but that  as "just in case” info.

Best regards
Alex



Reply | Threaded
Open this post in threaded view
|

Re: Multiple Spark Versions / Interpreter on CDH 5.9 Cluster

Jongyoul Lee
Hi,

You can set SPARK_HOME in your interpreter tab instead of setting in zeppelin-env.sh. Remove SPARK_HOME in your zeppelin-env.sh and set it up interpreter tabs with two different spark interpreters.

On Tue, Oct 24, 2017 at 11:39 AM, Jeff Zhang <[hidden email]> wrote:

Take a look at spark submit command

Refer this link for details



Meier, Alexander <[hidden email]>于2017年10月24日周二 上午2:54写道:
Hi there

Any ideas why the new interpreter is not using the specified SPARK_HOME?

Best regards
Alex Meier


On 18 Oct 2017, at 11:14, Meier, Alexander <[hidden email]> wrote:

Hi there

We have a CDH 5.9 Cluster where zeppelin 0.7.1is running fine with spark 1.6, impala, hive, hbase and apache phoenix. As we have both, Spark 1.6 and Spark 2 distributed and activated on the cluster, we now also want to use a spark2 interpreter in zeppelin.

I created a new interpreter %spark2 in zeppelin belonging to the spark interpreter group. 
Also I added an interpreter property “SPARK_HOME” with value /opt/cloudera/parcels/SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904/lib/spark2 
(I have also tried to set SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2 with the same result.)

When testing the interpreter:
%spark2
sc.version
spark.version 
we get
res0: String = 1.6.0
<console>:28: error: not found: value spark
              spark.version

Why is the interpreter still using Spark 1.6? I thought the interpreter property would override the SPARK_HOME setting in zeppelin-env.sh

Any ideas?

btw: we also use impersonation and run the interpreter instantiated per user in isolated mode, but that  as "just in case” info.

Best regards
Alex






--
이종열, Jongyoul Lee, 李宗烈