Cloudera Spark 2.2

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Cloudera Spark 2.2

Benjamin Kim
Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something.

Cheers,
Ben
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cloudera Spark 2.2

Jeff Zhang

What's the error you see in log ?


Benjamin Kim <[hidden email]>于2017年8月2日周三 上午8:18写道:
Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something.

Cheers,
Ben
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cloudera Spark 2.2

Jeff Zhang

By default, 0.7.1 doesn't support spark 2.2. But you can set zeppelin.spark.enableSupportedVersionCheck in interpreter setting to disable the supported version check.


Jeff Zhang <[hidden email]>于2017年8月2日周三 上午8:18写道:

What's the error you see in log ?


Benjamin Kim <[hidden email]>于2017年8月2日周三 上午8:18写道:
Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something.

Cheers,
Ben
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cloudera Spark 2.2

Benjamin Kim
Here is the error that was sent to me.

org.apache.zeppelin.interpreter.InterpreterException: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream

Cheers,
Ben


On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang <[hidden email]> wrote:

By default, 0.7.1 doesn't support spark 2.2. But you can set zeppelin.spark.enableSupportedVersionCheck in interpreter setting to disable the supported version check.


Jeff Zhang <[hidden email]>于2017年8月2日周三 上午8:18写道:

What's the error you see in log ?


Benjamin Kim <[hidden email]>于2017年8月2日周三 上午8:18写道:
Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something.

Cheers,
Ben
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cloudera Spark 2.2

Jeff Zhang

Then it is due to some classpath issue. I am not sure familiar with CDH, please check whether spark of CDH include hadoop jar with it. 


Benjamin Kim <[hidden email]>于2017年8月2日周三 上午8:22写道:
Here is the error that was sent to me.

org.apache.zeppelin.interpreter.InterpreterException: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream

Cheers,
Ben


On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang <[hidden email]> wrote:

By default, 0.7.1 doesn't support spark 2.2. But you can set zeppelin.spark.enableSupportedVersionCheck in interpreter setting to disable the supported version check.


Jeff Zhang <[hidden email]>于2017年8月2日周三 上午8:18写道:

What's the error you see in log ?


Benjamin Kim <[hidden email]>于2017年8月2日周三 上午8:18写道:
Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something.

Cheers,
Ben
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cloudera Spark 2.2

Benjamin Kim
Here is more.

org.apache.zeppelin.interpreter.InterpreterException: WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2).
WARNING: Running spark-class from user-defined location.
Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
at org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103)
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:110)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Cheers,
Ben


On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang <[hidden email]> wrote:

Then it is due to some classpath issue. I am not sure familiar with CDH, please check whether spark of CDH include hadoop jar with it. 


Benjamin Kim <[hidden email]>于2017年8月2日周三 上午8:22写道:
Here is the error that was sent to me.

org.apache.zeppelin.interpreter.InterpreterException: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream

Cheers,
Ben


On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang <[hidden email]> wrote:

By default, 0.7.1 doesn't support spark 2.2. But you can set zeppelin.spark.enableSupportedVersionCheck in interpreter setting to disable the supported version check.


Jeff Zhang <[hidden email]>于2017年8月2日周三 上午8:18写道:

What's the error you see in log ?


Benjamin Kim <[hidden email]>于2017年8月2日周三 上午8:18写道:
Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something.

Cheers,
Ben
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cloudera Spark 2.2

Ruslan Dautkhanov
Might need to recompile Zeppelin with Scala 2.11?
Also Spark 2.2 now requires JDK8 I believe.



--
Ruslan Dautkhanov

On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim <[hidden email]> wrote:
Here is more.

org.apache.zeppelin.interpreter.InterpreterException: WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2).
WARNING: Running spark-class from user-defined location.
Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
at org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103)
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:110)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Cheers,
Ben


On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang <[hidden email]> wrote:

Then it is due to some classpath issue. I am not sure familiar with CDH, please check whether spark of CDH include hadoop jar with it. 


Benjamin Kim <[hidden email]>于2017年8月2日周三 上午8:22写道:
Here is the error that was sent to me.

org.apache.zeppelin.interpreter.InterpreterException: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream

Cheers,
Ben


On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang <[hidden email]> wrote:

By default, 0.7.1 doesn't support spark 2.2. But you can set zeppelin.spark.enableSupportedVersionCheck in interpreter setting to disable the supported version check.


Jeff Zhang <[hidden email]>于2017年8月2日周三 上午8:18写道:

What's the error you see in log ?


Benjamin Kim <[hidden email]>于2017年8月2日周三 上午8:18写道:
Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something.

Cheers,
Ben

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cloudera Spark 2.2

Benjamin Kim
According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera. For some reason, Cloudera defaults to Spark 1.6 and so does the calls to spark-shell and spark-submit. To force the use of Spark 2.x, the calls need to be spark2-shell and spark2-submit. I wonder if this is causing the problem. By the way, we are using Java8 corporate wide, and there seems to be no problems using Zeppelin.

Cheers,
Ben

On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov <[hidden email]> wrote:
Might need to recompile Zeppelin with Scala 2.11?
Also Spark 2.2 now requires JDK8 I believe.



--
Ruslan Dautkhanov

On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim <[hidden email]> wrote:
Here is more.

org.apache.zeppelin.interpreter.InterpreterException: WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2).
WARNING: Running spark-class from user-defined location.
Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
at org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103)
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:110)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Cheers,
Ben


On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang <[hidden email]> wrote:

Then it is due to some classpath issue. I am not sure familiar with CDH, please check whether spark of CDH include hadoop jar with it. 


Benjamin Kim <[hidden email]>于2017年8月2日周三 上午8:22写道:
Here is the error that was sent to me.

org.apache.zeppelin.interpreter.InterpreterException: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream

Cheers,
Ben


On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang <[hidden email]> wrote:

By default, 0.7.1 doesn't support spark 2.2. But you can set zeppelin.spark.enableSupportedVersionCheck in interpreter setting to disable the supported version check.


Jeff Zhang <[hidden email]>于2017年8月2日周三 上午8:18写道:

What's the error you see in log ?


Benjamin Kim <[hidden email]>于2017年8月2日周三 上午8:18写道:
Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something.

Cheers,
Ben

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cloudera Spark 2.2

Ruslan Dautkhanov
You don't have to use spark2-shell and spark2-submit to use Spark 2.
That can be controled by setting SPARK_HOME using regular spark-submit/spark-shell.

$ which spark-submit
/usr/bin/spark-submit
$ which spark-shell
/usr/bin/spark-shell

$ spark-shell
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.6.0
      /_/



$ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2

$ spark-shell
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.0.cloudera1
      /_/


spark-submit and spark-shell are just shell script wrappers.


--
Ruslan Dautkhanov

On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim <[hidden email]> wrote:
According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera. For some reason, Cloudera defaults to Spark 1.6 and so does the calls to spark-shell and spark-submit. To force the use of Spark 2.x, the calls need to be spark2-shell and spark2-submit. I wonder if this is causing the problem. By the way, we are using Java8 corporate wide, and there seems to be no problems using Zeppelin.

Cheers,
Ben

On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov <[hidden email]> wrote:
Might need to recompile Zeppelin with Scala 2.11?
Also Spark 2.2 now requires JDK8 I believe.



--
Ruslan Dautkhanov

On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim <[hidden email]> wrote:
Here is more.

org.apache.zeppelin.interpreter.InterpreterException: WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2).
WARNING: Running spark-class from user-defined location.
Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
at org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103)
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:110)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Cheers,
Ben


On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang <[hidden email]> wrote:

Then it is due to some classpath issue. I am not sure familiar with CDH, please check whether spark of CDH include hadoop jar with it. 


Benjamin Kim <[hidden email]>于2017年8月2日周三 上午8:22写道:
Here is the error that was sent to me.

org.apache.zeppelin.interpreter.InterpreterException: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream

Cheers,
Ben


On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang <[hidden email]> wrote:

By default, 0.7.1 doesn't support spark 2.2. But you can set zeppelin.spark.enableSupportedVersionCheck in interpreter setting to disable the supported version check.


Jeff Zhang <[hidden email]>于2017年8月2日周三 上午8:18写道:

What's the error you see in log ?


Benjamin Kim <[hidden email]>于2017年8月2日周三 上午8:18写道:
Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something.

Cheers,
Ben


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cloudera Spark 2.2

Benjamin Kim
Does this work with Zeppelin 0.7.1? We an error when setting SPARK_HOME in zeppelin-env.sh to what you have below.

On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov <[hidden email]> wrote:
You don't have to use spark2-shell and spark2-submit to use Spark 2.
That can be controled by setting SPARK_HOME using regular spark-submit/spark-shell.

$ which spark-submit
/usr/bin/spark-submit
$ which spark-shell
/usr/bin/spark-shell

$ spark-shell
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.6.0
      /_/



$ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2

$ spark-shell
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.0.cloudera1
      /_/


spark-submit and spark-shell are just shell script wrappers.



--
Ruslan Dautkhanov

On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim <[hidden email]> wrote:
According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera. For some reason, Cloudera defaults to Spark 1.6 and so does the calls to spark-shell and spark-submit. To force the use of Spark 2.x, the calls need to be spark2-shell and spark2-submit. I wonder if this is causing the problem. By the way, we are using Java8 corporate wide, and there seems to be no problems using Zeppelin.

Cheers,
Ben

On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov <[hidden email]> wrote:
Might need to recompile Zeppelin with Scala 2.11?
Also Spark 2.2 now requires JDK8 I believe.



--
Ruslan Dautkhanov

On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim <[hidden email]> wrote:
Here is more.

org.apache.zeppelin.interpreter.InterpreterException: WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2).
WARNING: Running spark-class from user-defined location.
Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
at org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103)
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:110)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Cheers,
Ben


On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang <[hidden email]> wrote:

Then it is due to some classpath issue. I am not sure familiar with CDH, please check whether spark of CDH include hadoop jar with it. 


Benjamin Kim <[hidden email]>于2017年8月2日周三 上午8:22写道:
Here is the error that was sent to me.

org.apache.zeppelin.interpreter.InterpreterException: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream

Cheers,
Ben


On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang <[hidden email]> wrote:

By default, 0.7.1 doesn't support spark 2.2. But you can set zeppelin.spark.enableSupportedVersionCheck in interpreter setting to disable the supported version check.


Jeff Zhang <[hidden email]>于2017年8月2日周三 上午8:18写道:

What's the error you see in log ?


Benjamin Kim <[hidden email]>于2017年8月2日周三 上午8:18写道:
Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something.

Cheers,
Ben


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cloudera Spark 2.2

Ruslan Dautkhanov
We're using an ~April snapshot of Zeppelin, so not sure about 0.7.1.

Yes, we have that spark home in zeppelin-env.sh



--
Ruslan Dautkhanov

On Wed, Aug 2, 2017 at 4:31 PM, Benjamin Kim <[hidden email]> wrote:
Does this work with Zeppelin 0.7.1? We an error when setting SPARK_HOME in zeppelin-env.sh to what you have below.

On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov <[hidden email]> wrote:
You don't have to use spark2-shell and spark2-submit to use Spark 2.
That can be controled by setting SPARK_HOME using regular spark-submit/spark-shell.

$ which spark-submit
/usr/bin/spark-submit
$ which spark-shell
/usr/bin/spark-shell

$ spark-shell
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.6.0
      /_/



$ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2

$ spark-shell
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.0.cloudera1
      /_/


spark-submit and spark-shell are just shell script wrappers.



--
Ruslan Dautkhanov

On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim <[hidden email]> wrote:
According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera. For some reason, Cloudera defaults to Spark 1.6 and so does the calls to spark-shell and spark-submit. To force the use of Spark 2.x, the calls need to be spark2-shell and spark2-submit. I wonder if this is causing the problem. By the way, we are using Java8 corporate wide, and there seems to be no problems using Zeppelin.

Cheers,
Ben

On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov <[hidden email]> wrote:
Might need to recompile Zeppelin with Scala 2.11?
Also Spark 2.2 now requires JDK8 I believe.



--
Ruslan Dautkhanov

On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim <[hidden email]> wrote:
Here is more.

org.apache.zeppelin.interpreter.InterpreterException: WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2).
WARNING: Running spark-class from user-defined location.
Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
at org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103)
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:110)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Cheers,
Ben


On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang <[hidden email]> wrote:

Then it is due to some classpath issue. I am not sure familiar with CDH, please check whether spark of CDH include hadoop jar with it. 


Benjamin Kim <[hidden email]>于2017年8月2日周三 上午8:22写道:
Here is the error that was sent to me.

org.apache.zeppelin.interpreter.InterpreterException: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream

Cheers,
Ben


On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang <[hidden email]> wrote:

By default, 0.7.1 doesn't support spark 2.2. But you can set zeppelin.spark.enableSupportedVersionCheck in interpreter setting to disable the supported version check.


Jeff Zhang <[hidden email]>于2017年8月2日周三 上午8:18写道:

What's the error you see in log ?


Benjamin Kim <[hidden email]>于2017年8月2日周三 上午8:18写道:
Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something.

Cheers,
Ben



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cloudera Spark 2.2

Benjamin Kim
Did you build Zeppelin or download the binary?

On Wed, Aug 2, 2017 at 3:40 PM Ruslan Dautkhanov <[hidden email]> wrote:
We're using an ~April snapshot of Zeppelin, so not sure about 0.7.1.

Yes, we have that spark home in zeppelin-env.sh



--
Ruslan Dautkhanov

On Wed, Aug 2, 2017 at 4:31 PM, Benjamin Kim <[hidden email]> wrote:
Does this work with Zeppelin 0.7.1? We an error when setting SPARK_HOME in zeppelin-env.sh to what you have below.

On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov <[hidden email]> wrote:
You don't have to use spark2-shell and spark2-submit to use Spark 2.
That can be controled by setting SPARK_HOME using regular spark-submit/spark-shell.

$ which spark-submit
/usr/bin/spark-submit
$ which spark-shell
/usr/bin/spark-shell

$ spark-shell
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.6.0
      /_/



$ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2

$ spark-shell
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.0.cloudera1
      /_/


spark-submit and spark-shell are just shell script wrappers.



--
Ruslan Dautkhanov

On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim <[hidden email]> wrote:
According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera. For some reason, Cloudera defaults to Spark 1.6 and so does the calls to spark-shell and spark-submit. To force the use of Spark 2.x, the calls need to be spark2-shell and spark2-submit. I wonder if this is causing the problem. By the way, we are using Java8 corporate wide, and there seems to be no problems using Zeppelin.

Cheers,
Ben

On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov <[hidden email]> wrote:
Might need to recompile Zeppelin with Scala 2.11?
Also Spark 2.2 now requires JDK8 I believe.



--
Ruslan Dautkhanov

On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim <[hidden email]> wrote:
Here is more.

org.apache.zeppelin.interpreter.InterpreterException: WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2).
WARNING: Running spark-class from user-defined location.
Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
at org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103)
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:110)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Cheers,
Ben


On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang <[hidden email]> wrote:

Then it is due to some classpath issue. I am not sure familiar with CDH, please check whether spark of CDH include hadoop jar with it. 


Benjamin Kim <[hidden email]>于2017年8月2日周三 上午8:22写道:
Here is the error that was sent to me.

org.apache.zeppelin.interpreter.InterpreterException: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream

Cheers,
Ben


On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang <[hidden email]> wrote:

By default, 0.7.1 doesn't support spark 2.2. But you can set zeppelin.spark.enableSupportedVersionCheck in interpreter setting to disable the supported version check.


Jeff Zhang <[hidden email]>于2017年8月2日周三 上午8:18写道:

What's the error you see in log ?


Benjamin Kim <[hidden email]>于2017年8月2日周三 上午8:18写道:
Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something.

Cheers,
Ben



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cloudera Spark 2.2

Ruslan Dautkhanov
It was built. I think binaries are only available for official releases?



--
Ruslan Dautkhanov

On Wed, Aug 2, 2017 at 4:41 PM, Benjamin Kim <[hidden email]> wrote:
Did you build Zeppelin or download the binary?

On Wed, Aug 2, 2017 at 3:40 PM Ruslan Dautkhanov <[hidden email]> wrote:
We're using an ~April snapshot of Zeppelin, so not sure about 0.7.1.

Yes, we have that spark home in zeppelin-env.sh



--
Ruslan Dautkhanov

On Wed, Aug 2, 2017 at 4:31 PM, Benjamin Kim <[hidden email]> wrote:
Does this work with Zeppelin 0.7.1? We an error when setting SPARK_HOME in zeppelin-env.sh to what you have below.

On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov <[hidden email]> wrote:
You don't have to use spark2-shell and spark2-submit to use Spark 2.
That can be controled by setting SPARK_HOME using regular spark-submit/spark-shell.

$ which spark-submit
/usr/bin/spark-submit
$ which spark-shell
/usr/bin/spark-shell

$ spark-shell
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.6.0
      /_/



$ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2

$ spark-shell
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.0.cloudera1
      /_/


spark-submit and spark-shell are just shell script wrappers.



--
Ruslan Dautkhanov

On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim <[hidden email]> wrote:
According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera. For some reason, Cloudera defaults to Spark 1.6 and so does the calls to spark-shell and spark-submit. To force the use of Spark 2.x, the calls need to be spark2-shell and spark2-submit. I wonder if this is causing the problem. By the way, we are using Java8 corporate wide, and there seems to be no problems using Zeppelin.

Cheers,
Ben

On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov <[hidden email]> wrote:
Might need to recompile Zeppelin with Scala 2.11?
Also Spark 2.2 now requires JDK8 I believe.



--
Ruslan Dautkhanov

On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim <[hidden email]> wrote:
Here is more.

org.apache.zeppelin.interpreter.InterpreterException: WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2).
WARNING: Running spark-class from user-defined location.
Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
at org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103)
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:110)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Cheers,
Ben


On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang <[hidden email]> wrote:

Then it is due to some classpath issue. I am not sure familiar with CDH, please check whether spark of CDH include hadoop jar with it. 


Benjamin Kim <[hidden email]>于2017年8月2日周三 上午8:22写道:
Here is the error that was sent to me.

org.apache.zeppelin.interpreter.InterpreterException: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream

Cheers,
Ben


On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang <[hidden email]> wrote:

By default, 0.7.1 doesn't support spark 2.2. But you can set zeppelin.spark.enableSupportedVersionCheck in interpreter setting to disable the supported version check.


Jeff Zhang <[hidden email]>于2017年8月2日周三 上午8:18写道:

What's the error you see in log ?


Benjamin Kim <[hidden email]>于2017年8月2日周三 上午8:18写道:
Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something.

Cheers,
Ben




Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cloudera Spark 2.2

Benjamin Kim
Hi Ruslan,

Can you send me the steps you used to build it, especially the Maven command with the arguments? I will try to build it also.

I do believe that the binaries are for official releases.

Cheers,
Ben


On Wed, Aug 2, 2017 at 3:44 PM Ruslan Dautkhanov <[hidden email]> wrote:
It was built. I think binaries are only available for official releases?



--
Ruslan Dautkhanov

On Wed, Aug 2, 2017 at 4:41 PM, Benjamin Kim <[hidden email]> wrote:
Did you build Zeppelin or download the binary?

On Wed, Aug 2, 2017 at 3:40 PM Ruslan Dautkhanov <[hidden email]> wrote:
We're using an ~April snapshot of Zeppelin, so not sure about 0.7.1.

Yes, we have that spark home in zeppelin-env.sh



--
Ruslan Dautkhanov

On Wed, Aug 2, 2017 at 4:31 PM, Benjamin Kim <[hidden email]> wrote:
Does this work with Zeppelin 0.7.1? We an error when setting SPARK_HOME in zeppelin-env.sh to what you have below.

On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov <[hidden email]> wrote:
You don't have to use spark2-shell and spark2-submit to use Spark 2.
That can be controled by setting SPARK_HOME using regular spark-submit/spark-shell.

$ which spark-submit
/usr/bin/spark-submit
$ which spark-shell
/usr/bin/spark-shell

$ spark-shell
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.6.0
      /_/



$ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2

$ spark-shell
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.0.cloudera1
      /_/


spark-submit and spark-shell are just shell script wrappers.



--
Ruslan Dautkhanov

On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim <[hidden email]> wrote:
According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera. For some reason, Cloudera defaults to Spark 1.6 and so does the calls to spark-shell and spark-submit. To force the use of Spark 2.x, the calls need to be spark2-shell and spark2-submit. I wonder if this is causing the problem. By the way, we are using Java8 corporate wide, and there seems to be no problems using Zeppelin.

Cheers,
Ben

On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov <[hidden email]> wrote:
Might need to recompile Zeppelin with Scala 2.11?
Also Spark 2.2 now requires JDK8 I believe.



--
Ruslan Dautkhanov

On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim <[hidden email]> wrote:
Here is more.

org.apache.zeppelin.interpreter.InterpreterException: WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2).
WARNING: Running spark-class from user-defined location.
Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
at org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103)
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:110)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Cheers,
Ben


On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang <[hidden email]> wrote:

Then it is due to some classpath issue. I am not sure familiar with CDH, please check whether spark of CDH include hadoop jar with it. 


Benjamin Kim <[hidden email]>于2017年8月2日周三 上午8:22写道:
Here is the error that was sent to me.

org.apache.zeppelin.interpreter.InterpreterException: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream

Cheers,
Ben


On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang <[hidden email]> wrote:

By default, 0.7.1 doesn't support spark 2.2. But you can set zeppelin.spark.enableSupportedVersionCheck in interpreter setting to disable the supported version check.


Jeff Zhang <[hidden email]>于2017年8月2日周三 上午8:18写道:

What's the error you see in log ?


Benjamin Kim <[hidden email]>于2017年8月2日周三 上午8:18写道:
Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something.

Cheers,
Ben




Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cloudera Spark 2.2

Ruslan Dautkhanov
This should do:
 
export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2
export HIVE_HOME=/opt/cloudera/parcels/CDH/lib/hive
export HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop
export HADOOP_CONF_DIR=/etc/hadoop/conf
export HIVE_CONF_DIR=/etc/hive/conf
 

mvn clean package -DskipTests -Pspark-2.1 -Dhadoop.version=2.6.0-cdh5.10.1 -Phadoop-2.6 -Pvendor-repo -Pscala-2.10 -Psparkr -pl '!alluxio,!flink,!ignite,!lens,!cassandra,!bigquery,!scio' -e

You may needs additional steps depending which interpreters you use (like R etc).


--
Ruslan Dautkhanov

On Fri, Aug 4, 2017 at 8:31 AM, Benjamin Kim <[hidden email]> wrote:
Hi Ruslan,

Can you send me the steps you used to build it, especially the Maven command with the arguments? I will try to build it also.

I do believe that the binaries are for official releases.

Cheers,
Ben


On Wed, Aug 2, 2017 at 3:44 PM Ruslan Dautkhanov <[hidden email]> wrote:
It was built. I think binaries are only available for official releases?



--
Ruslan Dautkhanov

On Wed, Aug 2, 2017 at 4:41 PM, Benjamin Kim <[hidden email]> wrote:
Did you build Zeppelin or download the binary?

On Wed, Aug 2, 2017 at 3:40 PM Ruslan Dautkhanov <[hidden email]> wrote:
We're using an ~April snapshot of Zeppelin, so not sure about 0.7.1.

Yes, we have that spark home in zeppelin-env.sh



--
Ruslan Dautkhanov

On Wed, Aug 2, 2017 at 4:31 PM, Benjamin Kim <[hidden email]> wrote:
Does this work with Zeppelin 0.7.1? We an error when setting SPARK_HOME in zeppelin-env.sh to what you have below.

On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov <[hidden email]> wrote:
You don't have to use spark2-shell and spark2-submit to use Spark 2.
That can be controled by setting SPARK_HOME using regular spark-submit/spark-shell.

$ which spark-submit
/usr/bin/spark-submit
$ which spark-shell
/usr/bin/spark-shell

$ spark-shell
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.6.0
      /_/



$ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2

$ spark-shell
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.0.cloudera1
      /_/


spark-submit and spark-shell are just shell script wrappers.



--
Ruslan Dautkhanov

On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim <[hidden email]> wrote:
According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera. For some reason, Cloudera defaults to Spark 1.6 and so does the calls to spark-shell and spark-submit. To force the use of Spark 2.x, the calls need to be spark2-shell and spark2-submit. I wonder if this is causing the problem. By the way, we are using Java8 corporate wide, and there seems to be no problems using Zeppelin.

Cheers,
Ben

On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov <[hidden email]> wrote:
Might need to recompile Zeppelin with Scala 2.11?
Also Spark 2.2 now requires JDK8 I believe.



--
Ruslan Dautkhanov

On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim <[hidden email]> wrote:
Here is more.

org.apache.zeppelin.interpreter.InterpreterException: WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2).
WARNING: Running spark-class from user-defined location.
Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
at org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103)
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:110)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Cheers,
Ben


On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang <[hidden email]> wrote:

Then it is due to some classpath issue. I am not sure familiar with CDH, please check whether spark of CDH include hadoop jar with it. 


Benjamin Kim <[hidden email]>于2017年8月2日周三 上午8:22写道:
Here is the error that was sent to me.

org.apache.zeppelin.interpreter.InterpreterException: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream

Cheers,
Ben


On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang <[hidden email]> wrote:

By default, 0.7.1 doesn't support spark 2.2. But you can set zeppelin.spark.enableSupportedVersionCheck in interpreter setting to disable the supported version check.


Jeff Zhang <[hidden email]>于2017年8月2日周三 上午8:18写道:

What's the error you see in log ?


Benjamin Kim <[hidden email]>于2017年8月2日周三 上午8:18写道:
Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something.

Cheers,
Ben





Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Cloudera Spark 2.2

Benjamin Kim
Ruslan,

Quick observation. I see that you put Scala 2.10. How come you didn't use Scala 2.11?

Just curious.

Cheers,
Ben


On Fri, Aug 4, 2017 at 8:36 AM Ruslan Dautkhanov <[hidden email]> wrote:
This should do:
 
export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2
export HIVE_HOME=/opt/cloudera/parcels/CDH/lib/hive
export HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop
export HADOOP_CONF_DIR=/etc/hadoop/conf
export HIVE_CONF_DIR=/etc/hive/conf
 

mvn clean package -DskipTests -Pspark-2.1 -Dhadoop.version=2.6.0-cdh5.10.1 -Phadoop-2.6 -Pvendor-repo -Pscala-2.10 -Psparkr -pl '!alluxio,!flink,!ignite,!lens,!cassandra,!bigquery,!scio' -e

You may needs additional steps depending which interpreters you use (like R etc).


--
Ruslan Dautkhanov

On Fri, Aug 4, 2017 at 8:31 AM, Benjamin Kim <[hidden email]> wrote:
Hi Ruslan,

Can you send me the steps you used to build it, especially the Maven command with the arguments? I will try to build it also.

I do believe that the binaries are for official releases.

Cheers,
Ben


On Wed, Aug 2, 2017 at 3:44 PM Ruslan Dautkhanov <[hidden email]> wrote:
It was built. I think binaries are only available for official releases?



--
Ruslan Dautkhanov

On Wed, Aug 2, 2017 at 4:41 PM, Benjamin Kim <[hidden email]> wrote:
Did you build Zeppelin or download the binary?

On Wed, Aug 2, 2017 at 3:40 PM Ruslan Dautkhanov <[hidden email]> wrote:
We're using an ~April snapshot of Zeppelin, so not sure about 0.7.1.

Yes, we have that spark home in zeppelin-env.sh



--
Ruslan Dautkhanov

On Wed, Aug 2, 2017 at 4:31 PM, Benjamin Kim <[hidden email]> wrote:
Does this work with Zeppelin 0.7.1? We an error when setting SPARK_HOME in zeppelin-env.sh to what you have below.

On Wed, Aug 2, 2017 at 3:24 PM Ruslan Dautkhanov <[hidden email]> wrote:
You don't have to use spark2-shell and spark2-submit to use Spark 2.
That can be controled by setting SPARK_HOME using regular spark-submit/spark-shell.

$ which spark-submit
/usr/bin/spark-submit
$ which spark-shell
/usr/bin/spark-shell

$ spark-shell
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.6.0
      /_/



$ export SPARK_HOME=/opt/cloudera/parcels/SPARK2/lib/spark2

$ spark-shell
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.0.cloudera1
      /_/


spark-submit and spark-shell are just shell script wrappers.



--
Ruslan Dautkhanov

On Wed, Aug 2, 2017 at 10:22 AM, Benjamin Kim <[hidden email]> wrote:
According to the Zeppelin documentation, Zeppelin 0.7.1 supports Spark 2.1. But, I don't know if it supports Spark 2.2 or even 2.1 from Cloudera. For some reason, Cloudera defaults to Spark 1.6 and so does the calls to spark-shell and spark-submit. To force the use of Spark 2.x, the calls need to be spark2-shell and spark2-submit. I wonder if this is causing the problem. By the way, we are using Java8 corporate wide, and there seems to be no problems using Zeppelin.

Cheers,
Ben

On Tue, Aug 1, 2017 at 7:05 PM Ruslan Dautkhanov <[hidden email]> wrote:
Might need to recompile Zeppelin with Scala 2.11?
Also Spark 2.2 now requires JDK8 I believe.



--
Ruslan Dautkhanov

On Tue, Aug 1, 2017 at 6:26 PM, Benjamin Kim <[hidden email]> wrote:
Here is more.

org.apache.zeppelin.interpreter.InterpreterException: WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/SPARK2-2.2.0.cloudera1-1.cdh5.12.0.p0.142354/lib/spark2) overrides detected (/opt/cloudera/parcels/SPARK2/lib/spark2).
WARNING: Running spark-class from user-defined location.
Exception in thread "main" java.lang.NoSuchMethodError: scala.Predef$.$conforms()Lscala/Predef$$less$colon$less;
at org.apache.spark.util.Utils$.getDefaultPropertiesFile(Utils.scala:2103)
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:124)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:124)
at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:110)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Cheers,
Ben


On Tue, Aug 1, 2017 at 5:24 PM Jeff Zhang <[hidden email]> wrote:

Then it is due to some classpath issue. I am not sure familiar with CDH, please check whether spark of CDH include hadoop jar with it. 


Benjamin Kim <[hidden email]>于2017年8月2日周三 上午8:22写道:
Here is the error that was sent to me.

org.apache.zeppelin.interpreter.InterpreterException: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream

Cheers,
Ben


On Tue, Aug 1, 2017 at 5:20 PM Jeff Zhang <[hidden email]> wrote:

By default, 0.7.1 doesn't support spark 2.2. But you can set zeppelin.spark.enableSupportedVersionCheck in interpreter setting to disable the supported version check.


Jeff Zhang <[hidden email]>于2017年8月2日周三 上午8:18写道:

What's the error you see in log ?


Benjamin Kim <[hidden email]>于2017年8月2日周三 上午8:18写道:
Has anyone configured Zeppelin 0.7.1 for Cloudera's release of Spark 2.2? I can't get it to work. I downloaded the binary and set SPARK_HOME to /opt/cloudera/parcels/SPARK2/lib/spark2. I must be missing something.

Cheers,
Ben





Loading...