InvalidClassException using Zeppelin (master) and spark-2.1 on a standalone spark cluster

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

InvalidClassException using Zeppelin (master) and spark-2.1 on a standalone spark cluster

Antoine
Hei,

I'm trying to use Zeppelin from the master git branch with spark-2.1 and I get an invalid class exeption when I use a standalone spark cluster.

java.io.InvalidClassException: org.apache.commons.lang3.time.FastDateParser; local class incompatible: stream classdesc serialVersionUID = 2, local class serialVersionUID = 3

To reproduce the error, I configure the spark interpreter to connect to a standalone cluster, it works with a local spark, and to load a file. For example, spark.read.json("/data/file.json").

I'm building and running Zeppelin with docker, based on the dylanmei/docker-zeppelin docker image, but with a few changes to build the master branch (npm must be installed and bower needs to be configured to run with the root user), and using spark 2.1 instead of spark 2.0.

Can you reproduce the error ? Is there something I'm missing when I build Zeppelin ?

Thanks
Reply | Threaded
Open this post in threaded view
|

Re: InvalidClassException using Zeppelin (master) and spark-2.1 on a standalone spark cluster

Jonathan
Hi, Antoine, this issue was being tracked in https://issues.apache.org/jira/browse/ZEPPELIN-1977, but it is now resolved as of yesterday (looks like about 18 hours ago). Maybe you need to pull from master again and rebuild?

~ Jonathan

On Fri, Jan 20, 2017 at 1:19 PM Antoine <[hidden email]> wrote:
Hei,

I'm trying to use Zeppelin from the master git branch with spark-2.1 and I get an invalid class exeption when I use a standalone spark cluster.

java.io.InvalidClassException: org.apache.commons.lang3.time.FastDateParser; local class incompatible: stream classdesc serialVersionUID = 2, local class serialVersionUID = 3

To reproduce the error, I configure the spark interpreter to connect to a standalone cluster, it works with a local spark, and to load a file. For example, spark.read.json("/data/file.json").

I'm building and running Zeppelin with docker, based on the dylanmei/docker-zeppelin docker image, but with a few changes to build the master branch (npm must be installed and bower needs to be configured to run with the root user), and using spark 2.1 instead of spark 2.0.

Can you reproduce the error ? Is there something I'm missing when I build Zeppelin ?

Thanks
Reply | Threaded
Open this post in threaded view
|

Re: InvalidClassException using Zeppelin (master) and spark-2.1 on a standalone spark cluster

Antoine
Thanks Jonathan, I rebuilt the image and it now works perfectly. My image was built a few hours before the fix. Sorry for the noise.

2017-01-20 23:32 GMT+02:00 Jonathan Kelly <[hidden email]>:
Hi, Antoine, this issue was being tracked in https://issues.apache.org/jira/browse/ZEPPELIN-1977, but it is now resolved as of yesterday (looks like about 18 hours ago). Maybe you need to pull from master again and rebuild?

~ Jonathan

On Fri, Jan 20, 2017 at 1:19 PM Antoine <[hidden email]> wrote:
Hei,

I'm trying to use Zeppelin from the master git branch with spark-2.1 and I get an invalid class exeption when I use a standalone spark cluster.

java.io.InvalidClassException: org.apache.commons.lang3.time.FastDateParser; local class incompatible: stream classdesc serialVersionUID = 2, local class serialVersionUID = 3

To reproduce the error, I configure the spark interpreter to connect to a standalone cluster, it works with a local spark, and to load a file. For example, spark.read.json("/data/file.json").

I'm building and running Zeppelin with docker, based on the dylanmei/docker-zeppelin docker image, but with a few changes to build the master branch (npm must be installed and bower needs to be configured to run with the root user), and using spark 2.1 instead of spark 2.0.

Can you reproduce the error ? Is there something I'm missing when I build Zeppelin ?

Thanks

Reply | Threaded
Open this post in threaded view
|

Re: InvalidClassException using Zeppelin (master) and spark-2.1 on a standalone spark cluster

David Howell
Hi,
I know this issue is resolved for reading from json, and tested for that use case, but I'm seeing the exact same error message when writing to json.

java.io.InvalidClassException: org.apache.commons.lang3.time.FastDateParser; local class incompatible: stream classdesc serialVersionUID = 2, local class serialVersionUID = 3

Easy to reproduce on AWS (also happens writing to HDFS)

val dfyo = List((1,"hi"),(2,"there"),(3,"yo"))
            .toDF()
            .write
            .json(f"s3n://ReplaceWithBucketName/test/")


I know writing to json is not a common use case, so not an urgent issue.

I see the git commit looks like it is mostly replacing the import of
-import org.apache.commons.lang3.StringUtils;
+import org.apache.commons.lang.StringUtils;

But the issue is with org.apache.commons.lang3.time.FastDateParser

I had a quick search in the codebase and couldn't find any imports of that class directly or indirectly so I'm not really sure what would fix it.
Reply | Threaded
Open this post in threaded view
|

Re: InvalidClassException using Zeppelin (master) and spark-2.1 on a standalone spark cluster

Jianfeng (Jeff) Zhang

It is fixed in https://issues.apache.org/jira/browse/ZEPPELIN-1977


Best Regard,
Jeff Zhang





On 6/27/17, 12:46 PM, "David Howell" <[hidden email]> wrote:

>Hi,
>I know this issue is resolved for reading from json, and tested for that
>use
>case, but I'm seeing the exact same error message when writing to json.
>
>java.io.InvalidClassException:
>org.apache.commons.lang3.time.FastDateParser;
>local class incompatible: stream classdesc serialVersionUID = 2, local
>class
>serialVersionUID = 3
>
>Easy to reproduce on AWS (also happens writing to HDFS)
>
>val dfyo = List((1,"hi"),(2,"there"),(3,"yo"))
>            .toDF()
>            .write
>            .json(f"s3n://ReplaceWithBucketName/test/")
>
>
>I know writing to json is not a common use case, so not an urgent issue.
>
>I see the git commit looks like it is mostly replacing the import of
>-import org.apache.commons.lang3.StringUtils;
>+import org.apache.commons.lang.StringUtils;
>
>But the issue is with org.apache.commons.lang3.time.FastDateParser
>
>I had a quick search in the codebase and couldn't find any imports of that
>class directly or indirectly so I'm not really sure what would fix it.
>
>
>
>
>--
>View this message in context:
>http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/I
>nvalidClassException-using-Zeppelin-master-and-spark-2-1-on-a-standalone-s
>park-cluster-tp4900p5854.html
>Sent from the Apache Zeppelin Users (incubating) mailing list mailing
>list archive at Nabble.com.
>

Reply | Threaded
Open this post in threaded view
|

RE: InvalidClassException using Zeppelin (master) and spark-2.1 on a standalone spark cluster

David Howell
Hi Jeff,
The ticket says it is fixed from Zeppelin 0.7.0

I am running Zeppelin 0.7.1 , and yes it is fixed for reading from json, but it still throws an error for writing to json.

See my repro example.


-----Original Message-----
From: Jianfeng (Jeff) Zhang [mailto:[hidden email]]
Sent: Tuesday, 27 June 2017 4:40 PM
To: [hidden email]; [hidden email]
Subject: Re: InvalidClassException using Zeppelin (master) and spark-2.1 on a standalone spark cluster


It is fixed in https://issues.apache.org/jira/browse/ZEPPELIN-1977


Best Regard,
Jeff Zhang





On 6/27/17, 12:46 PM, "David Howell" <[hidden email]> wrote:

>Hi,
>I know this issue is resolved for reading from json, and tested for
>that use case, but I'm seeing the exact same error message when writing
>to json.
>
>java.io.InvalidClassException:
>org.apache.commons.lang3.time.FastDateParser;
>local class incompatible: stream classdesc serialVersionUID = 2, local
>class serialVersionUID = 3
>
>Easy to reproduce on AWS (also happens writing to HDFS)
>
>val dfyo = List((1,"hi"),(2,"there"),(3,"yo"))
>            .toDF()
>            .write
>            .json(f"s3n://ReplaceWithBucketName/test/")
>
>
>I know writing to json is not a common use case, so not an urgent issue.
>
>I see the git commit looks like it is mostly replacing the import of
>-import org.apache.commons.lang3.StringUtils;
>+import org.apache.commons.lang.StringUtils;
>
>But the issue is with org.apache.commons.lang3.time.FastDateParser
>
>I had a quick search in the codebase and couldn't find any imports of
>that class directly or indirectly so I'm not really sure what would fix it.
>
>
>
>
>--
>View this message in context:
>http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.co
>m/I
>nvalidClassException-using-Zeppelin-master-and-spark-2-1-on-a-standalon
>e-s
>park-cluster-tp4900p5854.html
>Sent from the Apache Zeppelin Users (incubating) mailing list mailing
>list archive at Nabble.com.
>

Reply | Threaded
Open this post in threaded view
|

Re: InvalidClassException using Zeppelin (master) and spark-2.1 on a standalone spark cluster

Jianfeng (Jeff) Zhang

It works for me in master branch.

Maybe the issue is due to ZEPPELIN-2375

Could you try 0.7.2 ?



Best Regard,
Jeff Zhang





On 6/27/17, 2:43 PM, "David Howell" <[hidden email]> wrote:

>Hi Jeff,
>The ticket says it is fixed from Zeppelin 0.7.0
>
>I am running Zeppelin 0.7.1 , and yes it is fixed for reading from json,
>but it still throws an error for writing to json.
>
>See my repro example.
>
>
>-----Original Message-----
>From: Jianfeng (Jeff) Zhang [mailto:[hidden email]]
>Sent: Tuesday, 27 June 2017 4:40 PM
>To: [hidden email]; [hidden email]
>Subject: Re: InvalidClassException using Zeppelin (master) and spark-2.1
>on a standalone spark cluster
>
>
>It is fixed in https://issues.apache.org/jira/browse/ZEPPELIN-1977
>
>
>Best Regard,
>Jeff Zhang
>
>
>
>
>
>On 6/27/17, 12:46 PM, "David Howell" <[hidden email]> wrote:
>
>>Hi,
>>I know this issue is resolved for reading from json, and tested for
>>that use case, but I'm seeing the exact same error message when writing
>>to json.
>>
>>java.io.InvalidClassException:
>>org.apache.commons.lang3.time.FastDateParser;
>>local class incompatible: stream classdesc serialVersionUID = 2, local
>>class serialVersionUID = 3
>>
>>Easy to reproduce on AWS (also happens writing to HDFS)
>>
>>val dfyo = List((1,"hi"),(2,"there"),(3,"yo"))
>>            .toDF()
>>            .write
>>            .json(f"s3n://ReplaceWithBucketName/test/")
>>
>>
>>I know writing to json is not a common use case, so not an urgent issue.
>>
>>I see the git commit looks like it is mostly replacing the import of
>>-import org.apache.commons.lang3.StringUtils;
>>+import org.apache.commons.lang.StringUtils;
>>
>>But the issue is with org.apache.commons.lang3.time.FastDateParser
>>
>>I had a quick search in the codebase and couldn't find any imports of
>>that class directly or indirectly so I'm not really sure what would fix
>>it.
>>
>>
>>
>>
>>--
>>View this message in context:
>>http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.co
>>m/I
>>nvalidClassException-using-Zeppelin-master-and-spark-2-1-on-a-standalon
>>e-s
>>park-cluster-tp4900p5854.html
>>Sent from the Apache Zeppelin Users (incubating) mailing list mailing
>>list archive at Nabble.com.
>>
>

Reply | Threaded
Open this post in threaded view
|

RE: InvalidClassException using Zeppelin (master) and spark-2.1 on a standalone spark cluster

David Howell
In reply to this post by David Howell
Hi Jeff,
My apologies, I realise this was caused by an uber jar I had loaded as a spark dependency which contained a conflicting library. Removing the dependency fixed the issue.

 I've shaded & rebuilt the jar and no longer seeing the issue.

Sorry for the confusion!

-----Original Message-----
From: David Howell
Sent: Tuesday, 27 June 2017 4:44 PM
To: [hidden email]; [hidden email]
Subject: RE: InvalidClassException using Zeppelin (master) and spark-2.1 on a standalone spark cluster

Hi Jeff,
The ticket says it is fixed from Zeppelin 0.7.0

I am running Zeppelin 0.7.1 , and yes it is fixed for reading from json, but it still throws an error for writing to json.

See my repro example.


-----Original Message-----
From: Jianfeng (Jeff) Zhang [mailto:[hidden email]]
Sent: Tuesday, 27 June 2017 4:40 PM
To: [hidden email]; [hidden email]
Subject: Re: InvalidClassException using Zeppelin (master) and spark-2.1 on a standalone spark cluster


It is fixed in https://issues.apache.org/jira/browse/ZEPPELIN-1977


Best Regard,
Jeff Zhang





On 6/27/17, 12:46 PM, "David Howell" <[hidden email]> wrote:

>Hi,
>I know this issue is resolved for reading from json, and tested for
>that use case, but I'm seeing the exact same error message when writing
>to json.
>
>java.io.InvalidClassException:
>org.apache.commons.lang3.time.FastDateParser;
>local class incompatible: stream classdesc serialVersionUID = 2, local
>class serialVersionUID = 3
>
>Easy to reproduce on AWS (also happens writing to HDFS)
>
>val dfyo = List((1,"hi"),(2,"there"),(3,"yo"))
>            .toDF()
>            .write
>            .json(f"s3n://ReplaceWithBucketName/test/")
>
>
>I know writing to json is not a common use case, so not an urgent issue.
>
>I see the git commit looks like it is mostly replacing the import of
>-import org.apache.commons.lang3.StringUtils;
>+import org.apache.commons.lang.StringUtils;
>
>But the issue is with org.apache.commons.lang3.time.FastDateParser
>
>I had a quick search in the codebase and couldn't find any imports of
>that class directly or indirectly so I'm not really sure what would fix it.
>
>
>
>
>--
>View this message in context:
>http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.co
>m/I
>nvalidClassException-using-Zeppelin-master-and-spark-2-1-on-a-standalon
>e-s
>park-cluster-tp4900p5854.html
>Sent from the Apache Zeppelin Users (incubating) mailing list mailing
>list archive at Nabble.com.
>