Livy - add external libraries from additional maven repo

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Livy - add external libraries from additional maven repo

Theofilos Kakantousis
Hi everyone,

I'm using Zeppelin with Livy 0.4 and trying to add external libraries from an additional maven repo to my application according to the documentation available here. The example works fine, but when I set the livy.spark.jars.packages to my library the interpreter throws an unresolved dependency error.

I have added the additional maven repository in the interpreter settings and have also tried setting livy.spark.jars.ivy but without luck. However, if I use the Spark interpreter with the following code it works fine.

"%dep
z.reset();
z.addRepo("my repo").url("http://myrepo").snapshot
z.load("mygroup:myartifact:myversion");

Has anyone managed to do that with Livy? Thanks!

Cheers,
Theo
Reply | Threaded
Open this post in threaded view
|

Re: Livy - add external libraries from additional maven repo

Felix Cheung
if I recall, %dep only works with the built in Spark interpreter and not the Livy interpreter.

To manage dependency win Livy you will need to set Spark conf with Livy.


From: Theofilos Kakantousis <[hidden email]>
Sent: Tuesday, May 30, 2017 9:05:15 AM
To: [hidden email]
Subject: Livy - add external libraries from additional maven repo
 
Hi everyone,

I'm using Zeppelin with Livy 0.4 and trying to add external libraries from an additional maven repo to my application according to the documentation available here. The example works fine, but when I set the livy.spark.jars.packages to my library the interpreter throws an unresolved dependency error.

I have added the additional maven repository in the interpreter settings and have also tried setting livy.spark.jars.ivy but without luck. However, if I use the Spark interpreter with the following code it works fine.

"%dep
z.reset();
z.addRepo("my repo").url("http://myrepo").snapshot
z.load("mygroup:myartifact:myversion");

Has anyone managed to do that with Livy? Thanks!

Cheers,
Theo
Reply | Threaded
Open this post in threaded view
|

Re: Livy - add external libraries from additional maven repo

Felix Cheung
To add, this might be an issue with Livy.

I'm seeing something similar as well.

If you can get a repo with calling the Livy REST API directly it will be worthwhile to follow up with the Livy community separately.



From: Felix Cheung <[hidden email]>
Sent: Tuesday, May 30, 2017 11:34:31 AM
To: [hidden email]; [hidden email]
Subject: Re: Livy - add external libraries from additional maven repo
 
if I recall, %dep only works with the built in Spark interpreter and not the Livy interpreter.

To manage dependency win Livy you will need to set Spark conf with Livy.


From: Theofilos Kakantousis <[hidden email]>
Sent: Tuesday, May 30, 2017 9:05:15 AM
To: [hidden email]
Subject: Livy - add external libraries from additional maven repo
 
Hi everyone,

I'm using Zeppelin with Livy 0.4 and trying to add external libraries from an additional maven repo to my application according to the documentation available here. The example works fine, but when I set the livy.spark.jars.packages to my library the interpreter throws an unresolved dependency error.

I have added the additional maven repository in the interpreter settings and have also tried setting livy.spark.jars.ivy but without luck. However, if I use the Spark interpreter with the following code it works fine.

"%dep
z.reset();
z.addRepo("my repo").url("http://myrepo").snapshot
z.load("mygroup:myartifact:myversion");

Has anyone managed to do that with Livy? Thanks!

Cheers,
Theo
Reply | Threaded
Open this post in threaded view
|

Re: Livy - add external libraries from additional maven repo

Ben Vogan
For what it's worth I have successfully added jar files and maven packages to sessions using zeppelin & livy 0.3 - although not using %dep.  In the interpreter settings I set the livy.spark.jars setting for jars that are on my HDFS cluster, and livy.spark.jars.packages for maven packages - although only using maven central and not a local repo.

--Ben

On Tue, May 30, 2017 at 12:36 PM, Felix Cheung <[hidden email]> wrote:
To add, this might be an issue with Livy.

I'm seeing something similar as well.

If you can get a repo with calling the Livy REST API directly it will be worthwhile to follow up with the Livy community separately.



From: Felix Cheung <[hidden email]>
Sent: Tuesday, May 30, 2017 11:34:31 AM
To: [hidden email]; [hidden email]
Subject: Re: Livy - add external libraries from additional maven repo
 
if I recall, %dep only works with the built in Spark interpreter and not the Livy interpreter.

To manage dependency win Livy you will need to set Spark conf with Livy.


From: Theofilos Kakantousis <[hidden email]>
Sent: Tuesday, May 30, 2017 9:05:15 AM
To: [hidden email]
Subject: Livy - add external libraries from additional maven repo
 
Hi everyone,

I'm using Zeppelin with Livy 0.4 and trying to add external libraries from an additional maven repo to my application according to the documentation available here. The example works fine, but when I set the livy.spark.jars.packages to my library the interpreter throws an unresolved dependency error.

I have added the additional maven repository in the interpreter settings and have also tried setting livy.spark.jars.ivy but without luck. However, if I use the Spark interpreter with the following code it works fine.

"%dep
z.reset();
z.addRepo("my repo").url("http://myrepo").snapshot
z.load("mygroup:myartifact:myversion");

Has anyone managed to do that with Livy? Thanks!

Cheers,
Theo



--
BENJAMIN VOGAN | Data Platform Team Lead

Reply | Threaded
Open this post in threaded view
|

Re: Livy - add external libraries from additional maven repo

Theofilos Kakantousis
Thanks everyone for the feedback!

Indeed %dep works only for Spark interpreter, just mentioned it to show the interpreter behavior I expected with Livy.
When setting my local maven repo and the "groupId:artifactId:version" in the interpreter settings, I can see the dependency (i.e. a jar file) being downloaded to the local OS under "local-repo" directory but the dependency is not deployed with the Spark application in YARN.

Cheers,
Theo

On 2017-05-31 01:19, Ben Vogan wrote:
For what it's worth I have successfully added jar files and maven packages to sessions using zeppelin & livy 0.3 - although not using %dep.  In the interpreter settings I set the livy.spark.jars setting for jars that are on my HDFS cluster, and livy.spark.jars.packages for maven packages - although only using maven central and not a local repo.

--Ben

On Tue, May 30, 2017 at 12:36 PM, Felix Cheung <[hidden email]> wrote:
To add, this might be an issue with Livy.

I'm seeing something similar as well.

If you can get a repo with calling the Livy REST API directly it will be worthwhile to follow up with the Livy community separately.



From: Felix Cheung <[hidden email]>
Sent: Tuesday, May 30, 2017 11:34:31 AM
To: [hidden email]; [hidden email]
Subject: Re: Livy - add external libraries from additional maven repo
 
if I recall, %dep only works with the built in Spark interpreter and not the Livy interpreter.

To manage dependency win Livy you will need to set Spark conf with Livy.


From: Theofilos Kakantousis <[hidden email]>
Sent: Tuesday, May 30, 2017 9:05:15 AM
To: [hidden email]
Subject: Livy - add external libraries from additional maven repo
 
Hi everyone,

I'm using Zeppelin with Livy 0.4 and trying to add external libraries from an additional maven repo to my application according to the documentation available here. The example works fine, but when I set the livy.spark.jars.packages to my library the interpreter throws an unresolved dependency error.

I have added the additional maven repository in the interpreter settings and have also tried setting livy.spark.jars.ivy but without luck. However, if I use the Spark interpreter with the following code it works fine.

"%dep
z.reset();
z.addRepo("my repo").url("http://myrepo").snapshot
z.load("mygroup:myartifact:myversion");

Has anyone managed to do that with Livy? Thanks!

Cheers,
Theo



--
BENJAMIN VOGAN | Data Platform Team Lead


Reply | Threaded
Open this post in threaded view
|

Re: Livy - add external libraries from additional maven repo

Felix Cheung
How are you setting this:
When setting my local maven repo and the "groupId:artifactId:version" in the interpreter settings


From: Theofilos Kakantousis <[hidden email]>
Sent: Wednesday, May 31, 2017 1:56:07 AM
To: [hidden email]
Subject: Re: Livy - add external libraries from additional maven repo
 
Thanks everyone for the feedback!

Indeed %dep works only for Spark interpreter, just mentioned it to show the interpreter behavior I expected with Livy.
When setting my local maven repo and the "groupId:artifactId:version" in the interpreter settings, I can see the dependency (i.e. a jar file) being downloaded to the local OS under "local-repo" directory but the dependency is not deployed with the Spark application in YARN.

Cheers,
Theo

On 2017-05-31 01:19, Ben Vogan wrote:
For what it's worth I have successfully added jar files and maven packages to sessions using zeppelin & livy 0.3 - although not using %dep.  In the interpreter settings I set the livy.spark.jars setting for jars that are on my HDFS cluster, and livy.spark.jars.packages for maven packages - although only using maven central and not a local repo.

--Ben

On Tue, May 30, 2017 at 12:36 PM, Felix Cheung <[hidden email]> wrote:
To add, this might be an issue with Livy.

I'm seeing something similar as well.

If you can get a repo with calling the Livy REST API directly it will be worthwhile to follow up with the Livy community separately.



From: Felix Cheung <[hidden email]>
Sent: Tuesday, May 30, 2017 11:34:31 AM
To: [hidden email]; [hidden email]
Subject: Re: Livy - add external libraries from additional maven repo
 
if I recall, %dep only works with the built in Spark interpreter and not the Livy interpreter.

To manage dependency win Livy you will need to set Spark conf with Livy.


From: Theofilos Kakantousis <[hidden email]>
Sent: Tuesday, May 30, 2017 9:05:15 AM
To: [hidden email]
Subject: Livy - add external libraries from additional maven repo
 
Hi everyone,

I'm using Zeppelin with Livy 0.4 and trying to add external libraries from an additional maven repo to my application according to the documentation available here. The example works fine, but when I set the livy.spark.jars.packages to my library the interpreter throws an unresolved dependency error.

I have added the additional maven repository in the interpreter settings and have also tried setting livy.spark.jars.ivy but without luck. However, if I use the Spark interpreter with the following code it works fine.

"%dep
z.reset();
z.addRepo("my repo").url("http://myrepo").snapshot
z.load("mygroup:myartifact:myversion");

Has anyone managed to do that with Livy? Thanks!

Cheers,
Theo



--
BENJAMIN VOGAN | Data Platform Team Lead


Reply | Threaded
Open this post in threaded view
|

Re: Livy - add external libraries from additional maven repo

Jeff Zhang
Ben's right, for livy interpreter, you need to specify livy.spark.jars.packages.

Specifying interpreter dependency doesn't work for livy interpreter, because livy run in yarn-cluster mode, while interpreter dependencies is downloaded in the client JVM.




Felix Cheung <[hidden email]>于2017年6月1日周四 上午2:52写道:
How are you setting this:
When setting my local maven repo and the "groupId:artifactId:version" in the interpreter settings


From: Theofilos Kakantousis <[hidden email]>
Sent: Wednesday, May 31, 2017 1:56:07 AM
To: [hidden email]

Subject: Re: Livy - add external libraries from additional maven repo
Thanks everyone for the feedback!

Indeed %dep works only for Spark interpreter, just mentioned it to show the interpreter behavior I expected with Livy.
When setting my local maven repo and the "groupId:artifactId:version" in the interpreter settings, I can see the dependency (i.e. a jar file) being downloaded to the local OS under "local-repo" directory but the dependency is not deployed with the Spark application in YARN.

Cheers,
Theo

On 2017-05-31 01:19, Ben Vogan wrote:
For what it's worth I have successfully added jar files and maven packages to sessions using zeppelin & livy 0.3 - although not using %dep.  In the interpreter settings I set the livy.spark.jars setting for jars that are on my HDFS cluster, and livy.spark.jars.packages for maven packages - although only using maven central and not a local repo.

--Ben

On Tue, May 30, 2017 at 12:36 PM, Felix Cheung <[hidden email]> wrote:
To add, this might be an issue with Livy.

I'm seeing something similar as well.

If you can get a repo with calling the Livy REST API directly it will be worthwhile to follow up with the Livy community separately.



From: Felix Cheung <[hidden email]>
Sent: Tuesday, May 30, 2017 11:34:31 AM
To: [hidden email]; [hidden email]
Subject: Re: Livy - add external libraries from additional maven repo
 
if I recall, %dep only works with the built in Spark interpreter and not the Livy interpreter.

To manage dependency win Livy you will need to set Spark conf with Livy.


From: Theofilos Kakantousis <[hidden email]>
Sent: Tuesday, May 30, 2017 9:05:15 AM
To: [hidden email]
Subject: Livy - add external libraries from additional maven repo
 
Hi everyone,

I'm using Zeppelin with Livy 0.4 and trying to add external libraries from an additional maven repo to my application according to the documentation available here. The example works fine, but when I set the livy.spark.jars.packages to my library the interpreter throws an unresolved dependency error.

I have added the additional maven repository in the interpreter settings and have also tried setting livy.spark.jars.ivy but without luck. However, if I use the Spark interpreter with the following code it works fine.

"%dep
z.reset();
z.addRepo("my repo").url("http://myrepo").snapshot
z.load("mygroup:myartifact:myversion");

Has anyone managed to do that with Livy? Thanks!

Cheers,
Theo



--
BENJAMIN VOGAN | Data Platform Team Lead


Reply | Threaded
Open this post in threaded view
|

Re: Livy - add external libraries from additional maven repo

Theofilos Kakantousis
Hi Jeff,

That is the first thing I tried, as per the documentation as well, but it only works for the main maven repo and not my custom maven repo. The documentation is not clear if it is supposed to work for custom maven repos. Have you tried it?

Kind regards,
Theo

On 2017-06-01 01:45, Jeff Zhang wrote:
Ben's right, for livy interpreter, you need to specify livy.spark.jars.packages.

Specifying interpreter dependency doesn't work for livy interpreter, because livy run in yarn-cluster mode, while interpreter dependencies is downloaded in the client JVM.




Felix Cheung <[hidden email]>于2017年6月1日周四 上午2:52写道:
How are you setting this:
When setting my local maven repo and the "groupId:artifactId:version" in the interpreter settings


From: Theofilos Kakantousis <[hidden email]>
Sent: Wednesday, May 31, 2017 1:56:07 AM
To: [hidden email]

Subject: Re: Livy - add external libraries from additional maven repo
Thanks everyone for the feedback!

Indeed %dep works only for Spark interpreter, just mentioned it to show the interpreter behavior I expected with Livy.
When setting my local maven repo and the "groupId:artifactId:version" in the interpreter settings, I can see the dependency (i.e. a jar file) being downloaded to the local OS under "local-repo" directory but the dependency is not deployed with the Spark application in YARN.

Cheers,
Theo

On 2017-05-31 01:19, Ben Vogan wrote:
For what it's worth I have successfully added jar files and maven packages to sessions using zeppelin & livy 0.3 - although not using %dep.  In the interpreter settings I set the livy.spark.jars setting for jars that are on my HDFS cluster, and livy.spark.jars.packages for maven packages - although only using maven central and not a local repo.

--Ben

On Tue, May 30, 2017 at 12:36 PM, Felix Cheung <[hidden email]> wrote:
To add, this might be an issue with Livy.

I'm seeing something similar as well.

If you can get a repo with calling the Livy REST API directly it will be worthwhile to follow up with the Livy community separately.



From: Felix Cheung <[hidden email]>
Sent: Tuesday, May 30, 2017 11:34:31 AM
To: [hidden email]; [hidden email]
Subject: Re: Livy - add external libraries from additional maven repo
 
if I recall, %dep only works with the built in Spark interpreter and not the Livy interpreter.

To manage dependency win Livy you will need to set Spark conf with Livy.


From: Theofilos Kakantousis <[hidden email]>
Sent: Tuesday, May 30, 2017 9:05:15 AM
To: [hidden email]
Subject: Livy - add external libraries from additional maven repo
 
Hi everyone,

I'm using Zeppelin with Livy 0.4 and trying to add external libraries from an additional maven repo to my application according to the documentation available here. The example works fine, but when I set the livy.spark.jars.packages to my library the interpreter throws an unresolved dependency error.

I have added the additional maven repository in the interpreter settings and have also tried setting livy.spark.jars.ivy but without luck. However, if I use the Spark interpreter with the following code it works fine.

"%dep
z.reset();
z.addRepo("my repo").url("http://myrepo").snapshot
z.load("mygroup:myartifact:myversion");

Has anyone managed to do that with Livy? Thanks!

Cheers,
Theo



--
BENJAMIN VOGAN | Data Platform Team Lead



Reply | Threaded
Open this post in threaded view
|

Re: Livy - add external libraries from additional maven repo

Jeff Zhang

I am afraid for now only official maven repo is supported.



Theofilos Kakantousis <[hidden email]>于2017年6月1日周四 下午4:04写道:
Hi Jeff,

That is the first thing I tried, as per the documentation as well, but it only works for the main maven repo and not my custom maven repo. The documentation is not clear if it is supposed to work for custom maven repos. Have you tried it?

Kind regards,
Theo


On 2017-06-01 01:45, Jeff Zhang wrote:
Ben's right, for livy interpreter, you need to specify livy.spark.jars.packages.

Specifying interpreter dependency doesn't work for livy interpreter, because livy run in yarn-cluster mode, while interpreter dependencies is downloaded in the client JVM.




Felix Cheung <[hidden email]>于2017年6月1日周四 上午2:52写道:
How are you setting this:
When setting my local maven repo and the "groupId:artifactId:version" in the interpreter settings


From: Theofilos Kakantousis <[hidden email]>
Sent: Wednesday, May 31, 2017 1:56:07 AM
To: [hidden email]

Subject: Re: Livy - add external libraries from additional maven repo
Thanks everyone for the feedback!

Indeed %dep works only for Spark interpreter, just mentioned it to show the interpreter behavior I expected with Livy.
When setting my local maven repo and the "groupId:artifactId:version" in the interpreter settings, I can see the dependency (i.e. a jar file) being downloaded to the local OS under "local-repo" directory but the dependency is not deployed with the Spark application in YARN.

Cheers,
Theo

On 2017-05-31 01:19, Ben Vogan wrote:
For what it's worth I have successfully added jar files and maven packages to sessions using zeppelin & livy 0.3 - although not using %dep.  In the interpreter settings I set the livy.spark.jars setting for jars that are on my HDFS cluster, and livy.spark.jars.packages for maven packages - although only using maven central and not a local repo.

--Ben

On Tue, May 30, 2017 at 12:36 PM, Felix Cheung <[hidden email]> wrote:
To add, this might be an issue with Livy.

I'm seeing something similar as well.

If you can get a repo with calling the Livy REST API directly it will be worthwhile to follow up with the Livy community separately.



From: Felix Cheung <[hidden email]>
Sent: Tuesday, May 30, 2017 11:34:31 AM
To: [hidden email]; [hidden email]
Subject: Re: Livy - add external libraries from additional maven repo
 
if I recall, %dep only works with the built in Spark interpreter and not the Livy interpreter.

To manage dependency win Livy you will need to set Spark conf with Livy.


From: Theofilos Kakantousis <[hidden email]>
Sent: Tuesday, May 30, 2017 9:05:15 AM
To: [hidden email]
Subject: Livy - add external libraries from additional maven repo
 
Hi everyone,

I'm using Zeppelin with Livy 0.4 and trying to add external libraries from an additional maven repo to my application according to the documentation available here. The example works fine, but when I set the livy.spark.jars.packages to my library the interpreter throws an unresolved dependency error.

I have added the additional maven repository in the interpreter settings and have also tried setting livy.spark.jars.ivy but without luck. However, if I use the Spark interpreter with the following code it works fine.

"%dep
z.reset();
z.addRepo("my repo").url("http://myrepo").snapshot
z.load("mygroup:myartifact:myversion");

Has anyone managed to do that with Livy? Thanks!

Cheers,
Theo



--
BENJAMIN VOGAN | Data Platform Team Lead