Can't load a dependency (because I'm behind a proxy?)

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Can't load a dependency (because I'm behind a proxy?)

Partridge, Lucas (GE Aviation)

Hi,

 

In a Zeppelin notebook I do this in the first paragraph:

 

%dep

z.reset()

z.load("com.quantifind:wisp_2.10:0.0.4")

 

- this appears to work ok when I run it (e.g., it says “Took 16 seconds”)

 

But when I try this in a second paragraph:

 

import com.quantifind.charts.Highcharts._

 

The result is:

 

<console>:21: error: object quantifind is not a member of package com

       import com.quantifind.charts.Highcharts._

 

So it looks like the dependency loading failed.  Two points here:

 

1)      I’m behind a corporate firewall. I was a bit alarmed to see https://issues.apache.org/jira/browse/ZEPPELIN-169 . Does this mean that it’s impossible for me to fetch a dependency from behind a proxy?!  If not, please tell me how I can configure Zeppelin to use a proxy.

2)      If z.load() fails (e.g., because it’s behind a proxy!) it would be great if it could report some kind of error message. Otherwise a user won’t know it’s failed until they try running an import statement later on.

 

BTW I’m using zeppelin-0.5.0-incubating-bin-spark-1.3.1_hadoop-2.3.tgz

 

Thanks, Lucas.

Reply | Threaded
Open this post in threaded view
|

RE: Can't load a dependency (because I'm behind a proxy?)

Partridge, Lucas (GE Aviation)

BTW I found this in the logs (*.out) at about the same time I ran the first paragraph:

 

java.lang.NullPointerException

                at org.sonatype.aether.impl.internal.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:352)

                at org.apache.zeppelin.spark.dep.DependencyContext.fetchArtifactWithDep(DependencyContext.java:141)

                at org.apache.zeppelin.spark.dep.DependencyContext.fetch(DependencyContext.java:98)

                at org.apache.zeppelin.spark.DepInterpreter.interpret(DepInterpreter.java:189)

                at org.apache.zeppelin.interpreter.ClassloaderInterpreter.interpret(ClassloaderInterpreter.java:57)

                at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:93)

                at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:277)

                at org.apache.zeppelin.scheduler.Job.run(Job.java:170)

                at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:118)

                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

                at java.util.concurrent.FutureTask.run(FutureTask.java:262)

                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)

                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)

                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

                at java.lang.Thread.run(Thread.java:745)

 

Should I raise an issue for this?

 

From: Partridge, Lucas (GE Aviation)
Sent: 17 September 2015 14:25
To: [hidden email]
Subject: Can't load a dependency (because I'm behind a proxy?)

 

Hi,

 

In a Zeppelin notebook I do this in the first paragraph:

 

%dep

z.reset()

z.load("com.quantifind:wisp_2.10:0.0.4")

 

- this appears to work ok when I run it (e.g., it says “Took 16 seconds”)

 

But when I try this in a second paragraph:

 

import com.quantifind.charts.Highcharts._

 

The result is:

 

<console>:21: error: object quantifind is not a member of package com

       import com.quantifind.charts.Highcharts._

 

So it looks like the dependency loading failed.  Two points here:

 

1)      I’m behind a corporate firewall. I was a bit alarmed to see https://issues.apache.org/jira/browse/ZEPPELIN-169 . Does this mean that it’s impossible for me to fetch a dependency from behind a proxy?!  If not, please tell me how I can configure Zeppelin to use a proxy.

2)      If z.load() fails (e.g., because it’s behind a proxy!) it would be great if it could report some kind of error message. Otherwise a user won’t know it’s failed until they try running an import statement later on.

 

BTW I’m using zeppelin-0.5.0-incubating-bin-spark-1.3.1_hadoop-2.3.tgz

 

Thanks, Lucas.

Reply | Threaded
Open this post in threaded view
|

RE: Can't load a dependency (because I'm behind a proxy?)

Partridge, Lucas (GE Aviation)

I’d forgotten to say that I’d already managed to get the ‘Zeppelin Tutorial’ notebook to run successfully behind the proxy. I had put this in conf/zeppelin-env.sh before starting my standalone instance of Zeppelin for the first time:

 

export ZEPPELIN_JAVA_OPTS=$JAVA_OPTS

 

where JAVA_OPTS was defined in ~/.profile as:

 

export JAVA_OPTS="-Dhttp.proxyHost=<proxy_server_name> -Dhttp.proxyPort=<proxy_server_port> -Dhttp.proxyUser=<my_user_id> -Dhttp.proxyPassword=<my_password>"

export JAVA_OPTS="$JAVA_OPTS -Dhttps.proxyHost=<proxy_server_name> -Dhttps.proxyPort=<proxy_server_port> -Dhttps.proxyUser=<my_user_id> -Dhttps.proxyPassword=<my_password>"

 

Also, I have just tried running the same z.load("com.quantifind:wisp_2.10:0.0.4") statement in another standalone instance of Zeppelin (same version) outside the corporate firewall, and that completed successfully (I could do the import too).  So it really looks like this is some sort of proxy issue.

 

Can someone please tell me how to get z.load() to load remote dependencies that are outside a corporate firewall?  Is it even possible in Zeppelin?  This is a blocking issue for me.

 

Many thanks in advance,

Lucas.

 

 

From: Partridge, Lucas (GE Aviation)
Sent: 17 September 2015 14:44
To: [hidden email]
Subject: RE: Can't load a dependency (because I'm behind a proxy?)

 

BTW I found this in the logs (*.out) at about the same time I ran the first paragraph:

 

java.lang.NullPointerException

                at org.sonatype.aether.impl.internal.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:352)

                at org.apache.zeppelin.spark.dep.DependencyContext.fetchArtifactWithDep(DependencyContext.java:141)

                at org.apache.zeppelin.spark.dep.DependencyContext.fetch(DependencyContext.java:98)

                at org.apache.zeppelin.spark.DepInterpreter.interpret(DepInterpreter.java:189)

                at org.apache.zeppelin.interpreter.ClassloaderInterpreter.interpret(ClassloaderInterpreter.java:57)

                at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:93)

                at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:277)

                at org.apache.zeppelin.scheduler.Job.run(Job.java:170)

                at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:118)

                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

                at java.util.concurrent.FutureTask.run(FutureTask.java:262)

                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)

                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)

                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

                at java.lang.Thread.run(Thread.java:745)

 

Should I raise an issue for this?

 

From: Partridge, Lucas (GE Aviation)
Sent: 17 September 2015 14:25
To: [hidden email]
Subject: Can't load a dependency (because I'm behind a proxy?)

 

Hi,

 

In a Zeppelin notebook I do this in the first paragraph:

 

%dep

z.reset()

z.load("com.quantifind:wisp_2.10:0.0.4")

 

- this appears to work ok when I run it (e.g., it says “Took 16 seconds”)

 

But when I try this in a second paragraph:

 

import com.quantifind.charts.Highcharts._

 

The result is:

 

<console>:21: error: object quantifind is not a member of package com

       import com.quantifind.charts.Highcharts._

 

So it looks like the dependency loading failed.  Two points here:

 

1)      I’m behind a corporate firewall. I was a bit alarmed to see https://issues.apache.org/jira/browse/ZEPPELIN-169 . Does this mean that it’s impossible for me to fetch a dependency from behind a proxy?!  If not, please tell me how I can configure Zeppelin to use a proxy.

2)      If z.load() fails (e.g., because it’s behind a proxy!) it would be great if it could report some kind of error message. Otherwise a user won’t know it’s failed until they try running an import statement later on.

 

BTW I’m using zeppelin-0.5.0-incubating-bin-spark-1.3.1_hadoop-2.3.tgz

 

Thanks, Lucas.

Reply | Threaded
Open this post in threaded view
|

RE: Can't load a dependency (because I'm behind a proxy?)

Rick Moritz
In reply to this post by Partridge, Lucas (GE Aviation)
Hello Lucas, hello list,

hopefully this message will thread properly.

This problem can actually be reproduced by the corresponding unit tests - at least on my "disconnected" system, the corresponding tests for the SparkInterpreter fail in exactly the same way as your code does. This is also an issue for me, since I will probably have to get those tests to pass, in order to deploy Zeppelin on our production system.

Since this actually fails unit tests, I think creating a corresponding issue is a logical next step.

I'm currently looking at the code of the test to figure out which component is responsible for directing the dependency lookup to the target, and how this can be overridden, but there's probably some implicit default in use, which makes figuring the root out slightly more tricky.

Have you had a look at where this could be overridden yet? Filed an issue already?

Unless we get some Progress going in this thread, we should start the usual procedures...

Thanks and Best,

Rick
Reply | Threaded
Open this post in threaded view
|

RE: Can't load a dependency (because I'm behind a proxy?)

Partridge, Lucas (GE Aviation)

Hi Rick,

 

I’m glad I’m not the only one suffering from thisJ.

 

No, I haven’t filed an issue yet but I have just added some details to the feature request at https://issues.apache.org/jira/browse/ZEPPELIN-169, asking for a feasible workaround.  I still don’t have a solution and am currently looking to see how I can hack the zeppelin-daemon.sh script on my box to call the addJarInDir function for each directory under my wisp’s lib_managed folder.  However this is going to be painful because there are many folders with jars under there; I don’t know which ones are actually required dependencies (all of them?!)…  I’m open to better suggestions.

 

Unfortunately I am new to Scala so I don’t even know what ‘implicit default’ means, if indeed you’re referring to Scala!  All I know is that the dependency-resolving code at https://github.com/apache/incubator-zeppelin/blob/v0.5.0/spark/src/main/java/org/apache/zeppelin/spark/dep/DependencyResolver.java could potentially call http://sonatype.github.io/sonatype-aether/apidocs/org/sonatype/aether/repository/RemoteRepository.html#setProxy%28org.sonatype.aether.repository.Proxy%29 . Maybe this is what Jeremy Subitil was implying at https://issues.apache.org/jira/browse/ZEPPELIN-169?focusedCommentId=14628118&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14628118 .

 

If you have failing unit tests then it probably makes sense for you to raise an issue, please.  Then I can always add to it.

Thanks!  It would be fantastic if Zeppelin could be used behind a corporate firewallJ.

Lucas.

 

From: Rick Moritz [mailto:[hidden email]]
Sent: 21 September 2015 15:19
To: [hidden email]
Cc: Partridge, Lucas (GE Aviation)
Subject: RE: Can't load a dependency (because I'm behind a proxy?)

 

Hello Lucas, hello list,

hopefully this message will thread properly.

This problem can actually be reproduced by the corresponding unit tests - at least on my "disconnected" system, the corresponding tests for the SparkInterpreter fail in exactly the same way as your code does. This is also an issue for me, since I will probably have to get those tests to pass, in order to deploy Zeppelin on our production system.

Since this actually fails unit tests, I think creating a corresponding issue is a logical next step.

I'm currently looking at the code of the test to figure out which component is responsible for directing the dependency lookup to the target, and how this can be overridden, but there's probably some implicit default in use, which makes figuring the root out slightly more tricky.

Have you had a look at where this could be overridden yet? Filed an issue already?

Unless we get some Progress going in this thread, we should start the usual procedures...

Thanks and Best,

Rick

Reply | Threaded
Open this post in threaded view
|

Re: Can't load a dependency (because I'm behind a proxy?)

Rick Moritz

Hi,

Unfortunately I am new to Scala so I don’t even know what ‘implicit default’ means, if indeed you’re referring to Scala! 

 


I was merely referring to the fact, that maven-central is often set as default repository somewhere internally (i.e. source) or externally (Zeppelin config, Maven/aether config...), and figuring out  where this setting comes from is going to take some....patience.
Nothing scala-specific in this case, but rather something inherited from sonatype-aether.

Also, for info/comparison, the trace from the crashing unit tests:

Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 102.584 sec <<< FAILURE! - in org.apache.zeppelin.spark.SparkInterpreterTest
testZContextDependencyLoading(org.apache.zeppelin.spark.SparkInterpreterTest)  Time elapsed: 63.612 sec  <<< FAILURE!
java.lang.AssertionError: expected:<SUCCESS> but was:<ERROR>
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.failNotEquals(Assert.java:743)
        at org.junit.Assert.assertEquals(Assert.java:118)
        at org.junit.Assert.assertEquals(Assert.java:144)
        at org.apache.zeppelin.spark.SparkInterpreterTest.testZContextDependencyLoading(SparkInterpreterTest.java:159)

which is followed up by

Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 69.667 sec <<< FAILURE! - in org.apache.zeppelin.spark.DepInterpreterTest
testDefault(org.apache.zeppelin.spark.DepInterpreterTest)  Time elapsed: 69.528 sec  <<< ERROR!
java.lang.NullPointerException: null
        at org.sonatype.aether.impl.internal.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:352)
        at org.apache.zeppelin.spark.dep.DependencyContext.fetchArtifactWithDep(DependencyContext.java:141)
        at org.apache.zeppelin.spark.dep.DependencyContext.fetch(DependencyContext.java:98)
        at org.apache.zeppelin.spark.DepInterpreter.interpret(DepInterpreter.java:189)
        at org.apache.zeppelin.spark.DepInterpreterTest.testDefault(DepInterpreterTest.java:88)


I'll give this a few hours yet, before opening an issue, maybe I'm just "holding it wrong". I doubt an issue will push this along much faster either, unless one of us actually submits a patch/PR to go along with it ;)

Best,

Rick

 

From: Rick Moritz [mailto:[hidden email]]
Sent: 21 September 2015 15:19
To: [hidden email]
Cc: Partridge, Lucas (GE Aviation)
Subject: RE: Can't load a dependency (because I'm behind a proxy?)

 

Hello Lucas, hello list,

hopefully this message will thread properly.

This problem can actually be reproduced by the corresponding unit tests - at least on my "disconnected" system, the corresponding tests for the SparkInterpreter fail in exactly the same way as your code does. This is also an issue for me, since I will probably have to get those tests to pass, in order to deploy Zeppelin on our production system.

Since this actually fails unit tests, I think creating a corresponding issue is a logical next step.

I'm currently looking at the code of the test to figure out which component is responsible for directing the dependency lookup to the target, and how this can be overridden, but there's probably some implicit default in use, which makes figuring the root out slightly more tricky.

Have you had a look at where this could be overridden yet? Filed an issue already?

Unless we get some Progress going in this thread, we should start the usual procedures...

Thanks and Best,

Rick


Reply | Threaded
Open this post in threaded view
|

RE: Can't load a dependency (because I'm behind a proxy?)

Partridge, Lucas (GE Aviation)

Hi Rick,

 

Ok, I misunderstood your original statementJ.  As you know, I’ve already answered some of your questions about repository lookup at https://issues.apache.org/jira/browse/ZEPPELIN-169.

 

I’ve tried using multiple z.load() statements to load wisp and all its 30 to 40 dependencies directly from the file system, but although the subsequent import statement succeeds the actual wisp commands fail with things like NoClassDefFoundError. So now I’m trying to package wisp as a fat jar and then try doing a z.load() of that…

 

Please go ahead and raise that issue if you think it will help, although there is already the feature request at ZEPPELIN-169. Regardless of whether this is a feature request or a bug this renders Zeppelin pretty useless to me behind the corporate firewall…

 

Thanks, Lucas.

 

From: Rick Moritz [mailto:[hidden email]]
Sent: 21 September 2015 17:00
To: Partridge, Lucas (GE Aviation)
Cc: [hidden email]
Subject: Re: Can't load a dependency (because I'm behind a proxy?)

 

 

Hi,

 

Unfortunately I am new to Scala so I don’t even know what ‘implicit default’ means, if indeed you’re referring to Scala! 

 

 

I was merely referring to the fact, that maven-central is often set as default repository somewhere internally (i.e. source) or externally (Zeppelin config, Maven/aether config...), and figuring out  where this setting comes from is going to take some....patience.

Nothing scala-specific in this case, but rather something inherited from sonatype-aether.

Also, for info/comparison, the trace from the crashing unit tests:

Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 102.584 sec <<< FAILURE! - in org.apache.zeppelin.spark.SparkInterpreterTest
testZContextDependencyLoading(org.apache.zeppelin.spark.SparkInterpreterTest)  Time elapsed: 63.612 sec  <<< FAILURE!
java.lang.AssertionError: expected:<SUCCESS> but was:<ERROR>
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.failNotEquals(Assert.java:743)
        at org.junit.Assert.assertEquals(Assert.java:118)
        at org.junit.Assert.assertEquals(Assert.java:144)
        at org.apache.zeppelin.spark.SparkInterpreterTest.testZContextDependencyLoading(SparkInterpreterTest.java:159)

which is followed up by

Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 69.667 sec <<< FAILURE! - in org.apache.zeppelin.spark.DepInterpreterTest
testDefault(org.apache.zeppelin.spark.DepInterpreterTest)  Time elapsed: 69.528 sec  <<< ERROR!
java.lang.NullPointerException: null
        at org.sonatype.aether.impl.internal.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:352)
        at org.apache.zeppelin.spark.dep.DependencyContext.fetchArtifactWithDep(DependencyContext.java:141)
        at org.apache.zeppelin.spark.dep.DependencyContext.fetch(DependencyContext.java:98)
        at org.apache.zeppelin.spark.DepInterpreter.interpret(DepInterpreter.java:189)
        at org.apache.zeppelin.spark.DepInterpreterTest.testDefault(DepInterpreterTest.java:88)

 

I'll give this a few hours yet, before opening an issue, maybe I'm just "holding it wrong". I doubt an issue will push this along much faster either, unless one of us actually submits a patch/PR to go along with it ;)

Best,

Rick

 

 

From: Rick Moritz [mailto:[hidden email]]
Sent: 21 September 2015 15:19
To: [hidden email]
Cc: Partridge, Lucas (GE Aviation)
Subject: RE: Can't load a dependency (because I'm behind a proxy?)

 

Hello Lucas, hello list,

hopefully this message will thread properly.

This problem can actually be reproduced by the corresponding unit tests - at least on my "disconnected" system, the corresponding tests for the SparkInterpreter fail in exactly the same way as your code does. This is also an issue for me, since I will probably have to get those tests to pass, in order to deploy Zeppelin on our production system.

Since this actually fails unit tests, I think creating a corresponding issue is a logical next step.

I'm currently looking at the code of the test to figure out which component is responsible for directing the dependency lookup to the target, and how this can be overridden, but there's probably some implicit default in use, which makes figuring the root out slightly more tricky.

Have you had a look at where this could be overridden yet? Filed an issue already?

Unless we get some Progress going in this thread, we should start the usual procedures...

Thanks and Best,

Rick

 

Reply | Threaded
Open this post in threaded view
|

Re: Can't load a dependency (because I'm behind a proxy?)

Rick Moritz
I actually got the unit test to pass, by manually downloading the dependency into the local repo. Using an artifact that isn't in the local repo after building Zeppelin makes for an unintuitive unit test. On the other hand, required ones may already be available in the class path and make for a bad test. I'll have to think a bit about what makes sense here.

A workaround I can propose to you, is to build dummy poms with your required dependencies and use those to load the deps into the local repo with a mvn resolve-dependencies call. That should get you up and running, if you have the required access.

Alternatively, I know that spark-notebook had issues supporting proxy-lookups as well,and the corresponding issue is still pending as well. In that case I used the same strategy of manually pushing dependencies into the local repo, and then adding that as a source (hence my frantic attempts to do this in Zeppelin as well ;) )

I think having a test that needs internet access is pretty much a no-go, and a different issue from the lack of proxy support. I'll definitely keep an eye on ZEPPELIN-169 since it's relevant for our dev environments, and file a new issue once I have a solid proposition on how to go ahead with testing this feature - a unit test should be self-contained. Maybe packaging the dependency as a resource and adding a local repo is a way to test this feature in isolation. at the very least, the dep should be added as a test-scoped dependency in the pom somewhere...

Thanks again for your comments @ZEPPELIN-169, at least for my test-case issue that pointed to some magic-string culprits.

Best,

Rick

On Tue, Sep 22, 2015 at 3:05 PM, Partridge, Lucas (GE Aviation) <[hidden email]> wrote:

Hi Rick,

 

Ok, I misunderstood your original statementJ.  As you know, I’ve already answered some of your questions about repository lookup at https://issues.apache.org/jira/browse/ZEPPELIN-169.

 

I’ve tried using multiple z.load() statements to load wisp and all its 30 to 40 dependencies directly from the file system, but although the subsequent import statement succeeds the actual wisp commands fail with things like NoClassDefFoundError. So now I’m trying to package wisp as a fat jar and then try doing a z.load() of that…

 

Please go ahead and raise that issue if you think it will help, although there is already the feature request at ZEPPELIN-169. Regardless of whether this is a feature request or a bug this renders Zeppelin pretty useless to me behind the corporate firewall…

 

Thanks, Lucas.

 

From: Rick Moritz [mailto:[hidden email]]
Sent: 21 September 2015 17:00
To: Partridge, Lucas (GE Aviation)
Cc: [hidden email]
Subject: Re: Can't load a dependency (because I'm behind a proxy?)

 

 

Hi,

 

Unfortunately I am new to Scala so I don’t even know what ‘implicit default’ means, if indeed you’re referring to Scala! 

 

 

I was merely referring to the fact, that maven-central is often set as default repository somewhere internally (i.e. source) or externally (Zeppelin config, Maven/aether config...), and figuring out  where this setting comes from is going to take some....patience.

Nothing scala-specific in this case, but rather something inherited from sonatype-aether.

Also, for info/comparison, the trace from the crashing unit tests:

Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 102.584 sec <<< FAILURE! - in org.apache.zeppelin.spark.SparkInterpreterTest
testZContextDependencyLoading(org.apache.zeppelin.spark.SparkInterpreterTest)  Time elapsed: 63.612 sec  <<< FAILURE!
java.lang.AssertionError: expected:<SUCCESS> but was:<ERROR>
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.failNotEquals(Assert.java:743)
        at org.junit.Assert.assertEquals(Assert.java:118)
        at org.junit.Assert.assertEquals(Assert.java:144)
        at org.apache.zeppelin.spark.SparkInterpreterTest.testZContextDependencyLoading(SparkInterpreterTest.java:159)

which is followed up by

Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 69.667 sec <<< FAILURE! - in org.apache.zeppelin.spark.DepInterpreterTest
testDefault(org.apache.zeppelin.spark.DepInterpreterTest)  Time elapsed: 69.528 sec  <<< ERROR!
java.lang.NullPointerException: null
        at org.sonatype.aether.impl.internal.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:352)
        at org.apache.zeppelin.spark.dep.DependencyContext.fetchArtifactWithDep(DependencyContext.java:141)
        at org.apache.zeppelin.spark.dep.DependencyContext.fetch(DependencyContext.java:98)
        at org.apache.zeppelin.spark.DepInterpreter.interpret(DepInterpreter.java:189)
        at org.apache.zeppelin.spark.DepInterpreterTest.testDefault(DepInterpreterTest.java:88)

 

I'll give this a few hours yet, before opening an issue, maybe I'm just "holding it wrong". I doubt an issue will push this along much faster either, unless one of us actually submits a patch/PR to go along with it ;)

Best,

Rick

 

 

From: Rick Moritz [mailto:[hidden email]]
Sent: 21 September 2015 15:19
To: [hidden email]
Cc: Partridge, Lucas (GE Aviation)
Subject: RE: Can't load a dependency (because I'm behind a proxy?)

 

Hello Lucas, hello list,

hopefully this message will thread properly.

This problem can actually be reproduced by the corresponding unit tests - at least on my "disconnected" system, the corresponding tests for the SparkInterpreter fail in exactly the same way as your code does. This is also an issue for me, since I will probably have to get those tests to pass, in order to deploy Zeppelin on our production system.

Since this actually fails unit tests, I think creating a corresponding issue is a logical next step.

I'm currently looking at the code of the test to figure out which component is responsible for directing the dependency lookup to the target, and how this can be overridden, but there's probably some implicit default in use, which makes figuring the root out slightly more tricky.

Have you had a look at where this could be overridden yet? Filed an issue already?

Unless we get some Progress going in this thread, we should start the usual procedures...

Thanks and Best,

Rick

 


Reply | Threaded
Open this post in threaded view
|

RE: Can't load a dependency (because I'm behind a proxy?)

Partridge, Lucas (GE Aviation)

Thanks for the tip, Rick.  I haven’t actually built Zeppelin yet; I’ve just been using the v0.5.0 binary.

 

As far as I know I already have all the wisp dependencies locally, in which case z.load-ing them all inside Zeppelin should work. But it doesn’t for reasons unknown.  Maybe I don’t have *all* the dependencies; or their order on the classpath is important; or…? Hence the fat jar approach.

 

Thanks for the warning about spark-notebook!  I appreciate testing access through a proxy might be difficult (maybe Apache offers one for testing?).

Thanks, Lucas.

 

 

From: Rick Moritz [mailto:[hidden email]]
Sent: 22 September 2015 14:23
To: Partridge, Lucas (GE Aviation)
Cc: [hidden email]
Subject: Re: Can't load a dependency (because I'm behind a proxy?)

 

I actually got the unit test to pass, by manually downloading the dependency into the local repo. Using an artifact that isn't in the local repo after building Zeppelin makes for an unintuitive unit test. On the other hand, required ones may already be available in the class path and make for a bad test. I'll have to think a bit about what makes sense here.

A workaround I can propose to you, is to build dummy poms with your required dependencies and use those to load the deps into the local repo with a mvn resolve-dependencies call. That should get you up and running, if you have the required access.

Alternatively, I know that spark-notebook had issues supporting proxy-lookups as well,and the corresponding issue is still pending as well. In that case I used the same strategy of manually pushing dependencies into the local repo, and then adding that as a source (hence my frantic attempts to do this in Zeppelin as well ;) )

I think having a test that needs internet access is pretty much a no-go, and a different issue from the lack of proxy support. I'll definitely keep an eye on ZEPPELIN-169 since it's relevant for our dev environments, and file a new issue once I have a solid proposition on how to go ahead with testing this feature - a unit test should be self-contained. Maybe packaging the dependency as a resource and adding a local repo is a way to test this feature in isolation. at the very least, the dep should be added as a test-scoped dependency in the pom somewhere...

Thanks again for your comments @ZEPPELIN-169, at least for my test-case issue that pointed to some magic-string culprits.

Best,

Rick

 

On Tue, Sep 22, 2015 at 3:05 PM, Partridge, Lucas (GE Aviation) <[hidden email]> wrote:

Hi Rick,

 

Ok, I misunderstood your original statementJ.  As you know, I’ve already answered some of your questions about repository lookup at https://issues.apache.org/jira/browse/ZEPPELIN-169.

 

I’ve tried using multiple z.load() statements to load wisp and all its 30 to 40 dependencies directly from the file system, but although the subsequent import statement succeeds the actual wisp commands fail with things like NoClassDefFoundError. So now I’m trying to package wisp as a fat jar and then try doing a z.load() of that…

 

Please go ahead and raise that issue if you think it will help, although there is already the feature request at ZEPPELIN-169. Regardless of whether this is a feature request or a bug this renders Zeppelin pretty useless to me behind the corporate firewall…

 

Thanks, Lucas.

 

From: Rick Moritz [mailto:[hidden email]]
Sent: 21 September 2015 17:00
To: Partridge, Lucas (GE Aviation)
Cc: [hidden email]
Subject: Re: Can't load a dependency (because I'm behind a proxy?)

 

 

Hi,

 

Unfortunately I am new to Scala so I don’t even know what ‘implicit default’ means, if indeed you’re referring to Scala! 

 

 

I was merely referring to the fact, that maven-central is often set as default repository somewhere internally (i.e. source) or externally (Zeppelin config, Maven/aether config...), and figuring out  where this setting comes from is going to take some....patience.

Nothing scala-specific in this case, but rather something inherited from sonatype-aether.

Also, for info/comparison, the trace from the crashing unit tests:

Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 102.584 sec <<< FAILURE! - in org.apache.zeppelin.spark.SparkInterpreterTest
testZContextDependencyLoading(org.apache.zeppelin.spark.SparkInterpreterTest)  Time elapsed: 63.612 sec  <<< FAILURE!
java.lang.AssertionError: expected:<SUCCESS> but was:<ERROR>
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.failNotEquals(Assert.java:743)
        at org.junit.Assert.assertEquals(Assert.java:118)
        at org.junit.Assert.assertEquals(Assert.java:144)
        at org.apache.zeppelin.spark.SparkInterpreterTest.testZContextDependencyLoading(SparkInterpreterTest.java:159)

which is followed up by

Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 69.667 sec <<< FAILURE! - in org.apache.zeppelin.spark.DepInterpreterTest
testDefault(org.apache.zeppelin.spark.DepInterpreterTest)  Time elapsed: 69.528 sec  <<< ERROR!
java.lang.NullPointerException: null
        at org.sonatype.aether.impl.internal.DefaultRepositorySystem.resolveDependencies(DefaultRepositorySystem.java:352)
        at org.apache.zeppelin.spark.dep.DependencyContext.fetchArtifactWithDep(DependencyContext.java:141)
        at org.apache.zeppelin.spark.dep.DependencyContext.fetch(DependencyContext.java:98)
        at org.apache.zeppelin.spark.DepInterpreter.interpret(DepInterpreter.java:189)
        at org.apache.zeppelin.spark.DepInterpreterTest.testDefault(DepInterpreterTest.java:88)

 

I'll give this a few hours yet, before opening an issue, maybe I'm just "holding it wrong". I doubt an issue will push this along much faster either, unless one of us actually submits a patch/PR to go along with it ;)

Best,

Rick

 

 

From: Rick Moritz [mailto:[hidden email]]
Sent: 21 September 2015 15:19
To: [hidden email]
Cc: Partridge, Lucas (GE Aviation)
Subject: RE: Can't load a dependency (because I'm behind a proxy?)

 

Hello Lucas, hello list,

hopefully this message will thread properly.

This problem can actually be reproduced by the corresponding unit tests - at least on my "disconnected" system, the corresponding tests for the SparkInterpreter fail in exactly the same way as your code does. This is also an issue for me, since I will probably have to get those tests to pass, in order to deploy Zeppelin on our production system.

Since this actually fails unit tests, I think creating a corresponding issue is a logical next step.

I'm currently looking at the code of the test to figure out which component is responsible for directing the dependency lookup to the target, and how this can be overridden, but there's probably some implicit default in use, which makes figuring the root out slightly more tricky.

Have you had a look at where this could be overridden yet? Filed an issue already?

Unless we get some Progress going in this thread, we should start the usual procedures...

Thanks and Best,

Rick