Zeppelin failing?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Zeppelin failing?

Michael Segel
Hi,

I’m running 7.2 compiled against MapR 5.2

I have nginx in front of Zeppelin so I run zeppelin on the local host port and use nginx as a proxy w ssh.

After running zeppelin for approximately 24 hours, if I login and try to run a paragraph in any notebook, I get a java error, connection refused.

Even trying to restart the interpreter (any interpreter results in a failure.)

I’m going to assume its human error on setting up the configuration.

Any suggestions?

Thx

-Mike

Reply | Threaded
Open this post in threaded view
|

Re: Zeppelin failing?

Michael Segel
The error:
ERROR [2017-10-18 00:00:00,003] ({pool-2-thread-20} Job.java[run]:188) - Job failed
org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
        at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:354)
        at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:97)
        at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:406)
        at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
        at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:329)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

And this in the zeppelin spark interpreter log:
ERROR [2017-10-18 10:39:29,582] ({BoneCP-pool-watch-thread} PoolWatchThread.java[fillConnections]:118) - Error in trying to obtain a connection. Retrying in 7000ms
java.sql.SQLException: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.
        at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection.setReadOnly(Unknown Source)
        at com.jolbox.bonecp.ConnectionHandle.setReadOnly(ConnectionHandle.java:1324)
        at com.jolbox.bonecp.ConnectionHandle.<init>(ConnectionHandle.java:262)
        at com.jolbox.bonecp.PoolWatchThread.fillConnections(PoolWatchThread.java:115)
        at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:82)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLException: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.
        at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)


> On Oct 18, 2017, at 10:36 AM, Michael Segel <[hidden email]> wrote:
>
> Hi,
>
> I’m running 7.2 compiled against MapR 5.2
>
> I have nginx in front of Zeppelin so I run zeppelin on the local host port and use nginx as a proxy w ssh.
>
> After running zeppelin for approximately 24 hours, if I login and try to run a paragraph in any notebook, I get a java error, connection refused.
>
> Even trying to restart the interpreter (any interpreter results in a failure.)
>
> I’m going to assume its human error on setting up the configuration.
>
> Any suggestions?
>
> Thx
>
> -Mike
>

Reply | Threaded
Open this post in threaded view
|

Re: Zeppelin failing?

Michael Segel
I went back and rebuilt 0.7.2 and it seems that the issue has gone away.

Not sure what changed or which setting could have been causing the issue with socket connections.


> On Oct 18, 2017, at 10:40 AM, Michael Segel <[hidden email]> wrote:
>
> The error:
> ERROR [2017-10-18 00:00:00,003] ({pool-2-thread-20} Job.java[run]:188) - Job failed
> org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
>        at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.interpret(RemoteInterpreter.java:354)
>        at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter.java:97)
>        at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:406)
>        at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
>        at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:329)
>        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
>        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>        at java.lang.Thread.run(Thread.java:748)
>
> And this in the zeppelin spark interpreter log:
> ERROR [2017-10-18 10:39:29,582] ({BoneCP-pool-watch-thread} PoolWatchThread.java[fillConnections]:118) - Error in trying to obtain a connection. Retrying in 7000ms
> java.sql.SQLException: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.
>        at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
>        at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
>        at org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
>        at org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
>        at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
>        at org.apache.derby.impl.jdbc.EmbedConnection.setReadOnly(Unknown Source)
>        at com.jolbox.bonecp.ConnectionHandle.setReadOnly(ConnectionHandle.java:1324)
>        at com.jolbox.bonecp.ConnectionHandle.<init>(ConnectionHandle.java:262)
>        at com.jolbox.bonecp.PoolWatchThread.fillConnections(PoolWatchThread.java:115)
>        at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:82)
>        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>        at java.lang.Thread.run(Thread.java:748)
> Caused by: java.sql.SQLException: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.
>        at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
>
>
>> On Oct 18, 2017, at 10:36 AM, Michael Segel <[hidden email]> wrote:
>>
>> Hi,
>>
>> I’m running 7.2 compiled against MapR 5.2
>>
>> I have nginx in front of Zeppelin so I run zeppelin on the local host port and use nginx as a proxy w ssh.
>>
>> After running zeppelin for approximately 24 hours, if I login and try to run a paragraph in any notebook, I get a java error, connection refused.
>>
>> Even trying to restart the interpreter (any interpreter results in a failure.)
>>
>> I’m going to assume its human error on setting up the configuration.
>>
>> Any suggestions?
>>
>> Thx
>>
>> -Mike
>>
>