Thrift server for call back will start. Port:

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Thrift server for call back will start. Port:

小野圭二
Hi there,

I am so confusing now, let me know why if You know this, please. 

The env:
   Mac:  High Sierra ver.10.13.3
   Zep: 0.7.3
   Spark: 2.2.0
   Java: 1.8
   notebook script:  %pyspark   print 'aaa'              (just it for test)

In this env,  i got the error in log as 

ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed

org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)



Indeed, before running notebook, RemoteInterpreterManagedProcess.java prepares a Port for callback to Thrift.
 

INFO ({pool-2-thread-1} RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for callback will start. Port: 63039



I guess, the reason why the error was Zeppelin could not attache to the Thrift PORT.


And my confusion is, when i run the same module on CentOS, it worked fine.
    CentOS:  7
    Zep: same
    Spark: same
    Java: same
     notebook script: same

And there was no the Thift port description in the log file. From this consequence, the fine working was cause of did not use Thrift port on CentOS.


Do You know actually there is different behavior between Mac and CentOS? And i am so appreciated if You know the workaround on Mac.


Best,
Ono
Reply | Threaded
Open this post in threaded view
|

Re: Thrift server for call back will start. Port:

Jeff Zhang

It is possible that 63039 is used by other processes, as Mac can not detect whether this port is used or not from my experience.

And can you reproduce this issue consistently ? Because each time zeppelin will chose a random port as long as you don't specify the portRange, so if you reproduce it consistently, it might due to other issues.




小野圭二 <[hidden email]>于2018年3月2日周五 下午1:32写道:
Hi there,

I am so confusing now, let me know why if You know this, please. 

The env:
   Mac:  High Sierra ver.10.13.3
   Zep: 0.7.3
   Spark: 2.2.0
   Java: 1.8
   notebook script:  %pyspark   print 'aaa'              (just it for test)

In this env,  i got the error in log as 

ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed

org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)



Indeed, before running notebook, RemoteInterpreterManagedProcess.java prepares a Port for callback to Thrift.
 

INFO ({pool-2-thread-1} RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for callback will start. Port: 63039



I guess, the reason why the error was Zeppelin could not attache to the Thrift PORT.


And my confusion is, when i run the same module on CentOS, it worked fine.
    CentOS:  7
    Zep: same
    Spark: same
    Java: same
     notebook script: same

And there was no the Thift port description in the log file. From this consequence, the fine working was cause of did not use Thrift port on CentOS.


Do You know actually there is different behavior between Mac and CentOS? And i am so appreciated if You know the workaround on Mac.


Best,
Ono
Reply | Threaded
Open this post in threaded view
|

Re: Thrift server for call back will start. Port:

小野圭二
Hi Jeff,

No one graved 63039 so far on Mac.
Yes, this issue has been reproduced as changing the port NO randomly.

Ono


2018-03-02 14:49 GMT+09:00 Jeff Zhang <[hidden email]>:

It is possible that 63039 is used by other processes, as Mac can not detect whether this port is used or not from my experience.

And can you reproduce this issue consistently ? Because each time zeppelin will chose a random port as long as you don't specify the portRange, so if you reproduce it consistently, it might due to other issues.




小野圭二 <[hidden email]>于2018年3月2日周五 下午1:32写道:
Hi there,

I am so confusing now, let me know why if You know this, please. 

The env:
   Mac:  High Sierra ver.10.13.3
   Zep: 0.7.3
   Spark: 2.2.0
   Java: 1.8
   notebook script:  %pyspark   print 'aaa'              (just it for test)

In this env,  i got the error in log as 

ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed

org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)



Indeed, before running notebook, RemoteInterpreterManagedProcess.java prepares a Port for callback to Thrift.
 

INFO ({pool-2-thread-1} RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for callback will start. Port: 63039



I guess, the reason why the error was Zeppelin could not attache to the Thrift PORT.


And my confusion is, when i run the same module on CentOS, it worked fine.
    CentOS:  7
    Zep: same
    Spark: same
    Java: same
     notebook script: same

And there was no the Thift port description in the log file. From this consequence, the fine working was cause of did not use Thrift port on CentOS.


Do You know actually there is different behavior between Mac and CentOS? And i am so appreciated if You know the workaround on Mac.


Best,
Ono

Reply | Threaded
Open this post in threaded view
|

Re: Thrift server for call back will start. Port:

小野圭二
Hi,

I show the all log on this error.There are 2 type error. The 1 is at the first run. The 2 is at the second run. Seems like could not established to Thrift callback port 51348 that was set at the first run.
First of all, does it need to establish the callback port of Thrift server?   
The notebook script was just 
    %pyspark   
        print 'aaa' 
Any advice appreciate.

1.
INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05 13:11:13,934] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109) - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2} Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using pyspark RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2} ManagedInterpreterGroup.java[getOrCreateInterpreterProcess]:57) - Create InterperterProcess for InterpreterGroup: 2CV1983WX:shared_process WARN [2018-03-05 13:11:14,010] ({pool-2-thread-2} InterpreterSetting.java[setupPropertiesForSparkR]:796) - sparkr.zip is not found, SparkR may not work. INFO [2018-03-05 13:11:14,017] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:123) - Choose port 51347 for RemoteInterpreterProcess INFO [2018-03-05 13:11:14,023] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for callback will start. Port: 51348 INFO [2018-03-05 13:11:14,049] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:205) - Run interpreter process [/Users/onokeiji/zeppelin/zeppelin/bin/interpreter.sh, -d, /Users/onokeiji/zeppelin/zeppelin/interpreter/spark, -c, 172.16.20.10, -p, 51348, -l, /Users/onokeiji/zeppelin/zeppelin/local-repo/2CV1983WX, -g, spark] INFO [2018-03-05 13:11:14,238] ({Exec Default Executor} RemoteInterpreterManagedProcess.java[onProcessComplete]:258) - Interpreter process exited 0 ERROR [2018-03-05 13:12:14,058] ({pool-2-thread-2} Job.java[run]:184) - Job failed org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at org.apache.zeppelin.scheduler.Job.run(Job.java:182) at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ERROR [2018-03-05 13:12:14,075] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at org.apache.zeppelin.scheduler.Job.run(Job.java:182) at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) WARN [2018-03-05 13:12:14,075] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227) - Job 20180302-135047_99102973 is finished, status: ERROR, exception: org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter, result: Cannot run interpreter INFO [2018-03-05 13:12:14,103] ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05 13:12:14,109] ({pool-2-thread-2} SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session

2.
INFO [2018-03-05 13:15:32,736] ({qtp1651855867-13} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
 INFO [2018-03-05 13:15:32,751] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109) - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
 INFO [2018-03-05 13:15:32,753] ({pool-2-thread-2} Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using pyspark RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
ERROR [2018-03-05 13:15:32,882] ({pool-2-thread-2} Job.java[run]:184) - Job failed
org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
... 13 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 20 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 21 more
ERROR [2018-03-05 13:15:32,884] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error
org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
... 13 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 20 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 21 more
 WARN [2018-03-05 13:15:32,885] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227) - Job 20180302-135047_99102973 is finished, status: ERROR, exception: org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused), result: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
 INFO [2018-03-05 13:15:32,910] ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
 INFO [2018-03-05 13:15:32,914] ({pool-2-thread-2} SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session

2018-03-02 15:07 GMT+09:00 小野圭二 <[hidden email]>:
Hi Jeff,

No one graved 63039 so far on Mac.
Yes, this issue has been reproduced as changing the port NO randomly.

Ono


2018-03-02 14:49 GMT+09:00 Jeff Zhang <[hidden email]>:

It is possible that 63039 is used by other processes, as Mac can not detect whether this port is used or not from my experience.

And can you reproduce this issue consistently ? Because each time zeppelin will chose a random port as long as you don't specify the portRange, so if you reproduce it consistently, it might due to other issues.




小野圭二 <[hidden email]>于2018年3月2日周五 下午1:32写道:
Hi there,

I am so confusing now, let me know why if You know this, please. 

The env:
   Mac:  High Sierra ver.10.13.3
   Zep: 0.7.3
   Spark: 2.2.0
   Java: 1.8
   notebook script:  %pyspark   print 'aaa'              (just it for test)

In this env,  i got the error in log as 

ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed

org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)



Indeed, before running notebook, RemoteInterpreterManagedProcess.java prepares a Port for callback to Thrift.
 

INFO ({pool-2-thread-1} RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for callback will start. Port: 63039



I guess, the reason why the error was Zeppelin could not attache to the Thrift PORT.


And my confusion is, when i run the same module on CentOS, it worked fine.
    CentOS:  7
    Zep: same
    Spark: same
    Java: same
     notebook script: same

And there was no the Thift port description in the log file. From this consequence, the fine working was cause of did not use Thrift port on CentOS.


Do You know actually there is different behavior between Mac and CentOS? And i am so appreciated if You know the workaround on Mac.


Best,
Ono


Reply | Threaded
Open this post in threaded view
|

Re: Thrift server for call back will start. Port:

Jeff Zhang

Could you check the interpreter log ? The issue of not be able to connecting to thrift port could happen due to network issue, like firewall or you use VPN, etc.



小野圭二 <[hidden email]>于2018年3月5日周一 下午12:32写道:
Hi,

I show the all log on this error.There are 2 type error. The 1 is at the first run. The 2 is at the second run. Seems like could not established to Thrift callback port 51348 that was set at the first run.
First of all, does it need to establish the callback port of Thrift server?   
The notebook script was just 
    %pyspark   
        print 'aaa' 
Any advice appreciate.

1.
INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05 13:11:13,934] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109) - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2} Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using pyspark RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2} ManagedInterpreterGroup.java[getOrCreateInterpreterProcess]:57) - Create InterperterProcess for InterpreterGroup: 2CV1983WX:shared_process WARN [2018-03-05 13:11:14,010] ({pool-2-thread-2} InterpreterSetting.java[setupPropertiesForSparkR]:796) - sparkr.zip is not found, SparkR may not work. INFO [2018-03-05 13:11:14,017] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:123) - Choose port 51347 for RemoteInterpreterProcess INFO [2018-03-05 13:11:14,023] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for callback will start. Port: 51348 INFO [2018-03-05 13:11:14,049] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:205) - Run interpreter process [/Users/onokeiji/zeppelin/zeppelin/bin/interpreter.sh, -d, /Users/onokeiji/zeppelin/zeppelin/interpreter/spark, -c, 172.16.20.10, -p, 51348, -l, /Users/onokeiji/zeppelin/zeppelin/local-repo/2CV1983WX, -g, spark] INFO [2018-03-05 13:11:14,238] ({Exec Default Executor} RemoteInterpreterManagedProcess.java[onProcessComplete]:258) - Interpreter process exited 0 ERROR [2018-03-05 13:12:14,058] ({pool-2-thread-2} Job.java[run]:184) - Job failed org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at org.apache.zeppelin.scheduler.Job.run(Job.java:182) at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ERROR [2018-03-05 13:12:14,075] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at org.apache.zeppelin.scheduler.Job.run(Job.java:182) at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) WARN [2018-03-05 13:12:14,075] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227) - Job 20180302-135047_99102973 is finished, status: ERROR, exception: org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter, result: Cannot run interpreter INFO [2018-03-05 13:12:14,103] ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05 13:12:14,109] ({pool-2-thread-2} SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session

2.
INFO [2018-03-05 13:15:32,736] ({qtp1651855867-13} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
 INFO [2018-03-05 13:15:32,751] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109) - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
 INFO [2018-03-05 13:15:32,753] ({pool-2-thread-2} Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using pyspark RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
ERROR [2018-03-05 13:15:32,882] ({pool-2-thread-2} Job.java[run]:184) - Job failed
org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
... 13 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 20 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 21 more
ERROR [2018-03-05 13:15:32,884] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error
org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
... 13 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 20 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 21 more
 WARN [2018-03-05 13:15:32,885] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227) - Job 20180302-135047_99102973 is finished, status: ERROR, exception: org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused), result: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
 INFO [2018-03-05 13:15:32,910] ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
 INFO [2018-03-05 13:15:32,914] ({pool-2-thread-2} SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session

2018-03-02 15:07 GMT+09:00 小野圭二 <[hidden email]>:
Hi Jeff,

No one graved 63039 so far on Mac.
Yes, this issue has been reproduced as changing the port NO randomly.

Ono


2018-03-02 14:49 GMT+09:00 Jeff Zhang <[hidden email]>:

It is possible that 63039 is used by other processes, as Mac can not detect whether this port is used or not from my experience.

And can you reproduce this issue consistently ? Because each time zeppelin will chose a random port as long as you don't specify the portRange, so if you reproduce it consistently, it might due to other issues.




小野圭二 <[hidden email]>于2018年3月2日周五 下午1:32写道:
Hi there,

I am so confusing now, let me know why if You know this, please. 

The env:
   Mac:  High Sierra ver.10.13.3
   Zep: 0.7.3
   Spark: 2.2.0
   Java: 1.8
   notebook script:  %pyspark   print 'aaa'              (just it for test)

In this env,  i got the error in log as 

ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed

org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)



Indeed, before running notebook, RemoteInterpreterManagedProcess.java prepares a Port for callback to Thrift.
 

INFO ({pool-2-thread-1} RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for callback will start. Port: 63039



I guess, the reason why the error was Zeppelin could not attache to the Thrift PORT.


And my confusion is, when i run the same module on CentOS, it worked fine.
    CentOS:  7
    Zep: same
    Spark: same
    Java: same
     notebook script: same

And there was no the Thift port description in the log file. From this consequence, the fine working was cause of did not use Thrift port on CentOS.


Do You know actually there is different behavior between Mac and CentOS? And i am so appreciated if You know the workaround on Mac.


Best,
Ono


Reply | Threaded
Open this post in threaded view
|

Re: Thrift server for call back will start. Port:

小野圭二
Actually, there was not created the interpreter log file as said 'Cannot run interpreter' in the log. And it was run on local machine as MAC, i mean stand alone. i stopped every firewall to make it easy. And again,this happened just on Mac. On CentOS, it works fine.
That makes me confusion. :-b

2018-03-05 14:06 GMT+09:00 Jeff Zhang <[hidden email]>:

Could you check the interpreter log ? The issue of not be able to connecting to thrift port could happen due to network issue, like firewall or you use VPN, etc.



小野圭二 <[hidden email]>于2018年3月5日周一 下午12:32写道:
Hi,

I show the all log on this error.There are 2 type error. The 1 is at the first run. The 2 is at the second run. Seems like could not established to Thrift callback port 51348 that was set at the first run.
First of all, does it need to establish the callback port of Thrift server?   
The notebook script was just 
    %pyspark   
        print 'aaa' 
Any advice appreciate.

1.
INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05 13:11:13,934] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109) - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2} Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using pyspark RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2} ManagedInterpreterGroup.java[getOrCreateInterpreterProcess]:57) - Create InterperterProcess for InterpreterGroup: 2CV1983WX:shared_process WARN [2018-03-05 13:11:14,010] ({pool-2-thread-2} InterpreterSetting.java[setupPropertiesForSparkR]:796) - sparkr.zip is not found, SparkR may not work. INFO [2018-03-05 13:11:14,017] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:123) - Choose port 51347 for RemoteInterpreterProcess INFO [2018-03-05 13:11:14,023] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for callback will start. Port: 51348 INFO [2018-03-05 13:11:14,049] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:205) - Run interpreter process [/Users/onokeiji/zeppelin/zeppelin/bin/interpreter.sh, -d, /Users/onokeiji/zeppelin/zeppelin/interpreter/spark, -c, 172.16.20.10, -p, 51348, -l, /Users/onokeiji/zeppelin/zeppelin/local-repo/2CV1983WX, -g, spark] INFO [2018-03-05 13:11:14,238] ({Exec Default Executor} RemoteInterpreterManagedProcess.java[onProcessComplete]:258) - Interpreter process exited 0 ERROR [2018-03-05 13:12:14,058] ({pool-2-thread-2} Job.java[run]:184) - Job failed org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at org.apache.zeppelin.scheduler.Job.run(Job.java:182) at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ERROR [2018-03-05 13:12:14,075] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at org.apache.zeppelin.scheduler.Job.run(Job.java:182) at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) WARN [2018-03-05 13:12:14,075] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227) - Job 20180302-135047_99102973 is finished, status: ERROR, exception: org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter, result: Cannot run interpreter INFO [2018-03-05 13:12:14,103] ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05 13:12:14,109] ({pool-2-thread-2} SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session

2.
INFO [2018-03-05 13:15:32,736] ({qtp1651855867-13} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
 INFO [2018-03-05 13:15:32,751] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109) - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
 INFO [2018-03-05 13:15:32,753] ({pool-2-thread-2} Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using pyspark RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
ERROR [2018-03-05 13:15:32,882] ({pool-2-thread-2} Job.java[run]:184) - Job failed
org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
... 13 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 20 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 21 more
ERROR [2018-03-05 13:15:32,884] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error
org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
... 13 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 20 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 21 more
 WARN [2018-03-05 13:15:32,885] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227) - Job 20180302-135047_99102973 is finished, status: ERROR, exception: org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused), result: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
 INFO [2018-03-05 13:15:32,910] ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
 INFO [2018-03-05 13:15:32,914] ({pool-2-thread-2} SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session

2018-03-02 15:07 GMT+09:00 小野圭二 <[hidden email]>:
Hi Jeff,

No one graved 63039 so far on Mac.
Yes, this issue has been reproduced as changing the port NO randomly.

Ono


2018-03-02 14:49 GMT+09:00 Jeff Zhang <[hidden email]>:

It is possible that 63039 is used by other processes, as Mac can not detect whether this port is used or not from my experience.

And can you reproduce this issue consistently ? Because each time zeppelin will chose a random port as long as you don't specify the portRange, so if you reproduce it consistently, it might due to other issues.




小野圭二 <[hidden email]>于2018年3月2日周五 下午1:32写道:
Hi there,

I am so confusing now, let me know why if You know this, please. 

The env:
   Mac:  High Sierra ver.10.13.3
   Zep: 0.7.3
   Spark: 2.2.0
   Java: 1.8
   notebook script:  %pyspark   print 'aaa'              (just it for test)

In this env,  i got the error in log as 

ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed

org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)



Indeed, before running notebook, RemoteInterpreterManagedProcess.java prepares a Port for callback to Thrift.
 

INFO ({pool-2-thread-1} RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for callback will start. Port: 63039



I guess, the reason why the error was Zeppelin could not attache to the Thrift PORT.


And my confusion is, when i run the same module on CentOS, it worked fine.
    CentOS:  7
    Zep: same
    Spark: same
    Java: same
     notebook script: same

And there was no the Thift port description in the log file. From this consequence, the fine working was cause of did not use Thrift port on CentOS.


Do You know actually there is different behavior between Mac and CentOS? And i am so appreciated if You know the workaround on Mac.


Best,
Ono



Reply | Threaded
Open this post in threaded view
|

Re: Thrift server for call back will start. Port:

Jeff Zhang

Try to add the following to log4j.properties for more log

log4j.logger.org.apache.zeppelin.interpreter=DEBUG



小野圭二 <[hidden email]>于2018年3月5日周一 下午5:13写道:
Actually, there was not created the interpreter log file as said 'Cannot run interpreter' in the log. And it was run on local machine as MAC, i mean stand alone. i stopped every firewall to make it easy. And again,this happened just on Mac. On CentOS, it works fine.
That makes me confusion. :-b

2018-03-05 14:06 GMT+09:00 Jeff Zhang <[hidden email]>:

Could you check the interpreter log ? The issue of not be able to connecting to thrift port could happen due to network issue, like firewall or you use VPN, etc.



小野圭二 <[hidden email]>于2018年3月5日周一 下午12:32写道:
Hi,

I show the all log on this error.There are 2 type error. The 1 is at the first run. The 2 is at the second run. Seems like could not established to Thrift callback port 51348 that was set at the first run.
First of all, does it need to establish the callback port of Thrift server?   
The notebook script was just 
    %pyspark   
        print 'aaa' 
Any advice appreciate.

1.
INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05 13:11:13,934] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109) - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2} Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using pyspark RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2} ManagedInterpreterGroup.java[getOrCreateInterpreterProcess]:57) - Create InterperterProcess for InterpreterGroup: 2CV1983WX:shared_process WARN [2018-03-05 13:11:14,010] ({pool-2-thread-2} InterpreterSetting.java[setupPropertiesForSparkR]:796) - sparkr.zip is not found, SparkR may not work. INFO [2018-03-05 13:11:14,017] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:123) - Choose port 51347 for RemoteInterpreterProcess INFO [2018-03-05 13:11:14,023] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for callback will start. Port: 51348 INFO [2018-03-05 13:11:14,049] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:205) - Run interpreter process [/Users/onokeiji/zeppelin/zeppelin/bin/interpreter.sh, -d, /Users/onokeiji/zeppelin/zeppelin/interpreter/spark, -c, 172.16.20.10, -p, 51348, -l, /Users/onokeiji/zeppelin/zeppelin/local-repo/2CV1983WX, -g, spark] INFO [2018-03-05 13:11:14,238] ({Exec Default Executor} RemoteInterpreterManagedProcess.java[onProcessComplete]:258) - Interpreter process exited 0 ERROR [2018-03-05 13:12:14,058] ({pool-2-thread-2} Job.java[run]:184) - Job failed org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at org.apache.zeppelin.scheduler.Job.run(Job.java:182) at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ERROR [2018-03-05 13:12:14,075] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at org.apache.zeppelin.scheduler.Job.run(Job.java:182) at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) WARN [2018-03-05 13:12:14,075] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227) - Job 20180302-135047_99102973 is finished, status: ERROR, exception: org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter, result: Cannot run interpreter INFO [2018-03-05 13:12:14,103] ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05 13:12:14,109] ({pool-2-thread-2} SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session

2.
INFO [2018-03-05 13:15:32,736] ({qtp1651855867-13} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
 INFO [2018-03-05 13:15:32,751] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109) - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
 INFO [2018-03-05 13:15:32,753] ({pool-2-thread-2} Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using pyspark RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
ERROR [2018-03-05 13:15:32,882] ({pool-2-thread-2} Job.java[run]:184) - Job failed
org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
... 13 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 20 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 21 more
ERROR [2018-03-05 13:15:32,884] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error
org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
... 13 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 20 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 21 more
 WARN [2018-03-05 13:15:32,885] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227) - Job 20180302-135047_99102973 is finished, status: ERROR, exception: org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused), result: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
 INFO [2018-03-05 13:15:32,910] ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
 INFO [2018-03-05 13:15:32,914] ({pool-2-thread-2} SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session

2018-03-02 15:07 GMT+09:00 小野圭二 <[hidden email]>:
Hi Jeff,

No one graved 63039 so far on Mac.
Yes, this issue has been reproduced as changing the port NO randomly.

Ono


2018-03-02 14:49 GMT+09:00 Jeff Zhang <[hidden email]>:

It is possible that 63039 is used by other processes, as Mac can not detect whether this port is used or not from my experience.

And can you reproduce this issue consistently ? Because each time zeppelin will chose a random port as long as you don't specify the portRange, so if you reproduce it consistently, it might due to other issues.




小野圭二 <[hidden email]>于2018年3月2日周五 下午1:32写道:
Hi there,

I am so confusing now, let me know why if You know this, please. 

The env:
   Mac:  High Sierra ver.10.13.3
   Zep: 0.7.3
   Spark: 2.2.0
   Java: 1.8
   notebook script:  %pyspark   print 'aaa'              (just it for test)

In this env,  i got the error in log as 

ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed

org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)



Indeed, before running notebook, RemoteInterpreterManagedProcess.java prepares a Port for callback to Thrift.
 

INFO ({pool-2-thread-1} RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for callback will start. Port: 63039



I guess, the reason why the error was Zeppelin could not attache to the Thrift PORT.


And my confusion is, when i run the same module on CentOS, it worked fine.
    CentOS:  7
    Zep: same
    Spark: same
    Java: same
     notebook script: same

And there was no the Thift port description in the log file. From this consequence, the fine working was cause of did not use Thrift port on CentOS.


Do You know actually there is different behavior between Mac and CentOS? And i am so appreciated if You know the workaround on Mac.


Best,
Ono



Reply | Threaded
Open this post in threaded view
|

Re: Thrift server for call back will start. Port:

小野圭二
Nothing changed. I guess this property does not effect cause of failing to call the interpreter.
i challenged it as log4j.......zeppelin=DEBUG, but no way.

2018-03-05 18:24 GMT+09:00 Jeff Zhang <[hidden email]>:

Try to add the following to log4j.properties for more log

log4j.logger.org.apache.zeppelin.interpreter=DEBUG



小野圭二 <[hidden email]>于2018年3月5日周一 下午5:13写道:
Actually, there was not created the interpreter log file as said 'Cannot run interpreter' in the log. And it was run on local machine as MAC, i mean stand alone. i stopped every firewall to make it easy. And again,this happened just on Mac. On CentOS, it works fine.
That makes me confusion. :-b

2018-03-05 14:06 GMT+09:00 Jeff Zhang <[hidden email]>:

Could you check the interpreter log ? The issue of not be able to connecting to thrift port could happen due to network issue, like firewall or you use VPN, etc.



小野圭二 <[hidden email]>于2018年3月5日周一 下午12:32写道:
Hi,

I show the all log on this error.There are 2 type error. The 1 is at the first run. The 2 is at the second run. Seems like could not established to Thrift callback port 51348 that was set at the first run.
First of all, does it need to establish the callback port of Thrift server?   
The notebook script was just 
    %pyspark   
        print 'aaa' 
Any advice appreciate.

1.
INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05 13:11:13,934] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109) - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2} Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using pyspark RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2} ManagedInterpreterGroup.java[getOrCreateInterpreterProcess]:57) - Create InterperterProcess for InterpreterGroup: 2CV1983WX:shared_process WARN [2018-03-05 13:11:14,010] ({pool-2-thread-2} InterpreterSetting.java[setupPropertiesForSparkR]:796) - sparkr.zip is not found, SparkR may not work. INFO [2018-03-05 13:11:14,017] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:123) - Choose port 51347 for RemoteInterpreterProcess INFO [2018-03-05 13:11:14,023] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for callback will start. Port: 51348 INFO [2018-03-05 13:11:14,049] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:205) - Run interpreter process [/Users/onokeiji/zeppelin/zeppelin/bin/interpreter.sh, -d, /Users/onokeiji/zeppelin/zeppelin/interpreter/spark, -c, 172.16.20.10, -p, 51348, -l, /Users/onokeiji/zeppelin/zeppelin/local-repo/2CV1983WX, -g, spark] INFO [2018-03-05 13:11:14,238] ({Exec Default Executor} RemoteInterpreterManagedProcess.java[onProcessComplete]:258) - Interpreter process exited 0 ERROR [2018-03-05 13:12:14,058] ({pool-2-thread-2} Job.java[run]:184) - Job failed org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at org.apache.zeppelin.scheduler.Job.run(Job.java:182) at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ERROR [2018-03-05 13:12:14,075] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at org.apache.zeppelin.scheduler.Job.run(Job.java:182) at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) WARN [2018-03-05 13:12:14,075] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227) - Job 20180302-135047_99102973 is finished, status: ERROR, exception: org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter, result: Cannot run interpreter INFO [2018-03-05 13:12:14,103] ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05 13:12:14,109] ({pool-2-thread-2} SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session

2.
INFO [2018-03-05 13:15:32,736] ({qtp1651855867-13} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
 INFO [2018-03-05 13:15:32,751] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109) - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
 INFO [2018-03-05 13:15:32,753] ({pool-2-thread-2} Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using pyspark RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
ERROR [2018-03-05 13:15:32,882] ({pool-2-thread-2} Job.java[run]:184) - Job failed
org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
... 13 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 20 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 21 more
ERROR [2018-03-05 13:15:32,884] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error
org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
... 13 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 20 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 21 more
 WARN [2018-03-05 13:15:32,885] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227) - Job 20180302-135047_99102973 is finished, status: ERROR, exception: org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused), result: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
 INFO [2018-03-05 13:15:32,910] ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
 INFO [2018-03-05 13:15:32,914] ({pool-2-thread-2} SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session

2018-03-02 15:07 GMT+09:00 小野圭二 <[hidden email]>:
Hi Jeff,

No one graved 63039 so far on Mac.
Yes, this issue has been reproduced as changing the port NO randomly.

Ono


2018-03-02 14:49 GMT+09:00 Jeff Zhang <[hidden email]>:

It is possible that 63039 is used by other processes, as Mac can not detect whether this port is used or not from my experience.

And can you reproduce this issue consistently ? Because each time zeppelin will chose a random port as long as you don't specify the portRange, so if you reproduce it consistently, it might due to other issues.




小野圭二 <[hidden email]>于2018年3月2日周五 下午1:32写道:
Hi there,

I am so confusing now, let me know why if You know this, please. 

The env:
   Mac:  High Sierra ver.10.13.3
   Zep: 0.7.3
   Spark: 2.2.0
   Java: 1.8
   notebook script:  %pyspark   print 'aaa'              (just it for test)

In this env,  i got the error in log as 

ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed

org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)



Indeed, before running notebook, RemoteInterpreterManagedProcess.java prepares a Port for callback to Thrift.
 

INFO ({pool-2-thread-1} RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for callback will start. Port: 63039



I guess, the reason why the error was Zeppelin could not attache to the Thrift PORT.


And my confusion is, when i run the same module on CentOS, it worked fine.
    CentOS:  7
    Zep: same
    Spark: same
    Java: same
     notebook script: same

And there was no the Thift port description in the log file. From this consequence, the fine working was cause of did not use Thrift port on CentOS.


Do You know actually there is different behavior between Mac and CentOS? And i am so appreciated if You know the workaround on Mac.


Best,
Ono




Reply | Threaded
Open this post in threaded view
|

Re: Thrift server for call back will start. Port:

小野圭二
I wonder why, but it seems work fine, and would like to share how to fix it.

First of all, this issue happened just on MAC.
[THE ENV WAS]
  macOS High Sierra v 10.13.3
  cpu core i5
  mem  8GB
  Zeppelin 0.8.0 snapshot
  Spark 2.2.0
[ISSUE]
   see this previous mails
[SOLUTION]
   According to the above ref, i removed 'SPARK_HOME', 'SPARK_SUBMIT_OPTIONS' from my environment parameters and conf/zeppelin-env.sh(set it as same like CentOS). Then restart zeppelin, that's all.
[TO DO]
   Seems like it is not good chemistry between Spark and Zeppelin on Mac. From the reason that the consequence of removing the environment parameters, Zeppelin headed to own spark lib. 
   Maybe should write this in the document as caution. 


Cheers, 

Ono Keiji


 

2018-03-06 9:15 GMT+09:00 小野圭二 <[hidden email]>:
Nothing changed. I guess this property does not effect cause of failing to call the interpreter.
i challenged it as log4j.......zeppelin=DEBUG, but no way.

2018-03-05 18:24 GMT+09:00 Jeff Zhang <[hidden email]>:

Try to add the following to log4j.properties for more log

log4j.logger.org.apache.zeppelin.interpreter=DEBUG



小野圭二 <[hidden email]>于2018年3月5日周一 下午5:13写道:
Actually, there was not created the interpreter log file as said 'Cannot run interpreter' in the log. And it was run on local machine as MAC, i mean stand alone. i stopped every firewall to make it easy. And again,this happened just on Mac. On CentOS, it works fine.
That makes me confusion. :-b

2018-03-05 14:06 GMT+09:00 Jeff Zhang <[hidden email]>:

Could you check the interpreter log ? The issue of not be able to connecting to thrift port could happen due to network issue, like firewall or you use VPN, etc.



小野圭二 <[hidden email]>于2018年3月5日周一 下午12:32写道:
Hi,

I show the all log on this error.There are 2 type error. The 1 is at the first run. The 2 is at the second run. Seems like could not established to Thrift callback port 51348 that was set at the first run.
First of all, does it need to establish the callback port of Thrift server?   
The notebook script was just 
    %pyspark   
        print 'aaa' 
Any advice appreciate.

1.
INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05 13:11:13,934] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109) - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2} Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using pyspark RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2} ManagedInterpreterGroup.java[getOrCreateInterpreterProcess]:57) - Create InterperterProcess for InterpreterGroup: 2CV1983WX:shared_process WARN [2018-03-05 13:11:14,010] ({pool-2-thread-2} InterpreterSetting.java[setupPropertiesForSparkR]:796) - sparkr.zip is not found, SparkR may not work. INFO [2018-03-05 13:11:14,017] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:123) - Choose port 51347 for RemoteInterpreterProcess INFO [2018-03-05 13:11:14,023] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for callback will start. Port: 51348 INFO [2018-03-05 13:11:14,049] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:205) - Run interpreter process [/Users/onokeiji/zeppelin/zeppelin/bin/interpreter.sh, -d, /Users/onokeiji/zeppelin/zeppelin/interpreter/spark, -c, 172.16.20.10, -p, 51348, -l, /Users/onokeiji/zeppelin/zeppelin/local-repo/2CV1983WX, -g, spark] INFO [2018-03-05 13:11:14,238] ({Exec Default Executor} RemoteInterpreterManagedProcess.java[onProcessComplete]:258) - Interpreter process exited 0 ERROR [2018-03-05 13:12:14,058] ({pool-2-thread-2} Job.java[run]:184) - Job failed org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at org.apache.zeppelin.scheduler.Job.run(Job.java:182) at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ERROR [2018-03-05 13:12:14,075] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at org.apache.zeppelin.scheduler.Job.run(Job.java:182) at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) WARN [2018-03-05 13:12:14,075] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227) - Job 20180302-135047_99102973 is finished, status: ERROR, exception: org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter, result: Cannot run interpreter INFO [2018-03-05 13:12:14,103] ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05 13:12:14,109] ({pool-2-thread-2} SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session

2.
INFO [2018-03-05 13:15:32,736] ({qtp1651855867-13} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
 INFO [2018-03-05 13:15:32,751] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109) - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
 INFO [2018-03-05 13:15:32,753] ({pool-2-thread-2} Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using pyspark RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
ERROR [2018-03-05 13:15:32,882] ({pool-2-thread-2} Job.java[run]:184) - Job failed
org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
... 13 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 20 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 21 more
ERROR [2018-03-05 13:15:32,884] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error
org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
... 13 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 20 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 21 more
 WARN [2018-03-05 13:15:32,885] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227) - Job 20180302-135047_99102973 is finished, status: ERROR, exception: org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused), result: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
 INFO [2018-03-05 13:15:32,910] ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
 INFO [2018-03-05 13:15:32,914] ({pool-2-thread-2} SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session

2018-03-02 15:07 GMT+09:00 小野圭二 <[hidden email]>:
Hi Jeff,

No one graved 63039 so far on Mac.
Yes, this issue has been reproduced as changing the port NO randomly.

Ono


2018-03-02 14:49 GMT+09:00 Jeff Zhang <[hidden email]>:

It is possible that 63039 is used by other processes, as Mac can not detect whether this port is used or not from my experience.

And can you reproduce this issue consistently ? Because each time zeppelin will chose a random port as long as you don't specify the portRange, so if you reproduce it consistently, it might due to other issues.




小野圭二 <[hidden email]>于2018年3月2日周五 下午1:32写道:
Hi there,

I am so confusing now, let me know why if You know this, please. 

The env:
   Mac:  High Sierra ver.10.13.3
   Zep: 0.7.3
   Spark: 2.2.0
   Java: 1.8
   notebook script:  %pyspark   print 'aaa'              (just it for test)

In this env,  i got the error in log as 

ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed

org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)



Indeed, before running notebook, RemoteInterpreterManagedProcess.java prepares a Port for callback to Thrift.
 

INFO ({pool-2-thread-1} RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for callback will start. Port: 63039



I guess, the reason why the error was Zeppelin could not attache to the Thrift PORT.


And my confusion is, when i run the same module on CentOS, it worked fine.
    CentOS:  7
    Zep: same
    Spark: same
    Java: same
     notebook script: same

And there was no the Thift port description in the log file. From this consequence, the fine working was cause of did not use Thrift port on CentOS.


Do You know actually there is different behavior between Mac and CentOS? And i am so appreciated if You know the workaround on Mac.


Best,
Ono





Reply | Threaded
Open this post in threaded view
|

Re: Thrift server for call back will start. Port:

Jeff Zhang
>>> Seems like it is not good chemistry between Spark and Zeppelin on Mac. From the reason that the consequence of removing the environment parameters, Zeppelin headed to own spark lib. 
 
Could you specify SPARK_HOME in your spark interpreter to use external spark instead of zeppelin's own spark lib ?




小野圭二 <[hidden email]>于2018年3月6日周二 上午10:06写道:
I wonder why, but it seems work fine, and would like to share how to fix it.

First of all, this issue happened just on MAC.
[THE ENV WAS]
  macOS High Sierra v 10.13.3
  cpu core i5
  mem  8GB
  Zeppelin 0.8.0 snapshot
  Spark 2.2.0
[ISSUE]
   see this previous mails
[SOLUTION]
   According to the above ref, i removed 'SPARK_HOME', 'SPARK_SUBMIT_OPTIONS' from my environment parameters and conf/zeppelin-env.sh(set it as same like CentOS). Then restart zeppelin, that's all.
[TO DO]
   Seems like it is not good chemistry between Spark and Zeppelin on Mac. From the reason that the consequence of removing the environment parameters, Zeppelin headed to own spark lib. 
   Maybe should write this in the document as caution. 


Cheers, 

Ono Keiji


 

2018-03-06 9:15 GMT+09:00 小野圭二 <[hidden email]>:
Nothing changed. I guess this property does not effect cause of failing to call the interpreter.
i challenged it as log4j.......zeppelin=DEBUG, but no way.

2018-03-05 18:24 GMT+09:00 Jeff Zhang <[hidden email]>:

Try to add the following to log4j.properties for more log

log4j.logger.org.apache.zeppelin.interpreter=DEBUG



小野圭二 <[hidden email]>于2018年3月5日周一 下午5:13写道:
Actually, there was not created the interpreter log file as said 'Cannot run interpreter' in the log. And it was run on local machine as MAC, i mean stand alone. i stopped every firewall to make it easy. And again,this happened just on Mac. On CentOS, it works fine.
That makes me confusion. :-b

2018-03-05 14:06 GMT+09:00 Jeff Zhang <[hidden email]>:

Could you check the interpreter log ? The issue of not be able to connecting to thrift port could happen due to network issue, like firewall or you use VPN, etc.



小野圭二 <[hidden email]>于2018年3月5日周一 下午12:32写道:
Hi,

I show the all log on this error.There are 2 type error. The 1 is at the first run. The 2 is at the second run. Seems like could not established to Thrift callback port 51348 that was set at the first run.
First of all, does it need to establish the callback port of Thrift server?   
The notebook script was just 
    %pyspark   
        print 'aaa' 
Any advice appreciate.

1.
INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05 13:11:13,934] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109) - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2} Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using pyspark RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2} ManagedInterpreterGroup.java[getOrCreateInterpreterProcess]:57) - Create InterperterProcess for InterpreterGroup: 2CV1983WX:shared_process WARN [2018-03-05 13:11:14,010] ({pool-2-thread-2} InterpreterSetting.java[setupPropertiesForSparkR]:796) - sparkr.zip is not found, SparkR may not work. INFO [2018-03-05 13:11:14,017] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:123) - Choose port 51347 for RemoteInterpreterProcess INFO [2018-03-05 13:11:14,023] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for callback will start. Port: 51348 INFO [2018-03-05 13:11:14,049] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:205) - Run interpreter process [/Users/onokeiji/zeppelin/zeppelin/bin/interpreter.sh, -d, /Users/onokeiji/zeppelin/zeppelin/interpreter/spark, -c, 172.16.20.10, -p, 51348, -l, /Users/onokeiji/zeppelin/zeppelin/local-repo/2CV1983WX, -g, spark] INFO [2018-03-05 13:11:14,238] ({Exec Default Executor} RemoteInterpreterManagedProcess.java[onProcessComplete]:258) - Interpreter process exited 0 ERROR [2018-03-05 13:12:14,058] ({pool-2-thread-2} Job.java[run]:184) - Job failed org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at org.apache.zeppelin.scheduler.Job.run(Job.java:182) at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ERROR [2018-03-05 13:12:14,075] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at org.apache.zeppelin.scheduler.Job.run(Job.java:182) at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) WARN [2018-03-05 13:12:14,075] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227) - Job 20180302-135047_99102973 is finished, status: ERROR, exception: org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter, result: Cannot run interpreter INFO [2018-03-05 13:12:14,103] ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05 13:12:14,109] ({pool-2-thread-2} SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session

2.
INFO [2018-03-05 13:15:32,736] ({qtp1651855867-13} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
 INFO [2018-03-05 13:15:32,751] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109) - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
 INFO [2018-03-05 13:15:32,753] ({pool-2-thread-2} Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using pyspark RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
ERROR [2018-03-05 13:15:32,882] ({pool-2-thread-2} Job.java[run]:184) - Job failed
org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
... 13 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 20 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 21 more
ERROR [2018-03-05 13:15:32,884] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error
org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
... 13 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 20 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 21 more
 WARN [2018-03-05 13:15:32,885] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227) - Job 20180302-135047_99102973 is finished, status: ERROR, exception: org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused), result: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
 INFO [2018-03-05 13:15:32,910] ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
 INFO [2018-03-05 13:15:32,914] ({pool-2-thread-2} SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session

2018-03-02 15:07 GMT+09:00 小野圭二 <[hidden email]>:
Hi Jeff,

No one graved 63039 so far on Mac.
Yes, this issue has been reproduced as changing the port NO randomly.

Ono


2018-03-02 14:49 GMT+09:00 Jeff Zhang <[hidden email]>:

It is possible that 63039 is used by other processes, as Mac can not detect whether this port is used or not from my experience.

And can you reproduce this issue consistently ? Because each time zeppelin will chose a random port as long as you don't specify the portRange, so if you reproduce it consistently, it might due to other issues.




小野圭二 <[hidden email]>于2018年3月2日周五 下午1:32写道:
Hi there,

I am so confusing now, let me know why if You know this, please. 

The env:
   Mac:  High Sierra ver.10.13.3
   Zep: 0.7.3
   Spark: 2.2.0
   Java: 1.8
   notebook script:  %pyspark   print 'aaa'              (just it for test)

In this env,  i got the error in log as 

ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed

org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)



Indeed, before running notebook, RemoteInterpreterManagedProcess.java prepares a Port for callback to Thrift.
 

INFO ({pool-2-thread-1} RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for callback will start. Port: 63039



I guess, the reason why the error was Zeppelin could not attache to the Thrift PORT.


And my confusion is, when i run the same module on CentOS, it worked fine.
    CentOS:  7
    Zep: same
    Spark: same
    Java: same
     notebook script: same

And there was no the Thift port description in the log file. From this consequence, the fine working was cause of did not use Thrift port on CentOS.


Do You know actually there is different behavior between Mac and CentOS? And i am so appreciated if You know the workaround on Mac.


Best,
Ono





Reply | Threaded
Open this post in threaded view
|

Re: Thrift server for call back will start. Port:

小野圭二
Hi Jeff,

I did set SPARK_HOME to use external spark at first. And that was fine work on CentOS, but not on Mac.
Do you mean try it again?

2018-03-06 11:12 GMT+09:00 Jeff Zhang <[hidden email]>:
>>> Seems like it is not good chemistry between Spark and Zeppelin on Mac. From the reason that the consequence of removing the environment parameters, Zeppelin headed to own spark lib. 
 
Could you specify SPARK_HOME in your spark interpreter to use external spark instead of zeppelin's own spark lib ?




小野圭二 <[hidden email]>于2018年3月6日周二 上午10:06写道:
I wonder why, but it seems work fine, and would like to share how to fix it.

First of all, this issue happened just on MAC.
[THE ENV WAS]
  macOS High Sierra v 10.13.3
  cpu core i5
  mem  8GB
  Zeppelin 0.8.0 snapshot
  Spark 2.2.0
[ISSUE]
   see this previous mails
[SOLUTION]
   According to the above ref, i removed 'SPARK_HOME', 'SPARK_SUBMIT_OPTIONS' from my environment parameters and conf/zeppelin-env.sh(set it as same like CentOS). Then restart zeppelin, that's all.
[TO DO]
   Seems like it is not good chemistry between Spark and Zeppelin on Mac. From the reason that the consequence of removing the environment parameters, Zeppelin headed to own spark lib. 
   Maybe should write this in the document as caution. 


Cheers, 

Ono Keiji


 

2018-03-06 9:15 GMT+09:00 小野圭二 <[hidden email]>:
Nothing changed. I guess this property does not effect cause of failing to call the interpreter.
i challenged it as log4j.......zeppelin=DEBUG, but no way.

2018-03-05 18:24 GMT+09:00 Jeff Zhang <[hidden email]>:

Try to add the following to log4j.properties for more log

log4j.logger.org.apache.zeppelin.interpreter=DEBUG



小野圭二 <[hidden email]>于2018年3月5日周一 下午5:13写道:
Actually, there was not created the interpreter log file as said 'Cannot run interpreter' in the log. And it was run on local machine as MAC, i mean stand alone. i stopped every firewall to make it easy. And again,this happened just on Mac. On CentOS, it works fine.
That makes me confusion. :-b

2018-03-05 14:06 GMT+09:00 Jeff Zhang <[hidden email]>:

Could you check the interpreter log ? The issue of not be able to connecting to thrift port could happen due to network issue, like firewall or you use VPN, etc.



小野圭二 <[hidden email]>于2018年3月5日周一 下午12:32写道:
Hi,

I show the all log on this error.There are 2 type error. The 1 is at the first run. The 2 is at the second run. Seems like could not established to Thrift callback port 51348 that was set at the first run.
First of all, does it need to establish the callback port of Thrift server?   
The notebook script was just 
    %pyspark   
        print 'aaa' 
Any advice appreciate.

1.
INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05 13:11:13,934] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109) - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2} Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using pyspark RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2} ManagedInterpreterGroup.java[getOrCreateInterpreterProcess]:57) - Create InterperterProcess for InterpreterGroup: 2CV1983WX:shared_process WARN [2018-03-05 13:11:14,010] ({pool-2-thread-2} InterpreterSetting.java[setupPropertiesForSparkR]:796) - sparkr.zip is not found, SparkR may not work. INFO [2018-03-05 13:11:14,017] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:123) - Choose port 51347 for RemoteInterpreterProcess INFO [2018-03-05 13:11:14,023] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for callback will start. Port: 51348 INFO [2018-03-05 13:11:14,049] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:205) - Run interpreter process [/Users/onokeiji/zeppelin/zeppelin/bin/interpreter.sh, -d, /Users/onokeiji/zeppelin/zeppelin/interpreter/spark, -c, 172.16.20.10, -p, 51348, -l, /Users/onokeiji/zeppelin/zeppelin/local-repo/2CV1983WX, -g, spark] INFO [2018-03-05 13:11:14,238] ({Exec Default Executor} RemoteInterpreterManagedProcess.java[onProcessComplete]:258) - Interpreter process exited 0 ERROR [2018-03-05 13:12:14,058] ({pool-2-thread-2} Job.java[run]:184) - Job failed org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at org.apache.zeppelin.scheduler.Job.run(Job.java:182) at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ERROR [2018-03-05 13:12:14,075] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at org.apache.zeppelin.scheduler.Job.run(Job.java:182) at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) WARN [2018-03-05 13:12:14,075] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227) - Job 20180302-135047_99102973 is finished, status: ERROR, exception: org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter, result: Cannot run interpreter INFO [2018-03-05 13:12:14,103] ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05 13:12:14,109] ({pool-2-thread-2} SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session

2.
INFO [2018-03-05 13:15:32,736] ({qtp1651855867-13} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
 INFO [2018-03-05 13:15:32,751] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109) - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
 INFO [2018-03-05 13:15:32,753] ({pool-2-thread-2} Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using pyspark RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
ERROR [2018-03-05 13:15:32,882] ({pool-2-thread-2} Job.java[run]:184) - Job failed
org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
... 13 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 20 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 21 more
ERROR [2018-03-05 13:15:32,884] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error
org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
... 13 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 20 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 21 more
 WARN [2018-03-05 13:15:32,885] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227) - Job 20180302-135047_99102973 is finished, status: ERROR, exception: org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused), result: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
 INFO [2018-03-05 13:15:32,910] ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
 INFO [2018-03-05 13:15:32,914] ({pool-2-thread-2} SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session

2018-03-02 15:07 GMT+09:00 小野圭二 <[hidden email]>:
Hi Jeff,

No one graved 63039 so far on Mac.
Yes, this issue has been reproduced as changing the port NO randomly.

Ono


2018-03-02 14:49 GMT+09:00 Jeff Zhang <[hidden email]>:

It is possible that 63039 is used by other processes, as Mac can not detect whether this port is used or not from my experience.

And can you reproduce this issue consistently ? Because each time zeppelin will chose a random port as long as you don't specify the portRange, so if you reproduce it consistently, it might due to other issues.




小野圭二 <[hidden email]>于2018年3月2日周五 下午1:32写道:
Hi there,

I am so confusing now, let me know why if You know this, please. 

The env:
   Mac:  High Sierra ver.10.13.3
   Zep: 0.7.3
   Spark: 2.2.0
   Java: 1.8
   notebook script:  %pyspark   print 'aaa'              (just it for test)

In this env,  i got the error in log as 

ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed

org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)



Indeed, before running notebook, RemoteInterpreterManagedProcess.java prepares a Port for callback to Thrift.
 

INFO ({pool-2-thread-1} RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for callback will start. Port: 63039



I guess, the reason why the error was Zeppelin could not attache to the Thrift PORT.


And my confusion is, when i run the same module on CentOS, it worked fine.
    CentOS:  7
    Zep: same
    Spark: same
    Java: same
     notebook script: same

And there was no the Thift port description in the log file. From this consequence, the fine working was cause of did not use Thrift port on CentOS.


Do You know actually there is different behavior between Mac and CentOS? And i am so appreciated if You know the workaround on Mac.


Best,
Ono






Reply | Threaded
Open this post in threaded view
|

Re: Thrift server for call back will start. Port:

Jeff Zhang

If you set SPARK_HOME in spark interpreter setting, then please remove SPARK_HOME in spark-env.sh, otherwise you do will hit weird issue.




小野圭二 <[hidden email]>于2018年3月6日周二 上午10:56写道:
Hi Jeff,

I did set SPARK_HOME to use external spark at first. And that was fine work on CentOS, but not on Mac.
Do you mean try it again?

2018-03-06 11:12 GMT+09:00 Jeff Zhang <[hidden email]>:
>>> Seems like it is not good chemistry between Spark and Zeppelin on Mac. From the reason that the consequence of removing the environment parameters, Zeppelin headed to own spark lib. 
 
Could you specify SPARK_HOME in your spark interpreter to use external spark instead of zeppelin's own spark lib ?




小野圭二 <[hidden email]>于2018年3月6日周二 上午10:06写道:
I wonder why, but it seems work fine, and would like to share how to fix it.

First of all, this issue happened just on MAC.
[THE ENV WAS]
  macOS High Sierra v 10.13.3
  cpu core i5
  mem  8GB
  Zeppelin 0.8.0 snapshot
  Spark 2.2.0
[ISSUE]
   see this previous mails
[SOLUTION]
   According to the above ref, i removed 'SPARK_HOME', 'SPARK_SUBMIT_OPTIONS' from my environment parameters and conf/zeppelin-env.sh(set it as same like CentOS). Then restart zeppelin, that's all.
[TO DO]
   Seems like it is not good chemistry between Spark and Zeppelin on Mac. From the reason that the consequence of removing the environment parameters, Zeppelin headed to own spark lib. 
   Maybe should write this in the document as caution. 


Cheers, 

Ono Keiji


 

2018-03-06 9:15 GMT+09:00 小野圭二 <[hidden email]>:
Nothing changed. I guess this property does not effect cause of failing to call the interpreter.
i challenged it as log4j.......zeppelin=DEBUG, but no way.

2018-03-05 18:24 GMT+09:00 Jeff Zhang <[hidden email]>:

Try to add the following to log4j.properties for more log

log4j.logger.org.apache.zeppelin.interpreter=DEBUG



小野圭二 <[hidden email]>于2018年3月5日周一 下午5:13写道:
Actually, there was not created the interpreter log file as said 'Cannot run interpreter' in the log. And it was run on local machine as MAC, i mean stand alone. i stopped every firewall to make it easy. And again,this happened just on Mac. On CentOS, it works fine.
That makes me confusion. :-b

2018-03-05 14:06 GMT+09:00 Jeff Zhang <[hidden email]>:

Could you check the interpreter log ? The issue of not be able to connecting to thrift port could happen due to network issue, like firewall or you use VPN, etc.



小野圭二 <[hidden email]>于2018年3月5日周一 下午12:32写道:
Hi,

I show the all log on this error.There are 2 type error. The 1 is at the first run. The 2 is at the second run. Seems like could not established to Thrift callback port 51348 that was set at the first run.
First of all, does it need to establish the callback port of Thrift server?   
The notebook script was just 
    %pyspark   
        print 'aaa' 
Any advice appreciate.

1.
INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05 13:11:13,934] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109) - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2} Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using pyspark RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2} ManagedInterpreterGroup.java[getOrCreateInterpreterProcess]:57) - Create InterperterProcess for InterpreterGroup: 2CV1983WX:shared_process WARN [2018-03-05 13:11:14,010] ({pool-2-thread-2} InterpreterSetting.java[setupPropertiesForSparkR]:796) - sparkr.zip is not found, SparkR may not work. INFO [2018-03-05 13:11:14,017] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:123) - Choose port 51347 for RemoteInterpreterProcess INFO [2018-03-05 13:11:14,023] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for callback will start. Port: 51348 INFO [2018-03-05 13:11:14,049] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:205) - Run interpreter process [/Users/onokeiji/zeppelin/zeppelin/bin/interpreter.sh, -d, /Users/onokeiji/zeppelin/zeppelin/interpreter/spark, -c, 172.16.20.10, -p, 51348, -l, /Users/onokeiji/zeppelin/zeppelin/local-repo/2CV1983WX, -g, spark] INFO [2018-03-05 13:11:14,238] ({Exec Default Executor} RemoteInterpreterManagedProcess.java[onProcessComplete]:258) - Interpreter process exited 0 ERROR [2018-03-05 13:12:14,058] ({pool-2-thread-2} Job.java[run]:184) - Job failed org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at org.apache.zeppelin.scheduler.Job.run(Job.java:182) at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ERROR [2018-03-05 13:12:14,075] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at org.apache.zeppelin.scheduler.Job.run(Job.java:182) at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) WARN [2018-03-05 13:12:14,075] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227) - Job 20180302-135047_99102973 is finished, status: ERROR, exception: org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter, result: Cannot run interpreter INFO [2018-03-05 13:12:14,103] ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05 13:12:14,109] ({pool-2-thread-2} SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session

2.
INFO [2018-03-05 13:15:32,736] ({qtp1651855867-13} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
 INFO [2018-03-05 13:15:32,751] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109) - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
 INFO [2018-03-05 13:15:32,753] ({pool-2-thread-2} Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using pyspark RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
ERROR [2018-03-05 13:15:32,882] ({pool-2-thread-2} Job.java[run]:184) - Job failed
org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
... 13 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 20 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 21 more
ERROR [2018-03-05 13:15:32,884] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error
org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
... 13 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 20 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 21 more
 WARN [2018-03-05 13:15:32,885] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227) - Job 20180302-135047_99102973 is finished, status: ERROR, exception: org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused), result: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
 INFO [2018-03-05 13:15:32,910] ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
 INFO [2018-03-05 13:15:32,914] ({pool-2-thread-2} SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session

2018-03-02 15:07 GMT+09:00 小野圭二 <[hidden email]>:
Hi Jeff,

No one graved 63039 so far on Mac.
Yes, this issue has been reproduced as changing the port NO randomly.

Ono


2018-03-02 14:49 GMT+09:00 Jeff Zhang <[hidden email]>:

It is possible that 63039 is used by other processes, as Mac can not detect whether this port is used or not from my experience.

And can you reproduce this issue consistently ? Because each time zeppelin will chose a random port as long as you don't specify the portRange, so if you reproduce it consistently, it might due to other issues.




小野圭二 <[hidden email]>于2018年3月2日周五 下午1:32写道:
Hi there,

I am so confusing now, let me know why if You know this, please. 

The env:
   Mac:  High Sierra ver.10.13.3
   Zep: 0.7.3
   Spark: 2.2.0
   Java: 1.8
   notebook script:  %pyspark   print 'aaa'              (just it for test)

In this env,  i got the error in log as 

ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed

org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)



Indeed, before running notebook, RemoteInterpreterManagedProcess.java prepares a Port for callback to Thrift.
 

INFO ({pool-2-thread-1} RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for callback will start. Port: 63039



I guess, the reason why the error was Zeppelin could not attache to the Thrift PORT.


And my confusion is, when i run the same module on CentOS, it worked fine.
    CentOS:  7
    Zep: same
    Spark: same
    Java: same
     notebook script: same

And there was no the Thift port description in the log file. From this consequence, the fine working was cause of did not use Thrift port on CentOS.


Do You know actually there is different behavior between Mac and CentOS? And i am so appreciated if You know the workaround on Mac.


Best,
Ono






Reply | Threaded
Open this post in threaded view
|

Re: Thrift server for call back will start. Port:

小野圭二
Combinations: SPARK_HOME set in 'O'->point to external spark , no '-' ->head to internal spark lib

spark interpreter setting----
zeppelin -env.sh----
environment parameters---
Work Fine:〇 NG:××××

The consequences are zeppelin-env.sh has a top priority. And if you want to use external Spark on Mac, you should not set SPARK_HOME in zeppelin-env.sh. Set it as environment parameters or in spark interpreter.

Cheers, 

Ono Keiji



2018-03-06 12:23 GMT+09:00 Jeff Zhang <[hidden email]>:

If you set SPARK_HOME in spark interpreter setting, then please remove SPARK_HOME in spark-env.sh, otherwise you do will hit weird issue.




小野圭二 <[hidden email]>于2018年3月6日周二 上午10:56写道:
Hi Jeff,

I did set SPARK_HOME to use external spark at first. And that was fine work on CentOS, but not on Mac.
Do you mean try it again?

2018-03-06 11:12 GMT+09:00 Jeff Zhang <[hidden email]>:
>>> Seems like it is not good chemistry between Spark and Zeppelin on Mac. From the reason that the consequence of removing the environment parameters, Zeppelin headed to own spark lib. 
 
Could you specify SPARK_HOME in your spark interpreter to use external spark instead of zeppelin's own spark lib ?




小野圭二 <[hidden email]>于2018年3月6日周二 上午10:06写道:
I wonder why, but it seems work fine, and would like to share how to fix it.

First of all, this issue happened just on MAC.
[THE ENV WAS]
  macOS High Sierra v 10.13.3
  cpu core i5
  mem  8GB
  Zeppelin 0.8.0 snapshot
  Spark 2.2.0
[ISSUE]
   see this previous mails
[SOLUTION]
   According to the above ref, i removed 'SPARK_HOME', 'SPARK_SUBMIT_OPTIONS' from my environment parameters and conf/zeppelin-env.sh(set it as same like CentOS). Then restart zeppelin, that's all.
[TO DO]
   Seems like it is not good chemistry between Spark and Zeppelin on Mac. From the reason that the consequence of removing the environment parameters, Zeppelin headed to own spark lib. 
   Maybe should write this in the document as caution. 


Cheers, 

Ono Keiji


 

2018-03-06 9:15 GMT+09:00 小野圭二 <[hidden email]>:
Nothing changed. I guess this property does not effect cause of failing to call the interpreter.
i challenged it as log4j.......zeppelin=DEBUG, but no way.

2018-03-05 18:24 GMT+09:00 Jeff Zhang <[hidden email]>:

Try to add the following to log4j.properties for more log

log4j.logger.org.apache.zeppelin.interpreter=DEBUG



小野圭二 <[hidden email]>于2018年3月5日周一 下午5:13写道:
Actually, there was not created the interpreter log file as said 'Cannot run interpreter' in the log. And it was run on local machine as MAC, i mean stand alone. i stopped every firewall to make it easy. And again,this happened just on Mac. On CentOS, it works fine.
That makes me confusion. :-b

2018-03-05 14:06 GMT+09:00 Jeff Zhang <[hidden email]>:

Could you check the interpreter log ? The issue of not be able to connecting to thrift port could happen due to network issue, like firewall or you use VPN, etc.



小野圭二 <[hidden email]>于2018年3月5日周一 下午12:32写道:
Hi,

I show the all log on this error.There are 2 type error. The 1 is at the first run. The 2 is at the second run. Seems like could not established to Thrift callback port 51348 that was set at the first run.
First of all, does it need to establish the callback port of Thrift server?   
The notebook script was just 
    %pyspark   
        print 'aaa' 
Any advice appreciate.

1.
INFO [2018-03-05 13:11:13,914] ({qtp1651855867-13} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05 13:11:13,934] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109) - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2} Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using pyspark RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session INFO [2018-03-05 13:11:13,936] ({pool-2-thread-2} ManagedInterpreterGroup.java[getOrCreateInterpreterProcess]:57) - Create InterperterProcess for InterpreterGroup: 2CV1983WX:shared_process WARN [2018-03-05 13:11:14,010] ({pool-2-thread-2} InterpreterSetting.java[setupPropertiesForSparkR]:796) - sparkr.zip is not found, SparkR may not work. INFO [2018-03-05 13:11:14,017] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:123) - Choose port 51347 for RemoteInterpreterProcess INFO [2018-03-05 13:11:14,023] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for callback will start. Port: 51348 INFO [2018-03-05 13:11:14,049] ({pool-2-thread-2} RemoteInterpreterManagedProcess.java[start]:205) - Run interpreter process [/Users/onokeiji/zeppelin/zeppelin/bin/interpreter.sh, -d, /Users/onokeiji/zeppelin/zeppelin/interpreter/spark, -c, 172.16.20.10, -p, 51348, -l, /Users/onokeiji/zeppelin/zeppelin/local-repo/2CV1983WX, -g, spark] INFO [2018-03-05 13:11:14,238] ({Exec Default Executor} RemoteInterpreterManagedProcess.java[onProcessComplete]:258) - Interpreter process exited 0 ERROR [2018-03-05 13:12:14,058] ({pool-2-thread-2} Job.java[run]:184) - Job failed org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at org.apache.zeppelin.scheduler.Job.run(Job.java:182) at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ERROR [2018-03-05 13:12:14,075] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter at org.apache.zeppelin.interpreter.remote.RemoteInterpreterManagedProcess.start(RemoteInterpreterManagedProcess.java:220) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getOrCreateInterpreterProcess(RemoteInterpreter.java:101) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:153) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126) at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265) at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423) at org.apache.zeppelin.scheduler.Job.run(Job.java:182) at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) WARN [2018-03-05 13:12:14,075] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227) - Job 20180302-135047_99102973 is finished, status: ERROR, exception: org.apache.zeppelin.interpreter.InterpreterException: Cannot run interpreter, result: Cannot run interpreter INFO [2018-03-05 13:12:14,103] ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U INFO [2018-03-05 13:12:14,109] ({pool-2-thread-2} SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session

2.
INFO [2018-03-05 13:15:32,736] ({qtp1651855867-13} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
 INFO [2018-03-05 13:15:32,751] ({pool-2-thread-2} SchedulerFactory.java[jobStarted]:109) - Job 20180302-135047_99102973 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session
 INFO [2018-03-05 13:15:32,753] ({pool-2-thread-2} Paragraph.java[jobRun]:397) - run paragraph 20180302-135047_99102973 using pyspark RemoteInterpreter_org.apache.zeppelin.spark.PySparkInterpreter_shared_session
ERROR [2018-03-05 13:15:32,882] ({pool-2-thread-2} Job.java[run]:184) - Job failed
org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
... 13 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 20 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 21 more
ERROR [2018-03-05 13:15:32,884] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2219) - Error
org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:152)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.internal_create(RemoteInterpreter.java:154)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.open(RemoteInterpreter.java:126)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreter.getFormType(RemoteInterpreter.java:265)
at org.apache.zeppelin.notebook.Paragraph.jobRun(Paragraph.java:423)
at org.apache.zeppelin.scheduler.Job.run(Job.java:182)
at org.apache.zeppelin.scheduler.RemoteScheduler$JobRunner.run(RemoteScheduler.java:307)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:53)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:37)
at org.apache.commons.pool2.BasePooledObjectFactory.makeObject(BasePooledObjectFactory.java:60)
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:861)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:435)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.getClient(RemoteInterpreterProcess.java:73)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess.callRemoteFunction(RemoteInterpreterProcess.java:144)
... 13 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
at org.apache.thrift.transport.TSocket.open(TSocket.java:187)
at org.apache.zeppelin.interpreter.remote.ClientFactory.create(ClientFactory.java:51)
... 20 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:182)
... 21 more
 WARN [2018-03-05 13:15:32,885] ({pool-2-thread-2} NotebookServer.java[afterStatusChange]:2227) - Job 20180302-135047_99102973 is finished, status: ERROR, exception: org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused), result: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
 INFO [2018-03-05 13:15:32,910] ({pool-2-thread-2} VFSNotebookRepo.java[save]:221) - Saving note:2D6S6TB3U
 INFO [2018-03-05 13:15:32,914] ({pool-2-thread-2} SchedulerFactory.java[jobFinished]:115) - Job 20180302-135047_99102973 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpreter-shared_session

2018-03-02 15:07 GMT+09:00 小野圭二 <[hidden email]>:
Hi Jeff,

No one graved 63039 so far on Mac.
Yes, this issue has been reproduced as changing the port NO randomly.

Ono


2018-03-02 14:49 GMT+09:00 Jeff Zhang <[hidden email]>:

It is possible that 63039 is used by other processes, as Mac can not detect whether this port is used or not from my experience.

And can you reproduce this issue consistently ? Because each time zeppelin will chose a random port as long as you don't specify the portRange, so if you reproduce it consistently, it might due to other issues.




小野圭二 <[hidden email]>于2018年3月2日周五 下午1:32写道:
Hi there,

I am so confusing now, let me know why if You know this, please. 

The env:
   Mac:  High Sierra ver.10.13.3
   Zep: 0.7.3
   Spark: 2.2.0
   Java: 1.8
   notebook script:  %pyspark   print 'aaa'              (just it for test)

In this env,  i got the error in log as 

ERROR ({pool-2-thread-2} Job.java[run]:184) - Job failed

org.apache.zeppelin.interpreter.InterpreterException: org.apache.zeppelin.interpreter.InterpreterException: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)



Indeed, before running notebook, RemoteInterpreterManagedProcess.java prepares a Port for callback to Thrift.
 

INFO ({pool-2-thread-1} RemoteInterpreterManagedProcess.java[start]:130) - Thrift server for callback will start. Port: 63039



I guess, the reason why the error was Zeppelin could not attache to the Thrift PORT.


And my confusion is, when i run the same module on CentOS, it worked fine.
    CentOS:  7
    Zep: same
    Spark: same
    Java: same
     notebook script: same

And there was no the Thift port description in the log file. From this consequence, the fine working was cause of did not use Thrift port on CentOS.


Do You know actually there is different behavior between Mac and CentOS? And i am so appreciated if You know the workaround on Mac.


Best,
Ono