Interpreter zombie processes

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Interpreter zombie processes

Ruslan Dautkhanov
How to make Interpreter processes terminate if Zeppelin server exits?

When we restart Zeppelin server (main process), in many cases, Interpreter 
process keeps running, essential becoming a zombie processes.

In case of Spark interpreter, it also holds SparkContext - consuming server-side
resources too.

How do we configure Zeppelin to kill all interpreters before its main process terminates?
Or is this a bug?

We're running ~2 weeks old snapshot of 0.7.0 Zeppelin.


Thank you,
Ruslan Dautkhanov
Reply | Threaded
Open this post in threaded view
|

Re: Interpreter zombie processes

Ruslan Dautkhanov
Moon,

> ZeppelinServer try to terminate interpreter process when shutting down [1].

Unfortunatally, this does not happen all the time.
I have seen zombie spark interpreter processes many times.
As a double conirmation - I see spark yarn application was still running.

> Also bin/zeppelin-deamon.sh script clean up if some processes are left [2].

We were not using that script. We were just running like zeppelin.sh --config ..
Will try to switch to zeppelin-deamon.sh and see if it makes a difference.

Thanks!




--
Ruslan Dautkhanov

On Thu, Dec 15, 2016 at 12:46 PM, moon <[hidden email]> wrote:
Hi,

ZeppelinServer try to terminate interpreter process when shutting down [1].
Also bin/zeppelin-deamon.sh script clean up if some processes are left [2].

If some processes are remained after shutting down Zeppelin, that's not an
expected result.

Thanks,
moon

[1]
https://github.com/apache/zeppelin/blob/master/zeppelin-server/src/main/java/org/apache/zeppelin/server/ZeppelinServer.java#L159
[2]
https://github.com/apache/zeppelin/blob/master/bin/zeppelin-daemon.sh#L218

On Thu, Dec 15, 2016 at 11:25 AM Ruslan Dautkhanov <[hidden email]>
wrote:

> How to make Interpreter processes terminate if Zeppelin server exits?
>
> When we restart Zeppelin server (main process), in many cases, Interpreter
> process keeps running, essential becoming a zombie processes.
>
> In case of Spark interpreter, it also holds SparkContext - consuming
> server-side
> resources too.
>
> How do we configure Zeppelin to kill all interpreters before its main
> process terminates?
> Or is this a bug?
>
> We're running ~2 weeks old snapshot of 0.7.0 Zeppelin.
>
>
> Thank you,
> Ruslan Dautkhanov
>

Reply | Threaded
Open this post in threaded view
|

Re: Interpreter zombie processes

Luciano Resende
I have also seen similar issues even using zeppelin-ddeamon but didn't have much time to investigate the issue when it was happening to me.

On Thu, Dec 15, 2016 at 12:15 PM Ruslan Dautkhanov <[hidden email]> wrote:
Moon,

> ZeppelinServer try to terminate interpreter process when shutting down [1].

Unfortunatally, this does not happen all the time.
I have seen zombie spark interpreter processes many times.
As a double conirmation - I see spark yarn application was still running.

> Also bin/zeppelin-deamon.sh script clean up if some processes are left [2].

We were not using that script. We were just running like zeppelin.sh --config ..
Will try to switch to zeppelin-deamon.sh and see if it makes a difference.

Thanks!




--
Ruslan Dautkhanov



On Thu, Dec 15, 2016 at 12:46 PM, moon <[hidden email]> wrote:
Hi,





ZeppelinServer try to terminate interpreter process when shutting down [1].


Also bin/zeppelin-deamon.sh script clean up if some processes are left [2].





If some processes are remained after shutting down Zeppelin, that's not an


expected result.





Thanks,


moon





[1]


https://github.com/apache/zeppelin/blob/master/zeppelin-server/src/main/java/org/apache/zeppelin/server/ZeppelinServer.java#L159


[2]


https://github.com/apache/zeppelin/blob/master/bin/zeppelin-daemon.sh#L218





On Thu, Dec 15, 2016 at 11:25 AM Ruslan Dautkhanov <[hidden email]>


wrote:





> How to make Interpreter processes terminate if Zeppelin server exits?


>


> When we restart Zeppelin server (main process), in many cases, Interpreter


> process keeps running, essential becoming a zombie processes.


>


> In case of Spark interpreter, it also holds SparkContext - consuming


> server-side


> resources too.


>


> How do we configure Zeppelin to kill all interpreters before its main


> process terminates?


> Or is this a bug?


>


> We're running ~2 weeks old snapshot of 0.7.0 Zeppelin.


>


>


> Thank you,


> Ruslan Dautkhanov


>



--
Sent from my Mobile device
Reply | Threaded
Open this post in threaded view
|

Re: Interpreter zombie processes

Jeff Zhang

I believe I see this before too.


Luciano Resende <[hidden email]>于2016年12月16日周五 上午10:17写道:
I have also seen similar issues even using zeppelin-ddeamon but didn't have much time to investigate the issue when it was happening to me.

On Thu, Dec 15, 2016 at 12:15 PM Ruslan Dautkhanov <[hidden email]> wrote:
Moon,

> ZeppelinServer try to terminate interpreter process when shutting down [1].

Unfortunatally, this does not happen all the time.
I have seen zombie spark interpreter processes many times.
As a double conirmation - I see spark yarn application was still running.

> Also bin/zeppelin-deamon.sh script clean up if some processes are left [2].

We were not using that script. We were just running like zeppelin.sh --config ..
Will try to switch to zeppelin-deamon.sh and see if it makes a difference.

Thanks!




--
Ruslan Dautkhanov



On Thu, Dec 15, 2016 at 12:46 PM, moon <[hidden email]> wrote:
Hi,





ZeppelinServer try to terminate interpreter process when shutting down [1].


Also bin/zeppelin-deamon.sh script clean up if some processes are left [2].





If some processes are remained after shutting down Zeppelin, that's not an


expected result.





Thanks,


moon





[1]


https://github.com/apache/zeppelin/blob/master/zeppelin-server/src/main/java/org/apache/zeppelin/server/ZeppelinServer.java#L159


[2]


https://github.com/apache/zeppelin/blob/master/bin/zeppelin-daemon.sh#L218





On Thu, Dec 15, 2016 at 11:25 AM Ruslan Dautkhanov <[hidden email]>


wrote:





> How to make Interpreter processes terminate if Zeppelin server exits?


>


> When we restart Zeppelin server (main process), in many cases, Interpreter


> process keeps running, essential becoming a zombie processes.


>


> In case of Spark interpreter, it also holds SparkContext - consuming


> server-side


> resources too.


>


> How do we configure Zeppelin to kill all interpreters before its main


> process terminates?


> Or is this a bug?


>


> We're running ~2 weeks old snapshot of 0.7.0 Zeppelin.


>


>


> Thank you,


> Ruslan Dautkhanov


>



--
Sent from my Mobile device
Reply | Threaded
Open this post in threaded view
|

Re: Interpreter zombie processes

Alexander Bezzubov-2
Have similar experience, although hard to say what's the reason as all process supposed to killed, as Moon pointed out.
Also noticed that with `mvn tests`, after almost every run, there are 1-2 zombie RemoteInterpreter processes hanging around.

--
Alex

On Fri, Dec 16, 2016 at 11:24 AM, Jeff Zhang <[hidden email]> wrote:

I believe I see this before too.


Luciano Resende <[hidden email]>于2016年12月16日周五 上午10:17写道:
I have also seen similar issues even using zeppelin-ddeamon but didn't have much time to investigate the issue when it was happening to me.

On Thu, Dec 15, 2016 at 12:15 PM Ruslan Dautkhanov <[hidden email]> wrote:
Moon,

> ZeppelinServer try to terminate interpreter process when shutting down [1].

Unfortunatally, this does not happen all the time.
I have seen zombie spark interpreter processes many times.
As a double conirmation - I see spark yarn application was still running.

> Also bin/zeppelin-deamon.sh script clean up if some processes are left [2].

We were not using that script. We were just running like zeppelin.sh --config ..
Will try to switch to zeppelin-deamon.sh and see if it makes a difference.

Thanks!




--
Ruslan Dautkhanov



On Thu, Dec 15, 2016 at 12:46 PM, moon <[hidden email]> wrote:
Hi,





ZeppelinServer try to terminate interpreter process when shutting down [1].


Also bin/zeppelin-deamon.sh script clean up if some processes are left [2].





If some processes are remained after shutting down Zeppelin, that's not an


expected result.





Thanks,


moon





[1]


https://github.com/apache/zeppelin/blob/master/zeppelin-server/src/main/java/org/apache/zeppelin/server/ZeppelinServer.java#L159


[2]


https://github.com/apache/zeppelin/blob/master/bin/zeppelin-daemon.sh#L218





On Thu, Dec 15, 2016 at 11:25 AM Ruslan Dautkhanov <[hidden email]>


wrote:





> How to make Interpreter processes terminate if Zeppelin server exits?


>


> When we restart Zeppelin server (main process), in many cases, Interpreter


> process keeps running, essential becoming a zombie processes.


>


> In case of Spark interpreter, it also holds SparkContext - consuming


> server-side


> resources too.


>


> How do we configure Zeppelin to kill all interpreters before its main


> process terminates?


> Or is this a bug?


>


> We're running ~2 weeks old snapshot of 0.7.0 Zeppelin.


>


>


> Thank you,


> Ruslan Dautkhanov


>



--
Sent from my Mobile device

Reply | Threaded
Open this post in threaded view
|

Re: Interpreter zombie processes

blaubaer
In reply to this post by Ruslan Dautkhanov
We are seeing this problem as well, regularly actually. Especially in situations when we have many concurrent interpreters running.
Reply | Threaded
Open this post in threaded view
|

Re: Interpreter zombie processes

Ruslan Dautkhanov
Thank you everyone for confirming this issue.


Thanks again.



--
Ruslan Dautkhanov

On Fri, Dec 16, 2016 at 2:48 AM, blaubaer <[hidden email]> wrote:
We are seeing this problem as well, regularly actually. Especially in
situations when we have many concurrent interpreters running.



--
View this message in context: http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/Interpreter-zombie-processes-tp4738p4746.html
Sent from the Apache Zeppelin Users (incubating) mailing list mailing list archive at Nabble.com.