Zeppelin Interpreter end

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Zeppelin Interpreter end

Litt, Shaun

Hi,  I am new to zeppelin and just got it configured to run in my YARN cluster, but I was wondering if there is a configuration or even a hard setting that shuts down interpreters after in-activity.  It seems like the interpreter (and it’s yarn consumption) hang out indefinitely, ideally there would be a clean way (like logout or a shutdown button within the notebook) to shutdown these interpreters, but additionally there should be a way for an admin of zeppelin to impose and idle timeout.  As a note to the scope of this, is dynamic yarn resource allocation configured (such that once a paragraph finishes it can release vcores)?

 

Thanks,

Shaun

 

This email and any files included with it may contain privileged,
proprietary and/or confidential information that is for the sole use
of the intended recipient(s).  Any disclosure, copying, distribution,
posting, or use of the information contained in or attached to this
email is prohibited unless permitted by the sender.  If you have
received this email in error, please immediately notify the sender
via return email, telephone, or fax and destroy this original transmission
and its included files without reading or saving it in any manner.
Thank you.

Reply | Threaded
Open this post in threaded view
|

Re: Zeppelin Interpreter end

Alexander Bezzubov
Hi,

right now, I'm not aware of such configuration in Zeppelin (please,
feel free to open the issue\submit a patch).

AFAIK dynamic YARN resource allocation is up to the user and is not
configured by default right now, which looks like one possible
solution to the problem you describe (at least cpu-wise)

As a workaround for your usecase, you can manually re-start Spark
interpreter (Interpreter menu -> restart), which, because of
lazy-loading, will not occupy any resources until somebody actually
runs it.

Hope this helps!


On Tue, Jun 30, 2015 at 8:23 AM, Litt, Shaun <[hidden email]> wrote:

> Hi,  I am new to zeppelin and just got it configured to run in my YARN
> cluster, but I was wondering if there is a configuration or even a hard
> setting that shuts down interpreters after in-activity.  It seems like the
> interpreter (and it’s yarn consumption) hang out indefinitely, ideally there
> would be a clean way (like logout or a shutdown button within the notebook)
> to shutdown these interpreters, but additionally there should be a way for
> an admin of zeppelin to impose and idle timeout.  As a note to the scope of
> this, is dynamic yarn resource allocation configured (such that once a
> paragraph finishes it can release vcores)?
>
>
>
> Thanks,
>
> Shaun
>
>
>
> This email and any files included with it may contain privileged,
> proprietary and/or confidential information that is for the sole use
> of the intended recipient(s).  Any disclosure, copying, distribution,
> posting, or use of the information contained in or attached to this
> email is prohibited unless permitted by the sender.  If you have
> received this email in error, please immediately notify the sender
> via return email, telephone, or fax and destroy this original transmission
> and its included files without reading or saving it in any manner.
> Thank you.



--
--
Kind regards,
Alexander.
Reply | Threaded
Open this post in threaded view
|

Re: Zeppelin Interpreter end

IT CTO
If I am not mistaking Apache Slider is aim to handle dynamic growing and shrinking of applications on YARN but it is still incubating (as zeppelin) might be interesting writing a slider package for zeppeling.
Eran

On Tue, Jun 30, 2015 at 9:02 AM Alexander Bezzubov <[hidden email]> wrote:
Hi,

right now, I'm not aware of such configuration in Zeppelin (please,
feel free to open the issue\submit a patch).

AFAIK dynamic YARN resource allocation is up to the user and is not
configured by default right now, which looks like one possible
solution to the problem you describe (at least cpu-wise)

As a workaround for your usecase, you can manually re-start Spark
interpreter (Interpreter menu -> restart), which, because of
lazy-loading, will not occupy any resources until somebody actually
runs it.

Hope this helps!


On Tue, Jun 30, 2015 at 8:23 AM, Litt, Shaun <[hidden email]> wrote:
> Hi,  I am new to zeppelin and just got it configured to run in my YARN
> cluster, but I was wondering if there is a configuration or even a hard
> setting that shuts down interpreters after in-activity.  It seems like the
> interpreter (and it’s yarn consumption) hang out indefinitely, ideally there
> would be a clean way (like logout or a shutdown button within the notebook)
> to shutdown these interpreters, but additionally there should be a way for
> an admin of zeppelin to impose and idle timeout.  As a note to the scope of
> this, is dynamic yarn resource allocation configured (such that once a
> paragraph finishes it can release vcores)?
>
>
>
> Thanks,
>
> Shaun
>
>
>
> This email and any files included with it may contain privileged,
> proprietary and/or confidential information that is for the sole use
> of the intended recipient(s).  Any disclosure, copying, distribution,
> posting, or use of the information contained in or attached to this
> email is prohibited unless permitted by the sender.  If you have
> received this email in error, please immediately notify the sender
> via return email, telephone, or fax and destroy this original transmission
> and its included files without reading or saving it in any manner.
> Thank you.



--
--
Kind regards,
Alexander.
Reply | Threaded
Open this post in threaded view
|

Re: Zeppelin Interpreter end

IT CTO
Actually reading some more about slider, I am not sure it s what I thought it is... to fast on sending this mail.
Eran

On Tue, Jun 30, 2015 at 9:46 AM IT CTO <[hidden email]> wrote:
If I am not mistaking Apache Slider is aim to handle dynamic growing and shrinking of applications on YARN but it is still incubating (as zeppelin) might be interesting writing a slider package for zeppeling.
Eran

On Tue, Jun 30, 2015 at 9:02 AM Alexander Bezzubov <[hidden email]> wrote:
Hi,

right now, I'm not aware of such configuration in Zeppelin (please,
feel free to open the issue\submit a patch).

AFAIK dynamic YARN resource allocation is up to the user and is not
configured by default right now, which looks like one possible
solution to the problem you describe (at least cpu-wise)

As a workaround for your usecase, you can manually re-start Spark
interpreter (Interpreter menu -> restart), which, because of
lazy-loading, will not occupy any resources until somebody actually
runs it.

Hope this helps!


On Tue, Jun 30, 2015 at 8:23 AM, Litt, Shaun <[hidden email]> wrote:
> Hi,  I am new to zeppelin and just got it configured to run in my YARN
> cluster, but I was wondering if there is a configuration or even a hard
> setting that shuts down interpreters after in-activity.  It seems like the
> interpreter (and it’s yarn consumption) hang out indefinitely, ideally there
> would be a clean way (like logout or a shutdown button within the notebook)
> to shutdown these interpreters, but additionally there should be a way for
> an admin of zeppelin to impose and idle timeout.  As a note to the scope of
> this, is dynamic yarn resource allocation configured (such that once a
> paragraph finishes it can release vcores)?
>
>
>
> Thanks,
>
> Shaun
>
>
>
> This email and any files included with it may contain privileged,
> proprietary and/or confidential information that is for the sole use
> of the intended recipient(s).  Any disclosure, copying, distribution,
> posting, or use of the information contained in or attached to this
> email is prohibited unless permitted by the sender.  If you have
> received this email in error, please immediately notify the sender
> via return email, telephone, or fax and destroy this original transmission
> and its included files without reading or saving it in any manner.
> Thank you.



--
--
Kind regards,
Alexander.
Reply | Threaded
Open this post in threaded view
|

Re: Zeppelin Interpreter end

Rajat Gupta
We have implemented auto shutdown of spark application by adding an idle timeout setting directly in spark. We set the value of this setting in the zeppelin interpreter UI page and default it to 60 mins. For this fix we also added an auto lazy restart of interpreter in zeppelin if interpreter has shutdown because of inactivity. If you want I can send you patches for this both for spark and zeppelin. I did not submit the patch in zeppelin for this earlier because this makes sense only if spark has idle timeout in open source. 

Dynamic scaling of spark applications works fine on zeppelin with appropriate spark settings on interpreter page. Zeppelin just maintains handle to a spark context and that spark context is free to do anything afaik. We at Qubole are using it actively.

Rajat

On Tue, Jun 30, 2015 at 4:53 AM, Litt, Shaun <[hidden email]> wrote:

Hi,  I am new to zeppelin and just got it configured to run in my YARN cluster, but I was wondering if there is a configuration or even a hard setting that shuts down interpreters after in-activity.  It seems like the interpreter (and it’s yarn consumption) hang out indefinitely, ideally there would be a clean way (like logout or a shutdown button within the notebook) to shutdown these interpreters, but additionally there should be a way for an admin of zeppelin to impose and idle timeout.  As a note to the scope of this, is dynamic yarn resource allocation configured (such that once a paragraph finishes it can release vcores)?

 

Thanks,

Shaun

 

This email and any files included with it may contain privileged,
proprietary and/or confidential information that is for the sole use
of the intended recipient(s).  Any disclosure, copying, distribution,
posting, or use of the information contained in or attached to this
email is prohibited unless permitted by the sender.  If you have
received this email in error, please immediately notify the sender
via return email, telephone, or fax and destroy this original transmission
and its included files without reading or saving it in any manner.
Thank you.


Reply | Threaded
Open this post in threaded view
|

RE: Zeppelin Interpreter end

Litt, Shaun

Interesting and useful.  I would love to see your changes for reference.   Have you submitted this feature back to the SPARK community?  I think this would be useful for a lot of interactive spark use-cases in shared clusters.

 

Also I have a question regarding your experience maintaining your own SPARK fork.  We have also been forced to maintain a very minor spark fork to work around some bugs in our environment, but our internal goal is to try to get back to the main apache branch or a vendor fork.  Have you found keeping up with SPARK (which is changing lightning fast – pun intended) challenging?

 

Thanks,

Shaun Litt

 

 

From: Rajat Gupta [mailto:[hidden email]]
Sent: Wednesday, July 01, 2015 2:41 AM
To: [hidden email]
Subject: Re: Zeppelin Interpreter end

 

We have implemented auto shutdown of spark application by adding an idle timeout setting directly in spark. We set the value of this setting in the zeppelin interpreter UI page and default it to 60 mins. For this fix we also added an auto lazy restart of interpreter in zeppelin if interpreter has shutdown because of inactivity. If you want I can send you patches for this both for spark and zeppelin. I did not submit the patch in zeppelin for this earlier because this makes sense only if spark has idle timeout in open source. 

Dynamic scaling of spark applications works fine on zeppelin with appropriate spark settings on interpreter page. Zeppelin just maintains handle to a spark context and that spark context is free to do anything afaik. We at Qubole are using it actively.

Rajat

 

On Tue, Jun 30, 2015 at 4:53 AM, Litt, Shaun <[hidden email]> wrote:

Hi,  I am new to zeppelin and just got it configured to run in my YARN cluster, but I was wondering if there is a configuration or even a hard setting that shuts down interpreters after in-activity.  It seems like the interpreter (and it’s yarn consumption) hang out indefinitely, ideally there would be a clean way (like logout or a shutdown button within the notebook) to shutdown these interpreters, but additionally there should be a way for an admin of zeppelin to impose and idle timeout.  As a note to the scope of this, is dynamic yarn resource allocation configured (such that once a paragraph finishes it can release vcores)?

 

Thanks,

Shaun

 

This email and any files included with it may contain privileged,
proprietary and/or confidential information that is for the sole use
of the intended recipient(s).  Any disclosure, copying, distribution,
posting, or use of the information contained in or attached to this
email is prohibited unless permitted by the sender.  If you have
received this email in error, please immediately notify the sender
via return email, telephone, or fax and destroy this original transmission
and its included files without reading or saving it in any manner.
Thank you.

 

This email and any files included with it may contain privileged,
proprietary and/or confidential information that is for the sole use
of the intended recipient(s).  Any disclosure, copying, distribution,
posting, or use of the information contained in or attached to this
email is prohibited unless permitted by the sender.  If you have
received this email in error, please immediately notify the sender
via return email, telephone, or fax and destroy this original transmission
and its included files without reading or saving it in any manner.
Thank you.

Reply | Threaded
Open this post in threaded view
|

Re: Zeppelin Interpreter end

Rajat Gupta
No we have not submitted it back to spark community. I can send the patches for both spark and zeppelin. If it seems interesting enough then I can start a PR for zeppelin part at least. I will try to do this by weekend if it sounds interesting enough.

Rajat

On Wed, Jul 1, 2015 at 5:10 PM, Litt, Shaun <[hidden email]> wrote:

Interesting and useful.  I would love to see your changes for reference.   Have you submitted this feature back to the SPARK community?  I think this would be useful for a lot of interactive spark use-cases in shared clusters.

 

Also I have a question regarding your experience maintaining your own SPARK fork.  We have also been forced to maintain a very minor spark fork to work around some bugs in our environment, but our internal goal is to try to get back to the main apache branch or a vendor fork.  Have you found keeping up with SPARK (which is changing lightning fast – pun intended) challenging?

 

Thanks,

Shaun Litt

 

 

From: Rajat Gupta [mailto:[hidden email]]
Sent: Wednesday, July 01, 2015 2:41 AM
To: [hidden email]
Subject: Re: Zeppelin Interpreter end

 

We have implemented auto shutdown of spark application by adding an idle timeout setting directly in spark. We set the value of this setting in the zeppelin interpreter UI page and default it to 60 mins. For this fix we also added an auto lazy restart of interpreter in zeppelin if interpreter has shutdown because of inactivity. If you want I can send you patches for this both for spark and zeppelin. I did not submit the patch in zeppelin for this earlier because this makes sense only if spark has idle timeout in open source. 

Dynamic scaling of spark applications works fine on zeppelin with appropriate spark settings on interpreter page. Zeppelin just maintains handle to a spark context and that spark context is free to do anything afaik. We at Qubole are using it actively.

Rajat

 

On Tue, Jun 30, 2015 at 4:53 AM, Litt, Shaun <[hidden email]> wrote:

Hi,  I am new to zeppelin and just got it configured to run in my YARN cluster, but I was wondering if there is a configuration or even a hard setting that shuts down interpreters after in-activity.  It seems like the interpreter (and it’s yarn consumption) hang out indefinitely, ideally there would be a clean way (like logout or a shutdown button within the notebook) to shutdown these interpreters, but additionally there should be a way for an admin of zeppelin to impose and idle timeout.  As a note to the scope of this, is dynamic yarn resource allocation configured (such that once a paragraph finishes it can release vcores)?

 

Thanks,

Shaun

 

This email and any files included with it may contain privileged,
proprietary and/or confidential information that is for the sole use
of the intended recipient(s).  Any disclosure, copying, distribution,
posting, or use of the information contained in or attached to this
email is prohibited unless permitted by the sender.  If you have
received this email in error, please immediately notify the sender
via return email, telephone, or fax and destroy this original transmission
and its included files without reading or saving it in any manner.
Thank you.

 

This email and any files included with it may contain privileged,
proprietary and/or confidential information that is for the sole use
of the intended recipient(s).  Any disclosure, copying, distribution,
posting, or use of the information contained in or attached to this
email is prohibited unless permitted by the sender.  If you have
received this email in error, please immediately notify the sender
via return email, telephone, or fax and destroy this original transmission
and its included files without reading or saving it in any manner.
Thank you.