Stop execution if a paragraph errors

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Stop execution if a paragraph errors

Girish Reddy
Hello:

It seems the default behavior in Zeppelin is to continue execution of paragraph even if one of the paragraph errors out.  Is there a way to configure it to stop execution as soon as a paragraph errors out?  Or is this something that needs to be handled in the paragraph code?

Thanks,
-Girish


Reply | Threaded
Open this post in threaded view
|

Re: Stop execution if a paragraph errors

moon
Administrator
Hi Girish,

Currently, I think there're no simple way. Please feel free to file an issue on JIRA. 

Technically, RunNote function iterates runParagraph in front-end side [1].  To make execution stop on paragraph error, 

a) Need to iterate it on backend side by calling [2], otherwise closing browser window may stop notebook run.
b) [2] need to be modified to not use scheduler from individual interpreter but use another FIFO scheduler that run all job sequentially.
c) and maybe it's good to an option to choose, current behavior and this new behavior in front-end side.

Thanks,
moon



On Wed, Mar 9, 2016 at 7:44 AM Girish Reddy <[hidden email]> wrote:
Hello:

It seems the default behavior in Zeppelin is to continue execution of paragraph even if one of the paragraph errors out.  Is there a way to configure it to stop execution as soon as a paragraph errors out?  Or is this something that needs to be handled in the paragraph code?

Thanks,
-Girish


Reply | Threaded
Open this post in threaded view
|

Re: Stop execution if a paragraph errors

Skanda

Hi Moon & Girish

The note has a runAll method that is being used by the cron scheduler. This method iterates the paragraphs and submits each of them for execution in a single go. This can be made to run sequentially by checking the paragraph status before triggering the next paragraph. We can have a config in paragraph settings in ui say "enable skip on failure" which can be checked to see whether next  paragraph should be triggered based on the status of the current paragraph. Also the notebook run can be modified to use this runAll method. Your thoughts pls.

Regards
Skanda

On 09-Mar-2016 11:38 PM, "moon soo Lee" <[hidden email]> wrote:
Hi Girish,

Currently, I think there're no simple way. Please feel free to file an issue on JIRA. 

Technically, RunNote function iterates runParagraph in front-end side [1].  To make execution stop on paragraph error, 

a) Need to iterate it on backend side by calling [2], otherwise closing browser window may stop notebook run.
b) [2] need to be modified to not use scheduler from individual interpreter but use another FIFO scheduler that run all job sequentially.
c) and maybe it's good to an option to choose, current behavior and this new behavior in front-end side.

Thanks,
moon



On Wed, Mar 9, 2016 at 7:44 AM Girish Reddy <[hidden email]> wrote:
Hello:

It seems the default behavior in Zeppelin is to continue execution of paragraph even if one of the paragraph errors out.  Is there a way to configure it to stop execution as soon as a paragraph errors out?  Or is this something that needs to be handled in the paragraph code?

Thanks,
-Girish


Reply | Threaded
Open this post in threaded view
|

Re: Stop execution if a paragraph errors

moon
Administrator
Hi Skanda,

Thanks for sharing great idea. I think you idea is really make sense. :-)

Thanks,
moon

On Wed, Mar 9, 2016 at 11:04 AM Skanda <[hidden email]> wrote:

Hi Moon & Girish

The note has a runAll method that is being used by the cron scheduler. This method iterates the paragraphs and submits each of them for execution in a single go. This can be made to run sequentially by checking the paragraph status before triggering the next paragraph. We can have a config in paragraph settings in ui say "enable skip on failure" which can be checked to see whether next  paragraph should be triggered based on the status of the current paragraph. Also the notebook run can be modified to use this runAll method. Your thoughts pls.

Regards


Skanda

On 09-Mar-2016 11:38 PM, "moon soo Lee" <[hidden email]> wrote:
Hi Girish,

Currently, I think there're no simple way. Please feel free to file an issue on JIRA. 

Technically, RunNote function iterates runParagraph in front-end side [1].  To make execution stop on paragraph error, 

a) Need to iterate it on backend side by calling [2], otherwise closing browser window may stop notebook run.
b) [2] need to be modified to not use scheduler from individual interpreter but use another FIFO scheduler that run all job sequentially.
c) and maybe it's good to an option to choose, current behavior and this new behavior in front-end side.

Thanks,
moon



On Wed, Mar 9, 2016 at 7:44 AM Girish Reddy <[hidden email]> wrote:
Hello:

It seems the default behavior in Zeppelin is to continue execution of paragraph even if one of the paragraph errors out.  Is there a way to configure it to stop execution as soon as a paragraph errors out?  Or is this something that needs to be handled in the paragraph code?

Thanks,
-Girish


Reply | Threaded
Open this post in threaded view
|

RE: Stop execution if a paragraph errors

Guha, Ayan (AU - Melbourne)

Hi

 

This is my first mail to this group. I am writing this in order to get a fair understanding of how zeppelin can be integrated with Spark. 

 

Our use case is to load few tables from a DB to Spark, run some transformation. Once done, we want to expose data through Zeppelin for analytics. I have few question around that to sound off any gross architectural flaws. 

 

Questions:

 

1. How Zeppelin connects to Spark? Thrift JDBC? How is it different than JDBC server?

 

2. What is the scope of Spark application when it is used from Zeppelin? For example, if I have few subsequent actions in zeppelin like map,filter,reduceByKey, filter,collect. I assume this will translate to an application and get submitted to Spark. However, If I want to use reuse some part of the data (for example) after first map transformation in earlier application. Can I do it? Or will it be another application and another spark submit?

 

 In our use case data will already be loaded in RDDs. So how Zeppelin can access it? Is it even possible?

 

3. How can I control access on specific rdds to specific users in Zeppelin (assuming we have implemented some way of login mechanism in Zeppelin and we have a mapping between Zeppelin users and their LDAP accounts). Is it even possible? 

 

appreciate any help/pointers/guidance. 

https://ssl.gstatic.com/ui/v1/icons/mail/images/cleardot.gif

 

 

 




This e-mail and any attachments to it are confidential. You must not use, disclose or act on the e-mail if you are not the intended recipient. If you have

received this e-mail in error, please let us know by contacting the sender and deleting the original e-mail.
Liability limited by a scheme approved under Professional Standards Legislation.
Deloitte refers to one or more of

Deloitte Touche Tohmatsu Limited, a UK private company limited by guarantee, and its network of member firms, each of which is a legally separate and

independent entity. Please see www.deloitte.com.au/about for a detailed

description of the legal structure of Deloitte Touche Tohmatsu Limited and its member firms.


Reply | Threaded
Open this post in threaded view
|

Re: Stop execution if a paragraph errors

pmarini
This post has NOT been accepted by the mailing list yet.
In reply to this post by moon
Hello,
did someone create a Jira for this topic?
Thanks