Spark Interpreter: Change default scheduler pool

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Spark Interpreter: Change default scheduler pool

Fabian Böhnlein
Hi all,

how can I change (globally, for Zeppelin) the default scheduler pool which SparkInterpreter submits jobs to. Currently all jobs go into the pool 'default' but I want them to go into the pool 'fair'.
We use "Per Note" and "scoped" processes for best resource sharing.

"spark.scheduler.pool"="fair" in Interpreter Settings does not work, should it?

What works is
sc.setLocalProperty("spark.scheduler.pool","fair")
but it's required in every note (not just notebook) since it's on thread level.

Is there a possibility to globally/per notebook set the 'fair' pool as the default pool?

Zeppelin brings two (hardcoded?) sheduler pools 'default' and 'fair'. Between them, the scheduling is FAIR. 'default' is FIFO, 'fair' is FAIR.

This is awesome and together with dynamicAllocation allows for super flexible usage for multiple users but above behavior is a bit complicated.

Thanks,
Fabian



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Spark Interpreter: Change default scheduler pool

moon
Administrator
Hi Fabian,

Thanks for sharing the issue.
SparkSqlInterpreter set scheduler to "fair" depends on interpreter property [1]. I think we can do the similar for SparkInterpreter.
Do you mind file a new JIRA issue for it?

Regards,
moon



On Tue, Mar 28, 2017 at 5:24 AM Fabian Böhnlein <[hidden email]> wrote:
Hi all,

how can I change (globally, for Zeppelin) the default scheduler pool which SparkInterpreter submits jobs to. Currently all jobs go into the pool 'default' but I want them to go into the pool 'fair'.
We use "Per Note" and "scoped" processes for best resource sharing.

"spark.scheduler.pool"="fair" in Interpreter Settings does not work, should it?

What works is
sc.setLocalProperty("spark.scheduler.pool","fair")
but it's required in every note (not just notebook) since it's on thread level.

Is there a possibility to globally/per notebook set the 'fair' pool as the default pool?

Zeppelin brings two (hardcoded?) sheduler pools 'default' and 'fair'. Between them, the scheduling is FAIR. 'default' is FIFO, 'fair' is FAIR.

This is awesome and together with dynamicAllocation allows for super flexible usage for multiple users but above behavior is a bit complicated.

Thanks,
Fabian



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Spark Interpreter: Change default scheduler pool

Fabian Böhnlein
Hi moon,

exactly, thanks for the pointer.


Best,
Fabian


On Tue, 28 Mar 2017 at 15:48 moon soo Lee <[hidden email]> wrote:
Hi Fabian,

Thanks for sharing the issue.
SparkSqlInterpreter set scheduler to "fair" depends on interpreter property [1]. I think we can do the similar for SparkInterpreter.
Do you mind file a new JIRA issue for it?

Regards,
moon



On Tue, Mar 28, 2017 at 5:24 AM Fabian Böhnlein <[hidden email]> wrote:
Hi all,

how can I change (globally, for Zeppelin) the default scheduler pool which SparkInterpreter submits jobs to. Currently all jobs go into the pool 'default' but I want them to go into the pool 'fair'.
We use "Per Note" and "scoped" processes for best resource sharing.

"spark.scheduler.pool"="fair" in Interpreter Settings does not work, should it?

What works is
sc.setLocalProperty("spark.scheduler.pool","fair")
but it's required in every note (not just notebook) since it's on thread level.

Is there a possibility to globally/per notebook set the 'fair' pool as the default pool?

Zeppelin brings two (hardcoded?) sheduler pools 'default' and 'fair'. Between them, the scheduling is FAIR. 'default' is FIFO, 'fair' is FAIR.

This is awesome and together with dynamicAllocation allows for super flexible usage for multiple users but above behavior is a bit complicated.

Thanks,
Fabian



Loading...