Customizing sparkconfig before starting spark app

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Customizing sparkconfig before starting spark app

Serega Sheypak
Hi, I have few questions about spark application customization
1. Is it possible to set spark app name from notebook, not from zeppelin conf?
2. Is is possible to register custom kryo serializers?
3. Is it possible to configure user name? Right now I'm running zeppelin as root and all jobs are submitted as root. I want to use logged user name instead.

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Customizing sparkconfig before starting spark app

moon
Administrator
Hi,

For now, there's no special way to configure spark app name from notebook while Zeppelin initialize SparkContext before execute any code in the notebook.

Registering custom kryo serializer is no different from spark-shell, i think.

Submitting job as a logged in user is depends on how your configure Spark interpreter in the Zeppelin. You can configure Zeppelin to use each user uses their own SparkContext (and separate interpreter process), with combination of Interpreter impersonation.

Please feel free open JIRA issue if you have some feature that helps you and all other people.

Thanks,
moon

On Wed, Apr 26, 2017 at 1:27 PM Serega Sheypak <[hidden email]> wrote:
Hi, I have few questions about spark application customization
1. Is it possible to set spark app name from notebook, not from zeppelin conf?
2. Is is possible to register custom kryo serializers?
3. Is it possible to configure user name? Right now I'm running zeppelin as root and all jobs are submitted as root. I want to use logged user name instead.

Loading...