Livy interpreter Scala code string interpolation

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Livy interpreter Scala code string interpolation

Tan, Jialiang

Hi,

 

When writing Spark Scala code in zeppelin with Livy interpreter, string interpolation does not seem to work. For example:

 

val devices = sc.objectFile[(VertexId, DeviceAttr)](

s"s3a://${bucket}/${datasetS3Prefix}/${tableName} /dt=${end_dt}/tid=${tid}/${datasetS3Suffix}")

 

I think the dollar curly brackets were interpreted as Javascript template literals and hence disappear when Livy server gets the message and becomes:

 

val devices = sc.objectFile[(VertexId, DeviceAttr)](

   s"s3a://///dt= /tid= / ")

 

At the same time a couple of input boxes appears at the bottom of this paragraph in Zeppelin UI having the corresponding titles in the original dollar curly brackets.

 

Is this a bug? How can we overcome the string interpolation issue?

 

Thanks,

Jialiang

Reply | Threaded
Open this post in threaded view
|

Re: Livy interpreter Scala code string interpolation

Mohit Jaggi
I run into similar issues with shell scripts that use ${var}. Can we use a different magic notation for Z, one that has a lower chance of colliding with shell scripts and Scala code? This will be hard to do due to the variety of interpreters supported. Perhaps this can be made configurable?

On Fri, Nov 3, 2017 at 10:36 AM, Tan, Jialiang <[hidden email]> wrote:

Hi,

 

When writing Spark Scala code in zeppelin with Livy interpreter, string interpolation does not seem to work. For example:

 

val devices = sc.objectFile[(VertexId, DeviceAttr)](

s"s3a://${bucket}/${datasetS3Prefix}/${tableName} /dt=${end_dt}/tid=${tid}/${datasetS3Suffix}")

 

I think the dollar curly brackets were interpreted as Javascript template literals and hence disappear when Livy server gets the message and becomes:

 

val devices = sc.objectFile[(VertexId, DeviceAttr)](

   s"s3a://///dt= /tid= / ")

 

At the same time a couple of input boxes appears at the bottom of this paragraph in Zeppelin UI having the corresponding titles in the original dollar curly brackets.

 

Is this a bug? How can we overcome the string interpolation issue?

 

Thanks,

Jialiang


Reply | Threaded
Open this post in threaded view
|

Re: Livy interpreter Scala code string interpolation

Jeff Zhang

Yes, this is due to the dynamic forms of zeppelin. 


Mohit Jaggi <[hidden email]>于2017年11月4日周六 上午2:28写道:
I run into similar issues with shell scripts that use ${var}. Can we use a different magic notation for Z, one that has a lower chance of colliding with shell scripts and Scala code? This will be hard to do due to the variety of interpreters supported. Perhaps this can be made configurable?

On Fri, Nov 3, 2017 at 10:36 AM, Tan, Jialiang <[hidden email]> wrote:

Hi,

 

When writing Spark Scala code in zeppelin with Livy interpreter, string interpolation does not seem to work. For example:

 

val devices = sc.objectFile[(VertexId, DeviceAttr)](

s"s3a://${bucket}/${datasetS3Prefix}/${tableName} /dt=${end_dt}/tid=${tid}/${datasetS3Suffix}")

 

I think the dollar curly brackets were interpreted as Javascript template literals and hence disappear when Livy server gets the message and becomes:

 

val devices = sc.objectFile[(VertexId, DeviceAttr)](

   s"s3a://///dt= /tid= / ")

 

At the same time a couple of input boxes appears at the bottom of this paragraph in Zeppelin UI having the corresponding titles in the original dollar curly brackets.

 

Is this a bug? How can we overcome the string interpolation issue?

 

Thanks,

Jialiang


Reply | Threaded
Open this post in threaded view
|

Re: Livy interpreter Scala code string interpolation

Tan, Jialiang

But somehow Spark interpreter works without this problem. Why is that the case if the problem is on Zeppelin UI side?

 

From: Jeff Zhang <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Friday, November 3, 2017 at 4:48 PM
To: "[hidden email]" <[hidden email]>
Subject: Re: Livy interpreter Scala code string interpolation

 

 

Yes, this is due to the dynamic forms of zeppelin. 

 

 

Mohit Jaggi <[hidden email]>2017114日周六 上午2:28写道:

I run into similar issues with shell scripts that use ${var}. Can we use a different magic notation for Z, one that has a lower chance of colliding with shell scripts and Scala code? This will be hard to do due to the variety of interpreters supported. Perhaps this can be made configurable?

 

On Fri, Nov 3, 2017 at 10:36 AM, Tan, Jialiang <[hidden email]> wrote:

Hi,

 

When writing Spark Scala code in zeppelin with Livy interpreter, string interpolation does not seem to work. For example:

 

val devices = sc.objectFile[(VertexId, DeviceAttr)](

s"s3a://${bucket}/${datasetS3Prefix}/${tableName} /dt=${end_dt}/tid=${tid}/${datasetS3Suffix}")

 

I think the dollar curly brackets were interpreted as Javascript template literals and hence disappear when Livy server gets the message and becomes:

 

val devices = sc.objectFile[(VertexId, DeviceAttr)](

   s"s3a://///dt= /tid= / ")

 

At the same time a couple of input boxes appears at the bottom of this paragraph in Zeppelin UI having the corresponding titles in the original dollar curly brackets.

 

Is this a bug? How can we overcome the string interpolation issue?

 

Thanks,

Jialiang

 

Reply | Threaded
Open this post in threaded view
|

Re: Livy interpreter Scala code string interpolation

Jeff Zhang
Oops, this is a bug of livy interpreter, Could you help create a ticket ? Thanks


Tan, Jialiang <[hidden email]>于2017年11月4日周六 上午7:50写道:

But somehow Spark interpreter works without this problem. Why is that the case if the problem is on Zeppelin UI side?

 

From: Jeff Zhang <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Friday, November 3, 2017 at 4:48 PM
To: "[hidden email]" <[hidden email]>
Subject: Re: Livy interpreter Scala code string interpolation

 

 

Yes, this is due to the dynamic forms of zeppelin. 

 

 

Mohit Jaggi <[hidden email]>2017114日周六 上午2:28写道:

I run into similar issues with shell scripts that use ${var}. Can we use a different magic notation for Z, one that has a lower chance of colliding with shell scripts and Scala code? This will be hard to do due to the variety of interpreters supported. Perhaps this can be made configurable?

 

On Fri, Nov 3, 2017 at 10:36 AM, Tan, Jialiang <[hidden email]> wrote:

Hi,

 

When writing Spark Scala code in zeppelin with Livy interpreter, string interpolation does not seem to work. For example:

 

val devices = sc.objectFile[(VertexId, DeviceAttr)](

s"s3a://${bucket}/${datasetS3Prefix}/${tableName} /dt=${end_dt}/tid=${tid}/${datasetS3Suffix}")

 

I think the dollar curly brackets were interpreted as Javascript template literals and hence disappear when Livy server gets the message and becomes:

 

val devices = sc.objectFile[(VertexId, DeviceAttr)](

   s"s3a://///dt= /tid= / ")

 

At the same time a couple of input boxes appears at the bottom of this paragraph in Zeppelin UI having the corresponding titles in the original dollar curly brackets.

 

Is this a bug? How can we overcome the string interpolation issue?

 

Thanks,

Jialiang

 

Reply | Threaded
Open this post in threaded view
|

Re: Livy interpreter Scala code string interpolation

Mohit Jaggi
So for spark interpreter does ${x} create a dynamic textbox or is interpreted as string interpolation? Or is it context-sensitive and knows about s"...."?

On Fri, Nov 3, 2017 at 4:59 PM, Jeff Zhang <[hidden email]> wrote:
Oops, this is a bug of livy interpreter, Could you help create a ticket ? Thanks


Tan, Jialiang <[hidden email]>于2017年11月4日周六 上午7:50写道:

But somehow Spark interpreter works without this problem. Why is that the case if the problem is on Zeppelin UI side?

 

From: Jeff Zhang <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Friday, November 3, 2017 at 4:48 PM
To: "[hidden email]" <[hidden email]>
Subject: Re: Livy interpreter Scala code string interpolation

 

 

Yes, this is due to the dynamic forms of zeppelin. 

 

 

Mohit Jaggi <[hidden email]>2017114日周六 上午2:28写道:

I run into similar issues with shell scripts that use ${var}. Can we use a different magic notation for Z, one that has a lower chance of colliding with shell scripts and Scala code? This will be hard to do due to the variety of interpreters supported. Perhaps this can be made configurable?

 

On Fri, Nov 3, 2017 at 10:36 AM, Tan, Jialiang <[hidden email]> wrote:

Hi,

 

When writing Spark Scala code in zeppelin with Livy interpreter, string interpolation does not seem to work. For example:

 

val devices = sc.objectFile[(VertexId, DeviceAttr)](

s"s3a://${bucket}/${datasetS3Prefix}/${tableName} /dt=${end_dt}/tid=${tid}/${datasetS3Suffix}")

 

I think the dollar curly brackets were interpreted as Javascript template literals and hence disappear when Livy server gets the message and becomes:

 

val devices = sc.objectFile[(VertexId, DeviceAttr)](

   s"s3a://///dt= /tid= / ")

 

At the same time a couple of input boxes appears at the bottom of this paragraph in Zeppelin UI having the corresponding titles in the original dollar curly brackets.

 

Is this a bug? How can we overcome the string interpolation issue?

 

Thanks,

Jialiang

 


Reply | Threaded
Open this post in threaded view
|

Re: Livy interpreter Scala code string interpolation

Tan, Jialiang

I think for Spark interpreter it can properly process the string interpolation without any problems caused by dynamic form.

 

From: Mohit Jaggi <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Friday, November 3, 2017 at 5:20 PM
To: "[hidden email]" <[hidden email]>
Subject: Re: Livy interpreter Scala code string interpolation

 

So for spark interpreter does ${x} create a dynamic textbox or is interpreted as string interpolation? Or is it context-sensitive and knows about s"...."?

 

On Fri, Nov 3, 2017 at 4:59 PM, Jeff Zhang <[hidden email]> wrote:

Oops, this is a bug of livy interpreter, Could you help create a ticket ? Thanks

 

 

Tan, Jialiang <[hidden email]>2017114日周六 上午7:50写道:

But somehow Spark interpreter works without this problem. Why is that the case if the problem is on Zeppelin UI side?

 

From: Jeff Zhang <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Friday, November 3, 2017 at 4:48 PM
To: "[hidden email]" <[hidden email]>
Subject: Re: Livy interpreter Scala code string interpolation

 

 

Yes, this is due to the dynamic forms of zeppelin. 

 

 

Mohit Jaggi <[hidden email]>2017114日周六 上午2:28写道:

I run into similar issues with shell scripts that use ${var}. Can we use a different magic notation for Z, one that has a lower chance of colliding with shell scripts and Scala code? This will be hard to do due to the variety of interpreters supported. Perhaps this can be made configurable?

 

On Fri, Nov 3, 2017 at 10:36 AM, Tan, Jialiang <[hidden email]> wrote:

Hi,

 

When writing Spark Scala code in zeppelin with Livy interpreter, string interpolation does not seem to work. For example:

 

val devices = sc.objectFile[(VertexId, DeviceAttr)](

s"s3a://${bucket}/${datasetS3Prefix}/${tableName} /dt=${end_dt}/tid=${tid}/${datasetS3Suffix}")

 

I think the dollar curly brackets were interpreted as Javascript template literals and hence disappear when Livy server gets the message and becomes:

 

val devices = sc.objectFile[(VertexId, DeviceAttr)](

   s"s3a://///dt= /tid= / ")

 

At the same time a couple of input boxes appears at the bottom of this paragraph in Zeppelin UI having the corresponding titles in the original dollar curly brackets.

 

Is this a bug? How can we overcome the string interpolation issue?

 

Thanks,

Jialiang

 

 

Reply | Threaded
Open this post in threaded view
|

Re: Livy interpreter Scala code string interpolation

Jeff Zhang

Spark only enable dynamic form for sql interpreter, while livy enable it for interpreters. This is the bug of livy interpreter.



Tan, Jialiang <[hidden email]>于2017年11月4日周六 上午8:22写道:

I think for Spark interpreter it can properly process the string interpolation without any problems caused by dynamic form.

 

From: Mohit Jaggi <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Friday, November 3, 2017 at 5:20 PM


To: "[hidden email]" <[hidden email]>
Subject: Re: Livy interpreter Scala code string interpolation

 

So for spark interpreter does ${x} create a dynamic textbox or is interpreted as string interpolation? Or is it context-sensitive and knows about s"...."?

 

On Fri, Nov 3, 2017 at 4:59 PM, Jeff Zhang <[hidden email]> wrote:

Oops, this is a bug of livy interpreter, Could you help create a ticket ? Thanks

 

 

Tan, Jialiang <[hidden email]>2017114日周六 上午7:50写道:

But somehow Spark interpreter works without this problem. Why is that the case if the problem is on Zeppelin UI side?

 

From: Jeff Zhang <[hidden email]>
Reply-To: "[hidden email]" <[hidden email]>
Date: Friday, November 3, 2017 at 4:48 PM
To: "[hidden email]" <[hidden email]>
Subject: Re: Livy interpreter Scala code string interpolation

 

 

Yes, this is due to the dynamic forms of zeppelin. 

 

 

Mohit Jaggi <[hidden email]>2017114日周六 上午2:28写道:

I run into similar issues with shell scripts that use ${var}. Can we use a different magic notation for Z, one that has a lower chance of colliding with shell scripts and Scala code? This will be hard to do due to the variety of interpreters supported. Perhaps this can be made configurable?

 

On Fri, Nov 3, 2017 at 10:36 AM, Tan, Jialiang <[hidden email]> wrote:

Hi,

 

When writing Spark Scala code in zeppelin with Livy interpreter, string interpolation does not seem to work. For example:

 

val devices = sc.objectFile[(VertexId, DeviceAttr)](

s"s3a://${bucket}/${datasetS3Prefix}/${tableName} /dt=${end_dt}/tid=${tid}/${datasetS3Suffix}")

 

I think the dollar curly brackets were interpreted as Javascript template literals and hence disappear when Livy server gets the message and becomes:

 

val devices = sc.objectFile[(VertexId, DeviceAttr)](

   s"s3a://///dt= /tid= / ")

 

At the same time a couple of input boxes appears at the bottom of this paragraph in Zeppelin UI having the corresponding titles in the original dollar curly brackets.

 

Is this a bug? How can we overcome the string interpolation issue?

 

Thanks,

Jialiang