How to %sql interpreter

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

How to %sql interpreter

Petr Knez
Hi,
could you anyone help me how to force %sql (spark) interpreter not to trunc string parameters who are longer then 15 chars.

My case is to visualize data from dataframe but I can't use z.show (where is possible to set truncate option to false) because I use %livy.spark interpreter where z context doesn't work.

Only my current option is to create Spark temporary table from dataframe and use %sql interpreter.
But with above mentioned consequences.

Thanks
Petr



Reply | Threaded
Open this post in threaded view
|

Re: How to %sql interpreter

moon
Administrator
Hi,

Could you share little more about "trunc string parameters longer than 15 chars"? Which parameter are you talking about?

Thanks,
moon

On Tue, Aug 15, 2017 at 5:01 AM Petr Knez <[hidden email]> wrote:
Hi,
could you anyone help me how to force %sql (spark) interpreter not to trunc string parameters who are longer then 15 chars.

My case is to visualize data from dataframe but I can't use z.show (where is possible to set truncate option to false) because I use %livy.spark interpreter where z context doesn't work.

Only my current option is to create Spark temporary table from dataframe and use %sql interpreter.
But with above mentioned consequences.

Thanks
Petr



Reply | Threaded
Open this post in threaded view
|

Re: How to %sql interpreter

Petr Knez
Hi,
there is screenshot.
Red rectangles in attached picture show what I mean.

Text in those red highlighted columns is "trunctated" and there is no option to show full content (or I don't know how to do it)

Similar situation is when i show dataframe directly with spark API, but there is option "truncate" where is possible to set to false.

Similation behaviour without "truncate"
df.show()
----------------+--------------------+--------------------+---------------+
|GatewayTimeStamp| VehicleId| CurrentTripId|LastContactTime|
+----------------+--------------------+--------------------+---------------+
| 1502711324|d66395cc-4ae3-407...|00000000-0000-000...| 1502711291|
| 1502711325|d66395cc-4ae3-407...|00000000-0000-000...| 1502711326|
| 1502711440|d66395cc-4ae3-407...|00000000-0000-000...| 1502711439|
| 1502725991|d66395cc-4ae3-407...|00000000-0000-000...| 1502725991|
| 1502725876|d66395cc-4ae3-407...|00000000-0000-000...| 1502725842|

What I would like to see in %sql
df.show(false)
+----------------+------------------------------------+------------------------------------+---------------+
|GatewayTimeStamp|VehicleId |CurrentTripId |LastContactTime|
+----------------+------------------------------------+------------------------------------+---------------+
|1502711324 |d66395cc-4ae3-4079-adb9-4a3b59a311e8|00000000-0000-0000-0000-000000000000|1502711291 |
|1502711325 |d66395cc-4ae3-4079-adb9-4a3b59a311e8|00000000-0000-0000-0000-000000000000|1502711326 |
|1502711440 |d66395cc-4ae3-4079-adb9-4a3b59a311e8|00000000-0000-0000-0000-000000000000|1502711439 |
|1502725991 |d66395cc-4ae3-4079-adb9-4a3b59a311e8|00000000-0000-0000-0000-000000000000|1502725991 |


=> %sql default behaviour is like df.show(true)
Is there a way how to force %sql to work df.show(false)?

Petr




On Wed, Aug 16, 2017 at 6:08 PM, moon soo Lee <[hidden email]> wrote:
Hi,

Could you share little more about "trunc string parameters longer than 15 chars"? Which parameter are you talking about?

Thanks,
moon

On Tue, Aug 15, 2017 at 5:01 AM Petr Knez <[hidden email]> wrote:
Hi,
could you anyone help me how to force %sql (spark) interpreter not to trunc string parameters who are longer then 15 chars.

My case is to visualize data from dataframe but I can't use z.show (where is possible to set truncate option to false) because I use %livy.spark interpreter where z context doesn't work.

Only my current option is to create Spark temporary table from dataframe and use %sql interpreter.
But with above mentioned consequences.

Thanks
Petr





example.png (30K) Download Attachment