sql paragraph doesn't see my 3rd party jars

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

sql paragraph doesn't see my 3rd party jars

Serega Sheypak
Hi, I'm trying to use spark and sql paragraphs with 3rd party jars added to spark interpreter configuration.

My spark code works fine.


My sql paragraph fails with class not found exception
%sql
create external table MY_TABLE row format serde 'com.my.MyAvroSerde' 
with serdeproperties ('serialization.class'='com.my.ContainerProto') stored as inputformat 'com.my.ProtoAvroFileFormat'  
LOCATION 'hdfs://my/data'

Exception:
MetaException(message:org.apache.hadoop.hive.serde2.SerDeException java.lang.ClassNotFoundException: Class com.my.ContainerProto not found)


It's confusing since spark paragraph works well with
code

import com.my.ContainerProto
// bla-bla
rdd.map{bytes => ContainerProto.fromBytes(bytes)}

Code executed and result produced. Why sql paragraph doesn't see my 3rd party jars?



Reply | Threaded
Open this post in threaded view
|

Re: sql paragraph doesn't see my 3rd party jars

moon
Administrator
Could you share little more about how did you add 3rd party jars?

ie. through %spark.dep, Interpreter setting menu on GUI, environment variable, and so on.

Thanks,
moon

On Sat, Oct 7, 2017 at 3:47 AM Serega Sheypak <[hidden email]> wrote:
Hi, I'm trying to use spark and sql paragraphs with 3rd party jars added to spark interpreter configuration.

My spark code works fine.


My sql paragraph fails with class not found exception
%sql
create external table MY_TABLE row format serde 'com.my.MyAvroSerde' 
with serdeproperties ('serialization.class'='com.my.ContainerProto') stored as inputformat 'com.my.ProtoAvroFileFormat'  
LOCATION 'hdfs://my/data'

Exception:
MetaException(message:org.apache.hadoop.hive.serde2.SerDeException java.lang.ClassNotFoundException: Class com.my.ContainerProto not found)


It's confusing since spark paragraph works well with
code

import com.my.ContainerProto
// bla-bla
rdd.map{bytes => ContainerProto.fromBytes(bytes)}

Code executed and result produced. Why sql paragraph doesn't see my 3rd party jars?