Set spark.sql.extention property using Livy interpreter

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Set spark.sql.extention property using Livy interpreter

Fernando Antunes
Hi. I’m trying to use Hive warehouse connector with Livy Interpreter.  To do that, I must set the spark.sql.extention property before the spark context creation. How to do that using Livy Interpreter ?


My 1st paragraph is :

%livy.conf


I’e tried :

1) livy.spark.conf spark.sql.extention com.qubole.spark.hiveacid.HiveAcidAutoConvertExtension
2) spark.conf spark.sql.extention com.qubole.spark.hiveacid.HiveAcidAutoConvertExtension

But they didn’t work.

Set this property on the spark-default affects all other applications and notebook that does not use Hive warehouse connector.