How to load hive configuration in HiveContext?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

How to load hive configuration in HiveContext?

김동경
I am trying to access a hive table which points to directory with sub-directories via SparkSQL.
So I need to set "mapred.input.dir.recursive" as true for recursive exploration.

I noticed that I can explicitly set the configuration using following command:

val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
sqlContext.setConf("mapreduce.input.fileinputformat.input.dir.recursive","true")

It works as well.
However, I couldn`t find any way to load that configuration from files such as hive-site.xml or mapred-site.xml.

How to load configuration from file?

Please help me.

Thanks in advance.
Dongkyoung Kim
Reply | Threaded
Open this post in threaded view
|

Re: How to load hive configuration in HiveContext?

Mina Lee
You can load hive-site.xml and mapred-site.xml by setting HADOOP_CONF_DIR variable in ZEPPELIN_HOME/conf/zeppelin-env.sh.
Both xml files should be placed under your HADOOP_CONF_DIR.

FYI, Zeppelin creates HiveContext automatically by default to value name `sqlContext

On Mon, Sep 14, 2015 at 5:14 PM, 김동경 <[hidden email]> wrote:
I am trying to access a hive table which points to directory with sub-directories via SparkSQL.
So I need to set "mapred.input.dir.recursive" as true for recursive exploration.

I noticed that I can explicitly set the configuration using following command:

val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
sqlContext.setConf("mapreduce.input.fileinputformat.input.dir.recursive","true")

It works as well.
However, I couldn`t find any way to load that configuration from files such as hive-site.xml or mapred-site.xml.

How to load configuration from file?

Please help me.

Thanks in advance.
Dongkyoung Kim