Appreciate for sharing problem and solution!
Best,
moon
Dear all,
I found the reason.
After enabling the "spark.sql.parquet.useDataSourceApi" in sqlContext, the partition of parquet works correctly.
example code:
```
sqlContext.setConf("spark.sql.parquet.useDataSourceApi", "true")
val ecrtb20150622 = sqlContext.parquetFile("hdfs:///bwlogs/beta/archive/EC.RTB/_year=2015/_month=06/_day=22")
```
Hope this might help others in the future.
Best,