Re: Protobuf clash between 2.4 and 2.5 versions

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Re: Protobuf clash between 2.4 and 2.5 versions

Fernando Estevan Mesa
I've tried several build options but I cannot get zeppelin to compile with protobuf 2.5.0

Whenever I run a spark command against HDFS I get the error below. I understand it be due to a clash between protobuf 2.4 on zeppelin and protobuf 2.5 on spark.

The build command is below. Any help is appreciated.

sc.textFile("<a href="hdfs://quickstart.cloudera:8020/user/cloudera/201410.csv" class="">hdfs://quickstart.cloudera:8020/user/cloudera/201410.csv").collect().foreach(println)

java.io.IOException: Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: callId, status; Host Details : local host is: "fmesa-mbp/192.168.10.50"; destination host is: "quickstart.cloudera":8020;

mvn clean install -DskipTests -Pspark-1.5 -Phadoop-2.6 -Ppyspark -Dhadoop.version=2.6.0-cdh5.4.2 -Phadoop-2.6 -Dprotobuf.version=2.5.0