Can't run simple example with scala and spark SQL. Some non obvious syntax error in SQL

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Can't run simple example with scala and spark SQL. Some non obvious syntax error in SQL

Serega Sheypak
Here is my sample notebook:
%spark
val linesText = sc.textFile("hdfs://cluster/user/me/lines.txt")

  case class Line(id:Long, firstField:String, secondField:String)

  val lines = linesText.map{ line =>
  val splitted = line.split(" ")
    println("splitted => " + splitted)
    Line(splitted(0).toLong, splitted(1), splitted(2))
  }

  lines.toDF().registerTempTable("lines")

  %sql select firstField, secondField, count(1) from lines group by firstField, secondField order by firstField, secondField

1. I can see that spark job was started on my YARN cluster.
2. It failed
UI shows exception. Can't understand what do I do wrong:
<div class="gmail-tableDisplay gmail-ng-scope" src="&#39;app/notebook/paragraph/result/result.html?v=1492775620971&#39;" style="box-sizing:border-box;margin-top:2px">
%sql select firstField, secondField, count(1) from lines group by firstField, secondField order by firstField, secondField ^
3. There is suspicious output in zeppelin log:

 INFO [2017-05-02 11:50:02,846] ({pool-2-thread-8} SchedulerFactory.java[jobFinished]:137) - Job paragraph_1493724118696_868476558 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session1712472970
ERROR [2017-05-02 11:50:18,809] ({qtp1286783232-166} NotebookServer.java[onMessage]:380) - Can't handle message
java.lang.NullPointerException
at org.apache.zeppelin.socket.NotebookServer.addNewParagraphIfLastParagraphIsExecuted(NotebookServer.java:1713)
at org.apache.zeppelin.socket.NotebookServer.persistAndExecuteSingleParagraph(NotebookServer.java:1741)
at org.apache.zeppelin.socket.NotebookServer.runAllParagraphs(NotebookServer.java:1641)
at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:291)
at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:59)
at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128)
at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161)
at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309)
at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214)
at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
 INFO [2017-05-02 11:50:18,811] ({pool-2-thread-14} SchedulerFactory.java[jobStarted]:131) - Job paragraph_1493724118696_868476558 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session1712472970
 INFO [2017-05-02 11:50:18,812] ({pool-2-thread-14} Paragraph.java[jobRun]:363) - run paragraph 20170502-112158_458502255 using spark org.apache.zeppelin.interpreter.LazyOpenInterpreter@24ca4045
 WARN [2017-05-02 11:50:19,810] ({pool-2-thread-14} NotebookServer.java[afterStatusChange]:2162) - Job 20170502-112158_458502255 is finished, status: ERROR, exception: null, result: %text linesText: org.apache.spark.rdd.RDD[String] = hdfs://path/to/my/file.txt MapPartitionsRDD[21] at textFile at <console>:27
defined class Line
lines: org.apache.spark.rdd.RDD[Line] = MapPartitionsRDD[22] at map at <console>:31
warning: there was one deprecation warning; re-run with -deprecation for details
<console>:1: error: ';' expected but ',' found.
  %sql select firstField, secondField, count(1) from lines group by firstField, secondField order by firstField, secondField


4. Interpreter log is also confusing:
INFO [2017-05-02 11:41:52,706] ({pool-2-thread-2} Logging.scala[logInfo]:54) - Warehouse location for Hive client (version 1.1.0) is file:/spark-warehouse
INFO [2017-05-02 11:41:52,983] ({pool-2-thread-2} PerfLogger.java[PerfLogBegin]:122) - <PERFLOG method=create_database from=org.apache.hadoop.hive.metastore.RetryingHMSHandler>
INFO [2017-05-02 11:41:52,984] ({pool-2-thread-2} HiveMetaStore.java[logInfo]:795) - 0: create_database: Database(name:default, description:default database, locationUri:file:/spark-warehouse, parameters:{})
INFO [2017-05-02 11:41:52,984] ({pool-2-thread-2} HiveMetaStore.java[logAuditEvent]:388) - ugi=zblenessy ip=unknown-ip-addr cmd=create_database: Database(name:default, description:default database, locationUri:file:/spark-warehouse, parameters:{})
ERROR [2017-05-02 11:41:52,992] ({pool-2-thread-2} RetryingHMSHandler.java[invokeInternal]:189) - AlreadyExistsException(message:Database default already exists)
 at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:944)

What should I do with failed metastore DB creation? Is it fine?

I'm stuck, can you give me some ideas please?
Reply | Threaded
Open this post in threaded view
|

Re: Can't run simple example with scala and spark SQL. Some non obvious syntax error in SQL

Serega Sheypak
Just spark code works fine:
val linesText = sc.textFile("hdfs://my/file/onhdfs.txt")

  case class Line(id:Long, firstField:String, secondField:String)

  val lines = linesText.map{ line =>
  val splitted = line.split(" ")
    println("splitted => " + splitted)
    Line(splitted(0).toLong, splitted(1), splitted(2))
  }
  
lines.collect().foreach(println)

prints file contexts to UI. I have some trouble with sql...



2017-05-02 13:57 GMT+02:00 Serega Sheypak <[hidden email]>:
Here is my sample notebook:
%spark
val linesText = sc.textFile("hdfs://cluster/user/me/lines.txt")

  case class Line(id:Long, firstField:String, secondField:String)

  val lines = linesText.map{ line =>
  val splitted = line.split(" ")
    println("splitted => " + splitted)
    Line(splitted(0).toLong, splitted(1), splitted(2))
  }

  lines.toDF().registerTempTable("lines")

  %sql select firstField, secondField, count(1) from lines group by firstField, secondField order by firstField, secondField

1. I can see that spark job was started on my YARN cluster.
2. It failed
UI shows exception. Can't understand what do I do wrong:
<div class="m_5858111259781881512gmail-tableDisplay m_5858111259781881512gmail-ng-scope" src="https://ci6.googleusercontent.com/proxy/XB7V7I05p3jC5YnshOrt2K9V9FE2z_jqfnjl5kceR8GIK804EgLZIDMcUaPS7iawZzLF06kIb_pD1BuMe_YSBcX8RCmSdXvLcwwRLBKswWxr9rhEg7P0Jg=s0-d-e1-ft#http://&#39;app/notebook/paragraph/result/result.html?v=1492775620971&#39;" style="box-sizing:border-box;margin-top:2px">
%sql select firstField, secondField, count(1) from lines group by firstField, secondField order by firstField, secondField ^
3. There is suspicious output in zeppelin log:

 INFO [2017-05-02 11:50:02,846] ({pool-2-thread-8} SchedulerFactory.java[jobFinished]:137) - Job paragraph_1493724118696_868476558 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session1712472970
ERROR [2017-05-02 11:50:18,809] ({qtp1286783232-166} NotebookServer.java[onMessage]:380) - Can't handle message
java.lang.NullPointerException
at org.apache.zeppelin.socket.NotebookServer.addNewParagraphIfLastParagraphIsExecuted(NotebookServer.java:1713)
at org.apache.zeppelin.socket.NotebookServer.persistAndExecuteSingleParagraph(NotebookServer.java:1741)
at org.apache.zeppelin.socket.NotebookServer.runAllParagraphs(NotebookServer.java:1641)
at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:291)
at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:59)
at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128)
at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161)
at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309)
at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214)
at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
 INFO [2017-05-02 11:50:18,811] ({pool-2-thread-14} SchedulerFactory.java[jobStarted]:131) - Job paragraph_1493724118696_868476558 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session1712472970
 INFO [2017-05-02 11:50:18,812] ({pool-2-thread-14} Paragraph.java[jobRun]:363) - run paragraph 20170502-112158_458502255 using spark org.apache.zeppelin.interpreter.LazyOpenInterpreter@24ca4045
 WARN [2017-05-02 11:50:19,810] ({pool-2-thread-14} NotebookServer.java[afterStatusChange]:2162) - Job 20170502-112158_458502255 is finished, status: ERROR, exception: null, result: %text linesText: org.apache.spark.rdd.RDD[String] = hdfs://path/to/my/file.txt MapPartitionsRDD[21] at textFile at <console>:27
defined class Line
lines: org.apache.spark.rdd.RDD[Line] = MapPartitionsRDD[22] at map at <console>:31
warning: there was one deprecation warning; re-run with -deprecation for details
<console>:1: error: ';' expected but ',' found.
  %sql select firstField, secondField, count(1) from lines group by firstField, secondField order by firstField, secondField


4. Interpreter log is also confusing:
INFO [2017-05-02 11:41:52,706] ({pool-2-thread-2} Logging.scala[logInfo]:54) - Warehouse location for Hive client (version 1.1.0) is file:/spark-warehouse
INFO [2017-05-02 11:41:52,983] ({pool-2-thread-2} PerfLogger.java[PerfLogBegin]:122) - <PERFLOG method=create_database from=org.apache.hadoop.hive.metastore.RetryingHMSHandler>
INFO [2017-05-02 11:41:52,984] ({pool-2-thread-2} HiveMetaStore.java[logInfo]:795) - 0: create_database: Database(name:default, description:default database, locationUri:file:/spark-warehouse, parameters:{})
INFO [2017-05-02 11:41:52,984] ({pool-2-thread-2} HiveMetaStore.java[logAuditEvent]:388) - ugi=zblenessy ip=unknown-ip-addr cmd=create_database: Database(name:default, description:default database, locationUri:file:/spark-warehouse, parameters:{})
ERROR [2017-05-02 11:41:52,992] ({pool-2-thread-2} RetryingHMSHandler.java[invokeInternal]:189) - AlreadyExistsException(message:Database default already exists)
 at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:944)

What should I do with failed metastore DB creation? Is it fine?

I'm stuck, can you give me some ideas please?

Reply | Threaded
Open this post in threaded view
|

RE: Can't run simple example with scala and spark SQL. Some non obvious syntax error in SQL

David Howell
In reply to this post by Serega Sheypak

Hi Serega,

I see this in the error log “error: ';' expected but ',' found.”

 

Are you running the %sql in the same paragraph as the %spark? I don’t think that is supported. I think you have to shift the %sql to a new paragraph, you can then run the spark and then the sql separately.

 

From: [hidden email]
Sent: Tuesday, 2 May 2017 9:58 PM
To: [hidden email]
Subject: Can't run simple example with scala and spark SQL. Some non obvious syntax error in SQL

 

Here is my sample notebook:
%spark
val linesText = sc.textFile("hdfs://cluster/user/me/lines.txt")

  case class Line(id:Long, firstField:String, secondField:String)

  val lines = linesText.map{ line =>
  val splitted = line.split(" ")
    println("splitted => " + splitted)
    Line(splitted(0).toLong, splitted(1), splitted(2))
  }

  lines.toDF().registerTempTable("lines")

  %sql select firstField, secondField, count(1) from lines group by firstField, secondField order by firstField, secondField

1. I can see that spark job was started on my YARN cluster.
2. It failed
UI shows exception. Can't understand what do I do wrong:
%sql select firstField, secondField, count(1) from lines group by firstField, secondField order by firstField, secondField ^
3. There is suspicious output in zeppelin log:

 INFO [2017-05-02 11:50:02,846] ({pool-2-thread-8} SchedulerFactory.java[jobFinished]:137) - Job paragraph_1493724118696_868476558 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session1712472970
ERROR [2017-05-02 11:50:18,809] ({qtp1286783232-166} NotebookServer.java[onMessage]:380) - Can't handle message
java.lang.NullPointerException
at org.apache.zeppelin.socket.NotebookServer.addNewParagraphIfLastParagraphIsExecuted(NotebookServer.java:1713)
at org.apache.zeppelin.socket.NotebookServer.persistAndExecuteSingleParagraph(NotebookServer.java:1741)
at org.apache.zeppelin.socket.NotebookServer.runAllParagraphs(NotebookServer.java:1641)
at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:291)
at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:59)
at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128)
at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161)
at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309)
at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214)
at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
 INFO [2017-05-02 11:50:18,811] ({pool-2-thread-14} SchedulerFactory.java[jobStarted]:131) - Job paragraph_1493724118696_868476558 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session1712472970
 INFO [2017-05-02 11:50:18,812] ({pool-2-thread-14} Paragraph.java[jobRun]:363) - run paragraph 20170502-112158_458502255 using spark org.apache.zeppelin.interpreter.LazyOpenInterpreter@24ca4045
 WARN [2017-05-02 11:50:19,810] ({pool-2-thread-14} NotebookServer.java[afterStatusChange]:2162) - Job 20170502-112158_458502255 is finished, status: ERROR, exception: null, result: %text linesText: org.apache.spark.rdd.RDD[String] = hdfs://path/to/my/file.txt MapPartitionsRDD[21] at textFile at <console>:27
defined class Line
lines: org.apache.spark.rdd.RDD[Line] = MapPartitionsRDD[22] at map at <console>:31
warning: there was one deprecation warning; re-run with -deprecation for details
<console>:1: error: ';' expected but ',' found.
  %sql select firstField, secondField, count(1) from lines group by firstField, secondField order by firstField, secondField


4. Interpreter log is also confusing:
INFO [2017-05-02 11:41:52,706] ({pool-2-thread-2} Logging.scala[logInfo]:54) - Warehouse location for Hive client (version 1.1.0) is file:/spark-warehouse
INFO [2017-05-02 11:41:52,983] ({pool-2-thread-2} PerfLogger.java[PerfLogBegin]:122) - <PERFLOG method=create_database from=org.apache.hadoop.hive.metastore.RetryingHMSHandler>
INFO [2017-05-02 11:41:52,984] ({pool-2-thread-2} HiveMetaStore.java[logInfo]:795) - 0: create_database: Database(name:default, description:default database, locationUri:file:/spark-warehouse, parameters:{})
INFO [2017-05-02 11:41:52,984] ({pool-2-thread-2} HiveMetaStore.java[logAuditEvent]:388) - ugi=zblenessy ip=unknown-ip-addr cmd=create_database: Database(name:default, description:default database, locationUri:file:/spark-warehouse, parameters:{})
ERROR [2017-05-02 11:41:52,992] ({pool-2-thread-2} RetryingHMSHandler.java[invokeInternal]:189) - AlreadyExistsException(message:Database default already exists)
 at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:944)

What should I do with failed metastore DB creation? Is it fine?

I'm stuck, can you give me some ideas please?
Reply | Threaded
Open this post in threaded view
|

Re: Can't run simple example with scala and spark SQL. Some non obvious syntax error in SQL

Serega Sheypak
Ah... sorry, so stupid. You are right. I've moved 
%sql select firstField, secondField, count(1) from lines group by firstField, secondField order by firstField, secondField

to the next paragraph and it works!

What does it mean?
zeppelin-interpreter-spark-spark interpreter complain every second on:

WARN [2017-05-02 17:05:42,396] ({dispatcher-event-loop-16} ScriptBasedMapping.java[runResolveCommand]:254) - Exception running /etc/hadoop/conf.cloudera.yarn7/topology.py 17.134.172.218 

java.io.IOException: Cannot run program "/etc/hadoop/conf.cloudera.yarn7/topology.py" (in directory "/"): error=2, No such file or directory


does spark really need it? What I'm supposed to do?


2017-05-02 15:16 GMT+02:00 David Howell <[hidden email]>:

Hi Serega,

I see this in the error log “error: ';' expected but ',' found.”

 

Are you running the %sql in the same paragraph as the %spark? I don’t think that is supported. I think you have to shift the %sql to a new paragraph, you can then run the spark and then the sql separately.

 

From: [hidden email]
Sent: Tuesday, 2 May 2017 9:58 PM
To: [hidden email]
Subject: Can't run simple example with scala and spark SQL. Some non obvious syntax error in SQL

 

Here is my sample notebook:
%spark
val linesText = sc.textFile("hdfs://cluster/user/me/lines.txt")

  case class Line(id:Long, firstField:String, secondField:String)

  val lines = linesText.map{ line =>
  val splitted = line.split(" ")
    println("splitted => " + splitted)
    Line(splitted(0).toLong, splitted(1), splitted(2))
  }

  lines.toDF().registerTempTable("lines")

  %sql select firstField, secondField, count(1) from lines group by firstField, secondField order by firstField, secondField

1. I can see that spark job was started on my YARN cluster.
2. It failed
UI shows exception. Can't understand what do I do wrong:
<div class="m_2241282945879730089gmail-tableDisplay m_2241282945879730089gmail-ng-scope" src="https://ci6.googleusercontent.com/proxy/XB7V7I05p3jC5YnshOrt2K9V9FE2z_jqfnjl5kceR8GIK804EgLZIDMcUaPS7iawZzLF06kIb_pD1BuMe_YSBcX8RCmSdXvLcwwRLBKswWxr9rhEg7P0Jg=s0-d-e1-ft#http://&#39;app/notebook/paragraph/result/result.html?v=1492775620971&#39;" style="box-sizing:border-box;margin-top:2px">
%sql select firstField, secondField, count(1) from lines group by firstField, secondField order by firstField, secondField ^
3. There is suspicious output in zeppelin log:

 INFO [2017-05-02 11:50:02,846] ({pool-2-thread-8} SchedulerFactory.java[jobFinished]:137) - Job paragraph_1493724118696_868476558 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session1712472970
ERROR [2017-05-02 11:50:18,809] ({qtp1286783232-166} NotebookServer.java[onMessage]:380) - Can't handle message
java.lang.NullPointerException
at org.apache.zeppelin.socket.NotebookServer.addNewParagraphIfLastParagraphIsExecuted(NotebookServer.java:1713)
at org.apache.zeppelin.socket.NotebookServer.persistAndExecuteSingleParagraph(NotebookServer.java:1741)
at org.apache.zeppelin.socket.NotebookServer.runAllParagraphs(NotebookServer.java:1641)
at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:291)
at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:59)
at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128)
at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161)
at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309)
at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214)
at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
 INFO [2017-05-02 11:50:18,811] ({pool-2-thread-14} SchedulerFactory.java[jobStarted]:131) - Job paragraph_1493724118696_868476558 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session1712472970
 INFO [2017-05-02 11:50:18,812] ({pool-2-thread-14} Paragraph.java[jobRun]:363) - run paragraph 20170502-112158_458502255 using spark org.apache.zeppelin.interpreter.LazyOpenInterpreter@24ca4045
 WARN [2017-05-02 11:50:19,810] ({pool-2-thread-14} NotebookServer.java[afterStatusChange]:2162) - Job 20170502-112158_458502255 is finished, status: ERROR, exception: null, result: %text linesText: org.apache.spark.rdd.RDD[String] = hdfs://path/to/my/file.txt MapPartitionsRDD[21] at textFile at <console>:27
defined class Line
lines: org.apache.spark.rdd.RDD[Line] = MapPartitionsRDD[22] at map at <console>:31
warning: there was one deprecation warning; re-run with -deprecation for details
<console>:1: error: ';' expected but ',' found.
  %sql select firstField, secondField, count(1) from lines group by firstField, secondField order by firstField, secondField


4. Interpreter log is also confusing:
INFO [2017-05-02 11:41:52,706] ({pool-2-thread-2} Logging.scala[logInfo]:54) - Warehouse location for Hive client (version 1.1.0) is file:/spark-warehouse
INFO [2017-05-02 11:41:52,983] ({pool-2-thread-2} PerfLogger.java[PerfLogBegin]:122) - <PERFLOG method=create_database from=org.apache.hadoop.hive.metastore.RetryingHMSHandler>
INFO [2017-05-02 11:41:52,984] ({pool-2-thread-2} HiveMetaStore.java[logInfo]:795) - 0: create_database: Database(name:default, description:default database, locationUri:file:/spark-warehouse, parameters:{})
INFO [2017-05-02 11:41:52,984] ({pool-2-thread-2} HiveMetaStore.java[logAuditEvent]:388) - ugi=zblenessy ip=unknown-ip-addr cmd=create_database: Database(name:default, description:default database, locationUri:file:/spark-warehouse, parameters:{})
ERROR [2017-05-02 11:41:52,992] ({pool-2-thread-2} RetryingHMSHandler.java[invokeInternal]:189) - AlreadyExistsException(message:Database default already exists)
 at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:944)

What should I do with failed metastore DB creation? Is it fine?

I'm stuck, can you give me some ideas please?

Reply | Threaded
Open this post in threaded view
|

RE: Can't run simple example with scala and spark SQL. Some non obvious syntax error in SQL

David Howell

That looks to be related to  Cloudera,  I'm sorry I can't provide any useful advice, I've not used that distro. Probably best to ask on a Cloudera forum, user group or the Cloudera support.

 

From: [hidden email]
Sent: Wednesday, 3 May 2017 3:11 AM
To: [hidden email]
Cc: [hidden email]
Subject: Re: Can't run simple example with scala and spark SQL. Some non obvious syntax error in SQL

 

Ah... sorry, so stupid. You are right. I've moved 
%sql select firstField, secondField, count(1) from lines group by firstField, secondField order by firstField, secondField

to the next paragraph and it works!

What does it mean?
zeppelin-interpreter-spark-spark interpreter complain every second on:

WARN [2017-05-02 17:05:42,396] ({dispatcher-event-loop-16} ScriptBasedMapping.java[runResolveCommand]:254) - Exception running /etc/hadoop/conf.cloudera.yarn7/topology.py 17.134.172.218 

java.io.IOException: Cannot run program "/etc/hadoop/conf.cloudera.yarn7/topology.py" (in directory "/"): error=2, No such file or directory


does spark really need it? What I'm supposed to do?


2017-05-02 15:16 GMT+02:00 David Howell <[hidden email]>:

Hi Serega,

I see this in the error log “error: ';' expected but ',' found.”

 

Are you running the %sql in the same paragraph as the %spark? I don’t think that is supported. I think you have to shift the %sql to a new paragraph, you can then run the spark and then the sql separately.

 

From: [hidden email]
Sent: Tuesday, 2 May 2017 9:58 PM
To: [hidden email]
Subject: Can't run simple example with scala and spark SQL. Some non obvious syntax error in SQL

 

Here is my sample notebook:
%spark
val linesText = sc.textFile("hdfs://cluster/user/me/lines.txt")

  case class Line(id:Long, firstField:String, secondField:String)

  val lines = linesText.map{ line =>
  val splitted = line.split(" ")
    println("splitted => " + splitted)
    Line(splitted(0).toLong, splitted(1), splitted(2))
  }

  lines.toDF().registerTempTable("lines")

  %sql select firstField, secondField, count(1) from lines group by firstField, secondField order by firstField, secondField

1. I can see that spark job was started on my YARN cluster.
2. It failed
UI shows exception. Can't understand what do I do wrong:
%sql select firstField, secondField, count(1) from lines group by firstField, secondField order by firstField, secondField ^
3. There is suspicious output in zeppelin log:

 INFO [2017-05-02 11:50:02,846] ({pool-2-thread-8} SchedulerFactory.java[jobFinished]:137) - Job paragraph_1493724118696_868476558 finished by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session1712472970
ERROR [2017-05-02 11:50:18,809] ({qtp1286783232-166} NotebookServer.java[onMessage]:380) - Can't handle message
java.lang.NullPointerException
at org.apache.zeppelin.socket.NotebookServer.addNewParagraphIfLastParagraphIsExecuted(NotebookServer.java:1713)
at org.apache.zeppelin.socket.NotebookServer.persistAndExecuteSingleParagraph(NotebookServer.java:1741)
at org.apache.zeppelin.socket.NotebookServer.runAllParagraphs(NotebookServer.java:1641)
at org.apache.zeppelin.socket.NotebookServer.onMessage(NotebookServer.java:291)
at org.apache.zeppelin.socket.NotebookSocket.onWebSocketText(NotebookSocket.java:59)
at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextMessage(JettyListenerEventDriver.java:128)
at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:65)
at org.eclipse.jetty.websocket.common.events.JettyListenerEventDriver.onTextFrame(JettyListenerEventDriver.java:122)
at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:161)
at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:309)
at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:214)
at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:220)
at org.eclipse.jetty.websocket.common.Parser.parse(Parser.java:258)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.readParse(AbstractWebSocketConnection.java:632)
at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:480)
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
 INFO [2017-05-02 11:50:18,811] ({pool-2-thread-14} SchedulerFactory.java[jobStarted]:131) - Job paragraph_1493724118696_868476558 started by scheduler org.apache.zeppelin.interpreter.remote.RemoteInterpretershared_session1712472970
 INFO [2017-05-02 11:50:18,812] ({pool-2-thread-14} Paragraph.java[jobRun]:363) - run paragraph 20170502-112158_458502255 using spark org.apache.zeppelin.interpreter.LazyOpenInterpreter@24ca4045
 WARN [2017-05-02 11:50:19,810] ({pool-2-thread-14} NotebookServer.java[afterStatusChange]:2162) - Job 20170502-112158_458502255 is finished, status: ERROR, exception: null, result: %text linesText: org.apache.spark.rdd.RDD[String] = hdfs://path/to/my/file.txt MapPartitionsRDD[21] at textFile at <console>:27
defined class Line
lines: org.apache.spark.rdd.RDD[Line] = MapPartitionsRDD[22] at map at <console>:31
warning: there was one deprecation warning; re-run with -deprecation for details
<console>:1: error: ';' expected but ',' found.
  %sql select firstField, secondField, count(1) from lines group by firstField, secondField order by firstField, secondField


4. Interpreter log is also confusing:
INFO [2017-05-02 11:41:52,706] ({pool-2-thread-2} Logging.scala[logInfo]:54) - Warehouse location for Hive client (version 1.1.0) is file:/spark-warehouse
INFO [2017-05-02 11:41:52,983] ({pool-2-thread-2} PerfLogger.java[PerfLogBegin]:122) - <PERFLOG method=create_database from=org.apache.hadoop.hive.metastore.RetryingHMSHandler>
INFO [2017-05-02 11:41:52,984] ({pool-2-thread-2} HiveMetaStore.java[logInfo]:795) - 0: create_database: Database(name:default, description:default database, locationUri:file:/spark-warehouse, parameters:{})
INFO [2017-05-02 11:41:52,984] ({pool-2-thread-2} HiveMetaStore.java[logAuditEvent]:388) - ugi=zblenessy ip=unknown-ip-addr cmd=create_database: Database(name:default, description:default database, locationUri:file:/spark-warehouse, parameters:{})
ERROR [2017-05-02 11:41:52,992] ({pool-2-thread-2} RetryingHMSHandler.java[invokeInternal]:189) - AlreadyExistsException(message:Database default already exists)
 at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_database(HiveMetaStore.java:944)

What should I do with failed metastore DB creation? Is it fine?

I'm stuck, can you give me some ideas please?