More than 2 notebooks in R failing with error sparkr intrepreter not responding

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

More than 2 notebooks in R failing with error sparkr intrepreter not responding

Meethu Mathew
Hi,

I have met with a strange issue in running R notebooks in zeppelin(0.7.2).
Spark intrepreter is in per note Scoped mode and spark version is 1.6.2

Please find the steps below to reproduce the issue:
1. Create a notebook (Note1) and run any r code in a paragraph. I ran the following code.
%r
rdf <- data.frame(c(1,2,3,4))
colnames(rdf) <- c("myCol")
sdf <- createDataFrame(sqlContext, rdf)  
withColumn(sdf, "newCol", sdf$myCol * 2.0)

2.  Create another notebook (Note2) and run any r code in a paragraph. I ran the same code as above.

Till now everything works fine.

3. Create third notebook (Note3) and run any r code in a paragraph. I ran the same code. This notebook fails with the error 
org.apache.zeppelin.interpreter.InterpreterException: sparkr is not responding

 What I understood from the analysis is that  the process created for sparkr interpreter is not getting killed properly and this makes every third model to throw an error while executing. The process will be killed on restarting the sparkr interpreter and another 2 models could be executed successfully. ie, For every third model run using the sparkr interpreter, the error is thrown. We suspect this as a limitation with zeppelin. 

Please help to solve this issue

Regards, 
Meethu Mathew

Reply | Threaded
Open this post in threaded view
|

Re: More than 2 notebooks in R failing with error sparkr intrepreter not responding

Jeff Zhang

Could you check the interpreter log ?

Meethu Mathew <[hidden email]>于2018年1月3日周三 下午3:05写道:
Hi,

I have met with a strange issue in running R notebooks in zeppelin(0.7.2).
Spark intrepreter is in per note Scoped mode and spark version is 1.6.2

Please find the steps below to reproduce the issue:
1. Create a notebook (Note1) and run any r code in a paragraph. I ran the following code.
%r
rdf <- data.frame(c(1,2,3,4))
colnames(rdf) <- c("myCol")
sdf <- createDataFrame(sqlContext, rdf)  
withColumn(sdf, "newCol", sdf$myCol * 2.0)

2.  Create another notebook (Note2) and run any r code in a paragraph. I ran the same code as above.

Till now everything works fine.

3. Create third notebook (Note3) and run any r code in a paragraph. I ran the same code. This notebook fails with the error 
org.apache.zeppelin.interpreter.InterpreterException: sparkr is not responding

 What I understood from the analysis is that  the process created for sparkr interpreter is not getting killed properly and this makes every third model to throw an error while executing. The process will be killed on restarting the sparkr interpreter and another 2 models could be executed successfully. ie, For every third model run using the sparkr interpreter, the error is thrown. We suspect this as a limitation with zeppelin. 

Please help to solve this issue

Regards, 


Meethu Mathew

Reply | Threaded
Open this post in threaded view
|

Re: More than 2 notebooks in R failing with error sparkr intrepreter not responding

Meethu Mathew
Hi Jeff,

PFB the interpreter log.
 
INFO [2018-01-03 12:10:05,960] ({pool-2-thread-9} Logging.scala[logInfo]:58) - Starting HTTP Server
 INFO [2018-01-03 12:10:05,961] ({pool-2-thread-9} Server.java[doStart]:272) - jetty-8.y.z-SNAPSHOT
 INFO [2018-01-03 12:10:05,963] ({pool-2-thread-9} AbstractConnector.java[doStart]:338) - Started SocketConnector@0.0.0.0:58989
 INFO [2018-01-03 12:10:05,963] ({pool-2-thread-9} Logging.scala[logInfo]:58) - Successfully started service 'HTTP class server' on port 58989.
 INFO [2018-01-03 12:10:06,094] ({dispatcher-event-loop-1} Logging.scala[logInfo]:58) - Removed broadcast_1_piece0 on localhost:42453 in memory (size: 854.0 B, free: 511.1 MB)
 INFO [2018-01-03 12:10:07,049] ({pool-2-thread-9} ZeppelinR.java[createRScript]:353) - File /tmp/zeppelin_sparkr-5046601627391341672.R created
ERROR [2018-01-03 12:10:17,051] ({pool-2-thread-9} Job.java[run]:188) - Job failed
org.apache.zeppelin.interpreter.InterpreterException: sparkr is not responding 


R version 3.4.1 (2017-06-30) -- "Single Candle"
Copyright (C) 2017 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
....
....
> args <- commandArgs(trailingOnly = TRUE)

> hashCode <- as.integer(args[1])

> port <- as.integer(args[2])

> libPath <- args[3]

> version <- as.integer(args[4])

> rm(args)

> print(paste("Port ", toString(port)))
[1]
 "Port  58063"

> print(paste("LibPath ", libPath))
[1]

 "LibPath  /home/meethu/spark-1.6.1-bin-hadoop2.6/R/lib"

> .libPaths(c(file.path(libPath), .libPaths()))

> library(SparkR)

Attaching package: ‘SparkR’

The following objects are masked from ‘package:stats’:

    cov, filter, lag, na.omit, predict, sd, var

The following objects are masked from ‘package:base’:

    colnames, colnames<-, endsWith, intersect, rank, rbind, sample,
    startsWith, subset, summary, table, transform

> SparkR:::connectBackend("localhost", port, 6000)
A connection with                            
description "->localhost:58063"
class      
 "sockconn"         
mode        "wb"               
text        "binary"           
opened      "opened"           
can read    "yes"              
can write   "yes"              

> # scStartTime is needed by R/pkg/R/sparkR.R

> assign(".scStartTime", as.integer(Sys.time()), envir = SparkR:::.sparkREnv)

> # getZeppelinR

> .zeppelinR = SparkR:::callJStatic("org.apache.zeppelin.spark.ZeppelinR", "getZeppelinR", hashCode)

at org.apache.zeppelin.spark.ZeppelinR.waitForRScriptInitialized(ZeppelinR.java:285)
at org.apache.zeppelin.spark.ZeppelinR.request(ZeppelinR.java:227)
at org.apache.zeppelin.spark.ZeppelinR.eval(ZeppelinR.java:176)
at org.apache.zeppelin.spark.ZeppelinR.open(ZeppelinR.java:165)
at org.apache.zeppelin.spark.SparkRInterpreter.open(SparkRInterpreter.java:90)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:491)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
 INFO [2018-01-03 12:10:17,070] ({pool-2-thread-9} SchedulerFactory.java[jobFinished]:137) - Job remoteInterpretJob_1514961605951 finished by scheduler org.apache.zeppelin.spark.SparkRInterpreter392022746
 INFO [2018-01-03 12:39:22,664] ({Spark Context Cleaner} Logging.scala[logInfo]:58) - Cleaned accumulator 2


PFB the output of the command  ps -ef | grep /usr/lib/R/bin/exec/R

meethu    6647  6470  0 12:09 pts/1    00:00:00 /usr/lib/R/bin/exec/R --no-save --no-restore -f /tmp/zeppelin_sparkr-1100854828050763213.R --args 214655664 58063 /home/meethu/spark-1.6.1-bin-hadoop2.6/R/lib 10601
meethu    6701  6470  0 12:09 pts/1    00:00:00 /usr/lib/R/bin/exec/R --no-save --no-restore -f /tmp/zeppelin_sparkr-4152305170353311178.R --args 1642312173 58063 /home/meethu/spark-1.6.1-bin-hadoop2.6/R/lib 10601
meethu    6745  6470  0 12:10 pts/1    00:00:00 /usr/lib/R/bin/exec/R --no-save --no-restore -f /tmp/zeppelin_sparkr-5046601627391341672.R --args 1158632477 58063 /home/meethu/spark-1.6.1-bin-hadoop2.6/R/lib 10601


Regards, 
Meethu Mathew


On Wed, Jan 3, 2018 at 12:56 PM, Jeff Zhang <[hidden email]> wrote:

Could you check the interpreter log ?

Meethu Mathew <[hidden email]>于2018年1月3日周三 下午3:05写道:
Hi,

I have met with a strange issue in running R notebooks in zeppelin(0.7.2).
Spark intrepreter is in per note Scoped mode and spark version is 1.6.2

Please find the steps below to reproduce the issue:
1. Create a notebook (Note1) and run any r code in a paragraph. I ran the following code.
%r
rdf <- data.frame(c(1,2,3,4))
colnames(rdf) <- c("myCol")
sdf <- createDataFrame(sqlContext, rdf)  
withColumn(sdf, "newCol", sdf$myCol * 2.0)

2.  Create another notebook (Note2) and run any r code in a paragraph. I ran the same code as above.

Till now everything works fine.

3. Create third notebook (Note3) and run any r code in a paragraph. I ran the same code. This notebook fails with the error 
org.apache.zeppelin.interpreter.InterpreterException: sparkr is not responding

 What I understood from the analysis is that  the process created for sparkr interpreter is not getting killed properly and this makes every third model to throw an error while executing. The process will be killed on restarting the sparkr interpreter and another 2 models could be executed successfully. ie, For every third model run using the sparkr interpreter, the error is thrown. We suspect this as a limitation with zeppelin. 

Please help to solve this issue

Regards, 


Meethu Mathew