Illegal Inheritance error

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Illegal Inheritance error

Ben Vogan
Hi all,

I've been using Zeppelin for a couple of weeks now with a stable configuration, but all of a sudden I am getting "Illegal inheritance" errors like so:

 INFO [2017-05-14 03:25:32,678] ({pool-2-thread-56} Paragraph.java[jobRun]:362) - run paragraph 20170514-032326_663206142 using livy org.apache.zeppelin.interpreter.LazyOpenInterpreter@505a171c
 WARN [2017-05-14 03:25:33,696] ({pool-2-thread-56} NotebookServer.java[afterStatusChange]:2058) - Job 20170514-032326_663206142 is finished, status: ERROR, exception: null, result: %text <console>:4: error: illegal inheritance;

It happens across multiple notebooks and across by my spark and livy interpreters.  I don't know where to look for more information about what is wrong.  I don't see any errors in spark/yarn at all.  The driver got created, but it looks like no jobs were ever submitted to spark.

Help would be greatly appreciated.

Thanks,

--
BENJAMIN VOGAN | Data Platform Team Lead

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Illegal Inheritance error

Ben Vogan
Hi all,

For some reason today I'm getting a stack:

org.apache.zeppelin.livy.LivyException: Fail to create SQLContext,<console>:4: error: illegal inheritance;
at org.apache.zeppelin.livy.LivySparkSQLInterpreter.open(LivySparkSQLInterpreter.java:76)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:483)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

On the Livy server I see no errors and there is an open session on yarn.

Some help on this would be greatly appreciated! 

--Ben

On Sun, May 14, 2017 at 6:16 AM, Ben Vogan <[hidden email]> wrote:
Hi all,

I've been using Zeppelin for a couple of weeks now with a stable configuration, but all of a sudden I am getting "Illegal inheritance" errors like so:

 INFO [2017-05-14 03:25:32,678] ({pool-2-thread-56} Paragraph.java[jobRun]:362) - run paragraph 20170514-032326_663206142 using livy org.apache.zeppelin.interpreter.LazyOpenInterpreter@505a171c
 WARN [2017-05-14 03:25:33,696] ({pool-2-thread-56} NotebookServer.java[afterStatusChange]:2058) - Job 20170514-032326_663206142 is finished, status: ERROR, exception: null, result: %text <console>:4: error: illegal inheritance;

It happens across multiple notebooks and across by my spark and livy interpreters.  I don't know where to look for more information about what is wrong.  I don't see any errors in spark/yarn at all.  The driver got created, but it looks like no jobs were ever submitted to spark.

Help would be greatly appreciated.

Thanks,

--
BENJAMIN VOGAN | Data Platform Team Lead




--
BENJAMIN VOGAN | Data Platform Team Lead

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Illegal Inheritance error

Jeff Zhang

Which version of zeppelin do you use ? And can you check the yarn app log ?


Ben Vogan <[hidden email]>于2017年5月15日周一 下午5:56写道:
Hi all,

For some reason today I'm getting a stack:

org.apache.zeppelin.livy.LivyException: Fail to create SQLContext,<console>:4: error: illegal inheritance;
at org.apache.zeppelin.livy.LivySparkSQLInterpreter.open(LivySparkSQLInterpreter.java:76)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:483)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

On the Livy server I see no errors and there is an open session on yarn.

Some help on this would be greatly appreciated! 

--Ben

On Sun, May 14, 2017 at 6:16 AM, Ben Vogan <[hidden email]> wrote:
Hi all,

I've been using Zeppelin for a couple of weeks now with a stable configuration, but all of a sudden I am getting "Illegal inheritance" errors like so:

 INFO [2017-05-14 03:25:32,678] ({pool-2-thread-56} Paragraph.java[jobRun]:362) - run paragraph 20170514-032326_663206142 using livy org.apache.zeppelin.interpreter.LazyOpenInterpreter@505a171c
 WARN [2017-05-14 03:25:33,696] ({pool-2-thread-56} NotebookServer.java[afterStatusChange]:2058) - Job 20170514-032326_663206142 is finished, status: ERROR, exception: null, result: %text <console>:4: error: illegal inheritance;

It happens across multiple notebooks and across by my spark and livy interpreters.  I don't know where to look for more information about what is wrong.  I don't see any errors in spark/yarn at all.  The driver got created, but it looks like no jobs were ever submitted to spark.

Help would be greatly appreciated.

Thanks,

--
BENJAMIN VOGAN | Data Platform Team Lead




--
BENJAMIN VOGAN | Data Platform Team Lead

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Illegal Inheritance error

Ben Vogan
I am using 0.7.1 and I checked the yarn app log and don't see any errors.  It looks like this:

17/05/16 00:04:12 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
17/05/16 00:04:13 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1494373289850_0336_000001
17/05/16 00:04:13 INFO spark.SecurityManager: Changing view acls to: yarn,hdfs
17/05/16 00:04:13 INFO spark.SecurityManager: Changing modify acls to: yarn,hdfs
17/05/16 00:04:13 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, hdfs); users with modify permissions: Set(yarn, hdfs)
17/05/16 00:04:13 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
17/05/16 00:04:13 INFO yarn.ApplicationMaster: Waiting for spark context initialization
17/05/16 00:04:13 INFO yarn.ApplicationMaster: Waiting for spark context initialization ... 
17/05/16 00:04:14 INFO driver.RSCDriver: Connecting to: jarvis-hue002.internal.shopkick.com:40819
17/05/16 00:04:14 INFO driver.RSCDriver: Starting RPC server...
17/05/16 00:04:14 WARN rsc.RSCConf: Your hostname, jarvis-yarn008.internal.shopkick.com, resolves to a loopback address, but we couldn't find any external IP address!
17/05/16 00:04:14 WARN rsc.RSCConf: Set livy.rsc.rpc.server.address if you need to bind to another address.
17/05/16 00:04:14 INFO driver.RSCDriver: Received job request cd7d1356-709d-4674-a85c-21edade2c38d
17/05/16 00:04:14 INFO driver.RSCDriver: SparkContext not yet up, queueing job request.
17/05/16 00:04:17 INFO spark.SparkContext: Running Spark version 1.6.0
17/05/16 00:04:17 INFO spark.SecurityManager: Changing view acls to: yarn,hdfs
17/05/16 00:04:17 INFO spark.SecurityManager: Changing modify acls to: yarn,hdfs
17/05/16 00:04:17 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, hdfs); users with modify permissions: Set(yarn, hdfs)
17/05/16 00:04:17 INFO util.Utils: Successfully started service 'sparkDriver' on port 53267.
17/05/16 00:04:18 INFO slf4j.Slf4jLogger: Slf4jLogger started
17/05/16 00:04:18 INFO Remoting: Starting remoting
17/05/16 00:04:18 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@10.19.194.147:38037]
17/05/16 00:04:18 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriverActorSystem@10.19.194.147:38037]
17/05/16 00:04:18 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 38037.
17/05/16 00:04:18 INFO spark.SparkEnv: Registering MapOutputTracker
17/05/16 00:04:18 INFO spark.SparkEnv: Registering BlockManagerMaster
17/05/16 00:04:18 INFO storage.DiskBlockManager: Created local directory at /yarn/nm/usercache/hdfs/appcache/application_1494373289850_0336/blockmgr-f46429a6-7466-42c1-bd79-9ddf6ec61cb4
17/05/16 00:04:18 INFO storage.MemoryStore: MemoryStore started with capacity 1966.1 MB
17/05/16 00:04:18 INFO spark.SparkEnv: Registering OutputCommitCoordinator
17/05/16 00:04:18 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
17/05/16 00:04:18 INFO util.Utils: Successfully started service 'SparkUI' on port 49024.
17/05/16 00:04:18 INFO ui.SparkUI: Started SparkUI at http://10.19.194.147:49024
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR file:/services/livy-server/livy-server-current/rsc-jars/livy-api-0.3.0.jar at spark://10.19.194.147:53267/jars/livy-api-0.3.0.jar with timestamp 1494893058608
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR file:/services/livy-server/livy-server-current/rsc-jars/livy-rsc-0.3.0.jar at spark://10.19.194.147:53267/jars/livy-rsc-0.3.0.jar with timestamp 1494893058609
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR file:/services/livy-server/livy-server-current/rsc-jars/netty-all-4.0.29.Final.jar at spark://10.19.194.147:53267/jars/netty-all-4.0.29.Final.jar with timestamp 1494893058609
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR hdfs://jarvis-nameservice001/jarvis_pipelines/vertica-jdbc-7.1.2-0.jar at hdfs://jarvis-nameservice001/jarvis_pipelines/vertica-jdbc-7.1.2-0.jar with timestamp 1494893058609
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR hdfs://jarvis-nameservice001/jarvis_pipelines/shopkick-data-pipeline.jar at hdfs://jarvis-nameservice001/jarvis_pipelines/shopkick-data-pipeline.jar with timestamp 1494893058609
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR file:/services/livy-server/livy-server-current/repl_2.10-jars/commons-codec-1.9.jar at spark://10.19.194.147:53267/jars/commons-codec-1.9.jar with timestamp 1494893058609
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR file:/services/livy-server/livy-server-current/repl_2.10-jars/livy-repl_2.10-0.3.0.jar at spark://10.19.194.147:53267/jars/livy-repl_2.10-0.3.0.jar with timestamp 1494893058609
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR file:/services/livy-server/livy-server-current/repl_2.10-jars/livy-core_2.10-0.3.0.jar at spark://10.19.194.147:53267/jars/livy-core_2.10-0.3.0.jar with timestamp 1494893058609
17/05/16 00:04:18 INFO cluster.YarnClusterScheduler: Created YarnClusterScheduler
17/05/16 00:04:18 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 57551.
17/05/16 00:04:18 INFO netty.NettyBlockTransferService: Server created on 57551
17/05/16 00:04:18 INFO storage.BlockManager: external shuffle service port = 7337
17/05/16 00:04:18 INFO storage.BlockManagerMaster: Trying to register BlockManager
17/05/16 00:04:18 INFO storage.BlockManagerMasterEndpoint: Registering block manager 10.19.194.147:57551 with 1966.1 MB RAM, BlockManagerId(driver, 10.19.194.147, 57551)
17/05/16 00:04:18 INFO storage.BlockManagerMaster: Registered BlockManager
17/05/16 00:04:19 INFO scheduler.EventLoggingListener: Logging events to hdfs://jarvis-nameservice001/user/spark/applicationHistory/application_1494373289850_0336_1
17/05/16 00:04:19 INFO cluster.YarnClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
17/05/16 00:04:19 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done
17/05/16 00:04:19 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark://YarnAM@10.19.194.147:53267)
17/05/16 00:04:19 INFO yarn.YarnRMClient: Registering the ApplicationMaster
17/05/16 00:04:19 INFO yarn.ApplicationMaster: Started progress reporter thread with (heartbeat : 3000, initial allocation : 200) intervals
17/05/16 00:04:19 INFO hive.HiveContext: Initializing execution hive, version 1.1.0
17/05/16 00:04:19 INFO client.ClientWrapper: Inspected Hadoop version: 2.6.0-cdh5.7.0
17/05/16 00:04:19 INFO client.ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0-cdh5.7.0
17/05/16 00:04:20 INFO hive.metastore: Trying to connect to metastore with URI thrift://jarvis-hdfs003.internal.shopkick.com:9083
17/05/16 00:04:20 INFO hive.metastore: Opened a connection to metastore, current connections: 1
17/05/16 00:04:20 INFO hive.metastore: Connected to metastore.
17/05/16 00:04:20 INFO session.SessionState: Created HDFS directory: file:/yarn/nm/usercache/hdfs/appcache/application_1494373289850_0336/container_e14_1494373289850_0336_01_000001/tmp/spark-2217d267-a3c0-4cf4-9565-45f80517d41c/scratch/hdfs
17/05/16 00:04:20 INFO session.SessionState: Created local directory: /yarn/nm/usercache/hdfs/appcache/application_1494373289850_0336/container_e14_1494373289850_0336_01_000001/tmp/yarn
17/05/16 00:04:20 INFO session.SessionState: Created local directory: /yarn/nm/usercache/hdfs/appcache/application_1494373289850_0336/container_e14_1494373289850_0336_01_000001/tmp/478f39e9-5295-4e8e-97aa-40b5828f9440_resources
17/05/16 00:04:20 INFO session.SessionState: Created HDFS directory: file:/yarn/nm/usercache/hdfs/appcache/application_1494373289850_0336/container_e14_1494373289850_0336_01_000001/tmp/spark-2217d267-a3c0-4cf4-9565-45f80517d41c/scratch/hdfs/478f39e9-5295-4e8e-97aa-40b5828f9440
17/05/16 00:04:20 INFO session.SessionState: Created local directory: /yarn/nm/usercache/hdfs/appcache/application_1494373289850_0336/container_e14_1494373289850_0336_01_000001/tmp/yarn/478f39e9-5295-4e8e-97aa-40b5828f9440
17/05/16 00:04:20 INFO session.SessionState: Created HDFS directory: file:/yarn/nm/usercache/hdfs/appcache/application_1494373289850_0336/container_e14_1494373289850_0336_01_000001/tmp/spark-2217d267-a3c0-4cf4-9565-45f80517d41c/scratch/hdfs/478f39e9-5295-4e8e-97aa-40b5828f9440/_tmp_space.db
17/05/16 00:04:20 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr.
17/05/16 00:04:20 INFO repl.SparkInterpreter: Created sql context (with Hive support).

On Mon, May 15, 2017 at 5:43 PM, Jeff Zhang <[hidden email]> wrote:

Which version of zeppelin do you use ? And can you check the yarn app log ?


Ben Vogan <[hidden email]>于2017年5月15日周一 下午5:56写道:
Hi all,

For some reason today I'm getting a stack:

org.apache.zeppelin.livy.LivyException: Fail to create SQLContext,<console>:4: error: illegal inheritance;
at org.apache.zeppelin.livy.LivySparkSQLInterpreter.open(LivySparkSQLInterpreter.java:76)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:483)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

On the Livy server I see no errors and there is an open session on yarn.

Some help on this would be greatly appreciated! 

--Ben

On Sun, May 14, 2017 at 6:16 AM, Ben Vogan <[hidden email]> wrote:
Hi all,

I've been using Zeppelin for a couple of weeks now with a stable configuration, but all of a sudden I am getting "Illegal inheritance" errors like so:

 INFO [2017-05-14 03:25:32,678] ({pool-2-thread-56} Paragraph.java[jobRun]:362) - run paragraph 20170514-032326_663206142 using livy org.apache.zeppelin.interpreter.LazyOpenInterpreter@505a171c
 WARN [2017-05-14 03:25:33,696] ({pool-2-thread-56} NotebookServer.java[afterStatusChange]:2058) - Job 20170514-032326_663206142 is finished, status: ERROR, exception: null, result: %text <console>:4: error: illegal inheritance;

It happens across multiple notebooks and across by my spark and livy interpreters.  I don't know where to look for more information about what is wrong.  I don't see any errors in spark/yarn at all.  The driver got created, but it looks like no jobs were ever submitted to spark.

Help would be greatly appreciated.

Thanks,

--
BENJAMIN VOGAN | Data Platform Team Lead




--
BENJAMIN VOGAN | Data Platform Team Lead




--
BENJAMIN VOGAN | Data Platform Team Lead

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Illegal Inheritance error

Jeff Zhang

It is weird that the yarn app log shows the SQLContext is created successfully, but in zeppelin side it shows error of "Fail to create SQLContext"

Ben Vogan <[hidden email]>于2017年5月15日周一 下午8:07写道:
I am using 0.7.1 and I checked the yarn app log and don't see any errors.  It looks like this:

17/05/16 00:04:12 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
17/05/16 00:04:13 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1494373289850_0336_000001
17/05/16 00:04:13 INFO spark.SecurityManager: Changing view acls to: yarn,hdfs
17/05/16 00:04:13 INFO spark.SecurityManager: Changing modify acls to: yarn,hdfs
17/05/16 00:04:13 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, hdfs); users with modify permissions: Set(yarn, hdfs)
17/05/16 00:04:13 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
17/05/16 00:04:13 INFO yarn.ApplicationMaster: Waiting for spark context initialization
17/05/16 00:04:13 INFO yarn.ApplicationMaster: Waiting for spark context initialization ... 
17/05/16 00:04:14 INFO driver.RSCDriver: Connecting to: jarvis-hue002.internal.shopkick.com:40819
17/05/16 00:04:14 INFO driver.RSCDriver: Starting RPC server...
17/05/16 00:04:14 WARN rsc.RSCConf: Your hostname, jarvis-yarn008.internal.shopkick.com, resolves to a loopback address, but we couldn't find any external IP address!
17/05/16 00:04:14 WARN rsc.RSCConf: Set livy.rsc.rpc.server.address if you need to bind to another address.
17/05/16 00:04:14 INFO driver.RSCDriver: Received job request cd7d1356-709d-4674-a85c-21edade2c38d
17/05/16 00:04:14 INFO driver.RSCDriver: SparkContext not yet up, queueing job request.
17/05/16 00:04:17 INFO spark.SparkContext: Running Spark version 1.6.0
17/05/16 00:04:17 INFO spark.SecurityManager: Changing view acls to: yarn,hdfs
17/05/16 00:04:17 INFO spark.SecurityManager: Changing modify acls to: yarn,hdfs
17/05/16 00:04:17 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, hdfs); users with modify permissions: Set(yarn, hdfs)
17/05/16 00:04:17 INFO util.Utils: Successfully started service 'sparkDriver' on port 53267.
17/05/16 00:04:18 INFO slf4j.Slf4jLogger: Slf4jLogger started
17/05/16 00:04:18 INFO Remoting: Starting remoting
17/05/16 00:04:18 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@10.19.194.147:38037]
17/05/16 00:04:18 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriverActorSystem@10.19.194.147:38037]
17/05/16 00:04:18 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 38037.
17/05/16 00:04:18 INFO spark.SparkEnv: Registering MapOutputTracker
17/05/16 00:04:18 INFO spark.SparkEnv: Registering BlockManagerMaster
17/05/16 00:04:18 INFO storage.DiskBlockManager: Created local directory at /yarn/nm/usercache/hdfs/appcache/application_1494373289850_0336/blockmgr-f46429a6-7466-42c1-bd79-9ddf6ec61cb4
17/05/16 00:04:18 INFO storage.MemoryStore: MemoryStore started with capacity 1966.1 MB
17/05/16 00:04:18 INFO spark.SparkEnv: Registering OutputCommitCoordinator
17/05/16 00:04:18 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
17/05/16 00:04:18 INFO util.Utils: Successfully started service 'SparkUI' on port 49024.
17/05/16 00:04:18 INFO ui.SparkUI: Started SparkUI at http://10.19.194.147:49024
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR file:/services/livy-server/livy-server-current/rsc-jars/livy-api-0.3.0.jar at spark://10.19.194.147:53267/jars/livy-api-0.3.0.jar with timestamp 1494893058608
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR file:/services/livy-server/livy-server-current/rsc-jars/livy-rsc-0.3.0.jar at spark://10.19.194.147:53267/jars/livy-rsc-0.3.0.jar with timestamp 1494893058609
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR file:/services/livy-server/livy-server-current/rsc-jars/netty-all-4.0.29.Final.jar at spark://10.19.194.147:53267/jars/netty-all-4.0.29.Final.jar with timestamp 1494893058609
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR hdfs://jarvis-nameservice001/jarvis_pipelines/vertica-jdbc-7.1.2-0.jar at hdfs://jarvis-nameservice001/jarvis_pipelines/vertica-jdbc-7.1.2-0.jar with timestamp 1494893058609
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR hdfs://jarvis-nameservice001/jarvis_pipelines/shopkick-data-pipeline.jar at hdfs://jarvis-nameservice001/jarvis_pipelines/shopkick-data-pipeline.jar with timestamp 1494893058609
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR file:/services/livy-server/livy-server-current/repl_2.10-jars/commons-codec-1.9.jar at spark://10.19.194.147:53267/jars/commons-codec-1.9.jar with timestamp 1494893058609
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR file:/services/livy-server/livy-server-current/repl_2.10-jars/livy-repl_2.10-0.3.0.jar at spark://10.19.194.147:53267/jars/livy-repl_2.10-0.3.0.jar with timestamp 1494893058609
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR file:/services/livy-server/livy-server-current/repl_2.10-jars/livy-core_2.10-0.3.0.jar at spark://10.19.194.147:53267/jars/livy-core_2.10-0.3.0.jar with timestamp 1494893058609
17/05/16 00:04:18 INFO cluster.YarnClusterScheduler: Created YarnClusterScheduler
17/05/16 00:04:18 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 57551.
17/05/16 00:04:18 INFO netty.NettyBlockTransferService: Server created on 57551
17/05/16 00:04:18 INFO storage.BlockManager: external shuffle service port = 7337
17/05/16 00:04:18 INFO storage.BlockManagerMaster: Trying to register BlockManager
17/05/16 00:04:18 INFO storage.BlockManagerMasterEndpoint: Registering block manager 10.19.194.147:57551 with 1966.1 MB RAM, BlockManagerId(driver, 10.19.194.147, 57551)
17/05/16 00:04:18 INFO storage.BlockManagerMaster: Registered BlockManager
17/05/16 00:04:19 INFO scheduler.EventLoggingListener: Logging events to hdfs://jarvis-nameservice001/user/spark/applicationHistory/application_1494373289850_0336_1
17/05/16 00:04:19 INFO cluster.YarnClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
17/05/16 00:04:19 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done
17/05/16 00:04:19 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark://YarnAM@10.19.194.147:53267)
17/05/16 00:04:19 INFO yarn.YarnRMClient: Registering the ApplicationMaster
17/05/16 00:04:19 INFO yarn.ApplicationMaster: Started progress reporter thread with (heartbeat : 3000, initial allocation : 200) intervals
17/05/16 00:04:19 INFO hive.HiveContext: Initializing execution hive, version 1.1.0
17/05/16 00:04:19 INFO client.ClientWrapper: Inspected Hadoop version: 2.6.0-cdh5.7.0
17/05/16 00:04:19 INFO client.ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0-cdh5.7.0
17/05/16 00:04:20 INFO hive.metastore: Trying to connect to metastore with URI thrift://jarvis-hdfs003.internal.shopkick.com:9083
17/05/16 00:04:20 INFO hive.metastore: Opened a connection to metastore, current connections: 1
17/05/16 00:04:20 INFO hive.metastore: Connected to metastore.
17/05/16 00:04:20 INFO session.SessionState: Created HDFS directory: file:/yarn/nm/usercache/hdfs/appcache/application_1494373289850_0336/container_e14_1494373289850_0336_01_000001/tmp/spark-2217d267-a3c0-4cf4-9565-45f80517d41c/scratch/hdfs
17/05/16 00:04:20 INFO session.SessionState: Created local directory: /yarn/nm/usercache/hdfs/appcache/application_1494373289850_0336/container_e14_1494373289850_0336_01_000001/tmp/yarn
17/05/16 00:04:20 INFO session.SessionState: Created local directory: /yarn/nm/usercache/hdfs/appcache/application_1494373289850_0336/container_e14_1494373289850_0336_01_000001/tmp/478f39e9-5295-4e8e-97aa-40b5828f9440_resources
17/05/16 00:04:20 INFO session.SessionState: Created HDFS directory: file:/yarn/nm/usercache/hdfs/appcache/application_1494373289850_0336/container_e14_1494373289850_0336_01_000001/tmp/spark-2217d267-a3c0-4cf4-9565-45f80517d41c/scratch/hdfs/478f39e9-5295-4e8e-97aa-40b5828f9440
17/05/16 00:04:20 INFO session.SessionState: Created local directory: /yarn/nm/usercache/hdfs/appcache/application_1494373289850_0336/container_e14_1494373289850_0336_01_000001/tmp/yarn/478f39e9-5295-4e8e-97aa-40b5828f9440
17/05/16 00:04:20 INFO session.SessionState: Created HDFS directory: file:/yarn/nm/usercache/hdfs/appcache/application_1494373289850_0336/container_e14_1494373289850_0336_01_000001/tmp/spark-2217d267-a3c0-4cf4-9565-45f80517d41c/scratch/hdfs/478f39e9-5295-4e8e-97aa-40b5828f9440/_tmp_space.db
17/05/16 00:04:20 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr.
17/05/16 00:04:20 INFO repl.SparkInterpreter: Created sql context (with Hive support).

On Mon, May 15, 2017 at 5:43 PM, Jeff Zhang <[hidden email]> wrote:

Which version of zeppelin do you use ? And can you check the yarn app log ?


Ben Vogan <[hidden email]>于2017年5月15日周一 下午5:56写道:
Hi all,

For some reason today I'm getting a stack:

org.apache.zeppelin.livy.LivyException: Fail to create SQLContext,<console>:4: error: illegal inheritance;
at org.apache.zeppelin.livy.LivySparkSQLInterpreter.open(LivySparkSQLInterpreter.java:76)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:483)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

On the Livy server I see no errors and there is an open session on yarn.

Some help on this would be greatly appreciated! 

--Ben

On Sun, May 14, 2017 at 6:16 AM, Ben Vogan <[hidden email]> wrote:
Hi all,

I've been using Zeppelin for a couple of weeks now with a stable configuration, but all of a sudden I am getting "Illegal inheritance" errors like so:

 INFO [2017-05-14 03:25:32,678] ({pool-2-thread-56} Paragraph.java[jobRun]:362) - run paragraph 20170514-032326_663206142 using livy org.apache.zeppelin.interpreter.LazyOpenInterpreter@505a171c
 WARN [2017-05-14 03:25:33,696] ({pool-2-thread-56} NotebookServer.java[afterStatusChange]:2058) - Job 20170514-032326_663206142 is finished, status: ERROR, exception: null, result: %text <console>:4: error: illegal inheritance;

It happens across multiple notebooks and across by my spark and livy interpreters.  I don't know where to look for more information about what is wrong.  I don't see any errors in spark/yarn at all.  The driver got created, but it looks like no jobs were ever submitted to spark.

Help would be greatly appreciated.

Thanks,

--
BENJAMIN VOGAN | Data Platform Team Lead




--
BENJAMIN VOGAN | Data Platform Team Lead




--
BENJAMIN VOGAN | Data Platform Team Lead

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Illegal Inheritance error

Ben Vogan
I know - this is driving me crazy.  It was working fine and without me touching any of it (zeppelin/livy/spark/yarn), it broke.  And with no errors in spark/yarn or livy.  I see a warning in the livy log about the hive-site.xml not being found.  In the interpreter configuration I have tried setting livy.repl.enableHiveContext to both true and false and it still appears to always create a hive context.  I'm not sure how to get rid of that warning - I tried putting the hive-site.xml into the spark conf directory on the livy host but that broke things entirely on the yarn side - claimed the file didn't exist (maybe it would have to be put on all the yarn executor hosts as well?).

17/05/16 19:47:39 WARN InteractiveSession$: Enable HiveContext but no hive-site.xml found under classpath or user request.
17/05/16 19:47:39 INFO InteractiveSession$: Creating LivyClient for sessionId: 26
17/05/16 19:47:39 WARN RSCConf: Your hostname, jarvis-hue002.internal.shopkick.com, resolves to a loopback address, but we couldn't find any external IP address!
17/05/16 19:47:39 WARN RSCConf: Set livy.rsc.rpc.server.address if you need to bind to another address.
17/05/16 19:47:39 INFO InteractiveSessionManager: Registering new session 26
17/05/16 19:47:39 INFO ContextLauncher: WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/spark) overrides detected (/opt/cloudera/parcels/CDH/lib/spark).
17/05/16 19:47:39 INFO ContextLauncher: WARNING: Running spark-class from user-defined location.
17/05/16 19:47:41 INFO ContextLauncher: 17/05/16 19:47:41 INFO yarn.Client: Requesting a new application from cluster with 12 NodeManagers
17/05/16 19:47:41 INFO ContextLauncher: 17/05/16 19:47:41 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (38912 MB per container)
17/05/16 19:47:41 INFO ContextLauncher: 17/05/16 19:47:41 INFO yarn.Client: Will allocate AM container, with 4505 MB memory including 409 MB overhead
17/05/16 19:47:41 INFO ContextLauncher: 17/05/16 19:47:41 INFO yarn.Client: Setting up container launch context for our AM
17/05/16 19:47:41 INFO ContextLauncher: 17/05/16 19:47:41 INFO yarn.Client: Setting up the launch environment for our AM container
17/05/16 19:47:41 INFO ContextLauncher: 17/05/16 19:47:41 INFO yarn.Client: Preparing resources for our AM container
17/05/16 19:47:41 INFO ContextLauncher: 17/05/16 19:47:41 INFO yarn.Client: Uploading resource file:/services/livy-server/livy-server-current/rsc-jars/livy-api-0.3.0.jar -> hdfs://jarvis-nameservice001/user/hdfs/.sparkStaging/application_1494373289850_0386/livy-api-0.3.0.jar
17/05/16 19:47:41 INFO ContextLauncher: 17/05/16 19:47:41 INFO yarn.Client: Uploading resource file:/services/livy-server/livy-server-current/rsc-jars/livy-rsc-0.3.0.jar -> hdfs://jarvis-nameservice001/user/hdfs/.sparkStaging/application_1494373289850_0386/livy-rsc-0.3.0.jar
17/05/16 19:47:42 INFO ContextLauncher: 17/05/16 19:47:42 INFO yarn.Client: Uploading resource file:/services/livy-server/livy-server-current/rsc-jars/netty-all-4.0.29.Final.jar -> hdfs://jarvis-nameservice001/user/hdfs/.sparkStaging/application_1494373289850_0386/netty-all-4.0.29.Final.jar
17/05/16 19:47:42 INFO ContextLauncher: 17/05/16 19:47:42 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://jarvis-nameservice001/jarvis_pipelines/vertica-jdbc-7.1.2-0.jar
17/05/16 19:47:42 INFO ContextLauncher: 17/05/16 19:47:42 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://jarvis-nameservice001/jarvis_pipelines/shopkick-data-pipeline.jar
17/05/16 19:47:42 INFO ContextLauncher: 17/05/16 19:47:42 INFO yarn.Client: Uploading resource file:/services/livy-server/livy-server-current/repl_2.10-jars/commons-codec-1.9.jar -> hdfs://jarvis-nameservice001/user/hdfs/.sparkStaging/application_1494373289850_0386/commons-codec-1.9.jar
17/05/16 19:47:42 INFO ContextLauncher: 17/05/16 19:47:42 INFO yarn.Client: Uploading resource file:/services/livy-server/livy-server-current/repl_2.10-jars/livy-repl_2.10-0.3.0.jar -> hdfs://jarvis-nameservice001/user/hdfs/.sparkStaging/application_1494373289850_0386/livy-repl_2.10-0.3.0.jar
17/05/16 19:47:42 INFO ContextLauncher: 17/05/16 19:47:42 INFO yarn.Client: Uploading resource file:/services/livy-server/livy-server-current/repl_2.10-jars/livy-core_2.10-0.3.0.jar -> hdfs://jarvis-nameservice001/user/hdfs/.sparkStaging/application_1494373289850_0386/livy-core_2.10-0.3.0.jar
17/05/16 19:47:42 INFO ContextLauncher: 17/05/16 19:47:42 INFO yarn.Client: Uploading resource file:/tmp/spark-48e81b56-c827-4227-a969-45022ec71175/__spark_conf__6757250222634589720.zip -> hdfs://jarvis-nameservice001/user/hdfs/.sparkStaging/application_1494373289850_0386/__spark_conf__6757250222634589720.zip
17/05/16 19:47:42 INFO ContextLauncher: 17/05/16 19:47:42 INFO spark.SecurityManager: Changing view acls to: hdfs
17/05/16 19:47:42 INFO ContextLauncher: 17/05/16 19:47:42 INFO spark.SecurityManager: Changing modify acls to: hdfs
17/05/16 19:47:42 INFO ContextLauncher: 17/05/16 19:47:42 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hdfs); users with modify permissions: Set(hdfs)
17/05/16 19:47:42 INFO ContextLauncher: 17/05/16 19:47:42 INFO yarn.Client: Submitting application 386 to ResourceManager
17/05/16 19:47:42 INFO ContextLauncher: 17/05/16 19:47:42 INFO impl.YarnClientImpl: Submitted application application_1494373289850_0386
17/05/16 19:47:42 INFO ContextLauncher: 17/05/16 19:47:42 INFO yarn.Client: Application report for application_1494373289850_0386 (state: ACCEPTED)
17/05/16 19:47:42 INFO ContextLauncher: 17/05/16 19:47:42 INFO yarn.Client:
17/05/16 19:47:42 INFO ContextLauncher:          client token: N/A
17/05/16 19:47:42 INFO ContextLauncher:          diagnostics: N/A
17/05/16 19:47:42 INFO ContextLauncher:          ApplicationMaster host: N/A
17/05/16 19:47:42 INFO ContextLauncher:          ApplicationMaster RPC port: -1
17/05/16 19:47:42 INFO ContextLauncher:          queue: root.hdfs
17/05/16 19:47:42 INFO ContextLauncher:          start time: 1494964062698
17/05/16 19:47:42 INFO ContextLauncher:          final status: UNDEFINED


On Mon, May 15, 2017 at 6:53 PM, Jeff Zhang <[hidden email]> wrote:

It is weird that the yarn app log shows the SQLContext is created successfully, but in zeppelin side it shows error of "Fail to create SQLContext"

Ben Vogan <[hidden email]>于2017年5月15日周一 下午8:07写道:
I am using 0.7.1 and I checked the yarn app log and don't see any errors.  It looks like this:

17/05/16 00:04:12 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
17/05/16 00:04:13 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1494373289850_0336_000001
17/05/16 00:04:13 INFO spark.SecurityManager: Changing view acls to: yarn,hdfs
17/05/16 00:04:13 INFO spark.SecurityManager: Changing modify acls to: yarn,hdfs
17/05/16 00:04:13 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, hdfs); users with modify permissions: Set(yarn, hdfs)
17/05/16 00:04:13 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
17/05/16 00:04:13 INFO yarn.ApplicationMaster: Waiting for spark context initialization
17/05/16 00:04:13 INFO yarn.ApplicationMaster: Waiting for spark context initialization ... 
17/05/16 00:04:14 INFO driver.RSCDriver: Connecting to: jarvis-hue002.internal.shopkick.com:40819
17/05/16 00:04:14 INFO driver.RSCDriver: Starting RPC server...
17/05/16 00:04:14 WARN rsc.RSCConf: Your hostname, jarvis-yarn008.internal.shopkick.com, resolves to a loopback address, but we couldn't find any external IP address!
17/05/16 00:04:14 WARN rsc.RSCConf: Set livy.rsc.rpc.server.address if you need to bind to another address.
17/05/16 00:04:14 INFO driver.RSCDriver: Received job request cd7d1356-709d-4674-a85c-21edade2c38d
17/05/16 00:04:14 INFO driver.RSCDriver: SparkContext not yet up, queueing job request.
17/05/16 00:04:17 INFO spark.SparkContext: Running Spark version 1.6.0
17/05/16 00:04:17 INFO spark.SecurityManager: Changing view acls to: yarn,hdfs
17/05/16 00:04:17 INFO spark.SecurityManager: Changing modify acls to: yarn,hdfs
17/05/16 00:04:17 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, hdfs); users with modify permissions: Set(yarn, hdfs)
17/05/16 00:04:17 INFO util.Utils: Successfully started service 'sparkDriver' on port 53267.
17/05/16 00:04:18 INFO slf4j.Slf4jLogger: Slf4jLogger started
17/05/16 00:04:18 INFO Remoting: Starting remoting
17/05/16 00:04:18 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@10.19.194.147:38037]
17/05/16 00:04:18 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriverActorSystem@10.19.194.147:38037]
17/05/16 00:04:18 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 38037.
17/05/16 00:04:18 INFO spark.SparkEnv: Registering MapOutputTracker
17/05/16 00:04:18 INFO spark.SparkEnv: Registering BlockManagerMaster
17/05/16 00:04:18 INFO storage.DiskBlockManager: Created local directory at /yarn/nm/usercache/hdfs/appcache/application_1494373289850_0336/blockmgr-f46429a6-7466-42c1-bd79-9ddf6ec61cb4
17/05/16 00:04:18 INFO storage.MemoryStore: MemoryStore started with capacity 1966.1 MB
17/05/16 00:04:18 INFO spark.SparkEnv: Registering OutputCommitCoordinator
17/05/16 00:04:18 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
17/05/16 00:04:18 INFO util.Utils: Successfully started service 'SparkUI' on port 49024.
17/05/16 00:04:18 INFO ui.SparkUI: Started SparkUI at http://10.19.194.147:49024
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR file:/services/livy-server/livy-server-current/rsc-jars/livy-api-0.3.0.jar at spark://10.19.194.147:53267/jars/livy-api-0.3.0.jar with timestamp 1494893058608
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR file:/services/livy-server/livy-server-current/rsc-jars/livy-rsc-0.3.0.jar at spark://10.19.194.147:53267/jars/livy-rsc-0.3.0.jar with timestamp 1494893058609
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR file:/services/livy-server/livy-server-current/rsc-jars/netty-all-4.0.29.Final.jar at spark://10.19.194.147:53267/jars/netty-all-4.0.29.Final.jar with timestamp 1494893058609
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR hdfs://jarvis-nameservice001/jarvis_pipelines/vertica-jdbc-7.1.2-0.jar at hdfs://jarvis-nameservice001/jarvis_pipelines/vertica-jdbc-7.1.2-0.jar with timestamp 1494893058609
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR hdfs://jarvis-nameservice001/jarvis_pipelines/shopkick-data-pipeline.jar at hdfs://jarvis-nameservice001/jarvis_pipelines/shopkick-data-pipeline.jar with timestamp 1494893058609
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR file:/services/livy-server/livy-server-current/repl_2.10-jars/commons-codec-1.9.jar at spark://10.19.194.147:53267/jars/commons-codec-1.9.jar with timestamp 1494893058609
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR file:/services/livy-server/livy-server-current/repl_2.10-jars/livy-repl_2.10-0.3.0.jar at spark://10.19.194.147:53267/jars/livy-repl_2.10-0.3.0.jar with timestamp 1494893058609
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR file:/services/livy-server/livy-server-current/repl_2.10-jars/livy-core_2.10-0.3.0.jar at spark://10.19.194.147:53267/jars/livy-core_2.10-0.3.0.jar with timestamp 1494893058609
17/05/16 00:04:18 INFO cluster.YarnClusterScheduler: Created YarnClusterScheduler
17/05/16 00:04:18 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 57551.
17/05/16 00:04:18 INFO netty.NettyBlockTransferService: Server created on 57551
17/05/16 00:04:18 INFO storage.BlockManager: external shuffle service port = 7337
17/05/16 00:04:18 INFO storage.BlockManagerMaster: Trying to register BlockManager
17/05/16 00:04:18 INFO storage.BlockManagerMasterEndpoint: Registering block manager 10.19.194.147:57551 with 1966.1 MB RAM, BlockManagerId(driver, 10.19.194.147, 57551)
17/05/16 00:04:18 INFO storage.BlockManagerMaster: Registered BlockManager
17/05/16 00:04:19 INFO scheduler.EventLoggingListener: Logging events to hdfs://jarvis-nameservice001/user/spark/applicationHistory/application_1494373289850_0336_1
17/05/16 00:04:19 INFO cluster.YarnClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
17/05/16 00:04:19 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done
17/05/16 00:04:19 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark://YarnAM@10.19.194.147:53267)
17/05/16 00:04:19 INFO yarn.YarnRMClient: Registering the ApplicationMaster
17/05/16 00:04:19 INFO yarn.ApplicationMaster: Started progress reporter thread with (heartbeat : 3000, initial allocation : 200) intervals
17/05/16 00:04:19 INFO hive.HiveContext: Initializing execution hive, version 1.1.0
17/05/16 00:04:19 INFO client.ClientWrapper: Inspected Hadoop version: 2.6.0-cdh5.7.0
17/05/16 00:04:19 INFO client.ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0-cdh5.7.0
17/05/16 00:04:20 INFO hive.metastore: Trying to connect to metastore with URI thrift://jarvis-hdfs003.internal.shopkick.com:9083
17/05/16 00:04:20 INFO hive.metastore: Opened a connection to metastore, current connections: 1
17/05/16 00:04:20 INFO hive.metastore: Connected to metastore.
17/05/16 00:04:20 INFO session.SessionState: Created HDFS directory: file:/yarn/nm/usercache/hdfs/appcache/application_1494373289850_0336/container_e14_1494373289850_0336_01_000001/tmp/spark-2217d267-a3c0-4cf4-9565-45f80517d41c/scratch/hdfs
17/05/16 00:04:20 INFO session.SessionState: Created local directory: /yarn/nm/usercache/hdfs/appcache/application_1494373289850_0336/container_e14_1494373289850_0336_01_000001/tmp/yarn
17/05/16 00:04:20 INFO session.SessionState: Created local directory: /yarn/nm/usercache/hdfs/appcache/application_1494373289850_0336/container_e14_1494373289850_0336_01_000001/tmp/478f39e9-5295-4e8e-97aa-40b5828f9440_resources
17/05/16 00:04:20 INFO session.SessionState: Created HDFS directory: file:/yarn/nm/usercache/hdfs/appcache/application_1494373289850_0336/container_e14_1494373289850_0336_01_000001/tmp/spark-2217d267-a3c0-4cf4-9565-45f80517d41c/scratch/hdfs/478f39e9-5295-4e8e-97aa-40b5828f9440
17/05/16 00:04:20 INFO session.SessionState: Created local directory: /yarn/nm/usercache/hdfs/appcache/application_1494373289850_0336/container_e14_1494373289850_0336_01_000001/tmp/yarn/478f39e9-5295-4e8e-97aa-40b5828f9440
17/05/16 00:04:20 INFO session.SessionState: Created HDFS directory: file:/yarn/nm/usercache/hdfs/appcache/application_1494373289850_0336/container_e14_1494373289850_0336_01_000001/tmp/spark-2217d267-a3c0-4cf4-9565-45f80517d41c/scratch/hdfs/478f39e9-5295-4e8e-97aa-40b5828f9440/_tmp_space.db
17/05/16 00:04:20 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr.
17/05/16 00:04:20 INFO repl.SparkInterpreter: Created sql context (with Hive support).

On Mon, May 15, 2017 at 5:43 PM, Jeff Zhang <[hidden email]> wrote:

Which version of zeppelin do you use ? And can you check the yarn app log ?


Ben Vogan <[hidden email]>于2017年5月15日周一 下午5:56写道:
Hi all,

For some reason today I'm getting a stack:

org.apache.zeppelin.livy.LivyException: Fail to create SQLContext,<console>:4: error: illegal inheritance;
at org.apache.zeppelin.livy.LivySparkSQLInterpreter.open(LivySparkSQLInterpreter.java:76)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:483)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

On the Livy server I see no errors and there is an open session on yarn.

Some help on this would be greatly appreciated! 

--Ben

On Sun, May 14, 2017 at 6:16 AM, Ben Vogan <[hidden email]> wrote:
Hi all,

I've been using Zeppelin for a couple of weeks now with a stable configuration, but all of a sudden I am getting "Illegal inheritance" errors like so:

 INFO [2017-05-14 03:25:32,678] ({pool-2-thread-56} Paragraph.java[jobRun]:362) - run paragraph 20170514-032326_663206142 using livy org.apache.zeppelin.interpreter.LazyOpenInterpreter@505a171c
 WARN [2017-05-14 03:25:33,696] ({pool-2-thread-56} NotebookServer.java[afterStatusChange]:2058) - Job 20170514-032326_663206142 is finished, status: ERROR, exception: null, result: %text <console>:4: error: illegal inheritance;

It happens across multiple notebooks and across by my spark and livy interpreters.  I don't know where to look for more information about what is wrong.  I don't see any errors in spark/yarn at all.  The driver got created, but it looks like no jobs were ever submitted to spark.

Help would be greatly appreciated.

Thanks,

--
BENJAMIN VOGAN | Data Platform Team Lead




--
BENJAMIN VOGAN | Data Platform Team Lead




--
BENJAMIN VOGAN | Data Platform Team Lead




--
BENJAMIN VOGAN | Data Platform Team Lead

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Illegal Inheritance error

Jeff Zhang

You should put hive-site.xml in SPARK_CONF_DIR, the can not find file bug is due to a spark bug


I have 1 workaround for you:

You need to install spark in all the nodes and put hive-site.xml in SPARK_CONF_DIR if your cluster is not large




Ben Vogan <[hidden email]>于2017年5月16日周二 下午3:52写道:
I know - this is driving me crazy.  It was working fine and without me touching any of it (zeppelin/livy/spark/yarn), it broke.  And with no errors in spark/yarn or livy.  I see a warning in the livy log about the hive-site.xml not being found.  In the interpreter configuration I have tried setting livy.repl.enableHiveContext to both true and false and it still appears to always create a hive context.  I'm not sure how to get rid of that warning - I tried putting the hive-site.xml into the spark conf directory on the livy host but that broke things entirely on the yarn side - claimed the file didn't exist (maybe it would have to be put on all the yarn executor hosts as well?).

17/05/16 19:47:39 WARN InteractiveSession$: Enable HiveContext but no hive-site.xml found under classpath or user request.
17/05/16 19:47:39 INFO InteractiveSession$: Creating LivyClient for sessionId: 26
17/05/16 19:47:39 WARN RSCConf: Your hostname, jarvis-hue002.internal.shopkick.com, resolves to a loopback address, but we couldn't find any external IP address!
17/05/16 19:47:39 WARN RSCConf: Set livy.rsc.rpc.server.address if you need to bind to another address.
17/05/16 19:47:39 INFO InteractiveSessionManager: Registering new session 26
17/05/16 19:47:39 INFO ContextLauncher: WARNING: User-defined SPARK_HOME (/opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/spark) overrides detected (/opt/cloudera/parcels/CDH/lib/spark).
17/05/16 19:47:39 INFO ContextLauncher: WARNING: Running spark-class from user-defined location.
17/05/16 19:47:41 INFO ContextLauncher: 17/05/16 19:47:41 INFO yarn.Client: Requesting a new application from cluster with 12 NodeManagers
17/05/16 19:47:41 INFO ContextLauncher: 17/05/16 19:47:41 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (38912 MB per container)
17/05/16 19:47:41 INFO ContextLauncher: 17/05/16 19:47:41 INFO yarn.Client: Will allocate AM container, with 4505 MB memory including 409 MB overhead
17/05/16 19:47:41 INFO ContextLauncher: 17/05/16 19:47:41 INFO yarn.Client: Setting up container launch context for our AM
17/05/16 19:47:41 INFO ContextLauncher: 17/05/16 19:47:41 INFO yarn.Client: Setting up the launch environment for our AM container
17/05/16 19:47:41 INFO ContextLauncher: 17/05/16 19:47:41 INFO yarn.Client: Preparing resources for our AM container
17/05/16 19:47:41 INFO ContextLauncher: 17/05/16 19:47:41 INFO yarn.Client: Uploading resource file:/services/livy-server/livy-server-current/rsc-jars/livy-api-0.3.0.jar -> hdfs://jarvis-nameservice001/user/hdfs/.sparkStaging/application_1494373289850_0386/livy-api-0.3.0.jar
17/05/16 19:47:41 INFO ContextLauncher: 17/05/16 19:47:41 INFO yarn.Client: Uploading resource file:/services/livy-server/livy-server-current/rsc-jars/livy-rsc-0.3.0.jar -> hdfs://jarvis-nameservice001/user/hdfs/.sparkStaging/application_1494373289850_0386/livy-rsc-0.3.0.jar
17/05/16 19:47:42 INFO ContextLauncher: 17/05/16 19:47:42 INFO yarn.Client: Uploading resource file:/services/livy-server/livy-server-current/rsc-jars/netty-all-4.0.29.Final.jar -> hdfs://jarvis-nameservice001/user/hdfs/.sparkStaging/application_1494373289850_0386/netty-all-4.0.29.Final.jar
17/05/16 19:47:42 INFO ContextLauncher: 17/05/16 19:47:42 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://jarvis-nameservice001/jarvis_pipelines/vertica-jdbc-7.1.2-0.jar
17/05/16 19:47:42 INFO ContextLauncher: 17/05/16 19:47:42 INFO yarn.Client: Source and destination file systems are the same. Not copying hdfs://jarvis-nameservice001/jarvis_pipelines/shopkick-data-pipeline.jar
17/05/16 19:47:42 INFO ContextLauncher: 17/05/16 19:47:42 INFO yarn.Client: Uploading resource file:/services/livy-server/livy-server-current/repl_2.10-jars/commons-codec-1.9.jar -> hdfs://jarvis-nameservice001/user/hdfs/.sparkStaging/application_1494373289850_0386/commons-codec-1.9.jar
17/05/16 19:47:42 INFO ContextLauncher: 17/05/16 19:47:42 INFO yarn.Client: Uploading resource file:/services/livy-server/livy-server-current/repl_2.10-jars/livy-repl_2.10-0.3.0.jar -> hdfs://jarvis-nameservice001/user/hdfs/.sparkStaging/application_1494373289850_0386/livy-repl_2.10-0.3.0.jar
17/05/16 19:47:42 INFO ContextLauncher: 17/05/16 19:47:42 INFO yarn.Client: Uploading resource file:/services/livy-server/livy-server-current/repl_2.10-jars/livy-core_2.10-0.3.0.jar -> hdfs://jarvis-nameservice001/user/hdfs/.sparkStaging/application_1494373289850_0386/livy-core_2.10-0.3.0.jar
17/05/16 19:47:42 INFO ContextLauncher: 17/05/16 19:47:42 INFO yarn.Client: Uploading resource file:/tmp/spark-48e81b56-c827-4227-a969-45022ec71175/__spark_conf__6757250222634589720.zip -> hdfs://jarvis-nameservice001/user/hdfs/.sparkStaging/application_1494373289850_0386/__spark_conf__6757250222634589720.zip
17/05/16 19:47:42 INFO ContextLauncher: 17/05/16 19:47:42 INFO spark.SecurityManager: Changing view acls to: hdfs
17/05/16 19:47:42 INFO ContextLauncher: 17/05/16 19:47:42 INFO spark.SecurityManager: Changing modify acls to: hdfs
17/05/16 19:47:42 INFO ContextLauncher: 17/05/16 19:47:42 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hdfs); users with modify permissions: Set(hdfs)
17/05/16 19:47:42 INFO ContextLauncher: 17/05/16 19:47:42 INFO yarn.Client: Submitting application 386 to ResourceManager
17/05/16 19:47:42 INFO ContextLauncher: 17/05/16 19:47:42 INFO impl.YarnClientImpl: Submitted application application_1494373289850_0386
17/05/16 19:47:42 INFO ContextLauncher: 17/05/16 19:47:42 INFO yarn.Client: Application report for application_1494373289850_0386 (state: ACCEPTED)
17/05/16 19:47:42 INFO ContextLauncher: 17/05/16 19:47:42 INFO yarn.Client:
17/05/16 19:47:42 INFO ContextLauncher:          client token: N/A
17/05/16 19:47:42 INFO ContextLauncher:          diagnostics: N/A
17/05/16 19:47:42 INFO ContextLauncher:          ApplicationMaster host: N/A
17/05/16 19:47:42 INFO ContextLauncher:          ApplicationMaster RPC port: -1
17/05/16 19:47:42 INFO ContextLauncher:          queue: root.hdfs
17/05/16 19:47:42 INFO ContextLauncher:          start time: 1494964062698
17/05/16 19:47:42 INFO ContextLauncher:          final status: UNDEFINED


On Mon, May 15, 2017 at 6:53 PM, Jeff Zhang <[hidden email]> wrote:

It is weird that the yarn app log shows the SQLContext is created successfully, but in zeppelin side it shows error of "Fail to create SQLContext"

Ben Vogan <[hidden email]>于2017年5月15日周一 下午8:07写道:
I am using 0.7.1 and I checked the yarn app log and don't see any errors.  It looks like this:

17/05/16 00:04:12 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
17/05/16 00:04:13 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1494373289850_0336_000001
17/05/16 00:04:13 INFO spark.SecurityManager: Changing view acls to: yarn,hdfs
17/05/16 00:04:13 INFO spark.SecurityManager: Changing modify acls to: yarn,hdfs
17/05/16 00:04:13 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, hdfs); users with modify permissions: Set(yarn, hdfs)
17/05/16 00:04:13 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
17/05/16 00:04:13 INFO yarn.ApplicationMaster: Waiting for spark context initialization
17/05/16 00:04:13 INFO yarn.ApplicationMaster: Waiting for spark context initialization ... 
17/05/16 00:04:14 INFO driver.RSCDriver: Connecting to: jarvis-hue002.internal.shopkick.com:40819
17/05/16 00:04:14 INFO driver.RSCDriver: Starting RPC server...
17/05/16 00:04:14 WARN rsc.RSCConf: Your hostname, jarvis-yarn008.internal.shopkick.com, resolves to a loopback address, but we couldn't find any external IP address!
17/05/16 00:04:14 WARN rsc.RSCConf: Set livy.rsc.rpc.server.address if you need to bind to another address.
17/05/16 00:04:14 INFO driver.RSCDriver: Received job request cd7d1356-709d-4674-a85c-21edade2c38d
17/05/16 00:04:14 INFO driver.RSCDriver: SparkContext not yet up, queueing job request.
17/05/16 00:04:17 INFO spark.SparkContext: Running Spark version 1.6.0
17/05/16 00:04:17 INFO spark.SecurityManager: Changing view acls to: yarn,hdfs
17/05/16 00:04:17 INFO spark.SecurityManager: Changing modify acls to: yarn,hdfs
17/05/16 00:04:17 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, hdfs); users with modify permissions: Set(yarn, hdfs)
17/05/16 00:04:17 INFO util.Utils: Successfully started service 'sparkDriver' on port 53267.
17/05/16 00:04:18 INFO slf4j.Slf4jLogger: Slf4jLogger started
17/05/16 00:04:18 INFO Remoting: Starting remoting
17/05/16 00:04:18 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@10.19.194.147:38037]
17/05/16 00:04:18 INFO Remoting: Remoting now listens on addresses: [akka.tcp://sparkDriverActorSystem@10.19.194.147:38037]
17/05/16 00:04:18 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 38037.
17/05/16 00:04:18 INFO spark.SparkEnv: Registering MapOutputTracker
17/05/16 00:04:18 INFO spark.SparkEnv: Registering BlockManagerMaster
17/05/16 00:04:18 INFO storage.DiskBlockManager: Created local directory at /yarn/nm/usercache/hdfs/appcache/application_1494373289850_0336/blockmgr-f46429a6-7466-42c1-bd79-9ddf6ec61cb4
17/05/16 00:04:18 INFO storage.MemoryStore: MemoryStore started with capacity 1966.1 MB
17/05/16 00:04:18 INFO spark.SparkEnv: Registering OutputCommitCoordinator
17/05/16 00:04:18 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
17/05/16 00:04:18 INFO util.Utils: Successfully started service 'SparkUI' on port 49024.
17/05/16 00:04:18 INFO ui.SparkUI: Started SparkUI at http://10.19.194.147:49024
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR file:/services/livy-server/livy-server-current/rsc-jars/livy-api-0.3.0.jar at spark://10.19.194.147:53267/jars/livy-api-0.3.0.jar with timestamp 1494893058608
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR file:/services/livy-server/livy-server-current/rsc-jars/livy-rsc-0.3.0.jar at spark://10.19.194.147:53267/jars/livy-rsc-0.3.0.jar with timestamp 1494893058609
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR file:/services/livy-server/livy-server-current/rsc-jars/netty-all-4.0.29.Final.jar at spark://10.19.194.147:53267/jars/netty-all-4.0.29.Final.jar with timestamp 1494893058609
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR hdfs://jarvis-nameservice001/jarvis_pipelines/vertica-jdbc-7.1.2-0.jar at hdfs://jarvis-nameservice001/jarvis_pipelines/vertica-jdbc-7.1.2-0.jar with timestamp 1494893058609
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR hdfs://jarvis-nameservice001/jarvis_pipelines/shopkick-data-pipeline.jar at hdfs://jarvis-nameservice001/jarvis_pipelines/shopkick-data-pipeline.jar with timestamp 1494893058609
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR file:/services/livy-server/livy-server-current/repl_2.10-jars/commons-codec-1.9.jar at spark://10.19.194.147:53267/jars/commons-codec-1.9.jar with timestamp 1494893058609
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR file:/services/livy-server/livy-server-current/repl_2.10-jars/livy-repl_2.10-0.3.0.jar at spark://10.19.194.147:53267/jars/livy-repl_2.10-0.3.0.jar with timestamp 1494893058609
17/05/16 00:04:18 INFO spark.SparkContext: Added JAR file:/services/livy-server/livy-server-current/repl_2.10-jars/livy-core_2.10-0.3.0.jar at spark://10.19.194.147:53267/jars/livy-core_2.10-0.3.0.jar with timestamp 1494893058609
17/05/16 00:04:18 INFO cluster.YarnClusterScheduler: Created YarnClusterScheduler
17/05/16 00:04:18 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 57551.
17/05/16 00:04:18 INFO netty.NettyBlockTransferService: Server created on 57551
17/05/16 00:04:18 INFO storage.BlockManager: external shuffle service port = 7337
17/05/16 00:04:18 INFO storage.BlockManagerMaster: Trying to register BlockManager
17/05/16 00:04:18 INFO storage.BlockManagerMasterEndpoint: Registering block manager 10.19.194.147:57551 with 1966.1 MB RAM, BlockManagerId(driver, 10.19.194.147, 57551)
17/05/16 00:04:18 INFO storage.BlockManagerMaster: Registered BlockManager
17/05/16 00:04:19 INFO scheduler.EventLoggingListener: Logging events to hdfs://jarvis-nameservice001/user/spark/applicationHistory/application_1494373289850_0336_1
17/05/16 00:04:19 INFO cluster.YarnClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
17/05/16 00:04:19 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done
17/05/16 00:04:19 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark://YarnAM@10.19.194.147:53267)
17/05/16 00:04:19 INFO yarn.YarnRMClient: Registering the ApplicationMaster
17/05/16 00:04:19 INFO yarn.ApplicationMaster: Started progress reporter thread with (heartbeat : 3000, initial allocation : 200) intervals
17/05/16 00:04:19 INFO hive.HiveContext: Initializing execution hive, version 1.1.0
17/05/16 00:04:19 INFO client.ClientWrapper: Inspected Hadoop version: 2.6.0-cdh5.7.0
17/05/16 00:04:19 INFO client.ClientWrapper: Loaded org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0-cdh5.7.0
17/05/16 00:04:20 INFO hive.metastore: Trying to connect to metastore with URI thrift://jarvis-hdfs003.internal.shopkick.com:9083
17/05/16 00:04:20 INFO hive.metastore: Opened a connection to metastore, current connections: 1
17/05/16 00:04:20 INFO hive.metastore: Connected to metastore.
17/05/16 00:04:20 INFO session.SessionState: Created HDFS directory: file:/yarn/nm/usercache/hdfs/appcache/application_1494373289850_0336/container_e14_1494373289850_0336_01_000001/tmp/spark-2217d267-a3c0-4cf4-9565-45f80517d41c/scratch/hdfs
17/05/16 00:04:20 INFO session.SessionState: Created local directory: /yarn/nm/usercache/hdfs/appcache/application_1494373289850_0336/container_e14_1494373289850_0336_01_000001/tmp/yarn
17/05/16 00:04:20 INFO session.SessionState: Created local directory: /yarn/nm/usercache/hdfs/appcache/application_1494373289850_0336/container_e14_1494373289850_0336_01_000001/tmp/478f39e9-5295-4e8e-97aa-40b5828f9440_resources
17/05/16 00:04:20 INFO session.SessionState: Created HDFS directory: file:/yarn/nm/usercache/hdfs/appcache/application_1494373289850_0336/container_e14_1494373289850_0336_01_000001/tmp/spark-2217d267-a3c0-4cf4-9565-45f80517d41c/scratch/hdfs/478f39e9-5295-4e8e-97aa-40b5828f9440
17/05/16 00:04:20 INFO session.SessionState: Created local directory: /yarn/nm/usercache/hdfs/appcache/application_1494373289850_0336/container_e14_1494373289850_0336_01_000001/tmp/yarn/478f39e9-5295-4e8e-97aa-40b5828f9440
17/05/16 00:04:20 INFO session.SessionState: Created HDFS directory: file:/yarn/nm/usercache/hdfs/appcache/application_1494373289850_0336/container_e14_1494373289850_0336_01_000001/tmp/spark-2217d267-a3c0-4cf4-9565-45f80517d41c/scratch/hdfs/478f39e9-5295-4e8e-97aa-40b5828f9440/_tmp_space.db
17/05/16 00:04:20 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr.
17/05/16 00:04:20 INFO repl.SparkInterpreter: Created sql context (with Hive support).

On Mon, May 15, 2017 at 5:43 PM, Jeff Zhang <[hidden email]> wrote:

Which version of zeppelin do you use ? And can you check the yarn app log ?


Ben Vogan <[hidden email]>于2017年5月15日周一 下午5:56写道:
Hi all,

For some reason today I'm getting a stack:

org.apache.zeppelin.livy.LivyException: Fail to create SQLContext,<console>:4: error: illegal inheritance;
at org.apache.zeppelin.livy.LivySparkSQLInterpreter.open(LivySparkSQLInterpreter.java:76)
at org.apache.zeppelin.interpreter.LazyOpenInterpreter.open(LazyOpenInterpreter.java:70)
at org.apache.zeppelin.interpreter.remote.RemoteInterpreterServer$InterpretJob.jobRun(RemoteInterpreterServer.java:483)
at org.apache.zeppelin.scheduler.Job.run(Job.java:175)
at org.apache.zeppelin.scheduler.FIFOScheduler$1.run(FIFOScheduler.java:139)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

On the Livy server I see no errors and there is an open session on yarn.

Some help on this would be greatly appreciated! 

--Ben

On Sun, May 14, 2017 at 6:16 AM, Ben Vogan <[hidden email]> wrote:
Hi all,

I've been using Zeppelin for a couple of weeks now with a stable configuration, but all of a sudden I am getting "Illegal inheritance" errors like so:

 INFO [2017-05-14 03:25:32,678] ({pool-2-thread-56} Paragraph.java[jobRun]:362) - run paragraph 20170514-032326_663206142 using livy org.apache.zeppelin.interpreter.LazyOpenInterpreter@505a171c
 WARN [2017-05-14 03:25:33,696] ({pool-2-thread-56} NotebookServer.java[afterStatusChange]:2058) - Job 20170514-032326_663206142 is finished, status: ERROR, exception: null, result: %text <console>:4: error: illegal inheritance;

It happens across multiple notebooks and across by my spark and livy interpreters.  I don't know where to look for more information about what is wrong.  I don't see any errors in spark/yarn at all.  The driver got created, but it looks like no jobs were ever submitted to spark.

Help would be greatly appreciated.

Thanks,

--
BENJAMIN VOGAN | Data Platform Team Lead




--
BENJAMIN VOGAN | Data Platform Team Lead




--
BENJAMIN VOGAN | Data Platform Team Lead




--
BENJAMIN VOGAN | Data Platform Team Lead

Loading...