Problem restarting Zeppelin daemon on EC2

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Problem restarting Zeppelin daemon on EC2

nirvanesque
This post has NOT been accepted by the mailing list yet.
Hello all,

In our Data Innovation Labs, we have been developing data transformation (using Spark and Scala) as well as data science applications (R- and based frameworks). Recently, after analysing different DataViz frameworks [Zeppelin, Jupyter, Spark Notebook, ... ] we settled on Zeppelin, which seems to address our polyglot requirements.

Nevertheless, we seem to be running into problems with the Zeppelin server, which does not start properly at times, for x reasons :-(
We have taken several measures, ranging from starting afresh with new installation, new notebooks, etc. But the trouble persists.

Following is a typical output of the logfile.
We are using Zeppelin 0.7.0 together with Spark 2.1.0 and Hadoop 2.7.3
Spark + Hadoop masters are collocated on the same machine as Zeppelin daemon.
Is it possible that the Jetty server instance (also used by Spark ?) is interfering with that of Zeppelin ?

Thanks in advance for all your advice and help,


Logfile :


 INFO [2017-04-18 16:45:02,588] ({Thread-10} ZeppelinServer.java[run]:196) - Shutting down Zeppelin Server ...
 INFO [2017-04-18 16:45:02,603] ({Thread-10} AbstractConnector.java[doStop]:306) - Stopped ServerConnector@5217f3d0{HTTP/1.1}{10.230.140.7:8090}
 INFO [2017-04-18 16:45:02,607] ({Thread-10} ContextHandler.java[doStop]:865) - Stopped o.e.j.w.WebAppContext@319b92f3{/,file:/usr/share/zeppelin/webapps/webapp/,UNAVAILABLE}{/usr/share/zeppelin/zeppelin-web-0.7.1.war}
ERROR [2017-04-18 16:45:03,542] ({Thread-10} ZeppelinServer.java[run]:203) - Error while stopping servlet container
java.lang.NullPointerException
        at org.apache.zeppelin.server.ZeppelinServer$1.run(ZeppelinServer.java:199)
 INFO [2017-04-18 16:45:03,542] ({Thread-10} ZeppelinServer.java[run]:205) - Bye
 INFO [2017-04-18 16:45:05,958] ({main} ZeppelinConfiguration.java[create]:101) - Load configuration from file:/usr/share/zeppelin/conf/zeppelin-site.xml
 INFO [2017-04-18 16:45:06,028] ({main} ZeppelinConfiguration.java[create]:109) - Server Host: 10.230.140.7
 INFO [2017-04-18 16:45:06,029] ({main} ZeppelinConfiguration.java[create]:111) - Server Port: 8090
 INFO [2017-04-18 16:45:06,029] ({main} ZeppelinConfiguration.java[create]:115) - Context Path: /
 INFO [2017-04-18 16:45:06,036] ({main} ZeppelinConfiguration.java[create]:116) - Zeppelin Version: 0.7.1
 INFO [2017-04-18 16:45:06,055] ({main} Log.java[initialized]:186) - Logging initialized @456ms
 INFO [2017-04-18 16:45:06,122] ({main} ZeppelinServer.java[setupWebAppContext]:341) - ZeppelinServer Webapp path: /usr/share/zeppelin/webapps
 INFO [2017-04-18 16:45:06,270] ({main} ZeppelinServer.java[main]:185) - Starting zeppelin server
 INFO [2017-04-18 16:45:06,272] ({main} Server.java[doStart]:327) - jetty-9.2.z-SNAPSHOT
 INFO [2017-04-18 16:45:09,173] ({main} StandardDescriptorProcessor.java[visitServlet]:297) - NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet
 WARN [2017-04-18 16:45:09,333] ({main} ContextHandler.java[log]:2062) - unavailable
javax.servlet.ServletException: Resource class org.apache.zeppelin.server.ZeppelinServer can not be instantiated due to InvocationTargetException
        at org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.createSingletonInstance(CXFNonSpringJaxrsServlet.java:396)
        at org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.createApplicationInstance(CXFNonSpringJaxrsServlet.java:454)
        at org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.createServerFromApplication(CXFNonSpringJaxrsServlet.java:432)
        at org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.init(CXFNonSpringJaxrsServlet.java:93)
        at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:612)
        at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:395)
        at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:871)
        at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:298)
        at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349)
        at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
        at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741)
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:163)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
        at org.eclipse.jetty.server.Server.start(Server.java:387)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
        at org.eclipse.jetty.server.Server.doStart(Server.java:354)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.apache.zeppelin.server.ZeppelinServer.main(ZeppelinServer.java:187)
 WARN [2017-04-18 16:45:10,231] ({main} WebAppContext.java[doStart]:514) - Failed startup of context o.e.j.w.WebAppContext@319b92f3{/,file:/usr/share/zeppelin/webapps/webapp/,STARTING}{/usr/share/zeppelin/zeppelin-web-0.7.1.war}
javax.servlet.ServletException: Resource class org.apache.zeppelin.server.ZeppelinServer can not be instantiated due to InvocationTargetException
        at org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.createSingletonInstance(CXFNonSpringJaxrsServlet.java:396)
        at org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.createApplicationInstance(CXFNonSpringJaxrsServlet.java:454)
        at org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.createServerFromApplication(CXFNonSpringJaxrsServlet.java:432)
        at org.apache.cxf.jaxrs.servlet.CXFNonSpringJaxrsServlet.init(CXFNonSpringJaxrsServlet.java:93)
        at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:612)
        at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:395)
        at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:871)
        at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:298)
        at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1349)
        at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1342)
        at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:741)
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:505)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:163)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
        at org.eclipse.jetty.server.Server.start(Server.java:387)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
        at org.eclipse.jetty.server.Server.doStart(Server.java:354)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.apache.zeppelin.server.ZeppelinServer.main(ZeppelinServer.java:187)
 INFO [2017-04-18 16:45:10,244] ({main} AbstractConnector.java[doStart]:266) - Started ServerConnector@5217f3d0{HTTP/1.1}{10.230.140.7:8090}
 INFO [2017-04-18 16:45:10,244] ({main} Server.java[doStart]:379) - Started @4648ms
 INFO [2017-04-18 16:45:10,244] ({main} ZeppelinServer.java[main]:192) - Done, zeppelin server started