Deadlock in Zeppelin 0.8.2

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Deadlock in Zeppelin 0.8.2

andreas.weise@gmail.com
Hi, today we faced a deadlock in our 0.8.2 production environment. Find the thread dump attached.
Found one Java-level deadlock:
=============================
"qtp459857341-184038":
  waiting to lock monitor 0x00007f0c58018238 (object 0x00000003f3079738, a java.util.ArrayList),
  which is held by "qtp459857341-183751"
"qtp459857341-183751":
  waiting to lock monitor 0x00007f0bcc044358 (object 0x0000000402537780, a java.util.ArrayList),
  which is held by "qtp459857341-183682"
"qtp459857341-183682":
  waiting to lock monitor 0x00007f0c2c19d348 (object 0x00000005d7d54268, a java.util.LinkedList),
  which is held by "qtp459857341-183610"
"qtp459857341-183610":
  waiting to lock monitor 0x00007f0bcc044358 (object 0x0000000402537780, a java.util.ArrayList),
  which is held by "qtp459857341-183682"

Java stack information for the threads listed above:
===================================================
"qtp459857341-184038":
        at org.apache.zeppelin.notebook.Note.getParagraph(Note.java:572)
        - waiting to lock <0x00000003f3079738> (a java.util.ArrayList)
        at org.apache.zeppelin.rest.HeliumRestApi.suggest(HeliumRestApi.java:117)
        at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)

....
"qtp459857341-183610":
        at org.apache.zeppelin.notebook.Note.getParagraphs(Note.java:750)
        - waiting to lock <0x0000000402537780> (a java.util.ArrayList)
        at org.apache.zeppelin.notebook.Notebook.getJobListByUnixTime(Notebook.java:872)
        at org.apache.zeppelin.socket.NotebookServer.broadcastUpdateNoteJobInfo(NotebookServer.java:568)
        at org.apache.zeppelin.socket.NotebookServer$NotebookInformationListener.onParagraphRemove(NotebookServer.java:2194)
        at org.apache.zeppelin.notebook.Notebook.onParagraphRemove(Notebook.java:1084)
        at org.apache.zeppelin.notebook.Note.removeParagraph(Note.java:446)
        - locked <0x00000005d7d54268> (a java.util.LinkedList)
        at org.apache.zeppelin.rest.NotebookRestApi.deleteParagraph(NotebookRestApi.java:620)
        at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
...

Please let me know, if that is already fixed in > 0.8.2 or requires a Jira Ticket? 

Best regards
Andreas

threadump.txt (1M) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Deadlock in Zeppelin 0.8.2

Jeff Zhang
I think this fixed in 0.9

Andreas Weise <[hidden email]> 于2020年7月16日周四 下午4:52写道:
Hi, today we faced a deadlock in our 0.8.2 production environment. Find the thread dump attached.
Found one Java-level deadlock:
=============================
"qtp459857341-184038":
  waiting to lock monitor 0x00007f0c58018238 (object 0x00000003f3079738, a java.util.ArrayList),
  which is held by "qtp459857341-183751"
"qtp459857341-183751":
  waiting to lock monitor 0x00007f0bcc044358 (object 0x0000000402537780, a java.util.ArrayList),
  which is held by "qtp459857341-183682"
"qtp459857341-183682":
  waiting to lock monitor 0x00007f0c2c19d348 (object 0x00000005d7d54268, a java.util.LinkedList),
  which is held by "qtp459857341-183610"
"qtp459857341-183610":
  waiting to lock monitor 0x00007f0bcc044358 (object 0x0000000402537780, a java.util.ArrayList),
  which is held by "qtp459857341-183682"

Java stack information for the threads listed above:
===================================================
"qtp459857341-184038":
        at org.apache.zeppelin.notebook.Note.getParagraph(Note.java:572)
        - waiting to lock <0x00000003f3079738> (a java.util.ArrayList)
        at org.apache.zeppelin.rest.HeliumRestApi.suggest(HeliumRestApi.java:117)
        at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)

....
"qtp459857341-183610":
        at org.apache.zeppelin.notebook.Note.getParagraphs(Note.java:750)
        - waiting to lock <0x0000000402537780> (a java.util.ArrayList)
        at org.apache.zeppelin.notebook.Notebook.getJobListByUnixTime(Notebook.java:872)
        at org.apache.zeppelin.socket.NotebookServer.broadcastUpdateNoteJobInfo(NotebookServer.java:568)
        at org.apache.zeppelin.socket.NotebookServer$NotebookInformationListener.onParagraphRemove(NotebookServer.java:2194)
        at org.apache.zeppelin.notebook.Notebook.onParagraphRemove(Notebook.java:1084)
        at org.apache.zeppelin.notebook.Note.removeParagraph(Note.java:446)
        - locked <0x00000005d7d54268> (a java.util.LinkedList)
        at org.apache.zeppelin.rest.NotebookRestApi.deleteParagraph(NotebookRestApi.java:620)
        at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source)
...

Please let me know, if that is already fixed in > 0.8.2 or requires a Jira Ticket? 

Best regards
Andreas


--
Best Regards

Jeff Zhang