Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

misfire 在一些条件下会陷入死循环,导致内存泄露,任务不能执行 #596

Closed
Ah39 opened this issue Nov 14, 2018 · 8 comments
Assignees
Labels

Comments

@Ah39
Copy link

Ah39 commented Nov 14, 2018

A,B两个节点执行任务,不断的对A,或者B执行失效,暂停,全部触发等操作,在一定条件下,会有节点misfire 会陷入死循环
jstack 日志如下
`
"job_demo444_QuartzSchedulerThread" #20 prio=5 os_prio=0 tid=0x00007fcb9c55e800 nid=0x314b in Object.wait() [0x00007fcb8491d000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)
- locked <0x000000071c3b9898> (a org.apache.zookeeper.ClientCnxn$Packet)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1097)
at org.apache.curator.framework.imps.ExistsBuilderImpl$3.call(ExistsBuilderImpl.java:237)
at org.apache.curator.framework.imps.ExistsBuilderImpl$3.call(ExistsBuilderImpl.java:226)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
at org.apache.curator.framework.imps.ExistsBuilderImpl.pathInForegroundStandard(ExistsBuilderImpl.java:223)
at org.apache.curator.framework.imps.ExistsBuilderImpl.pathInForeground(ExistsBuilderImpl.java:216)
at org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:175)
at org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:32)
at com.dangdang.ddframe.job.reg.zookeeper.ZookeeperRegistryCenter.isExisted(ZookeeperRegistryCenter.java:210)
at com.dangdang.ddframe.job.lite.internal.storage.JobNodeStorage.isJobNodeExisted(JobNodeStorage.java:66)
at com.dangdang.ddframe.job.lite.internal.storage.JobNodeStorage.createJobNodeIfNeeded(JobNodeStorage.java:107)
at com.dangdang.ddframe.job.lite.internal.execution.ExecutionService.setMisfire(ExecutionService.java:190)
at com.dangdang.ddframe.job.lite.internal.schedule.JobTriggerListener.triggerMisfired(JobTriggerListener.java:48)
at org.quartz.core.QuartzScheduler.notifyTriggerListenersMisfired(QuartzScheduler.java:1905)
at org.quartz.core.SchedulerSignalerImpl.notifyTriggerListenersMisfired(SchedulerSignalerImpl.java:74)
at org.quartz.simpl.RAMJobStore.applyMisfire(RAMJobStore.java:1354)
at org.quartz.simpl.RAMJobStore.acquireNextTriggers(RAMJobStore.java:1412)
- locked <0x00000005ca41d1f8> (a java.lang.Object)
at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:272)

Locked ownable synchronizers:
- None

"job_demo444_Worker-1" #19 prio=5 os_prio=0 tid=0x00007fcb9c54e000 nid=0x314a in Object.wait() [0x00007fcb84c1e000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:568)
- locked <0x00000005ca469a08> (a java.lang.Object)

Locked ownable synchronizers:
- None

"Curator-TreeCache-0" #17 daemon prio=5 os_prio=0 tid=0x00007fcb00006800 nid=0x3148 waiting for monitor entry [0x00007fcb84d1e000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.quartz.simpl.RAMJobStore.storeTrigger(RAMJobStore.java:413)
- waiting to lock <0x00000005ca41d1f8> (a java.lang.Object)
at org.quartz.core.QuartzScheduler.triggerJob(QuartzScheduler.java:1187)
at org.quartz.impl.StdScheduler.triggerJob(StdScheduler.java:341)
at org.quartz.impl.StdScheduler.triggerJob(StdScheduler.java:331)
at com.dangdang.ddframe.job.lite.internal.schedule.JobScheduleController.triggerJob(JobScheduleController.java:153)
at com.dangdang.ddframe.job.lite.internal.server.JobOperationListenerManager$JobTriggerStatusJobListener.dataChanged(JobOperationListenerManager.java:103)
at com.dangdang.ddframe.job.lite.internal.listener.AbstractJobListener.childEvent(AbstractJobListener.java:43)
at org.apache.curator.framework.recipes.cache.TreeCache$2.apply(TreeCache.java:741)
at org.apache.curator.framework.recipes.cache.TreeCache$2.apply(TreeCache.java:735)
at org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:93)
at org.apache.curator.shaded.com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
at org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:85)
at org.apache.curator.framework.recipes.cache.TreeCache.callListeners(TreeCache.java:734)
at org.apache.curator.framework.recipes.cache.TreeCache.access$1700(TreeCache.java:71)
at org.apache.curator.framework.recipes.cache.TreeCache$4.run(TreeCache.java:852)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Locked ownable synchronizers:
- <0x00000005ca443f18> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"Abandoned connection cleanup thread" #16 daemon prio=5 os_prio=0 tid=0x00007fcb9c42b800 nid=0x3147 in Object.wait() [0x00007fcb85020000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x00000005ca476a50> (a java.lang.ref.ReferenceQueue$Lock)
at com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:41)

Locked ownable synchronizers:
- None`

zk Debug 日志如下
16:05:59.801 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106557,3 replyHeader:: 1106557,6387909807,0 request:: '/zhuanzhuan/job_demo444,F response:: s{6359320524,6359320524,1541747750794,1541747750794,0,4,0,0,0,4,6359322628} 16:05:59.801 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106558,3 replyHeader:: 1106558,6387909807,0 request:: '/zhuanzhuan/job_demo444/execution/0/misfire,F response:: s{6387877478,6387877478,1542182287193,1542182287193,0,0,0,0,0,0,6387877478} trigger misfire Trigger 'DEFAULT.MT_1a9o3xuxgxkyd': triggerClass: 'org.quartz.impl.triggers.SimpleTriggerImpl calendar: 'null' misfireInstruction: 0 nextFireTime: Wed Nov 14 16:05:59 CST 2018 16:05:59.801 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106559,3 replyHeader:: 1106559,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} 16:05:59.802 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106560,4 replyHeader:: 1106560,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: #30,s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} set misfire0 ThreadThread[job_demo444_QuartzSchedulerThread,5,QuartzScheduler:job_demo444] 16:05:59.802 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106561,3 replyHeader:: 1106561,6387909807,0 request:: '/zhuanzhuan/job_demo444,F response:: s{6359320524,6359320524,1541747750794,1541747750794,0,4,0,0,0,4,6359322628} 16:05:59.802 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106562,3 replyHeader:: 1106562,6387909807,0 request:: '/zhuanzhuan/job_demo444/execution/0/misfire,F response:: s{6387877478,6387877478,1542182287193,1542182287193,0,0,0,0,0,0,6387877478} trigger misfire Trigger 'DEFAULT.MT_1vsq40tky2acq': triggerClass: 'org.quartz.impl.triggers.SimpleTriggerImpl calendar: 'null' misfireInstruction: 0 nextFireTime: Wed Nov 14 16:05:59 CST 2018 16:05:59.803 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106563,3 replyHeader:: 1106563,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} 16:05:59.803 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106564,4 replyHeader:: 1106564,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: #30,s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} set misfire0 ThreadThread[job_demo444_QuartzSchedulerThread,5,QuartzScheduler:job_demo444] 16:05:59.803 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106565,3 replyHeader:: 1106565,6387909807,0 request:: '/zhuanzhuan/job_demo444,F response:: s{6359320524,6359320524,1541747750794,1541747750794,0,4,0,0,0,4,6359322628} 16:05:59.804 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106566,3 replyHeader:: 1106566,6387909807,0 request:: '/zhuanzhuan/job_demo444/execution/0/misfire,F response:: s{6387877478,6387877478,1542182287193,1542182287193,0,0,0,0,0,0,6387877478} trigger misfire Trigger 'DEFAULT.MT_1d3sx2gq3sa9u': triggerClass: 'org.quartz.impl.triggers.SimpleTriggerImpl calendar: 'null' misfireInstruction: 0 nextFireTime: Wed Nov 14 16:05:59 CST 2018 16:05:59.804 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106567,3 replyHeader:: 1106567,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} 16:05:59.804 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106568,4 replyHeader:: 1106568,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: #30,s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} set misfire0 ThreadThread[job_demo444_QuartzSchedulerThread,5,QuartzScheduler:job_demo444] 16:05:59.805 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106569,3 replyHeader:: 1106569,6387909807,0 request:: '/zhuanzhuan/job_demo444,F response:: s{6359320524,6359320524,1541747750794,1541747750794,0,4,0,0,0,4,6359322628} 16:05:59.805 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106570,3 replyHeader:: 1106570,6387909807,0 request:: '/zhuanzhuan/job_demo444/execution/0/misfire,F response:: s{6387877478,6387877478,1542182287193,1542182287193,0,0,0,0,0,0,6387877478} trigger misfire Trigger 'DEFAULT.MT_4b4cdgajqh1f8': triggerClass: 'org.quartz.impl.triggers.SimpleTriggerImpl calendar: 'null' misfireInstruction: 0 nextFireTime: Wed Nov 14 16:05:59 CST 2018 16:05:59.805 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106571,3 replyHeader:: 1106571,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} 16:05:59.806 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106572,4 replyHeader:: 1106572,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: #30,s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} set misfire0 ThreadThread[job_demo444_QuartzSchedulerThread,5,QuartzScheduler:job_demo444] 16:05:59.806 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106573,3 replyHeader:: 1106573,6387909807,0 request:: '/zhuanzhuan/job_demo444,F response:: s{6359320524,6359320524,1541747750794,1541747750794,0,4,0,0,0,4,6359322628} 16:05:59.806 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106574,3 replyHeader:: 1106574,6387909807,0 request:: '/zhuanzhuan/job_demo444/execution/0/misfire,F response:: s{6387877478,6387877478,1542182287193,1542182287193,0,0,0,0,0,0,6387877478} trigger misfire Trigger 'DEFAULT.MT_7b8eseoqrqal6': triggerClass: 'org.quartz.impl.triggers.SimpleTriggerImpl calendar: 'null' misfireInstruction: 0 nextFireTime: Wed Nov 14 16:05:59 CST 2018 16:05:59.806 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106575,3 replyHeader:: 1106575,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} 16:05:59.807 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106576,4 replyHeader:: 1106576,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: #30,s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} set misfire0 ThreadThread[job_demo444_QuartzSchedulerThread,5,QuartzScheduler:job_demo444] 16:05:59.807 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106577,3 replyHeader:: 1106577,6387909807,0 request:: '/zhuanzhuan/job_demo444,F response:: s{6359320524,6359320524,1541747750794,1541747750794,0,4,0,0,0,4,6359322628} 16:05:59.807 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106578,3 replyHeader:: 1106578,6387909807,0 request:: '/zhuanzhuan/job_demo444/execution/0/misfire,F response:: s{6387877478,6387877478,1542182287193,1542182287193,0,0,0,0,0,0,6387877478} trigger misfire Trigger 'DEFAULT.MT_7o7frc6nj73r3': triggerClass: 'org.quartz.impl.triggers.SimpleTriggerImpl calendar: 'null' misfireInstruction: 0 nextFireTime: Wed Nov 14 16:05:59 CST 2018 16:05:59.808 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106579,3 replyHeader:: 1106579,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} 16:05:59.808 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106580,4 replyHeader:: 1106580,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: #30,s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} set misfire0 ThreadThread[job_demo444_QuartzSchedulerThread,5,QuartzScheduler:job_demo444] 16:05:59.808 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106581,3 replyHeader:: 1106581,6387909807,0 request:: '/zhuanzhuan/job_demo444,F response:: s{6359320524,6359320524,1541747750794,1541747750794,0,4,0,0,0,4,6359322628} 16:05:59.809 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106582,3 replyHeader:: 1106582,6387909807,0 request:: '/zhuanzhuan/job_demo444/execution/0/misfire,F response:: s{6387877478,6387877478,1542182287193,1542182287193,0,0,0,0,0,0,6387877478} trigger misfire Trigger 'DEFAULT.MT_14igpchin9avd': triggerClass: 'org.quartz.impl.triggers.SimpleTriggerImpl calendar: 'null' misfireInstruction: 0 nextFireTime: Wed Nov 14 16:05:59 CST 2018 16:05:59.809 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106583,3 replyHeader:: 1106583,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} 16:05:59.809 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106584,4 replyHeader:: 1106584,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: #30,s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} set misfire0 ThreadThread[job_demo444_QuartzSchedulerThread,5,QuartzScheduler:job_demo444] 16:05:59.810 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106585,3 replyHeader:: 1106585,6387909807,0 request:: '/zhuanzhuan/job_demo444,F response:: s{6359320524,6359320524,1541747750794,1541747750794,0,4,0,0,0,4,6359322628} 16:05:59.810 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106586,3 replyHeader:: 1106586,6387909807,0 request:: '/zhuanzhuan/job_demo444/execution/0/misfire,F response:: s{6387877478,6387877478,1542182287193,1542182287193,0,0,0,0,0,0,6387877478} trigger misfire Trigger 'DEFAULT.job_demo444': triggerClass: 'org.quartz.impl.triggers.CronTriggerImpl calendar: 'null' misfireInstruction: 1 nextFireTime: Wed Nov 14 16:05:59 CST 2018 16:05:59.810 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106587,3 replyHeader:: 1106587,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} 16:05:59.811 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106588,4 replyHeader:: 1106588,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: #30,s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} set misfire0 ThreadThread[job_demo444_QuartzSchedulerThread,5,QuartzScheduler:job_demo444] 16:05:59.811 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106589,3 replyHeader:: 1106589,6387909807,0 request:: '/zhuanzhuan/job_demo444,F response:: s{6359320524,6359320524,1541747750794,1541747750794,0,4,0,0,0,4,6359322628} 16:05:59.811 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106590,3 replyHeader:: 1106590,6387909807,0 request:: '/zhuanzhuan/job_demo444/execution/0/misfire,F response:: s{6387877478,6387877478,1542182287193,1542182287193,0,0,0,0,0,0,6387877478} trigger misfire Trigger 'DEFAULT.MT_1a9o3xuxgxkyd': triggerClass: 'org.quartz.impl.triggers.SimpleTriggerImpl calendar: 'null' misfireInstruction: 0 nextFireTime: Wed Nov 14 16:05:59 CST 2018 16:05:59.812 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106591,3 replyHeader:: 1106591,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} 16:05:59.812 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106592,4 replyHeader:: 1106592,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: #30,s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} set misfire0 ThreadThread[job_demo444_QuartzSchedulerThread,5,QuartzScheduler:job_demo444] 16:05:59.812 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106593,3 replyHeader:: 1106593,6387909807,0 request:: '/zhuanzhuan/job_demo444,F response:: s{6359320524,6359320524,1541747750794,1541747750794,0,4,0,0,0,4,6359322628} 16:05:59.813 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106594,3 replyHeader:: 1106594,6387909807,0 request:: '/zhuanzhuan/job_demo444/execution/0/misfire,F response:: s{6387877478,6387877478,1542182287193,1542182287193,0,0,0,0,0,0,6387877478} trigger misfire Trigger 'DEFAULT.MT_1vsq40tky2acq': triggerClass: 'org.quartz.impl.triggers.SimpleTriggerImpl calendar: 'null' misfireInstruction: 0 nextFireTime: Wed Nov 14 16:05:59 CST 2018 16:05:59.813 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106595,3 replyHeader:: 1106595,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} 16:05:59.813 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106596,4 replyHeader:: 1106596,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: #30,s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} set misfire0 ThreadThread[job_demo444_QuartzSchedulerThread,5,QuartzScheduler:job_demo444] 16:05:59.813 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106597,3 replyHeader:: 1106597,6387909807,0 request:: '/zhuanzhuan/job_demo444,F response:: s{6359320524,6359320524,1541747750794,1541747750794,0,4,0,0,0,4,6359322628} 16:05:59.814 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106598,3 replyHeader:: 1106598,6387909807,0 request:: '/zhuanzhuan/job_demo444/execution/0/misfire,F response:: s{6387877478,6387877478,1542182287193,1542182287193,0,0,0,0,0,0,6387877478} trigger misfire Trigger 'DEFAULT.MT_1d3sx2gq3sa9u': triggerClass: 'org.quartz.impl.triggers.SimpleTriggerImpl calendar: 'null' misfireInstruction: 0 nextFireTime: Wed Nov 14 16:05:59 CST 2018 16:05:59.814 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106599,3 replyHeader:: 1106599,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} 16:05:59.814 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106600,4 replyHeader:: 1106600,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: #30,s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} set misfire0 ThreadThread[job_demo444_QuartzSchedulerThread,5,QuartzScheduler:job_demo444] 16:05:59.815 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106601,3 replyHeader:: 1106601,6387909807,0 request:: '/zhuanzhuan/job_demo444,F response:: s{6359320524,6359320524,1541747750794,1541747750794,0,4,0,0,0,4,6359322628} 16:05:59.815 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106602,3 replyHeader:: 1106602,6387909807,0 request:: '/zhuanzhuan/job_demo444/execution/0/misfire,F response:: s{6387877478,6387877478,1542182287193,1542182287193,0,0,0,0,0,0,6387877478} trigger misfire Trigger 'DEFAULT.MT_4b4cdgajqh1f8': triggerClass: 'org.quartz.impl.triggers.SimpleTriggerImpl calendar: 'null' misfireInstruction: 0 nextFireTime: Wed Nov 14 16:05:59 CST 2018 16:05:59.815 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106603,3 replyHeader:: 1106603,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} 16:05:59.816 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106604,4 replyHeader:: 1106604,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: #30,s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} set misfire0 ThreadThread[job_demo444_QuartzSchedulerThread,5,QuartzScheduler:job_demo444] 16:05:59.816 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106605,3 replyHeader:: 1106605,6387909807,0 request:: '/zhuanzhuan/job_demo444,F response:: s{6359320524,6359320524,1541747750794,1541747750794,0,4,0,0,0,4,6359322628} 16:05:59.816 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106606,3 replyHeader:: 1106606,6387909807,0 request:: '/zhuanzhuan/job_demo444/execution/0/misfire,F response:: s{6387877478,6387877478,1542182287193,1542182287193,0,0,0,0,0,0,6387877478} trigger misfire Trigger 'DEFAULT.MT_7b8eseoqrqal6': triggerClass: 'org.quartz.impl.triggers.SimpleTriggerImpl calendar: 'null' misfireInstruction: 0 nextFireTime: Wed Nov 14 16:05:59 CST 2018 16:05:59.817 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106607,3 replyHeader:: 1106607,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} 16:05:59.817 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106608,4 replyHeader:: 1106608,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: #30,s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} set misfire0 ThreadThread[job_demo444_QuartzSchedulerThread,5,QuartzScheduler:job_demo444] 16:05:59.817 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106609,3 replyHeader:: 1106609,6387909807,0 request:: '/zhuanzhuan/job_demo444,F response:: s{6359320524,6359320524,1541747750794,1541747750794,0,4,0,0,0,4,6359322628} 16:05:59.818 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106610,3 replyHeader:: 1106610,6387909807,0 request:: '/zhuanzhuan/job_demo444/execution/0/misfire,F response:: s{6387877478,6387877478,1542182287193,1542182287193,0,0,0,0,0,0,6387877478} trigger misfire Trigger 'DEFAULT.MT_7o7frc6nj73r3': triggerClass: 'org.quartz.impl.triggers.SimpleTriggerImpl calendar: 'null' misfireInstruction: 0 nextFireTime: Wed Nov 14 16:05:59 CST 2018 16:05:59.818 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106611,3 replyHeader:: 1106611,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} 16:05:59.818 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106612,4 replyHeader:: 1106612,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: #30,s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} set misfire0 ThreadThread[job_demo444_QuartzSchedulerThread,5,QuartzScheduler:job_demo444] 16:05:59.819 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106613,3 replyHeader:: 1106613,6387909807,0 request:: '/zhuanzhuan/job_demo444,F response:: s{6359320524,6359320524,1541747750794,1541747750794,0,4,0,0,0,4,6359322628} 16:05:59.819 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106614,3 replyHeader:: 1106614,6387909807,0 request:: '/zhuanzhuan/job_demo444/execution/0/misfire,F response:: s{6387877478,6387877478,1542182287193,1542182287193,0,0,0,0,0,0,6387877478} trigger misfire Trigger 'DEFAULT.MT_14igpchin9avd': triggerClass: 'org.quartz.impl.triggers.SimpleTriggerImpl calendar: 'null' misfireInstruction: 0 nextFireTime: Wed Nov 14 16:05:59 CST 2018 16:05:59.819 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106615,3 replyHeader:: 1106615,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} 16:05:59.820 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106616,4 replyHeader:: 1106616,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: #30,s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} set misfire0 ThreadThread[job_demo444_QuartzSchedulerThread,5,QuartzScheduler:job_demo444] 16:05:59.820 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106617,3 replyHeader:: 1106617,6387909807,0 request:: '/zhuanzhuan/job_demo444,F response:: s{6359320524,6359320524,1541747750794,1541747750794,0,4,0,0,0,4,6359322628} 16:05:59.820 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106618,3 replyHeader:: 1106618,6387909807,0 request:: '/zhuanzhuan/job_demo444/execution/0/misfire,F response:: s{6387877478,6387877478,1542182287193,1542182287193,0,0,0,0,0,0,6387877478} trigger misfire Trigger 'DEFAULT.job_demo444': triggerClass: 'org.quartz.impl.triggers.CronTriggerImpl calendar: 'null' misfireInstruction: 1 nextFireTime: Wed Nov 14 16:05:59 CST 2018 16:05:59.820 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106619,3 replyHeader:: 1106619,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} 16:05:59.821 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106620,4 replyHeader:: 1106620,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: #30,s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} set misfire0 ThreadThread[job_demo444_QuartzSchedulerThread,5,QuartzScheduler:job_demo444] 16:05:59.821 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106621,3 replyHeader:: 1106621,6387909807,0 request:: '/zhuanzhuan/job_demo444,F response:: s{6359320524,6359320524,1541747750794,1541747750794,0,4,0,0,0,4,6359322628} 16:05:59.821 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106622,3 replyHeader:: 1106622,6387909807,0 request:: '/zhuanzhuan/job_demo444/execution/0/misfire,F response:: s{6387877478,6387877478,1542182287193,1542182287193,0,0,0,0,0,0,6387877478} trigger misfire Trigger 'DEFAULT.MT_1a9o3xuxgxkyd': triggerClass: 'org.quartz.impl.triggers.SimpleTriggerImpl calendar: 'null' misfireInstruction: 0 nextFireTime: Wed Nov 14 16:05:59 CST 2018 16:05:59.822 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106623,3 replyHeader:: 1106623,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} 16:05:59.822 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106624,4 replyHeader:: 1106624,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: #30,s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} set misfire0 ThreadThread[job_demo444_QuartzSchedulerThread,5,QuartzScheduler:job_demo444] 16:05:59.822 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106625,3 replyHeader:: 1106625,6387909807,0 request:: '/zhuanzhuan/job_demo444,F response:: s{6359320524,6359320524,1541747750794,1541747750794,0,4,0,0,0,4,6359322628} 16:05:59.823 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106626,3 replyHeader:: 1106626,6387909807,0 request:: '/zhuanzhuan/job_demo444/execution/0/misfire,F response:: s{6387877478,6387877478,1542182287193,1542182287193,0,0,0,0,0,0,6387877478} trigger misfire Trigger 'DEFAULT.MT_1vsq40tky2acq': triggerClass: 'org.quartz.impl.triggers.SimpleTriggerImpl calendar: 'null' misfireInstruction: 0 nextFireTime: Wed Nov 14 16:05:59 CST 2018 16:05:59.823 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106627,3 replyHeader:: 1106627,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} 16:05:59.823 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106628,4 replyHeader:: 1106628,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: #30,s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} set misfire0 ThreadThread[job_demo444_QuartzSchedulerThread,5,QuartzScheduler:job_demo444] 16:05:59.824 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106629,3 replyHeader:: 1106629,6387909807,0 request:: '/zhuanzhuan/job_demo444,F response:: s{6359320524,6359320524,1541747750794,1541747750794,0,4,0,0,0,4,6359322628} 16:05:59.824 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106630,3 replyHeader:: 1106630,6387909807,0 request:: '/zhuanzhuan/job_demo444/execution/0/misfire,F response:: s{6387877478,6387877478,1542182287193,1542182287193,0,0,0,0,0,0,6387877478} trigger misfire Trigger 'DEFAULT.MT_1d3sx2gq3sa9u': triggerClass: 'org.quartz.impl.triggers.SimpleTriggerImpl calendar: 'null' misfireInstruction: 0 nextFireTime: Wed Nov 14 16:05:59 CST 2018 16:05:59.824 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106631,3 replyHeader:: 1106631,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} 16:05:59.824 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106632,4 replyHeader:: 1106632,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: #30,s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} set misfire0 ThreadThread[job_demo444_QuartzSchedulerThread,5,QuartzScheduler:job_demo444] 16:05:59.825 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106633,3 replyHeader:: 1106633,6387909807,0 request:: '/zhuanzhuan/job_demo444,F response:: s{6359320524,6359320524,1541747750794,1541747750794,0,4,0,0,0,4,6359322628} 16:05:59.825 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106634,3 replyHeader:: 1106634,6387909807,0 request:: '/zhuanzhuan/job_demo444/execution/0/misfire,F response:: s{6387877478,6387877478,1542182287193,1542182287193,0,0,0,0,0,0,6387877478} trigger misfire Trigger 'DEFAULT.MT_4b4cdgajqh1f8': triggerClass: 'org.quartz.impl.triggers.SimpleTriggerImpl calendar: 'null' misfireInstruction: 0 nextFireTime: Wed Nov 14 16:05:59 CST 2018 16:05:59.825 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106635,3 replyHeader:: 1106635,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} 16:05:59.826 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106636,4 replyHeader:: 1106636,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: #30,s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} set misfire0 ThreadThread[job_demo444_QuartzSchedulerThread,5,QuartzScheduler:job_demo444] 16:05:59.826 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106637,3 replyHeader:: 1106637,6387909807,0 request:: '/zhuanzhuan/job_demo444,F response:: s{6359320524,6359320524,1541747750794,1541747750794,0,4,0,0,0,4,6359322628} 16:05:59.826 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106638,3 replyHeader:: 1106638,6387909807,0 request:: '/zhuanzhuan/job_demo444/execution/0/misfire,F response:: s{6387877478,6387877478,1542182287193,1542182287193,0,0,0,0,0,0,6387877478} trigger misfire Trigger 'DEFAULT.MT_7b8eseoqrqal6': triggerClass: 'org.quartz.impl.triggers.SimpleTriggerImpl calendar: 'null' misfireInstruction: 0 nextFireTime: Wed Nov 14 16:05:59 CST 2018 16:05:59.827 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106639,3 replyHeader:: 1106639,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} 16:05:59.827 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106640,4 replyHeader:: 1106640,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: #30,s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} set misfire0 ThreadThread[job_demo444_QuartzSchedulerThread,5,QuartzScheduler:job_demo444] 16:05:59.827 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106641,3 replyHeader:: 1106641,6387909807,0 request:: '/zhuanzhuan/job_demo444,F response:: s{6359320524,6359320524,1541747750794,1541747750794,0,4,0,0,0,4,6359322628} 16:05:59.827 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106642,3 replyHeader:: 1106642,6387909807,0 request:: '/zhuanzhuan/job_demo444/execution/0/misfire,F response:: s{6387877478,6387877478,1542182287193,1542182287193,0,0,0,0,0,0,6387877478} trigger misfire Trigger 'DEFAULT.MT_7o7frc6nj73r3': triggerClass: 'org.quartz.impl.triggers.SimpleTriggerImpl calendar: 'null' misfireInstruction: 0 nextFireTime: Wed Nov 14 16:05:59 CST 2018 16:05:59.828 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106643,3 replyHeader:: 1106643,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495} 16:05:59.828 [main-SendThread(tjtx-84-166.58os.org:2181)] DEBUG org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x166e210fe2f057b, packet:: clientPath:null serverPath:null finished:false header:: 1106644,4 replyHeader:: 1106644,6387909807,0 request:: '/zhuanzhuan/job_demo444/servers/10.132.78.16/sharding,F response:: #30,s{6387875495,6387875495,1542182261852,1542182261852,0,0,0,0,1,0,6387875495}

@Ah39 Ah39 changed the title misfire 在一些条件下会陷入死循环,导致内存泄露,任务只能执行 misfire 在一些条件下会陷入死循环,导致内存泄露,任务不能执行 Nov 14, 2018
@Ah39
Copy link
Author

Ah39 commented Nov 14, 2018

在execute 中obFacade.misfireIfRunning 会设置misfire标识,jobFacade.isExecuteMisfired 会判断是否要进入misfire过程。quartz本身执行,如果是不在当前时间,JobTriggerListener 也会设置misfire。
` public final void execute() {

    if (jobFacade.misfireIfRunning(shardingContexts.getShardingItemParameters().keySet())) {`

@Ah39
Copy link
Author

Ah39 commented Nov 14, 2018

#515
#324
全部是跟misfire 有关,我开始以为是zk,submitRequest没有返回,其实是zk client一直在submitRequest,所以多次jstack,看起来就像是zk client submitRequst 卡住了。

@Ah39
Copy link
Author

Ah39 commented Nov 14, 2018

并且创建了多个trigger
image

@Ah39
Copy link
Author

Ah39 commented Nov 14, 2018

在 org.quartz.simpl.RAMJobStore.acquireNextTriggers 这个方法里面陷入了死循环

@Ah39
Copy link
Author

Ah39 commented Nov 16, 2018

2.0.5 版本

@kekerzzZ
Copy link
Contributor

please assign it to me.

@kekerzzZ
Copy link
Contributor

@Ah39 Hi, I followed the steps and versions you said to simulate that there was no reappearance problem. Can you discuss it further?

@terrymanu
Copy link
Member

Closed because of no response.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants