[jboss-jira] [JBoss JIRA] (WFLY-9631) "Failed to reinstate timer" warning is shown when creating large number of EJB timers
Gonzalo Muñoz Fernández (Jira)
issues at jboss.org
Wed May 29 04:29:00 EDT 2019
[ https://issues.jboss.org/browse/WFLY-9631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13739805#comment-13739805 ]
Gonzalo Muñoz Fernández commented on WFLY-9631:
-----------------------------------------------
I've also tested with EAP 7.2.0.GA in Openshift, and same warning:
17:40:05,861 WARN [org.jboss.as.ejb3.timer] (Timer-1) WFLYEJB0161: Failed to reinstate timer 'ROOT.ROOT.EJBTimerScheduler' (id=6f066747-f57e-4ad3-85e5-2a6f5e20333f) from its persistent state: java.lang.IllegalStateException: WFLYCTL0075: Duplicate resource 6f066747-f57e-4ad3-85e5-2a6f5e20333f
at org.jboss.as.controller.registry.AbstractModelResource$DefaultResourceProvider.register(AbstractModelResource.java:299)
at org.jboss.as.controller.registry.AbstractModelResource.registerChild(AbstractModelResource.java:178)
at org.jboss.as.ejb3.subsystem.deployment.TimerServiceResource.timerCreated(TimerServiceResource.java:193)
at org.jboss.as.ejb3.timerservice.TimerServiceImpl.registerTimerResource(TimerServiceImpl.java:1126)
at org.jboss.as.ejb3.timerservice.TimerServiceImpl.startTimer(TimerServiceImpl.java:786)
at org.jboss.as.ejb3.timerservice.TimerServiceImpl$TimerRefreshListener.timerAdded(TimerServiceImpl.java:1319)
at org.jboss.as.ejb3.timerservice.persistence.database.DatabaseTimerPersistence$RefreshTask.run(DatabaseTimerPersistence.java:832)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Then, following traces are shown:
17:54:06,718 INFO [org.jboss.as.ejb3.timer] (Timer-1) WFLYEJB0017: Next expiration is null. No tasks will be scheduled for timer [ID=9B432AAB-6388-40B3-8E1E-BB41909AB742 TIMEDOBJECTID=ROOT.ROOT.EJBTIMERSCHEDULER AUTO-TIMER?:FALSE PERSISTENT?:TRUE TIMERSERVICE=ORG.JBOSS.AS.EJB3.TIMERSERVICE.TIMERSERVICEIMPL at 426E9346 INITIALEXPIRATION=2019-05-28 17:46:16.528 INTERVALDURATION(IN MILLI SEC)=0 NEXTEXPIRATION=NULL TIMERSTATE=ACTIVE INFO=EJBTIMERJOB [TIMERJOBINSTANCE=GLOBALJPATIMERJOBINSTANCE [TIMERSERVICEID=CONT-ID-TIMERSERVICEID, GETJOBHANDLE()=EJBGLOBALJOBHANDLE [UUID=2023-2022-1]]]]
I was testing a performance scenario with EJB Timers for RHPAM (thousand of them are scheduled to be triggered at same time inside their corresponding processes).
When "TIMER_SERVICE_DATA_STORE_REFRESH_INTERVAL" for the EJB timers is set up with a low value (120 seconds, for example), these timers are never triggered and our RHPAM processes do not finish.
This only happens with one pod and low values of "TIMER_SERVICE_DATA_STORE_REFRESH_INTERVAL" (not happening when this value is set up to 60000 seconds, for example)
> "Failed to reinstate timer" warning is shown when creating large number of EJB timers
> -------------------------------------------------------------------------------------
>
> Key: WFLY-9631
> URL: https://issues.jboss.org/browse/WFLY-9631
> Project: WildFly
> Issue Type: Bug
> Components: EJB
> Environment: EAP 7.0.8.GA
> Oracle 12 c
> [Configuration of datasources|https://github.com/MarianMacik/timers-testing/blob/c7dde64d27623949f3cafd0e11828bce8a93de21/jboss-eap-7.0/standalone/configuration/standalone-full.xml#L142-L205] - XA Datasource for jBPM - our application - and a separate XA Datasource for EJB Timers in a different schema.
> Reporter: Marian Macik
> Assignee: Michal Jurc
> Priority: Major
> Attachments: WFLY-9631-reproducer.zip
>
>
> Hi, [~manovotn],
> as we discussed before, I am creating a JIRA for an issue I experienced when testing EJB Timers with jBPM.
> What I did was that I started a large number of processes with jBPM, let's say 10 000 and more. Each process contains a timer which jBPM will create an EJB Timer for. Each timer was configured to fire at exactly the same time, e.g. 12:00 PM. During the test I got this warning displayed about 5 times for each 10 000 timers created:
> {code:java}
> WARN [org.jboss.as.ejb3] (Timer-1) WFLYEJB0161: Failed to reinstate timer 'kie-server.kie-server.EJBTimerScheduler' (id=09afab21-6b0f-41b0-9338-403b4a12e507) from its persistent state: java.lang.IllegalStateException: WFLYCTL0075: Duplicate resource 09afab21-6b0f-41b0-9338-403b4a12e507
> at org.jboss.as.controller.registry.AbstractModelResource$DefaultResourceProvider.register(AbstractModelResource.java:290)
> at org.jboss.as.controller.registry.AbstractModelResource.registerChild(AbstractModelResource.java:169)
> at org.jboss.as.ejb3.subsystem.deployment.TimerServiceResource.timerCreated(TimerServiceResource.java:193)
> at org.jboss.as.ejb3.timerservice.TimerServiceImpl.registerTimerResource(TimerServiceImpl.java:1094)
> at org.jboss.as.ejb3.timerservice.TimerServiceImpl.startTimer(TimerServiceImpl.java:767)
> at org.jboss.as.ejb3.timerservice.TimerServiceImpl$TimerRefreshListener.timerAdded(TimerServiceImpl.java:1235)
> at org.jboss.as.ejb3.timerservice.persistence.database.DatabaseTimerPersistence$RefreshTask.run(DatabaseTimerPersistence.java:820)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
> {code}
> The result of these warnings was that when I had only one EAP instance to fire these timers, it wouldn't fire exactly the timers with ids that were in those warnings, e.g. if it was id=09afab21-6b0f-41b0-9338-403b4a12e507 then this timer wouldn't fire. I had waited for minutes to be sure there is another tick of a refresh interval (configured to 1 min) but nothing happened. But when I started the second EAP instance, they were immediately picked up by that instance and fired. It seems that if there is this warning on a particular instance, the instance won't pick up the timer whatsoever. They will be picked up only by some other instances if there are any. With 2 EAP instances there were warnings too, but all timers fired since I guess affected timers were picked up each time by the other instance.
> Another interesting observation is that when I configured a refresh interval to be shorter, e.g. 5 seconds, these warnings were showing up almost every 5 seconds. When I set it to 30 minutes, I didn't get warnings at all, since all timers fired earlier than a refresh occurred.
> So I think it has something to do with refresh interval.
> Can you please have a look at it? For further configuration details, check the Environment field.
> Thank you very much!
> Marian Macik
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
More information about the jboss-jira
mailing list