[JBoss JIRA] (WFCORE-2089) Infinite wildfly boot on StartException in service start
by Brian Stansberry (JIRA)
[ https://issues.jboss.org/browse/WFCORE-2089?page=com.atlassian.jira.plugi... ]
Brian Stansberry reassigned WFCORE-2089:
----------------------------------------
Assignee: Brian Stansberry (was: Darran Lofthouse)
> Infinite wildfly boot on StartException in service start
> --------------------------------------------------------
>
> Key: WFCORE-2089
> URL: https://issues.jboss.org/browse/WFCORE-2089
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management, Security
> Reporter: Jan Kalina
> Assignee: Brian Stansberry
> Priority: Blocker
> Attachments: threads-2.txt, threads.txt
>
>
> Following exception (and probably similar too) will cause wildfly frozing during start. Visible especially during integration tests (which will never ends), but reproducible directly too (see steps).
> {code:java}
> 15:59:37,252 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-4) MSC000001: Failed to start service org.wildfly.security.security-realm.ManagementRealm: org.jboss.msc.service.StartException in service org.wildfly.security.security-realm.ManagementRealm: WFLYELY00025: Referenced property file is invalid: ELY01006: No realm name found in password property file
> at org.wildfly.extension.elytron.PropertiesRealmDefinition$1$1.get(PropertiesRealmDefinition.java:185)
> at org.wildfly.extension.elytron.PropertiesRealmDefinition$1$1.get(PropertiesRealmDefinition.java:164)
> at org.wildfly.extension.elytron.TrivialService.start(TrivialService.java:53)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1963)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1896)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> *Update:* This problem with infinite boot will occure everytime the start() method of some service throws StartException(). Not an Elytron problem.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years, 7 months
[JBoss JIRA] (DROOLS-1371) Too many classloaders use too much memory for a large rule set
by Bill Tuminaro (JIRA)
[ https://issues.jboss.org/browse/DROOLS-1371?page=com.atlassian.jira.plugi... ]
Bill Tuminaro commented on DROOLS-1371:
---------------------------------------
After looking at the Drools source code, I see why there are 2 classloaders for each rule.
Here is the object graph for the 2 classloaders per rule with variable names and class names:
1) kContainer (KieContainerImpl)
kBases (ConcurrentHashMap)
segments (ConcurrentHashMap$Segment)
segment[3]
value (KnowledgeBaseImpl)
pkgs (HashMap)
value (DialectRuntimeRegistry)
value( JavaDialectRuntimeData)
classloader (JavaDialectRuntimeData$PackageClassLoader)
2) (KieRepositoryImpl)
kieModuleRepo (KieRepositoryImpl$KieModuleRepo)
kieModules (ConcurrentHashMap)
segments
table [0] (ConcurrentHashMap$HashEntry)
value (TreeMap)
kBuilders(HashMap)
entrySet (HashMap$EntrySet)
table [1] (HashMap$Entry)
value (KnowledgeBuilderImp)
continued ============> pkgRegistryMap (LinkedHashMap)
values (HashMap$Values)
this$0 (LinkedHashMap)
table (Hashmap$Entry)
[5] (LinkedHashMap$Entry)
value (PackageRegistry)
dialectCompiletimeRegistry (DialectCompiletimeRegistry)
map (HashMap)
[10] (HashMap$Entry)
value (JavaDialect)
packageStoreWrapper (PackageStore)
javaDialectRuntimeData (JavaDialectRuntimeData)
continued ============> classloader (JavaDialectRuntimeData$PackageClassLoader)
I will close this request as it appears to be working as designed.
-BillT
> Too many classloaders use too much memory for a large rule set
> --------------------------------------------------------------
>
> Key: DROOLS-1371
> URL: https://issues.jboss.org/browse/DROOLS-1371
> Project: Drools
> Issue Type: Bug
> Components: core engine
> Affects Versions: 6.3.0.Final
> Reporter: Bill Tuminaro
> Assignee: Mario Fusco
> Attachments: WhyTwoClassoaders.java, drools-1371.PNG
>
>
> Why are there so many classloaders after I create a KieSession for a KieFileSystem and KieContainer that only contains one rule?
> The extra classloaders for each rule are consuming too much memory for a rule set that contains several thousand rules.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years, 7 months
[JBoss JIRA] (WFLY-7757) Infinispan JDBC cache stores missing read/write attribute operations for legacy attributes
by Paul Ferraro (JIRA)
Paul Ferraro created WFLY-7757:
----------------------------------
Summary: Infinispan JDBC cache stores missing read/write attribute operations for legacy attributes
Key: WFLY-7757
URL: https://issues.jboss.org/browse/WFLY-7757
Project: WildFly
Issue Type: Bug
Components: Clustering
Reporter: Paul Ferraro
Assignee: Paul Ferraro
Fix For: 11.0.0.Alpha1
In WF10, the following commands were valid:
/subsystem=infinispan/cache-container=foo/local-cache=bar/store=string-jdbc:read-attribute(name=datasource)
/subsystem=infinispan/cache-container=foo/local-cache=bar/store=string-jdbc:write-attribute(name=datasource, value=java:jboss/datasources/ExampleDS)
In WF11, these commands result in "Unknown attribute" errors. Even though the "datasource" attribute was deprecated in WF11, these operations should still work, and should return a value translated from the "data-source" attribute.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years, 7 months
[JBoss JIRA] (WFCORE-2089) Infinite wildfly boot on StartException in service start
by Brian Stansberry (JIRA)
[ https://issues.jboss.org/browse/WFCORE-2089?page=com.atlassian.jira.plugi... ]
Brian Stansberry edited comment on WFCORE-2089 at 12/7/16 5:47 PM:
-------------------------------------------------------------------
Not really. The reason the server was aborting is because MSC was locked up because a bunch of services were doing JMX calls and blocking. MSC being unable to stabilize triggers a failed boot and an abort. But my guess is your JMX change will remove the problem with MSC not stabilizing, and we'll just be left with this service not starting. And that alone won't trigger abort.
I think it's ok for AccessIdentityAddHandler to add a Stage.VERIFY step that checks that the IdentityService is up and calls context.setRollbackOnly() if not. That will ensure that boot rolls back and aborts if that service isn't up.
I think the key thing though is to get that blocking future.get() out of there. <vaguely-waving-hands>Perhaps use the IdentityService itself as the Supplier(s).</vaguely-waving-hands> It knows if it has completed start(), and if yes, its suppliers return the domain, if not return null. Until IdentityService completes start, this facility is unavailable, and if it doesn't complete start the server is going down.
was (Author: brian.stansberry):
Not really. The reason the server was aborting is because MSC was locked up because a bunch of services were doing JMX calls and blocking. MSC being unable to stabilize triggers a failed boot and an abort. But my guess is your JMX change will remove the problem with MSC not stabilizing, and we'll just be left with this service not starting. And that alone won't trigger abort.
I think it's ok for AccessIdentityAddHandler to add a Stage.VERIFY step that checks that the IdentityService is up and calls context.setRollbackOnly() if not. That will ensure that boot rolls back and aborts if that service isn't up.
I think the key thing though is to get that blocking future.get() out of there. <vaguely-waving-hands>Perhaps use the IdentityService itself as the Supplier(s).</vaguely-waving-hands> It knows if it has completed start(), and if yes, its suppliers return the domain, if not return null. Until IdentityService completes start, this facility is unavailable, and if it doesn't complete start the service is going down.
> Infinite wildfly boot on StartException in service start
> --------------------------------------------------------
>
> Key: WFCORE-2089
> URL: https://issues.jboss.org/browse/WFCORE-2089
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management, Security
> Reporter: Jan Kalina
> Assignee: Darran Lofthouse
> Priority: Blocker
> Attachments: threads-2.txt, threads.txt
>
>
> Following exception (and probably similar too) will cause wildfly frozing during start. Visible especially during integration tests (which will never ends), but reproducible directly too (see steps).
> {code:java}
> 15:59:37,252 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-4) MSC000001: Failed to start service org.wildfly.security.security-realm.ManagementRealm: org.jboss.msc.service.StartException in service org.wildfly.security.security-realm.ManagementRealm: WFLYELY00025: Referenced property file is invalid: ELY01006: No realm name found in password property file
> at org.wildfly.extension.elytron.PropertiesRealmDefinition$1$1.get(PropertiesRealmDefinition.java:185)
> at org.wildfly.extension.elytron.PropertiesRealmDefinition$1$1.get(PropertiesRealmDefinition.java:164)
> at org.wildfly.extension.elytron.TrivialService.start(TrivialService.java:53)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1963)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1896)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> *Update:* This problem with infinite boot will occure everytime the start() method of some service throws StartException(). Not an Elytron problem.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years, 7 months
[JBoss JIRA] (WFCORE-2089) Infinite wildfly boot on StartException in service start
by Brian Stansberry (JIRA)
[ https://issues.jboss.org/browse/WFCORE-2089?page=com.atlassian.jira.plugi... ]
Brian Stansberry commented on WFCORE-2089:
------------------------------------------
Not really. The reason the server was aborting is because MSC was locked up because a bunch of services were doing JMX calls and blocking. MSC being unable to stabilize triggers a failed boot and an abort. But my guess is your JMX change will remove the problem with MSC not stabilizing, and we'll just be left with this service not starting. And that alone won't trigger abort.
I think it's ok for AccessIdentityAddHandler to add a Stage.VERIFY step that checks that the IdentityService is up and calls context.setRollbackOnly() if not. That will ensure that boot rolls back and aborts if that service isn't up.
I think the key thing though is to get that blocking future.get() out of there. <vaguely-waving-hands>Perhaps use the IdentityService itself as the Supplier(s).</vaguely-waving-hands> It knows if it has completed start(), and if yes, its suppliers return the domain, if not return null. Until IdentityService completes start, this facility is unavailable, and if it doesn't complete start the service is going down.
> Infinite wildfly boot on StartException in service start
> --------------------------------------------------------
>
> Key: WFCORE-2089
> URL: https://issues.jboss.org/browse/WFCORE-2089
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management, Security
> Reporter: Jan Kalina
> Assignee: Darran Lofthouse
> Priority: Blocker
> Attachments: threads-2.txt, threads.txt
>
>
> Following exception (and probably similar too) will cause wildfly frozing during start. Visible especially during integration tests (which will never ends), but reproducible directly too (see steps).
> {code:java}
> 15:59:37,252 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-4) MSC000001: Failed to start service org.wildfly.security.security-realm.ManagementRealm: org.jboss.msc.service.StartException in service org.wildfly.security.security-realm.ManagementRealm: WFLYELY00025: Referenced property file is invalid: ELY01006: No realm name found in password property file
> at org.wildfly.extension.elytron.PropertiesRealmDefinition$1$1.get(PropertiesRealmDefinition.java:185)
> at org.wildfly.extension.elytron.PropertiesRealmDefinition$1$1.get(PropertiesRealmDefinition.java:164)
> at org.wildfly.extension.elytron.TrivialService.start(TrivialService.java:53)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1963)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1896)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> *Update:* This problem with infinite boot will occure everytime the start() method of some service throws StartException(). Not an Elytron problem.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years, 7 months
[JBoss JIRA] (WFCORE-2089) Infinite wildfly boot on StartException in service start
by Darran Lofthouse (JIRA)
[ https://issues.jboss.org/browse/WFCORE-2089?page=com.atlassian.jira.plugi... ]
Darran Lofthouse commented on WFCORE-2089:
------------------------------------------
If I understand what you are saying, it sounds like if we allow this to return quickly whilst booting the service dependency issue is already covered?
> Infinite wildfly boot on StartException in service start
> --------------------------------------------------------
>
> Key: WFCORE-2089
> URL: https://issues.jboss.org/browse/WFCORE-2089
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management, Security
> Reporter: Jan Kalina
> Assignee: Darran Lofthouse
> Priority: Blocker
> Attachments: threads-2.txt, threads.txt
>
>
> Following exception (and probably similar too) will cause wildfly frozing during start. Visible especially during integration tests (which will never ends), but reproducible directly too (see steps).
> {code:java}
> 15:59:37,252 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-4) MSC000001: Failed to start service org.wildfly.security.security-realm.ManagementRealm: org.jboss.msc.service.StartException in service org.wildfly.security.security-realm.ManagementRealm: WFLYELY00025: Referenced property file is invalid: ELY01006: No realm name found in password property file
> at org.wildfly.extension.elytron.PropertiesRealmDefinition$1$1.get(PropertiesRealmDefinition.java:185)
> at org.wildfly.extension.elytron.PropertiesRealmDefinition$1$1.get(PropertiesRealmDefinition.java:164)
> at org.wildfly.extension.elytron.TrivialService.start(TrivialService.java:53)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1963)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1896)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> *Update:* This problem with infinite boot will occure everytime the start() method of some service throws StartException(). Not an Elytron problem.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years, 7 months
[JBoss JIRA] (WFCORE-2089) Infinite wildfly boot on StartException in service start
by Brian Stansberry (JIRA)
[ https://issues.jboss.org/browse/WFCORE-2089?page=com.atlassian.jira.plugi... ]
Brian Stansberry commented on WFCORE-2089:
------------------------------------------
Actually, the second thread dump I've attached is a real case of this call path preventing a clean abort. If the ControllerBootThread had been allowed to continue the end result would have been an abort.
> Infinite wildfly boot on StartException in service start
> --------------------------------------------------------
>
> Key: WFCORE-2089
> URL: https://issues.jboss.org/browse/WFCORE-2089
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management, Security
> Reporter: Jan Kalina
> Assignee: Darran Lofthouse
> Priority: Blocker
> Attachments: threads-2.txt, threads.txt
>
>
> Following exception (and probably similar too) will cause wildfly frozing during start. Visible especially during integration tests (which will never ends), but reproducible directly too (see steps).
> {code:java}
> 15:59:37,252 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-4) MSC000001: Failed to start service org.wildfly.security.security-realm.ManagementRealm: org.jboss.msc.service.StartException in service org.wildfly.security.security-realm.ManagementRealm: WFLYELY00025: Referenced property file is invalid: ELY01006: No realm name found in password property file
> at org.wildfly.extension.elytron.PropertiesRealmDefinition$1$1.get(PropertiesRealmDefinition.java:185)
> at org.wildfly.extension.elytron.PropertiesRealmDefinition$1$1.get(PropertiesRealmDefinition.java:164)
> at org.wildfly.extension.elytron.TrivialService.start(TrivialService.java:53)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1963)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1896)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> *Update:* This problem with infinite boot will occure everytime the start() method of some service throws StartException(). Not an Elytron problem.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years, 7 months
[JBoss JIRA] (WFCORE-2089) Infinite wildfly boot on StartException in service start
by Darran Lofthouse (JIRA)
[ https://issues.jboss.org/browse/WFCORE-2089?page=com.atlassian.jira.plugi... ]
Darran Lofthouse commented on WFCORE-2089:
------------------------------------------
The problem case could be if the management interface do come up but this service doesn't come up.
> Infinite wildfly boot on StartException in service start
> --------------------------------------------------------
>
> Key: WFCORE-2089
> URL: https://issues.jboss.org/browse/WFCORE-2089
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management, Security
> Reporter: Jan Kalina
> Assignee: Darran Lofthouse
> Priority: Blocker
> Attachments: threads-2.txt, threads.txt
>
>
> Following exception (and probably similar too) will cause wildfly frozing during start. Visible especially during integration tests (which will never ends), but reproducible directly too (see steps).
> {code:java}
> 15:59:37,252 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-4) MSC000001: Failed to start service org.wildfly.security.security-realm.ManagementRealm: org.jboss.msc.service.StartException in service org.wildfly.security.security-realm.ManagementRealm: WFLYELY00025: Referenced property file is invalid: ELY01006: No realm name found in password property file
> at org.wildfly.extension.elytron.PropertiesRealmDefinition$1$1.get(PropertiesRealmDefinition.java:185)
> at org.wildfly.extension.elytron.PropertiesRealmDefinition$1$1.get(PropertiesRealmDefinition.java:164)
> at org.wildfly.extension.elytron.TrivialService.start(TrivialService.java:53)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1963)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1896)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> *Update:* This problem with infinite boot will occure everytime the start() method of some service throws StartException(). Not an Elytron problem.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years, 7 months
[JBoss JIRA] (WFCORE-2089) Infinite wildfly boot on StartException in service start
by Brian Stansberry (JIRA)
[ https://issues.jboss.org/browse/WFCORE-2089?page=com.atlassian.jira.plugi... ]
Brian Stansberry commented on WFCORE-2089:
------------------------------------------
The main problem I believe is internal calls by the server itself, where now we have a new requirement for some service that is easily misconfigured by the user, and that doesn't do anything obviously beneficial to the handling of internal calls. We've added fragility for little gain.
Presumably if the realm service fails to start, the management interface services depend on it so they won't start. So outside callers should not have access that way.
Even if we abort the start of the server, I am not confident this call path will not prevent a clean abort. Perhaps not now but it seems quite easy to inadvertently make some change and end up calling into this.
> Infinite wildfly boot on StartException in service start
> --------------------------------------------------------
>
> Key: WFCORE-2089
> URL: https://issues.jboss.org/browse/WFCORE-2089
> Project: WildFly Core
> Issue Type: Bug
> Components: Domain Management, Security
> Reporter: Jan Kalina
> Assignee: Darran Lofthouse
> Priority: Blocker
> Attachments: threads-2.txt, threads.txt
>
>
> Following exception (and probably similar too) will cause wildfly frozing during start. Visible especially during integration tests (which will never ends), but reproducible directly too (see steps).
> {code:java}
> 15:59:37,252 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-4) MSC000001: Failed to start service org.wildfly.security.security-realm.ManagementRealm: org.jboss.msc.service.StartException in service org.wildfly.security.security-realm.ManagementRealm: WFLYELY00025: Referenced property file is invalid: ELY01006: No realm name found in password property file
> at org.wildfly.extension.elytron.PropertiesRealmDefinition$1$1.get(PropertiesRealmDefinition.java:185)
> at org.wildfly.extension.elytron.PropertiesRealmDefinition$1$1.get(PropertiesRealmDefinition.java:164)
> at org.wildfly.extension.elytron.TrivialService.start(TrivialService.java:53)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1963)
> at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1896)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> *Update:* This problem with infinite boot will occure everytime the start() method of some service throws StartException(). Not an Elytron problem.
--
This message was sent by Atlassian JIRA
(v7.2.3#72005)
7 years, 7 months