LevelDB is initialized when a configuration template refers to it ----------------------------------------------------------------- Key: ISPN-9400 URL: https://issues.jboss.org/browse/ISPN-9400 Project: Infinispan Issue Type: Bug Components: Server Reporter: Alan Field Assignee: Will Burns Priority: Major Fix For: 10.1.0.Final For the scalability tests, I am running 25 instances of the server on a node. I have been using the information in JIRA JDG-1504 to make this work. I added the following in my server configuration file to keep the data directory unique for each instance running on a node: {noformat} <global-state> <persistent-location path="data-${jboss.node.name}" /> </global-state> {noformat} The server is starting up without errors for the global state, but it is showing the following error: {noformat} [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: 14:51:31,955 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-6) MSC000001: Failed to start service jboss.datagrid-infinispan-endpoint.hotrod.hotrod-connec ctor: org.jboss.msc.service.StartException in service jboss.datagrid-infinispan-endpoint.hotrod.hotrod-connector: DGENDPT10004: Failed to start HotRodServer [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.server.endpoint.subsystem.ProtocolServerService.start(ProtocolServerService.java:157) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:2032) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1955) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at java.lang.Thread.run(Thread.java:748) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on object o of type PersistenceManagerImplengineering.redhat.com] out: [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:172) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.factories.AbstractComponentRegistry$PrioritizedMethod.invoke(AbstractComponentRegistry.java:878) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.factories.AbstractComponentRegistry.invokePrioritizedMethods(AbstractComponentRegistry.java:643) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:630) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:549) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.factories.ComponentRegistry.start(ComponentRegistry.java:232) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:891) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:411) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:676) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:620) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:506) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:492) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:525) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:516) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:94) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.jboss.as.clustering.infinispan.DefaultCacheContainer.getCache(DefaultCacheContainer.java:85) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.security.actions.GetCacheAction.run(GetCacheAction.java:26) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.security.actions.GetCacheAction.run(GetCacheAction.java:14) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.security.Security.doPrivileged(Security.java:81) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.server.hotrod.SecurityActions.doPrivileged(SecurityActions.java:34) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.server.hotrod.SecurityActions.getCache(SecurityActions.java:51) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.server.hotrod.HotRodServer.getCacheInstance(HotRodServer.java:384) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.server.hotrod.HotRodServer.preStartCaches(HotRodServer.java:314) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.server.hotrod.HotRodServer.startTransport(HotRodServer.java:296) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.server.core.AbstractProtocolServer.startInternal(AbstractProtocolServer.java:61) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.server.hotrod.HotRodServer.startInternal(HotRodServer.java:230) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.server.hotrod.HotRodServer.startInternal(HotRodServer.java:95) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.server.core.AbstractProtocolServer.start(AbstractProtocolServer.java:79) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.server.endpoint.subsystem.SecurityActions$6.run(SecurityActions.java:136) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.server.endpoint.subsystem.SecurityActions$6.run(SecurityActions.java:133) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.security.Security.doPrivileged(Security.java:76) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.server.endpoint.subsystem.SecurityActions.doPrivileged(SecurityActions.java:42) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.server.endpoint.subsystem.SecurityActions.startProtocolServer(SecurityActions.java:140) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.server.endpoint.subsystem.ProtocolServerService.startProtocolServer(ProtocolServerService.java:191) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.server.endpoint.subsystem.ProtocolServerService.start(ProtocolServerService.java:149) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: ... 5 more [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: Caused by: org.infinispan.commons.CacheException: Unable to start cache loaders [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:199) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at sun.reflect.GeneratedMethodAccessor77.invoke(Unknown Source) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at java.lang.reflect.Method.invoke(Method.java:498) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: ... 39 more [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: Caused by: org.infinispan.commons.CacheConfigurationException: Unable to open database [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.persistence.leveldb.LevelDBStore.start(LevelDBStore.java:109) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.persistence.manager.PersistenceManagerImpl.lambda$start$0(PersistenceManagerImpl.java:163) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at java.util.ArrayList.forEach(ArrayList.java:1257) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.persistence.manager.PersistenceManagerImpl.start(PersistenceManagerImpl.java:179) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: ... 43 more [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: lock /root/jboss-datagrid-7.2.0-server/standalone/data/datagrid-infinispan/clustered/ /datapersistent-leveldb-store/LOCK: Resource temporarily unavailable [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.persistence.leveldb.LevelDBStore.openDatabase(LevelDBStore.java:154) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: at org.infinispan.persistence.leveldb.LevelDBStore.start(LevelDBStore.java:105) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: ... 46 more [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: 14:51:31,965 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([ [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: ("subsystem" => "datagrid-infinispan-endpoint"), [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: ("hotrod-connector" => "hotrod-connector") [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: ]) - failure description: {"WFLYCTL0080: Failed services" => {"jboss.datagrid-infinispan-endpoint.hotrod.hotrod-connector" => "org.jboss.msc.service.StartException n in service jboss.datagrid-infinispan-endpoint.hotrod.hotrod-connector: DGENDPT10004: Failed to start HotRodServer [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: Caused by: org.infinispan.commons.CacheException: Unable to invoke method public void org.infinispan.persistence.manager.PersistenceManagerImpl.start() on obje ect of type PersistenceManagerImplneering.redhat.com] out: [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: Caused by: org.infinispan.commons.CacheException: Unable to start cache loaders [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: Caused by: org.infinispan.commons.CacheConfigurationException: Unable to open database [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error: lock /root/jboss-datagrid-7.2.0-server/standalone/data/datagrid-infinispan/cluste ered/datapersistent-leveldb-store/LOCK: Resource temporarily unavailable"}} [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: 14:51:32,019 INFO [org.jboss.as.controller] (Controller Boot Thread) WFLYCTL0183: Service status report [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: WFLYCTL0186: Services which failed to start: service jboss.datagrid-infinispan-endpoint.hotrod.hotrod-connector: org.jboss.msc.service.StartException in ser rvice jboss.datagrid-infinispan-endpoint.hotrod.hotrod-connector: DGENDPT10004: Failed to start HotRodServer [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: 14:51:32,219 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://127.0.0.1:10013/management [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: 14:51:32,220 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:10013 [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: 14:51:32,220 ERROR [org.jboss.as] (Controller Boot Thread) WFLYSRV0026: Data Grid 7.2.0 (WildFly Core 2.1.18.Final-redhat-1) started (with errors) in 10819ms - Sta arted 199 of 243 services (1 services failed or missing dependencies, 126 services are lazy, passive or on-demand) [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: 14:51:32,221 WARN [org.radargun.service.InfinispanServerLifecycle] (Thread-2) Server started with errors [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: 14:51:32,267 INFO [org.radargun.stages.lifecycle.LifecycleHelper] (sc-main) No members found in the cluster. Expected: 256 [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: 14:51:33,296 INFO [org.radargun.stages.lifecycle.LifecycleHelper] (sc-main) No members found in the cluster. Expected: 256 [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: 14:51:33,958 INFO [org.radargun.service.Infinispan80ServerService] (Thread-2) All Java PIDs: '153566' [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: 14:51:33,958 INFO [org.radargun.service.Infinispan80ServerService] (Thread-2) Java PIDs: '153566' [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: 14:51:34,339 ERROR [org.radargun.service.InfinispanServerClustered] (sc-main) Failed to find CacheManager MBean using domain 'org.infinispan' and name 'default'. T Trying again using type.stack.engineering.redhat.com] out: [b05-h06-1029p.rdu.openstack.engineering.redhat.com] out: 14.482: [GC pause (G1 Evacuation Pause) (young), 0.2065067 secs] {noformat} The only reference to LevelDB is in this configuration template: {noformat} <distributed-cache-configuration name="persistent-leveldb-store"> <leveldb-store shared="false" fetch-state="true" passivation="false"/> </distributed-cache-configuration> {noformat} Commenting this template out makes the error message go away. I don't think that LevelDB should be initialized until it is being used.