[JBoss JIRA] (ISPN-6746) Server instances fail to boot with WARN messages
by Galder Zamarreño (JIRA)
Galder Zamarreño created ISPN-6746:
--------------------------------------
Summary: Server instances fail to boot with WARN messages
Key: ISPN-6746
URL: https://issues.jboss.org/browse/ISPN-6746
Project: Infinispan
Issue Type: Bug
Components: Server
Affects Versions: 8.2.2.Final, 9.0.0.Alpha2
Reporter: Galder Zamarreño
Fix For: 9.0.0.Alpha3, 9.0.0.Final
Attached can be found TRACE logs.
{code}
2016-05-31 16:20:06,460 WARN [org.jgroups.protocols.pbcast.GMS] (MSC service thread 1-2) node6: JOIN(node6) sent to node6 timed out (after 1000 ms), on try 1
2016-05-31 16:20:07,466 WARN [org.jgroups.protocols.pbcast.GMS] (MSC service thread 1-2) node6: JOIN(node6) sent to node6 timed out (after 1000 ms), on try 2
2016-05-31 16:20:08,513 WARN [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-8) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup
2016-05-31 16:20:08,514 WARN [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-8) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup
2016-05-31 16:20:08,656 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000078: Starting JGroups channel clustered
2016-05-31 16:20:08,673 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000094: Received new cluster view for channel clustered: [node6|5] (5) [node6, 1f1cd996-6d79-eb91-10bf-08e40a8efd75, node4, node6, node5]
2016-05-31 16:20:08,682 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000079: Channel clustered local address is node6, physical addresses are [127.0.0.1:55801]
2016-05-31 16:20:10,053 WARN [org.jgroups.protocols.pbcast.GMS] (Incoming-2,node6) node6: failed to create view from delta-view; dropping view: java.lang.IllegalStateException: the view-id of the delta view ([node6|3]) doesn't match the current view-id ([node6|5]); discarding delta view [node6|6], ref-view=[node6|3], joined=[node4, node6, node5]
2016-05-31 16:20:56,961 WARN [org.jgroups.protocols.pbcast.GMS] (Incoming-2,node6) node6: failed to create view from delta-view; dropping view: java.lang.IllegalStateException: the view-id of the delta view ([node6|3]) doesn't match the current view-id ([node6|5]); discarding delta view [node6|7], ref-view=[node6|3], left=[1f1cd996-6d79-eb91-10bf-08e40a8efd75], joined=[node4, node6, node5]
2016-05-31 16:21:00,026 INFO [org.jboss.as.server] (Thread-2) WFLYSRV0220: Server shutdown has been requested.
2016-05-31 16:21:00,038 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
2016-05-31 16:21:00,045 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-1) WFLYJCA0019: Stopped Driver service with driver-name = h2
2016-05-31 16:21:01,912 WARN [org.jgroups.protocols.pbcast.GMS] (Incoming-2,node6) node6: failed to create view from delta-view; dropping view: java.lang.IllegalStateException: the view-id of the delta view ([node6|3]) doesn't match the current view-id ([node6|5]); discarding delta view [node6|8], ref-view=[node6|3], left=[1f1cd996-6d79-eb91-10bf-08e40a8efd75], joined=[node4, node6, node5]
2016-05-31 16:21:06,915 WARN [org.jgroups.protocols.pbcast.GMS] (Incoming-2,node6) node6: failed to create view from delta-view; dropping view: java.lang.IllegalStateException: the view-id of the delta view ([node6|3]) doesn't match the current view-id ([node6|5]); discarding delta view [node6|9], ref-view=[node6|3], left=[1f1cd996-6d79-eb91-10bf-08e40a8efd75], joined=[node4, node6, node5]
2016-05-31 16:21:08,749 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (MSC service thread 1-8) ISPN000329: Unable to read rebalancing status from coordinator node6: org.infinispan.util.concurrent.TimeoutException: Replication timeout for node6
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:822)
at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$0(JGroupsTransport.java:647)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-05-31 16:21:08,754 INFO [org.infinispan.globalstate.impl.GlobalStateManagerImpl] (MSC service thread 1-8) ISPN000390: Persisted state, version=9.0.0-SNAPSHOT timestamp=2016-05-31T14:21:08.751Z
2016-05-31 16:21:08,754 INFO [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-8) ISPN000128: Infinispan version: Infinispan 'Chakra' 9.0.0-SNAPSHOT
2016-05-31 16:21:08,795 WARN [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-8) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup
2016-05-31 16:21:08,796 WARN [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-8) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup
2016-05-31 16:21:08,803 INFO [org.infinispan.globalstate.impl.GlobalStateManagerImpl] (MSC service thread 1-4) ISPN000390: Persisted state, version=9.0.0-SNAPSHOT timestamp=2016-05-31T14:21:08.803Z
2016-05-31 16:21:08,804 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000080: Disconnecting JGroups channel clustered
2016-05-31 16:21:08,805 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000082: Stopping the RpcDispatcher for channel clustered
2016-05-31 16:21:09,827 INFO [org.jboss.as] (MSC service thread 1-8) WFLYSRV0050: Infinispan Server 9.0.0-SNAPSHOT (WildFly Core 2.0.10.Final) stopped in 9791ms
{code}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 10 months
[JBoss JIRA] (ISPN-6746) Server instances fail to boot with WARN messages
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-6746?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño updated ISPN-6746:
-----------------------------------
Attachment: 6-timeout-booting-trace.tgz
> Server instances fail to boot with WARN messages
> ------------------------------------------------
>
> Key: ISPN-6746
> URL: https://issues.jboss.org/browse/ISPN-6746
> Project: Infinispan
> Issue Type: Bug
> Components: Server
> Affects Versions: 9.0.0.Alpha2, 8.2.2.Final
> Reporter: Galder Zamarreño
> Fix For: 9.0.0.Alpha3, 9.0.0.Final
>
> Attachments: 6-timeout-booting-trace.tgz
>
>
> Attached can be found TRACE logs.
> {code}
> 2016-05-31 16:20:06,460 WARN [org.jgroups.protocols.pbcast.GMS] (MSC service thread 1-2) node6: JOIN(node6) sent to node6 timed out (after 1000 ms), on try 1
> 2016-05-31 16:20:07,466 WARN [org.jgroups.protocols.pbcast.GMS] (MSC service thread 1-2) node6: JOIN(node6) sent to node6 timed out (after 1000 ms), on try 2
> 2016-05-31 16:20:08,513 WARN [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-8) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup
> 2016-05-31 16:20:08,514 WARN [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-8) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup
> 2016-05-31 16:20:08,656 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000078: Starting JGroups channel clustered
> 2016-05-31 16:20:08,673 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000094: Received new cluster view for channel clustered: [node6|5] (5) [node6, 1f1cd996-6d79-eb91-10bf-08e40a8efd75, node4, node6, node5]
> 2016-05-31 16:20:08,682 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-8) ISPN000079: Channel clustered local address is node6, physical addresses are [127.0.0.1:55801]
> 2016-05-31 16:20:10,053 WARN [org.jgroups.protocols.pbcast.GMS] (Incoming-2,node6) node6: failed to create view from delta-view; dropping view: java.lang.IllegalStateException: the view-id of the delta view ([node6|3]) doesn't match the current view-id ([node6|5]); discarding delta view [node6|6], ref-view=[node6|3], joined=[node4, node6, node5]
> 2016-05-31 16:20:56,961 WARN [org.jgroups.protocols.pbcast.GMS] (Incoming-2,node6) node6: failed to create view from delta-view; dropping view: java.lang.IllegalStateException: the view-id of the delta view ([node6|3]) doesn't match the current view-id ([node6|5]); discarding delta view [node6|7], ref-view=[node6|3], left=[1f1cd996-6d79-eb91-10bf-08e40a8efd75], joined=[node4, node6, node5]
> 2016-05-31 16:21:00,026 INFO [org.jboss.as.server] (Thread-2) WFLYSRV0220: Server shutdown has been requested.
> 2016-05-31 16:21:00,038 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
> 2016-05-31 16:21:00,045 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-1) WFLYJCA0019: Stopped Driver service with driver-name = h2
> 2016-05-31 16:21:01,912 WARN [org.jgroups.protocols.pbcast.GMS] (Incoming-2,node6) node6: failed to create view from delta-view; dropping view: java.lang.IllegalStateException: the view-id of the delta view ([node6|3]) doesn't match the current view-id ([node6|5]); discarding delta view [node6|8], ref-view=[node6|3], left=[1f1cd996-6d79-eb91-10bf-08e40a8efd75], joined=[node4, node6, node5]
> 2016-05-31 16:21:06,915 WARN [org.jgroups.protocols.pbcast.GMS] (Incoming-2,node6) node6: failed to create view from delta-view; dropping view: java.lang.IllegalStateException: the view-id of the delta view ([node6|3]) doesn't match the current view-id ([node6|5]); discarding delta view [node6|9], ref-view=[node6|3], left=[1f1cd996-6d79-eb91-10bf-08e40a8efd75], joined=[node4, node6, node5]
> 2016-05-31 16:21:08,749 WARN [org.infinispan.topology.ClusterTopologyManagerImpl] (MSC service thread 1-8) ISPN000329: Unable to read rebalancing status from coordinator node6: org.infinispan.util.concurrent.TimeoutException: Replication timeout for node6
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.checkRsp(JGroupsTransport.java:822)
> at org.infinispan.remoting.transport.jgroups.JGroupsTransport.lambda$invokeRemotelyAsync$0(JGroupsTransport.java:647)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:602)
> at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1962)
> at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:46)
> at org.infinispan.remoting.transport.jgroups.SingleResponseFuture.call(SingleResponseFuture.java:17)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2016-05-31 16:21:08,754 INFO [org.infinispan.globalstate.impl.GlobalStateManagerImpl] (MSC service thread 1-8) ISPN000390: Persisted state, version=9.0.0-SNAPSHOT timestamp=2016-05-31T14:21:08.751Z
> 2016-05-31 16:21:08,754 INFO [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-8) ISPN000128: Infinispan version: Infinispan 'Chakra' 9.0.0-SNAPSHOT
> 2016-05-31 16:21:08,795 WARN [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-8) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup
> 2016-05-31 16:21:08,796 WARN [org.infinispan.configuration.cache.AbstractStoreConfigurationBuilder] (MSC service thread 1-8) ISPN000149: Fetch persistent state and purge on startup are both disabled, cache may contain stale entries on startup
> 2016-05-31 16:21:08,803 INFO [org.infinispan.globalstate.impl.GlobalStateManagerImpl] (MSC service thread 1-4) ISPN000390: Persisted state, version=9.0.0-SNAPSHOT timestamp=2016-05-31T14:21:08.803Z
> 2016-05-31 16:21:08,804 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000080: Disconnecting JGroups channel clustered
> 2016-05-31 16:21:08,805 INFO [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (MSC service thread 1-4) ISPN000082: Stopping the RpcDispatcher for channel clustered
> 2016-05-31 16:21:09,827 INFO [org.jboss.as] (MSC service thread 1-8) WFLYSRV0050: Infinispan Server 9.0.0-SNAPSHOT (WildFly Core 2.0.10.Final) stopped in 9791ms
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 10 months
[JBoss JIRA] (ISPN-6740) Client topologies not updated when cache topology loaded from persistent state
by Galder Zamarreño (JIRA)
[ https://issues.jboss.org/browse/ISPN-6740?page=com.atlassian.jira.plugin.... ]
Galder Zamarreño commented on ISPN-6740:
----------------------------------------
Btw, the issue also happens when there's no persistent state to be loaded, since initial CacheTopology is initialised with topology id to 0.
> Client topologies not updated when cache topology loaded from persistent state
> ------------------------------------------------------------------------------
>
> Key: ISPN-6740
> URL: https://issues.jboss.org/browse/ISPN-6740
> Project: Infinispan
> Issue Type: Bug
> Components: Remote Protocols, State Transfer
> Affects Versions: 9.0.0.Alpha2, 8.2.2.Final
> Reporter: Galder Zamarreño
> Assignee: Dan Berindei
> Fix For: 9.0.0.Alpha3, 9.0.0.Final
>
>
> Infinispan Caches now support storing persistent views. When these are loaded, these might be loaded with topology ID 0:
> {code}
> 2016-05-31 10:20:04,254 INFO [org.infinispan.globalstate.impl.GlobalStateManagerImpl] (MSC service thread 1-3)
> ISPN000389: Loaded global state, version=9.0.0-SNAPSHOT timestamp=2016-05-30T12:03:33.822Z
> ....
> 2016-05-31 10:20:07,867 TRACE [org.infinispan.statetransfer.StateTransferManagerImpl] (MSC service thread 1-6)
> Installing new cache topology CacheTopology{id=0, rebalanceId=0, currentCH=DefaultConsistentHash{ns=20, owners = (3)[node5: 7+5, node4: 7+7, node6: 6+8]},
> pendingCH=null, unionCH=null, actualMembers=[node5, node4, node6], persistentUUIDs=[
> bb76729d-2b30-4e54-8108-4ac1db9a04cf, bb76729d-2b30-4e54-8108-4ac1db9a04cf, bb76729d-2b30-4e54-8108-4ac1db9a04cf]} on cache default
> {code}
> If there's no further view changes, the topology ID will remain 0. When a Hot Rod client first connects, it sends its view topology as 0 so that it receives a newly installed topology, but if the topology is already 0 in the server, the server won't send the installed topology, even though it should be newer than having no topology.
> We should start numbering topologies in server starting from 1 instead. That avoids this issue. This is easier than forcing clients to send -1 as initial topology because the topology ID is currently defined as VInt that can only 0 or positive number.
> Also, some extra log messages indicating that the cache topology installed comes from persisted state would be handy for debugging.
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 10 months
[JBoss JIRA] (ISPN-6745) Locks are lost in pessimistic cache
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-6745?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant reassigned ISPN-6745:
-------------------------------------
Assignee: Pedro Ruivo
> Locks are lost in pessimistic cache
> -----------------------------------
>
> Key: ISPN-6745
> URL: https://issues.jboss.org/browse/ISPN-6745
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 7.2.3.Final
> Environment: JBoss DataGrid 6.5.0 (6.3.1.Final-redhat-1)
> 3 nodes in REPL_SYNC mode
> pessimistic locking
> read committed isolation
> Reporter: Eugene Scripnik
> Assignee: Pedro Ruivo
> Attachments: InfinispanNodeFailureTest.java
>
>
> When you perform multiple TX write operations in one transaction (put, replace, lock, etc) and one of the nodes goes down, there is a slight chance that some locks will be lost and acquired by another transaction before current transaction ends.
> So client ends up with two transactions holding the same lock on pessimistic cache at the same time. Both transactions commit at the end successfully.
> I spent some time debugging infinispan code and found that PessimisticLockingInterceptor#releaseLocksOnFailureBeforePrepare releases all locks when OutdatedTopologyException occurs on remote node. But then StateTransferInterceptor#handleTxWriteCommand retries last command. This behavior produces inconsistent state - all locks before last command are released and any other transaction can acquire them.
> I am attaching Test which reproduces this problem
--
This message was sent by Atlassian JIRA
(v6.4.11#64026)
7 years, 10 months