[JBoss JIRA] (ISPN-10041) Locking interceptor should check the topology before acquiring locks
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-10041?page=com.atlassian.jira.plugin... ]
Tristan Tarrant updated ISPN-10041:
-----------------------------------
Fix Version/s: 10.0.0.Beta4
(was: 10.0.0.Beta3)
> Locking interceptor should check the topology before acquiring locks
> --------------------------------------------------------------------
>
> Key: ISPN-10041
> URL: https://issues.jboss.org/browse/ISPN-10041
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 8.2.11.Final, 9.3.6.Final, 10.0.0.Beta2, 9.4.9.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Fix For: 10.0.0.Beta4
>
>
> The distribution interceptors check the command topology is the same as the current topology before sending a command to remote nodes, but the locking interceptors do not have any check.
> On a remote node, this means the inbound invocation handler acquires some locks in topology {{T}}, then the locking interceptor acquires other locks in topology {{T+1}}, and finally the distribution interceptor throws {{OutdatedTopologyException}} and releases the locks. In older versions there is also a potential for blocking a remote executor thread while waiting for the lock, but luckily that is not a problem in 9.4+. It would be more efficient if the locking interceptor was throwing {{OutdatedTopologyException}} instead.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years
[JBoss JIRA] (ISPN-10040) Embedded and server thread pool defaults should be the same
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-10040?page=com.atlassian.jira.plugin... ]
Tristan Tarrant updated ISPN-10040:
-----------------------------------
Fix Version/s: 10.0.0.Beta4
(was: 10.0.0.Beta3)
> Embedded and server thread pool defaults should be the same
> -----------------------------------------------------------
>
> Key: ISPN-10040
> URL: https://issues.jboss.org/browse/ISPN-10040
> Project: Infinispan
> Issue Type: Bug
> Components: Configuration
> Affects Versions: 8.2.11.Final, 10.0.0.Beta2, 9.4.9.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Fix For: 10.0.0.Beta4, 9.4.12.Final
>
>
> Embedded:
> {code:java}
> DEFAULT_THREAD_COUNT.put(REMOTE_COMMAND_EXECUTOR, 200);
> DEFAULT_QUEUE_SIZE.put(REMOTE_COMMAND_EXECUTOR, 0);
> {code}
> Server:
> {code:java}
> REMOTE_COMMAND("remote-command", 25, 25, 100000, 60000),
> {code}
> Using a huge queue is not ok for the remote thread pool, but I missed it before because I assumed the server was using the {{KnownComponentNames}} defaults. We should unify the thread pool defaults in another class with a more topical name and remove {{KnownComponentNames}}.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years
[JBoss JIRA] (ISPN-9988) ScatteredStateConsumerImpl can leak the exclusive topology lock
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-9988?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-9988:
----------------------------------
Fix Version/s: 10.0.0.Beta4
(was: 10.0.0.Beta3)
> ScatteredStateConsumerImpl can leak the exclusive topology lock
> ---------------------------------------------------------------
>
> Key: ISPN-9988
> URL: https://issues.jboss.org/browse/ISPN-9988
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 9.4.7.Final, 10.0.0.Beta1
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Fix For: 10.0.0.Beta4
>
>
> When an exception happens in {{ScatteredStateConsumerImpl.beforeTopologyInstalled}}, the exclusive topology lock is not released in {{StateConsumerImpl.onTopologyUpdate}}:
> {noformat}
> 15:21:54,783 ERROR (transport-thread-FunctionalScatteredInMemoryTest-NodeA-p43135-t5:[Topology-scattered]) [LocalTopologyManagerImpl] ISPN000230: Failed to start rebalance for cache scattered
> java.lang.IllegalArgumentException: The task is already cancelled.
> at org.infinispan.statetransfer.InboundTransferTask.cancelSegments(InboundTransferTask.java:172) ~[classes/:?]
> at org.infinispan.statetransfer.StateConsumerImpl.cancelTransfers(StateConsumerImpl.java:959) ~[classes/:?]
> at org.infinispan.scattered.impl.ScatteredStateConsumerImpl.beforeTopologyInstalled(ScatteredStateConsumerImpl.java:115) ~[classes/:?]
> at org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:292) ~[classes/:?]
> at org.infinispan.scattered.impl.ScatteredStateConsumerImpl.onTopologyUpdate(ScatteredStateConsumerImpl.java:102) ~[classes/:?]
> at org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:200) ~[classes/:?]
> {noformat}
> Because the exclusive topology lock is not released, threads that try to apply a new topology update block forever. This causes random failures with the ISPN-9863 thread leak checker:
> {noformat}
> 15:26:25,922 WARN (testng-RehashClusterPublisherManagerTest:[]) [ThreadLeakChecker] Possible leaked thread:
> "transport-thread-FunctionalScatteredInMemoryTest-NodeA-p43135-t3" daemon prio=5 tid=0x236fd nid=NA waiting
> java.lang.Thread.State: WAITING
> java.base(a)11/jdk.internal.misc.Unsafe.park(Native Method)
> java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
> java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
> java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:917)
> java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1240)
> java.base@11/java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:959)
> app//org.infinispan.statetransfer.StateTransferLockImpl.acquireExclusiveTopologyLock(StateTransferLockImpl.java:42)
> app//org.infinispan.statetransfer.StateConsumerImpl.onTopologyUpdate(StateConsumerImpl.java:291)
> app//org.infinispan.scattered.impl.ScatteredStateConsumerImpl.onTopologyUpdate(ScatteredStateConsumerImpl.java:102)
> app//org.infinispan.statetransfer.StateTransferManagerImpl.doTopologyUpdate(StateTransferManagerImpl.java:200)
> app//org.infinispan.statetransfer.StateTransferManagerImpl.access$000(StateTransferManagerImpl.java:57)
> app//org.infinispan.statetransfer.StateTransferManagerImpl$1.updateConsistentHash(StateTransferManagerImpl.java:113)
> app//org.infinispan.topology.LocalTopologyManagerImpl.doHandleTopologyUpdate(LocalTopologyManagerImpl.java:353)
> app//org.infinispan.topology.LocalTopologyManagerImpl.lambda$handleTopologyUpdate$1(LocalTopologyManagerImpl.java:275)
> 15:26:25,923 ERROR (testng-RehashClusterPublisherManagerTest:[]) [TestSuiteProgress] Test configuration failed: org.infinispan.reactive.publisher.impl.RehashClusterPublisherManagerTest.testClassFinished
> java.lang.AssertionError: Leaked threads:
> {transport-thread-FunctionalScatteredInMemoryTest-NodeA-p43135-t3: possible sources [org.infinispan.functional.FunctionalScatteredInMemoryTest[bias=ON_WRITE], org.infinispan.statetransfer.ClusterTopologyManagerTest[SCATTERED_SYNC, tx=false], org.infinispan.functional.FunctionalCachestoreTest[passivation=true], org.infinispan.functional.distribution.rehash.FunctionalNonTxBackupOwnerBecomingPrimaryOwnerTest, org.infinispan.functional.distribution.rehash.FunctionalNonTxJoinerBecomingBackupOwnerTest, org.infinispan.api.mvcc.PutForExternalReadTest[REPL_SYNC, tx=false], org.infinispan.functional.distribution.rehash.FunctionalTxTest, org.infinispan.functional.FunctionalEncodingTypeTest[tx=true]]}
> at org.infinispan.commons.test.ThreadLeakChecker.performCheck(ThreadLeakChecker.java:148) ~[infinispan-commons-test-10.0.0-SNAPSHOT.jar:10.0.0-SNAPSHOT]
> at org.infinispan.commons.test.ThreadLeakChecker.testFinished(ThreadLeakChecker.java:109) ~[infinispan-commons-test-10.0.0-SNAPSHOT.jar:10.0.0-SNAPSHOT]
> at org.infinispan.test.fwk.TestResourceTracker.testFinished(TestResourceTracker.java:112) ~[test-classes/:?]
> at org.infinispan.test.AbstractInfinispanTest.testClassFinished(AbstractInfinispanTest.java:142) ~[test-classes/:?]
> {noformat}
> The fix should address both the exclusive topology lock itself, by releasing it in a finally block, and the {{IllegalArgumentException}}, either by ignoring already cancelled transfers or by only cancelling transfers while holding {{transferMapsLock}}.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years
[JBoss JIRA] (ISPN-9852) Invocations waiting for a new topology should resume in parallel
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-9852?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-9852:
----------------------------------
Fix Version/s: 10.0.0.Beta4
(was: 10.0.0.Beta3)
> Invocations waiting for a new topology should resume in parallel
> ----------------------------------------------------------------
>
> Key: ISPN-9852
> URL: https://issues.jboss.org/browse/ISPN-9852
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 8.2.11.Final, 10.0.0.Alpha2, 9.4.5.Final
> Reporter: Dan Berindei
> Assignee: Dan Berindei
> Priority: Major
> Fix For: 10.0.0.Beta4, 9.4.12.Final
>
>
> When a command requires a topology newer than the current topology, it uses {{StateTransferLock.transactionDataFuture()}} to wait for the newer topology. When the tx data is received for the newer topology, some (most?) of the waiting operations do not use a separate executor, instead they are all resumed on the thread that received the tx data. If some operations block (e.g. because of a store), it could take a very long time to finish all the blocked operations.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years
[JBoss JIRA] (ISPN-9624) Deprecate StreamingMarshaller interface
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-9624?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-9624:
----------------------------------
Fix Version/s: 10.0.0.Beta4
(was: 10.0.0.Beta3)
> Deprecate StreamingMarshaller interface
> ---------------------------------------
>
> Key: ISPN-9624
> URL: https://issues.jboss.org/browse/ISPN-9624
> Project: Infinispan
> Issue Type: Sub-task
> Components: Core
> Affects Versions: 9.4.0.Final
> Reporter: Ryan Emerson
> Assignee: Ryan Emerson
> Priority: Major
> Fix For: 10.0.0.Beta4
>
>
> The StreamingMarshaller interface is no longer fit for purpose once we move away from jboss-marshalling as the ObjectOutput abstraction does not fit with non-serialization based marshallers. We should provide a simpler interface that simply allows reading/writing objects to a provided stream. This interface should then be exposed via the InitializationContext to store implementations.
> {code:java}
> @ThreadSafe
> public interface StreamAwareMarshaller extends Marshaller {
> /**
> * Marshall an object to the {@link OutputStream}
> *
> * @param o the object to write to the {@link OutputStream}
> * @param out the {@link OutputStream} to write the object to
> * @throws IOException if the object cannot be marshalled to the {@link OutputStream} due to some I/O error
> */
> void writeObject(Object o, OutputStream out) throws IOException;
> /**
> * Unmarshall an object from the {@link InputStream}
> *
> * @param in the {@link InputStream} to unmarshall an object from
> * @return the unmarshalled object instance
> * @throws IOException if unmarshalling cannot complete due to some I/O error
> * @throws ClassNotFoundException if the class of the object trying to unmarshall is not found
> */
> Object readObject(InputStream in) throws ClassNotFoundException, IOException;
> }
> {code}
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years
[JBoss JIRA] (ISPN-9621) Split global marshaller into Persistence and Internal marshaller
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-9621?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-9621:
----------------------------------
Fix Version/s: 10.0.0.Beta4
(was: 10.0.0.Beta3)
> Split global marshaller into Persistence and Internal marshaller
> ----------------------------------------------------------------
>
> Key: ISPN-9621
> URL: https://issues.jboss.org/browse/ISPN-9621
> Project: Infinispan
> Issue Type: Sub-task
> Components: Core, Loaders and Stores
> Affects Versions: 9.4.0.Final
> Reporter: Ryan Emerson
> Assignee: Ryan Emerson
> Priority: Major
> Fix For: 10.0.0.Beta4
>
>
> Currently the internal marshaller is used for marshalling key/value/metadata in store implementations. This means that it is possible for internal classes to be marshalled to a user store, which can cause compatibility issues if an internal classes serialization format changes (due to marshaller changes or addition/removal of fields).
> We should separate the internal and persistence marshaller, only exposing the latter to stores via the InitializationContext. The persistence marshaller should only be able to serialize user key/values and the metadata/version data.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years
[JBoss JIRA] (ISPN-9620) Rolling Upgrade Marshaller Changes
by Tristan Tarrant (Jira)
[ https://issues.jboss.org/browse/ISPN-9620?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-9620:
----------------------------------
Fix Version/s: 10.0.0.Beta4
(was: 10.0.0.Beta3)
> Rolling Upgrade Marshaller Changes
> ----------------------------------
>
> Key: ISPN-9620
> URL: https://issues.jboss.org/browse/ISPN-9620
> Project: Infinispan
> Issue Type: Enhancement
> Components: Core, Loaders and Stores
> Affects Versions: 9.4.0.Final
> Reporter: Ryan Emerson
> Assignee: Ryan Emerson
> Priority: Major
> Fix For: 10.0.0.Beta4
>
>
> In order to allow for compatibility between infinispan versions it is necessary for us to utilise a marshalling implementation at both the cluster (internal node-to-node communication) and persistence layer that is strictly defined but allows for future changes. This is necessary in order to facilitate both rolling and start/stop upgrades. Protocol buffers should be utilised as the wire/storage format, with protostream providing the implementation.
--
This message was sent by Atlassian Jira
(v7.12.1#712002)
7 years