[JBoss JIRA] (ISPN-5570) Cross-site: retry backup commands
by Dan Berindei (JIRA)
Dan Berindei created ISPN-5570:
----------------------------------
Summary: Cross-site: retry backup commands
Key: ISPN-5570
URL: https://issues.jboss.org/browse/ISPN-5570
Project: Infinispan
Issue Type: Bug
Components: Core, Cross-Site Replication
Affects Versions: 7.2.3.Final
Reporter: Dan Berindei
Fix For: 8.0.0.Final
There are 3 phases in a backup RPC:
1. Sender -> Local site master: caused by the site master is shutting down or crashing, or by a network split.
2. Local site master -> Remote site master:
2.1. Local site master is no longer a site master, e.g. because it's shutting down or because it's no longer coordinator after a merge.
2.2. Remote site master is not longer a site master.
2.3. Link between local site and remote site is down.
3. Remote site master -> Backup targets
Replication failures in phase 3 are handled by retrying (except for TimeoutExceptions), because {{BaseBackupReceiver}} uses regular cache methods to perform the updates.
But replication failures in phases 1 and 2 are not handled in any way, except for causing the remote site to be taken offline after a certain number of replication failures (if backup is synchronous). We should instead retry backup RPCs when we get a {{SuspectException}} or {{UnreachableException}}, and perhaps even when we get no response (2.2?), and only stop when the timeout expires or when the backup is taken offline.
Async backup probably needs retrying as well, and perhaps even a more sophisticated approach like I-RAC (ISPN-2634).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years
[JBoss JIRA] (ISPN-5551) Infinispan AS modules artifact should include JBoss Marshalling
by Sanne Grinovero (JIRA)
[ https://issues.jboss.org/browse/ISPN-5551?page=com.atlassian.jira.plugin.... ]
Sanne Grinovero commented on ISPN-5551:
---------------------------------------
Thanks for explaining Horia! Looks like in this case this should be really included then, but it's hard to make a general rule about these things if we want to try reuse "most" of the existing modules.
One way would be to simply use the same versions of a target WildFly version, so we build and test with that set of versions, but that would bind Infinispan to a specific WildFly version, while we'll probably want some fleixibility.
Looks like the better way would be to re-bundle all modules in a private slot ?! That's not what people expect from layered products.
Also it would be unclear on were to draw the line. Looks like libraries which are used extensively by Infinispan like JBoss Marshalling and JGroups would better be bundled as private slot, but what about the various other integrations, like JBoss Logger or Narayana?
The line seems to be define just on a "more likely to create trouble" base.. we should rather be more careful in tracking versions, when for example someone upgrades jboss-marshalling-osgi to a version which is different from the target platform. The Maven plugin "Enforcer" has a "dependency convergence" tool - I'm not sure if it can track also dependencies from the container - but we might want to use something like that to verify.
> Infinispan AS modules artifact should include JBoss Marshalling
> ---------------------------------------------------------------
>
> Key: ISPN-5551
> URL: https://issues.jboss.org/browse/ISPN-5551
> Project: Infinispan
> Issue Type: Bug
> Affects Versions: 7.2.2.Final
> Reporter: Horia Chiorean
> Assignee: Tristan Tarrant
> Fix For: 8.0.0.Beta1, 7.2.4.Final, 8.0.0.Final
>
>
> Infinispan 7 provides the following Maven artifact:
> {code:xml}
> <dependency>
> <groupId>org.infinispan</groupId>
> <artifactId>infinispan-as-embedded-modules</artifactId>
> <type>zip</type>
> </dependency>
> {code}
> which can be used (and is used by ModeShape) when running in a JBoss AS instance.
> However, this artifact does not contain the version of JBoss Marshalling that a particular version of Infinspan requires but rather picks up the {{main}} version which is available in the container:
> {code:xml}
> <module name="org.jboss.marshalling"/>
> <module name="org.jboss.marshalling.river" services="import"/>
> {code}
> Because of this, when deploying Infinispan 7.2.x in Wildfly 8.2.0, Infinispan will pick up the version of marshalling delivered by Wildfly (1.4.9.Final instead of 1.4.10.Final) causing class cast exceptions like:
> {code:java}
> Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to org.jboss.marshalling.Externalizer
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:1012)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1256)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)
> at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)
> at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)
> at org.infinispan.commons.marshall.jboss.AbstractJBossMarshaller.objectFromObjectStream(AbstractJBossMarshaller.java:135) [infinispan-commons.jar:7.2.0.Final]
> at org.infinispan.marshall.core.VersionAwareMarshaller.objectFromByteBuffer(VersionAwareMarshaller.java:101) [infinispan-core.jar:7.2.0.Final]
> at org.infinispan.commons.marshall.AbstractDelegatingMarshaller.objectFromByteBuffer(AbstractDelegatingMarshaller.java:80) [infinispan-commons.jar:7.2.0.Final]
> at org.infinispan.marshall.core.MarshalledEntryImpl.unmarshall(MarshalledEntryImpl.java:114) [infinispan-core.jar:7.2.0.Final]
> ... 162 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years
[JBoss JIRA] (ISPN-5569) NullPointerException on JCache.putAll(ConcurrentHashMap)
by Michael Gysel (JIRA)
Michael Gysel created ISPN-5569:
-----------------------------------
Summary: NullPointerException on JCache.putAll(ConcurrentHashMap)
Key: ISPN-5569
URL: https://issues.jboss.org/browse/ISPN-5569
Project: Infinispan
Issue Type: Bug
Affects Versions: 7.2.2.Final
Reporter: Michael Gysel
Class {{org.infinispan.jcache.embedded.JCache<K, V>}}
{code:java}@Override
public void putAll(Map<? extends K, ? extends V> inputMap) {
checkNotClosed();
// spec required check
if (inputMap == null || inputMap.containsKey(null) || inputMap.containsValue(null)) {
throw new NullPointerException(
"inputMap is null or keys/values contain a null entry: " + inputMap);
}
// more code
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years
[JBoss JIRA] (ISPN-5568) KeyAffinityService race condition on view change
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-5568?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration commented on ISPN-5568:
-----------------------------------------------
baranowb <bbaranow(a)redhat.com> changed the Status of [bug 1233968|https://bugzilla.redhat.com/show_bug.cgi?id=1233968] from NEW to ASSIGNED
> KeyAffinityService race condition on view change
> ------------------------------------------------
>
> Key: ISPN-5568
> URL: https://issues.jboss.org/browse/ISPN-5568
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 5.2.11.Final
> Reporter: Dennis Reed
>
> KeyAffinityService#getKeyForAddress runs in a tight loop looking for keys:
> {noformat}
> queue = address2key.get(address)
> while (result == null)
> result = queue.poll()
> {noformat}
> KeyAffinityService#handleViewChange clears and resets the queue list on membership change:
> {noformat}
> address2key.clear()
> for each address
> map.put(address, new queue)
> {noformat}
> If a view change comes in after getKeyForAddress gets the queue, and the queue is empty, it will get stuck in a tight loop looking at the wrong queue forever while new keys are added to the new queue.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years
[JBoss JIRA] (ISPN-5568) KeyAffinityService race condition on view change
by RH Bugzilla Integration (JIRA)
[ https://issues.jboss.org/browse/ISPN-5568?page=com.atlassian.jira.plugin.... ]
RH Bugzilla Integration updated ISPN-5568:
------------------------------------------
Bugzilla Update: Perform
Bugzilla References: https://bugzilla.redhat.com/show_bug.cgi?id=1233968
> KeyAffinityService race condition on view change
> ------------------------------------------------
>
> Key: ISPN-5568
> URL: https://issues.jboss.org/browse/ISPN-5568
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 5.2.11.Final
> Reporter: Dennis Reed
>
> KeyAffinityService#getKeyForAddress runs in a tight loop looking for keys:
> {noformat}
> queue = address2key.get(address)
> while (result == null)
> result = queue.poll()
> {noformat}
> KeyAffinityService#handleViewChange clears and resets the queue list on membership change:
> {noformat}
> address2key.clear()
> for each address
> map.put(address, new queue)
> {noformat}
> If a view change comes in after getKeyForAddress gets the queue, and the queue is empty, it will get stuck in a tight loop looking at the wrong queue forever while new keys are added to the new queue.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years
[JBoss JIRA] (ISPN-5568) KeyAffinityService race condition on view change
by Dennis Reed (JIRA)
[ https://issues.jboss.org/browse/ISPN-5568?page=com.atlassian.jira.plugin.... ]
Dennis Reed updated ISPN-5568:
------------------------------
Description:
KeyAffinityService#getKeyForAddress runs in a tight loop looking for keys:
{noformat}
queue = address2key.get(address)
while (result == null)
result = queue.poll()
{noformat}
KeyAffinityService#handleViewChange clears and resets the queue list on membership change:
{noformat}
address2key.clear()
for each address
map.put(address, new queue)
{noformat}
If a view change comes in after getKeyForAddress gets the queue, and the queue is empty, it will get stuck in a tight loop looking at the wrong queue forever while new keys are added to the new queue.
was:
KeyAffinityService#getKeyForAddress runs in a tight loop looking for keys:
{{
queue = address2key.get(address)
while (result == null)
result = queue.poll()
}}
KeyAffinityService#handleViewChange clears and resets the queue list on membership change:
{{
address2key.clear()
for each address
map.put(address, new queue)
}}
If a view change comes in after getKeyForAddress gets the queue, and the queue is empty, it will get stuck in a tight loop looking at the wrong queue forever while new keys are added to the new queue.
> KeyAffinityService race condition on view change
> ------------------------------------------------
>
> Key: ISPN-5568
> URL: https://issues.jboss.org/browse/ISPN-5568
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 5.2.11.Final
> Reporter: Dennis Reed
>
> KeyAffinityService#getKeyForAddress runs in a tight loop looking for keys:
> {noformat}
> queue = address2key.get(address)
> while (result == null)
> result = queue.poll()
> {noformat}
> KeyAffinityService#handleViewChange clears and resets the queue list on membership change:
> {noformat}
> address2key.clear()
> for each address
> map.put(address, new queue)
> {noformat}
> If a view change comes in after getKeyForAddress gets the queue, and the queue is empty, it will get stuck in a tight loop looking at the wrong queue forever while new keys are added to the new queue.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years
[JBoss JIRA] (ISPN-5568) KeyAffinityService race condition on view change
by Dennis Reed (JIRA)
[ https://issues.jboss.org/browse/ISPN-5568?page=com.atlassian.jira.plugin.... ]
Dennis Reed updated ISPN-5568:
------------------------------
Description:
KeyAffinityService#getKeyForAddress runs in a tight loop looking for keys:
{{
queue = address2key.get(address)
while (result == null)
result = queue.poll()
}}
KeyAffinityService#handleViewChange clears and resets the queue list on membership change:
{{
address2key.clear()
for each address
map.put(address, new queue)
}}
If a view change comes in after getKeyForAddress gets the queue, and the queue is empty, it will get stuck in a tight loop looking at the wrong queue forever while new keys are added to the new queue.
was:
KeyAffinityService#getKeyForAddress runs in a tight loop looking for keys:
queue = address2key.get(address)
while (result == null)
result = queue.poll()
KeyAffinityService#handleViewChange clears and resets the queue list on membership change:
address2key.clear()
for each address
map.put(address, new queue)
If a view change comes in after getKeyForAddress gets the queue, and the queue is empty, it will get stuck in tight loop looking at the wrong queue forever while new keys are added to the new queue.
> KeyAffinityService race condition on view change
> ------------------------------------------------
>
> Key: ISPN-5568
> URL: https://issues.jboss.org/browse/ISPN-5568
> Project: Infinispan
> Issue Type: Bug
> Components: Core
> Affects Versions: 5.2.11.Final
> Reporter: Dennis Reed
>
> KeyAffinityService#getKeyForAddress runs in a tight loop looking for keys:
> {{
> queue = address2key.get(address)
> while (result == null)
> result = queue.poll()
> }}
> KeyAffinityService#handleViewChange clears and resets the queue list on membership change:
> {{
> address2key.clear()
> for each address
> map.put(address, new queue)
> }}
> If a view change comes in after getKeyForAddress gets the queue, and the queue is empty, it will get stuck in a tight loop looking at the wrong queue forever while new keys are added to the new queue.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years
[JBoss JIRA] (ISPN-5568) KeyAffinityService race condition on view change
by Dennis Reed (JIRA)
Dennis Reed created ISPN-5568:
---------------------------------
Summary: KeyAffinityService race condition on view change
Key: ISPN-5568
URL: https://issues.jboss.org/browse/ISPN-5568
Project: Infinispan
Issue Type: Bug
Components: Core
Affects Versions: 5.2.11.Final
Reporter: Dennis Reed
KeyAffinityService#getKeyForAddress runs in a tight loop looking for keys:
queue = address2key.get(address)
while (result == null)
result = queue.poll()
KeyAffinityService#handleViewChange clears and resets the queue list on membership change:
address2key.clear()
for each address
map.put(address, new queue)
If a view change comes in after getKeyForAddress gets the queue, and the queue is empty, it will get stuck in tight loop looking at the wrong queue forever while new keys are added to the new queue.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years
[JBoss JIRA] (ISPN-5523) Enabling eager caching can lead to sever throwing "OutOfMemoryError: Direct buffer memory"
by Tristan Tarrant (JIRA)
[ https://issues.jboss.org/browse/ISPN-5523?page=com.atlassian.jira.plugin.... ]
Tristan Tarrant updated ISPN-5523:
----------------------------------
Fix Version/s: 7.2.4.Final
(was: 7.2.3.Final)
> Enabling eager caching can lead to sever throwing "OutOfMemoryError: Direct buffer memory"
> -------------------------------------------------------------------------------------------
>
> Key: ISPN-5523
> URL: https://issues.jboss.org/browse/ISPN-5523
> Project: Infinispan
> Issue Type: Bug
> Components: Remote Protocols
> Affects Versions: 7.2.2.Final, 8.0.0.Alpha1
> Reporter: Galder Zamarreño
> Assignee: Galder Zamarreño
> Fix For: 8.0.0.Beta1, 7.2.4.Final, 8.0.0.Final
>
>
> Some near caching tests are throwing:
> {code}
> [0m[31m04:11:24,499 ERROR [org.infinispan.server.hotrod.CacheDecodeContext] (HotRodServerWorker-43) ISPN005009: Unexpected error before any request parameters read: java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:658) [rt.jar:1.7.0_75]
> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) [rt.jar:1.7.0_75]
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) [rt.jar:1.7.0_75]
> at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:433) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:179) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.buffer.PoolArena.allocate(PoolArena.java:168) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.buffer.PoolArena.allocate(PoolArena.java:98) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:241) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:146) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:107) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:106) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:494) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:461) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:378) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:350) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) [netty-all-4.0.18.Final-redhat-1.jar:4.0.18.Final-redhat-1]
> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_75]
> {code}
> KeyValueVersionConverter allocates a byte buffer but does not release it. It could be the cause...
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
9 years