December 2020 - infinispan-issues

[Red Hat JIRA] (ISPN-12598) Hot Rod java client retries too many times

by Dan Berindei (Jira)

[ https://issues.redhat.com/browse/ISPN-12598?page=com.atlassian.jira.plugi... ] Dan Berindei updated ISPN-12598: -------------------------------- Description: The Java Hot Rod client has a {{maxRetries}} configuration option which tells it how many times to retry an operation after a failure (default: 10). When the number of retries is exceeded, the client does not fail immediately: instead, it tries to switch to another site, and tries {{maxRetries}} times on the new site as well. The client doesn't keep track of the clusters it switched off of, so it seems possible to go in an infinite loop, switching from one site to the next. If the client cannot switch to another site (e.g. because it was configured with a single site), it logs a debug message (`Cluster might have completely shut down, try resetting transport layer and topology id`) and tries the current site again for {{maxRetries}} times. So the actual number of retries with a single site is {{2 * maxRetries}} (or 1, if {{maxRetries == 0}}). Maybe automatic site switching is a good idea in some cases, but I'm not convinced it should be the default behaviour. At the very least, site switching should be decided at the remote cache manager level, when the client fails to open a new connection to any server in the current site, and not based on the number of retries done for any particular operation. was: The Java Hot Rod client has a {{maxRetries}} configuration option which tells it how many times to retry an operation after a failure (default: 10). When the number of retries is exceeded, the client does not fail immediately: instead, it tries to switch to another site, and tries {{maxRetries}} times on the new site as well. The client doesn't keep track of the clusters it switched off of, so it seems possible to go in an infinite loop, switching from one site to the next. If the client cannot switch to another site (e.g. because it was configured with a single site), it logs a debug message (`Cluster might have completely shut down, try resetting transport layer and topology id`) and tries the current site again for {{maxRetries}} times. So the actual number of retries with a single site is {{2 * maxRetries}}. Maybe automatic site switching is a good idea in some cases, but I'm not convinced it should be the default behaviour. At the very least, site switching should be decided at the remote cache manager level, when the client fails to open a new connection to any server in the current site, and not based on the number of retries done for any particular operation. > Hot Rod java client retries too many times > ------------------------------------------ > > Key: ISPN-12598 > URL: https://issues.redhat.com/browse/ISPN-12598 > Project: Infinispan > Issue Type: Bug > Components: Hot Rod > Affects Versions: 12.0.0.CR1 > Reporter: Dan Berindei > Assignee: Dan Berindei > Priority: Major > Fix For: 12.0.0.Final > > > The Java Hot Rod client has a {{maxRetries}} configuration option which tells it how many times to retry an operation after a failure (default: 10). > When the number of retries is exceeded, the client does not fail immediately: instead, it tries to switch to another site, and tries {{maxRetries}} times on the new site as well. The client doesn't keep track of the clusters it switched off of, so it seems possible to go in an infinite loop, switching from one site to the next. > If the client cannot switch to another site (e.g. because it was configured with a single site), it logs a debug message (`Cluster might have completely shut down, try resetting transport layer and topology id`) and tries the current site again for {{maxRetries}} times. So the actual number of retries with a single site is {{2 * maxRetries}} (or 1, if {{maxRetries == 0}}). > Maybe automatic site switching is a good idea in some cases, but I'm not convinced it should be the default behaviour. At the very least, site switching should be decided at the remote cache manager level, when the client fails to open a new connection to any server in the current site, and not based on the number of retries done for any particular operation. -- This message was sent by Atlassian Jira (v7.13.8#713008)

5 years

1
0
0 / 0

[Red Hat JIRA] (ISPN-12596) Hotrod client tries to read one byte to much when Multimap returns KeyDoesNotExist

by Dan Berindei (Jira)

[ https://issues.redhat.com/browse/ISPN-12596?page=com.atlassian.jira.plugi... ] Dan Berindei updated ISPN-12596: -------------------------------- Fix Version/s: 12.0.0.Final > Hotrod client tries to read one byte to much when Multimap returns KeyDoesNotExist > ---------------------------------------------------------------------------------- > > Key: ISPN-12596 > URL: https://issues.redhat.com/browse/ISPN-12596 > Project: Infinispan > Issue Type: Bug > Components: Hot Rod > Affects Versions: 12.0.0.Dev07, 11.0.8.Final > Reporter: Stephan Köll > Assignee: Dan Berindei > Priority: Major > Fix For: 12.0.0.Final > > > When using a mutimap.get(key) and the key is not in the multi map, a warning is printed: > {{WARN [org.infinispan.HOTROD] (HotRod-client-async-pool-4-16) ISPN004098: Closing connection [...] due to transport error: org.infinispan.client.hotrod.exceptions.InvalidResponseException:: ISPN004003: Invalid magic number. Expected 0xa1 and received 0x0 ?}} > > This message could be traced back to here: > [https://github.com/infinispan/infinispan/blob/12.0.0.Dev07/client/hotrod-...] > the package looks like: /_magic:_/0xA1, /_VLong:messageId:_/0x22, /_GET_MULTIMAP_RESPONSE:_/0x68, /_OperationStatus.KeyDoesNotExist:_/0x02, /_no topology change:_/0x00, /_???_/0x00 > The problem is the extra 0x00 (compare to the warning) > The "extra bytes" are the results size, in the case of "NOT_EXIST" 0 > [https://github.com/infinispan/infinispan/blob/12.0.0.Dev07/server/hotrod/...] > But in the *KeyDoesNotExist* case Multimap does not read the extra bytes. > Related Issue: https://issues.redhat.com/browse/ISPN-11630 > Most likely introduced while fixing that issue: > [https://github.com/infinispan/infinispan/commit/70f40926971d4769b24b13c1b...] > > > Related Chat: https://infinispan.zulipchat.com/#narrow/stream/118645-infinispan/t... -- This message was sent by Atlassian Jira (v7.13.8#713008)

5 years

1
0
0 / 0

[Red Hat JIRA] (ISPN-12596) Hotrod client tries to read one byte to much when Multimap returns KeyDoesNotExist

by Dan Berindei (Jira)

[ https://issues.redhat.com/browse/ISPN-12596?page=com.atlassian.jira.plugi... ] Dan Berindei updated ISPN-12596: -------------------------------- Status: Pull Request Sent (was: Open) Git Pull Request: https://github.com/infinispan/infinispan/pull/8953 > Hotrod client tries to read one byte to much when Multimap returns KeyDoesNotExist > ---------------------------------------------------------------------------------- > > Key: ISPN-12596 > URL: https://issues.redhat.com/browse/ISPN-12596 > Project: Infinispan > Issue Type: Bug > Components: Hot Rod > Affects Versions: 12.0.0.Dev07, 11.0.8.Final > Reporter: Stephan Köll > Assignee: Dan Berindei > Priority: Major > > When using a mutimap.get(key) and the key is not in the multi map, a warning is printed: > {{WARN [org.infinispan.HOTROD] (HotRod-client-async-pool-4-16) ISPN004098: Closing connection [...] due to transport error: org.infinispan.client.hotrod.exceptions.InvalidResponseException:: ISPN004003: Invalid magic number. Expected 0xa1 and received 0x0 ?}} > > This message could be traced back to here: > [https://github.com/infinispan/infinispan/blob/12.0.0.Dev07/client/hotrod-...] > the package looks like: /_magic:_/0xA1, /_VLong:messageId:_/0x22, /_GET_MULTIMAP_RESPONSE:_/0x68, /_OperationStatus.KeyDoesNotExist:_/0x02, /_no topology change:_/0x00, /_???_/0x00 > The problem is the extra 0x00 (compare to the warning) > The "extra bytes" are the results size, in the case of "NOT_EXIST" 0 > [https://github.com/infinispan/infinispan/blob/12.0.0.Dev07/server/hotrod/...] > But in the *KeyDoesNotExist* case Multimap does not read the extra bytes. > Related Issue: https://issues.redhat.com/browse/ISPN-11630 > Most likely introduced while fixing that issue: > [https://github.com/infinispan/infinispan/commit/70f40926971d4769b24b13c1b...] > > > Related Chat: https://infinispan.zulipchat.com/#narrow/stream/118645-infinispan/t... -- This message was sent by Atlassian Jira (v7.13.8#713008)

5 years

1
0
0 / 0

[Red Hat JIRA] (ISPN-12598) Hot Rod java client retries too many times

by Dan Berindei (Jira)

[ https://issues.redhat.com/browse/ISPN-12598?page=com.atlassian.jira.plugi... ] Dan Berindei commented on ISPN-12598: ------------------------------------- Even with {{maxRetries = 0}}, the client retries operations at least once, and the extra retries hide errors in the handling of server responses like in ISPN-12596 and ISPN-11630. > Hot Rod java client retries too many times > ------------------------------------------ > > Key: ISPN-12598 > URL: https://issues.redhat.com/browse/ISPN-12598 > Project: Infinispan > Issue Type: Bug > Components: Hot Rod > Affects Versions: 12.0.0.CR1 > Reporter: Dan Berindei > Assignee: Dan Berindei > Priority: Major > Fix For: 12.0.0.Final > > > The Java Hot Rod client has a {{maxRetries}} configuration option which tells it how many times to retry an operation after a failure (default: 10). > When the number of retries is exceeded, the client does not fail immediately: instead, it tries to switch to another site, and tries {{maxRetries}} times on the new site as well. The client doesn't keep track of the clusters it switched off of, so it seems possible to go in an infinite loop, switching from one site to the next. > If the client cannot switch to another site (e.g. because it was configured with a single site), it logs a debug message (`Cluster might have completely shut down, try resetting transport layer and topology id`) and tries the current site again for {{maxRetries}} times. So the actual number of retries with a single site is {{2 * maxRetries}}. > Maybe automatic site switching is a good idea in some cases, but I'm not convinced it should be the default behaviour. At the very least, site switching should be decided at the remote cache manager level, when the client fails to open a new connection to any server in the current site, and not based on the number of retries done for any particular operation. -- This message was sent by Atlassian Jira (v7.13.8#713008)

5 years

1
0
0 / 0

[Red Hat JIRA] (ISPN-12598) Hot Rod java client retries too many times

by Dan Berindei (Jira)

[ https://issues.redhat.com/browse/ISPN-12598?page=com.atlassian.jira.plugi... ] Dan Berindei updated ISPN-12598: -------------------------------- Status: Open (was: New) > Hot Rod java client retries too many times > ------------------------------------------ > > Key: ISPN-12598 > URL: https://issues.redhat.com/browse/ISPN-12598 > Project: Infinispan > Issue Type: Bug > Components: Hot Rod > Affects Versions: 12.0.0.CR1 > Reporter: Dan Berindei > Assignee: Dan Berindei > Priority: Major > Fix For: 12.0.0.Final > > > The Java Hot Rod client has a {{maxRetries}} configuration option which tells it how many times to retry an operation after a failure (default: 10). > When the number of retries is exceeded, the client does not fail immediately: instead, it tries to switch to another site, and tries {{maxRetries}} times on the new site as well. The client doesn't keep track of the clusters it switched off of, so it seems possible to go in an infinite loop, switching from one site to the next. > If the client cannot switch to another site (e.g. because it was configured with a single site), it logs a debug message (`Cluster might have completely shut down, try resetting transport layer and topology id`) and tries the current site again for {{maxRetries}} times. So the actual number of retries with a single site is {{2 * maxRetries}}. > Maybe automatic site switching is a good idea in some cases, but I'm not convinced it should be the default behaviour. At the very least, site switching should be decided at the remote cache manager level, when the client fails to open a new connection to any server in the current site, and not based on the number of retries done for any particular operation. -- This message was sent by Atlassian Jira (v7.13.8#713008)

5 years

1
0
0 / 0

[Red Hat JIRA] (ISPN-12598) Hot Rod java client retries too many times

by Dan Berindei (Jira)

Dan Berindei created ISPN-12598: ----------------------------------- Summary: Hot Rod java client retries too many times Key: ISPN-12598 URL: https://issues.redhat.com/browse/ISPN-12598 Project: Infinispan Issue Type: Bug Components: Hot Rod Affects Versions: 12.0.0.CR1 Reporter: Dan Berindei Assignee: Dan Berindei Fix For: 12.0.0.Final The Java Hot Rod client has a {{maxRetries}} configuration option which tells it how many times to retry an operation after a failure (default: 10). When the number of retries is exceeded, the client does not fail immediately: instead, it tries to switch to another site, and tries {{maxRetries}} times on the new site as well. The client doesn't keep track of the clusters it switched off of, so it seems possible to go in an infinite loop, switching from one site to the next. If the client cannot switch to another site (e.g. because it was configured with a single site), it logs a debug message (`Cluster might have completely shut down, try resetting transport layer and topology id`) and tries the current site again for {{maxRetries}} times. So the actual number of retries with a single site is {{2 * maxRetries}}. Maybe automatic site switching is a good idea in some cases, but I'm not convinced it should be the default behaviour. At the very least, site switching should be decided at the remote cache manager level, when the client fails to open a new connection to any server in the current site, and not based on the number of retries done for any particular operation. -- This message was sent by Atlassian Jira (v7.13.8#713008)

5 years

1
0
0 / 0

[Red Hat JIRA] (ISPN-12597) Disable JGroups error JGRP000006 stacktarce when configuring xsite on OCP and AWS LB

by Dmitry Volodin (Jira)

Dmitry Volodin created ISPN-12597: ------------------------------------- Summary: Disable JGroups error JGRP000006 stacktarce when configuring xsite on OCP and AWS LB Key: ISPN-12597 URL: https://issues.redhat.com/browse/ISPN-12597 Project: Infinispan Issue Type: Enhancement Components: Cross-Site Replication, OpenShift, Operator Affects Versions: 11.0.7.Final Reporter: Dmitry Volodin Assignee: Pedro Ruivo When configuring Cross-Site on OCP with operator and LoadBalancer on AWS (probably, will occurs on Azure and others cloud platforms) there are a large number of stack traces places inside Pod logs {code:java} 07:51:13,524 WARN (TcpServer.Acceptor[7900]-2,relay-global,_example-infinispan-1-11053:site1) [org.jgroups.protocols.TCP] JGRP000006: failed accepting connection from peer Socket[addr=/10.131.0.1,port=11781,localport=7900]: java.io.EOFException java.io.EOFException at java.base/java.io.DataInputStream.readFully(DataInputStream.java:202) at org.jgroups.blocks.cs.TcpConnection.readPeerAddress(TcpConnection.java:247) at org.jgroups.blocks.cs.TcpConnection.<init>(TcpConnection.java:53) at org.jgroups.blocks.cs.TcpServer$Acceptor.handleAccept(TcpServer.java:126) at org.jgroups.blocks.cs.TcpServer$Acceptor.run(TcpServer.java:111) at java.base/java.lang.Thread.run(Thread.java:834){code} The reason of this logs that's AWS/OCP pings LoadBalancer with probe each time with interval. No the same errors while internal (Service) cross-site configuration was made. It would be nice to have an option to disable this kind of traces probably for LB configuration only, as customers confusing this traces as a problem -- This message was sent by Atlassian Jira (v7.13.8#713008)

5 years

1
0
0 / 0

[Red Hat JIRA] (ISPN-12596) Hotrod client tries to read one byte to much when Multimap returns KeyDoesNotExist

by Dan Berindei (Jira)

[ https://issues.redhat.com/browse/ISPN-12596?page=com.atlassian.jira.plugi... ] Dan Berindei updated ISPN-12596: -------------------------------- Status: Open (was: New) > Hotrod client tries to read one byte to much when Multimap returns KeyDoesNotExist > ---------------------------------------------------------------------------------- > > Key: ISPN-12596 > URL: https://issues.redhat.com/browse/ISPN-12596 > Project: Infinispan > Issue Type: Bug > Components: Hot Rod > Affects Versions: 12.0.0.Dev07, 11.0.8.Final > Reporter: Stephan Köll > Assignee: Dan Berindei > Priority: Major > > When using a mutimap.get(key) and the key is not in the multi map, a warning is printed: > {{WARN [org.infinispan.HOTROD] (HotRod-client-async-pool-4-16) ISPN004098: Closing connection [...] due to transport error: org.infinispan.client.hotrod.exceptions.InvalidResponseException:: ISPN004003: Invalid magic number. Expected 0xa1 and received 0x0 ?}} > > This message could be traced back to here: > [https://github.com/infinispan/infinispan/blob/12.0.0.Dev07/client/hotrod-...] > the package looks like: /_magic:_/0xA1, /_VLong:messageId:_/0x22, /_GET_MULTIMAP_RESPONSE:_/0x68, /_OperationStatus.KeyDoesNotExist:_/0x02, /_no topology change:_/0x00, /_???_/0x00 > The problem is the extra 0x00 (compare to the warning) > The "extra bytes" are the results size, in the case of "NOT_EXIST" 0 > [https://github.com/infinispan/infinispan/blob/12.0.0.Dev07/server/hotrod/...] > But in the *KeyDoesNotExist* case Multimap does not read the extra bytes. > Related Issue: https://issues.redhat.com/browse/ISPN-11630 > Most likely introduced while fixing that issue: > [https://github.com/infinispan/infinispan/commit/70f40926971d4769b24b13c1b...] > > > Related Chat: https://infinispan.zulipchat.com/#narrow/stream/118645-infinispan/t... -- This message was sent by Atlassian Jira (v7.13.8#713008)

5 years

1
0
0 / 0

[Red Hat JIRA] (ISPN-12596) Hotrod client tries to read one byte to much when Multimap returns KeyDoesNotExist

by Dan Berindei (Jira)

[ https://issues.redhat.com/browse/ISPN-12596?page=com.atlassian.jira.plugi... ] Dan Berindei reassigned ISPN-12596: ----------------------------------- Assignee: Dan Berindei > Hotrod client tries to read one byte to much when Multimap returns KeyDoesNotExist > ---------------------------------------------------------------------------------- > > Key: ISPN-12596 > URL: https://issues.redhat.com/browse/ISPN-12596 > Project: Infinispan > Issue Type: Bug > Components: Hot Rod > Affects Versions: 12.0.0.Dev07, 11.0.8.Final > Reporter: Stephan Köll > Assignee: Dan Berindei > Priority: Major > > When using a mutimap.get(key) and the key is not in the multi map, a warning is printed: > {{WARN [org.infinispan.HOTROD] (HotRod-client-async-pool-4-16) ISPN004098: Closing connection [...] due to transport error: org.infinispan.client.hotrod.exceptions.InvalidResponseException:: ISPN004003: Invalid magic number. Expected 0xa1 and received 0x0 ?}} > > This message could be traced back to here: > [https://github.com/infinispan/infinispan/blob/12.0.0.Dev07/client/hotrod-...] > the package looks like: /_magic:_/0xA1, /_VLong:messageId:_/0x22, /_GET_MULTIMAP_RESPONSE:_/0x68, /_OperationStatus.KeyDoesNotExist:_/0x02, /_no topology change:_/0x00, /_???_/0x00 > The problem is the extra 0x00 (compare to the warning) > The "extra bytes" are the results size, in the case of "NOT_EXIST" 0 > [https://github.com/infinispan/infinispan/blob/12.0.0.Dev07/server/hotrod/...] > But in the *KeyDoesNotExist* case Multimap does not read the extra bytes. > Related Issue: https://issues.redhat.com/browse/ISPN-11630 > Most likely introduced while fixing that issue: > [https://github.com/infinispan/infinispan/commit/70f40926971d4769b24b13c1b...] > > > Related Chat: https://infinispan.zulipchat.com/#narrow/stream/118645-infinispan/t... -- This message was sent by Atlassian Jira (v7.13.8#713008)

5 years

1
0
0 / 0

[Red Hat JIRA] (ISPN-11614) PersistenceMarshaller should utilise an independent SerializationContext for user types

by Tristan Tarrant (Jira)

[ https://issues.redhat.com/browse/ISPN-11614?page=com.atlassian.jira.plugi... ] Tristan Tarrant closed ISPN-11614. ---------------------------------- > PersistenceMarshaller should utilise an independent SerializationContext for user types > --------------------------------------------------------------------------------------- > > Key: ISPN-11614 > URL: https://issues.redhat.com/browse/ISPN-11614 > Project: Infinispan > Issue Type: Bug > Components: Core > Affects Versions: 10.1.5.Final, 11.0.0.Dev03 > Reporter: Ryan Emerson > Assignee: Ryan Emerson > Priority: Major > Fix For: 12.0.0.CR1 > > > If a user configures a SerializationContextInitializer it is currently registered to the same SerializationContext as the persistent internal types. This means that it is possible for a user to reference internal protobuf message types in their .proto files. > We should introduce the following: > # A "user" SerializationContext that is used exclusively for user types configured via xml/service-loader/programmatically: > User types are only registered with this context. The global and persistence contexts do not register these types. If a type does not exist in the global/persistence context, then it's wrapped with a MarshallableUserObject and marshalling is delegated to the user marshaller. Previously this only occurred if a custom marshaller was defined as user types were added to the global/persistence context, but as this is no longer the case all user types are wrapped. > # A USER_MARSHALLER created by MarshallerFactory, which if a custom marshaller is not defined creates a ProtoStream based marshaller that consumes the user SerializationContext, if a custom marshaller is defined then that is returned: > This logic was previously in the PersistenceMarshallerImpl, but this allows us to inject the marshaller directly into various places in the code. > # DelegatingUserMarshaller. A wrapper class to call the start/stop of the configured user marshaller and log the marshaller being used. -- This message was sent by Atlassian Jira (v7.13.8#713008)

5 years

1
0
0 / 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

infinispan-issues December 2020