5:05 p.m.
Galder Zamarreño created ISPN-2005:
--------------------------------------
Summary: Memcached server throwing UnknownOperationException with many
clients
Key: ISPN-2005
URL: https://issues.jboss.org/browse/ISPN-2005
Project: Infinispan
Issue Type: Bug
Components: Cache Server
Affects Versions: 5.1.3.FINAL
Reporter: Galder Zamarreño
Assignee: Galder Zamarreño
Fix For: 5.1.4.FINAL
With native memcached out of the box, I can execute such a test without any problems:
1000 clients (threads), 400 operations per thread, 34% writes
However, running Infinispan Memcached distribution with 1000 worker threads results in
exceptions such as:
{code}2012-04-23 16:46:59,893 ERROR (MemcachedServerWorker-1-957)
[org.infinispan.server.memcached.MemcachedDecoder] ISPN005003: Exception reported
org.infinispan.server.core.UnknownOperationException: Unknown operation:
at
org.infinispan.server.memcached.RequestResolver$.toRequest(MemcachedDecoder.scala:659)
at
org.infinispan.server.memcached.MemcachedDecoder.readHeader(MemcachedDecoder.scala:73)
at
org.infinispan.server.core.AbstractProtocolDecoder.decodeHeader(AbstractProtocolDecoder.scala:92)
at
org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:69)
at
org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:45)
at
org.infinispan.server.core.transport.CustomReplayingDecoder.callDecode(CustomReplayingDecoder.java:250)
at
org.infinispan.server.core.transport.CustomReplayingDecoder.messageReceived(CustomReplayingDecoder.java:223)
at
org.infinispan.server.core.AbstractProtocolDecoder.messageReceived(AbstractProtocolDecoder.scala:360)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680){code}
Client shows:
{code}2012-04-23 16:46:28.742 INFO net.spy.memcached.MemcachedConnection: Reconnecting
due to exception on {QA sa=/127.0.0.1:11211, #Rops=0, #Wops=0, #iq=0, topRop=null,
topWop=null, toWrite=0, interested=0}
java.net.SocketException: Connection reset by peer
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:369)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:242)
at net.spy.memcached.MemcachedConnection.run(MemcachedConnection.java:833){code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:30 p.m.
[
https://issues.jboss.org/browse/ISPN-2005?page=com.atlassian.jira.plugin....
]
Galder Zamarreño commented on ISPN-2005:
----------------------------------------
This can be replicated even with 300 threads.
Memcached server throwing UnknownOperationException with many
clients
---------------------------------------------------------------------
Key: ISPN-2005
URL: https://issues.jboss.org/browse/ISPN-2005
Project: Infinispan
Issue Type: Bug
Components: Cache Server
Affects Versions: 5.1.3.FINAL
Reporter: Galder Zamarreño
Assignee: Galder Zamarreño
Fix For: 5.1.4.FINAL
With native memcached out of the box, I can execute such a test without any problems:
1000 clients (threads), 400 operations per thread, 34% writes
However, running Infinispan Memcached distribution with 1000 worker threads results in
exceptions such as:
{code}2012-04-23 16:46:59,893 ERROR (MemcachedServerWorker-1-957)
[org.infinispan.server.memcached.MemcachedDecoder] ISPN005003: Exception reported
org.infinispan.server.core.UnknownOperationException: Unknown operation:
at
org.infinispan.server.memcached.RequestResolver$.toRequest(MemcachedDecoder.scala:659)
at
org.infinispan.server.memcached.MemcachedDecoder.readHeader(MemcachedDecoder.scala:73)
at
org.infinispan.server.core.AbstractProtocolDecoder.decodeHeader(AbstractProtocolDecoder.scala:92)
at
org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:69)
at
org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:45)
at
org.infinispan.server.core.transport.CustomReplayingDecoder.callDecode(CustomReplayingDecoder.java:250)
at
org.infinispan.server.core.transport.CustomReplayingDecoder.messageReceived(CustomReplayingDecoder.java:223)
at
org.infinispan.server.core.AbstractProtocolDecoder.messageReceived(AbstractProtocolDecoder.scala:360)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680){code}
Client shows:
{code}2012-04-23 16:46:28.742 INFO net.spy.memcached.MemcachedConnection: Reconnecting
due to exception on {QA sa=/127.0.0.1:11211, #Rops=0, #Wops=0, #iq=0, topRop=null,
topWop=null, toWrite=0, interested=0}
java.net.SocketException: Connection reset by peer
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:369)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:242)
at net.spy.memcached.MemcachedConnection.run(MemcachedConnection.java:833){code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:58 p.m.
[
https://issues.jboss.org/browse/ISPN-2005?page=com.atlassian.jira.plugin....
]
Galder Zamarreño commented on ISPN-2005:
----------------------------------------
I was using Spymemcached 2.8.1, but the issue is also present in Spymemcached 2.7 which is
the version used by QE. I'm surprised that they have not seen this before. Something
similar has been reported before but not sure if it's the same cause:
https://bugzilla.redhat.com/show_bug.cgi?id=801296
Memcached server throwing UnknownOperationException with many
clients
---------------------------------------------------------------------
Key: ISPN-2005
URL: https://issues.jboss.org/browse/ISPN-2005
Project: Infinispan
Issue Type: Bug
Components: Cache Server
Affects Versions: 5.1.3.FINAL
Reporter: Galder Zamarreño
Assignee: Galder Zamarreño
Fix For: 5.1.4.FINAL
With native memcached out of the box, I can execute such a test without any problems:
1000 clients (threads), 400 operations per thread, 34% writes
However, running Infinispan Memcached distribution with 1000 worker threads results in
exceptions such as:
{code}2012-04-23 16:46:59,893 ERROR (MemcachedServerWorker-1-957)
[org.infinispan.server.memcached.MemcachedDecoder] ISPN005003: Exception reported
org.infinispan.server.core.UnknownOperationException: Unknown operation:
at
org.infinispan.server.memcached.RequestResolver$.toRequest(MemcachedDecoder.scala:659)
at
org.infinispan.server.memcached.MemcachedDecoder.readHeader(MemcachedDecoder.scala:73)
at
org.infinispan.server.core.AbstractProtocolDecoder.decodeHeader(AbstractProtocolDecoder.scala:92)
at
org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:69)
at
org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:45)
at
org.infinispan.server.core.transport.CustomReplayingDecoder.callDecode(CustomReplayingDecoder.java:250)
at
org.infinispan.server.core.transport.CustomReplayingDecoder.messageReceived(CustomReplayingDecoder.java:223)
at
org.infinispan.server.core.AbstractProtocolDecoder.messageReceived(AbstractProtocolDecoder.scala:360)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680){code}
Client shows:
{code}2012-04-23 16:46:28.742 INFO net.spy.memcached.MemcachedConnection: Reconnecting
due to exception on {QA sa=/127.0.0.1:11211, #Rops=0, #Wops=0, #iq=0, topRop=null,
topWop=null, toWrite=0, interested=0}
java.net.SocketException: Connection reset by peer
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:369)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:242)
at net.spy.memcached.MemcachedConnection.run(MemcachedConnection.java:833){code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
4:42 p.m.
[
https://issues.jboss.org/browse/ISPN-2005?page=com.atlassian.jira.plugin....
]
Work on ISPN-2005 started by Galder Zamarreño.
Memcached server throwing UnknownOperationException with many
clients
---------------------------------------------------------------------
Key: ISPN-2005
URL: https://issues.jboss.org/browse/ISPN-2005
Project: Infinispan
Issue Type: Bug
Components: Cache Server
Affects Versions: 5.1.3.FINAL
Reporter: Galder Zamarreño
Assignee: Galder Zamarreño
Fix For: 5.1.4.FINAL
With native memcached out of the box, I can execute such a test without any problems:
1000 clients (threads), 400 operations per thread, 34% writes
However, running Infinispan Memcached distribution with 1000 worker threads results in
exceptions such as:
{code}2012-04-23 16:46:59,893 ERROR (MemcachedServerWorker-1-957)
[org.infinispan.server.memcached.MemcachedDecoder] ISPN005003: Exception reported
org.infinispan.server.core.UnknownOperationException: Unknown operation:
at
org.infinispan.server.memcached.RequestResolver$.toRequest(MemcachedDecoder.scala:659)
at
org.infinispan.server.memcached.MemcachedDecoder.readHeader(MemcachedDecoder.scala:73)
at
org.infinispan.server.core.AbstractProtocolDecoder.decodeHeader(AbstractProtocolDecoder.scala:92)
at
org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:69)
at
org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:45)
at
org.infinispan.server.core.transport.CustomReplayingDecoder.callDecode(CustomReplayingDecoder.java:250)
at
org.infinispan.server.core.transport.CustomReplayingDecoder.messageReceived(CustomReplayingDecoder.java:223)
at
org.infinispan.server.core.AbstractProtocolDecoder.messageReceived(AbstractProtocolDecoder.scala:360)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680){code}
Client shows:
{code}2012-04-23 16:46:28.742 INFO net.spy.memcached.MemcachedConnection: Reconnecting
due to exception on {QA sa=/127.0.0.1:11211, #Rops=0, #Wops=0, #iq=0, topRop=null,
topWop=null, toWrite=0, interested=0}
java.net.SocketException: Connection reset by peer
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:369)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:242)
at net.spy.memcached.MemcachedConnection.run(MemcachedConnection.java:833){code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
5:05 p.m.
[
https://issues.jboss.org/browse/ISPN-2005?page=com.atlassian.jira.plugin....
]
Galder Zamarreño commented on ISPN-2005:
----------------------------------------
Some interesting info: I added TRACE to server on org.infinispan and I replicated it with
300 threads, but couldn't find anything conclusive. Once I added debug logs for the
client (spymemcached), I cannot replicate the issue any more, not even with 500 or 1000
threads. So, this smells like a problem in spymemcached itself.
Memcached server throwing UnknownOperationException with many
clients
---------------------------------------------------------------------
Key: ISPN-2005
URL: https://issues.jboss.org/browse/ISPN-2005
Project: Infinispan
Issue Type: Bug
Components: Cache Server
Affects Versions: 5.1.3.FINAL
Reporter: Galder Zamarreño
Assignee: Galder Zamarreño
Fix For: 5.1.4.FINAL
With native memcached out of the box, I can execute such a test without any problems:
1000 clients (threads), 400 operations per thread, 34% writes
However, running Infinispan Memcached distribution with 1000 worker threads results in
exceptions such as:
{code}2012-04-23 16:46:59,893 ERROR (MemcachedServerWorker-1-957)
[org.infinispan.server.memcached.MemcachedDecoder] ISPN005003: Exception reported
org.infinispan.server.core.UnknownOperationException: Unknown operation:
at
org.infinispan.server.memcached.RequestResolver$.toRequest(MemcachedDecoder.scala:659)
at
org.infinispan.server.memcached.MemcachedDecoder.readHeader(MemcachedDecoder.scala:73)
at
org.infinispan.server.core.AbstractProtocolDecoder.decodeHeader(AbstractProtocolDecoder.scala:92)
at
org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:69)
at
org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:45)
at
org.infinispan.server.core.transport.CustomReplayingDecoder.callDecode(CustomReplayingDecoder.java:250)
at
org.infinispan.server.core.transport.CustomReplayingDecoder.messageReceived(CustomReplayingDecoder.java:223)
at
org.infinispan.server.core.AbstractProtocolDecoder.messageReceived(AbstractProtocolDecoder.scala:360)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680){code}
Client shows:
{code}2012-04-23 16:46:28.742 INFO net.spy.memcached.MemcachedConnection: Reconnecting
due to exception on {QA sa=/127.0.0.1:11211, #Rops=0, #Wops=0, #iq=0, topRop=null,
topWop=null, toWrite=0, interested=0}
java.net.SocketException: Connection reset by peer
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:369)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:242)
at net.spy.memcached.MemcachedConnection.run(MemcachedConnection.java:833){code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
6:51 p.m.
[
https://issues.jboss.org/browse/ISPN-2005?page=com.atlassian.jira.plugin....
]
Galder Zamarreño commented on ISPN-2005:
----------------------------------------
I've just tried with the Python Memcached client with 300 threads and showed no
problems whatsoever. This really smells like an issue in Spymemcached which is both
present in 2.7 and 2.8.1. I'll submit an issue in Spymemcached asap.
Memcached server throwing UnknownOperationException with many
clients
---------------------------------------------------------------------
Key: ISPN-2005
URL: https://issues.jboss.org/browse/ISPN-2005
Project: Infinispan
Issue Type: Bug
Components: Cache Server
Affects Versions: 5.1.3.FINAL
Reporter: Galder Zamarreño
Assignee: Galder Zamarreño
Fix For: 5.1.4.FINAL
With native memcached out of the box, I can execute such a test without any problems:
1000 clients (threads), 400 operations per thread, 34% writes
However, running Infinispan Memcached distribution with 1000 worker threads results in
exceptions such as:
{code}2012-04-23 16:46:59,893 ERROR (MemcachedServerWorker-1-957)
[org.infinispan.server.memcached.MemcachedDecoder] ISPN005003: Exception reported
org.infinispan.server.core.UnknownOperationException: Unknown operation:
at
org.infinispan.server.memcached.RequestResolver$.toRequest(MemcachedDecoder.scala:659)
at
org.infinispan.server.memcached.MemcachedDecoder.readHeader(MemcachedDecoder.scala:73)
at
org.infinispan.server.core.AbstractProtocolDecoder.decodeHeader(AbstractProtocolDecoder.scala:92)
at
org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:69)
at
org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:45)
at
org.infinispan.server.core.transport.CustomReplayingDecoder.callDecode(CustomReplayingDecoder.java:250)
at
org.infinispan.server.core.transport.CustomReplayingDecoder.messageReceived(CustomReplayingDecoder.java:223)
at
org.infinispan.server.core.AbstractProtocolDecoder.messageReceived(AbstractProtocolDecoder.scala:360)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680){code}
Client shows:
{code}2012-04-23 16:46:28.742 INFO net.spy.memcached.MemcachedConnection: Reconnecting
due to exception on {QA sa=/127.0.0.1:11211, #Rops=0, #Wops=0, #iq=0, topRop=null,
topWop=null, toWrite=0, interested=0}
java.net.SocketException: Connection reset by peer
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:369)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:242)
at net.spy.memcached.MemcachedConnection.run(MemcachedConnection.java:833){code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
7 p.m.
[
https://issues.jboss.org/browse/ISPN-2005?page=com.atlassian.jira.plugin....
]
Manik Surtani commented on ISPN-2005:
-------------------------------------
Pls link to the issue on Spymemcached's tracker as well...
Memcached server throwing UnknownOperationException with many
clients
---------------------------------------------------------------------
Key: ISPN-2005
URL: https://issues.jboss.org/browse/ISPN-2005
Project: Infinispan
Issue Type: Bug
Components: Cache Server
Affects Versions: 5.1.3.FINAL
Reporter: Galder Zamarreño
Assignee: Galder Zamarreño
Fix For: 5.1.4.FINAL
With native memcached out of the box, I can execute such a test without any problems:
1000 clients (threads), 400 operations per thread, 34% writes
However, running Infinispan Memcached distribution with 1000 worker threads results in
exceptions such as:
{code}2012-04-23 16:46:59,893 ERROR (MemcachedServerWorker-1-957)
[org.infinispan.server.memcached.MemcachedDecoder] ISPN005003: Exception reported
org.infinispan.server.core.UnknownOperationException: Unknown operation:
at
org.infinispan.server.memcached.RequestResolver$.toRequest(MemcachedDecoder.scala:659)
at
org.infinispan.server.memcached.MemcachedDecoder.readHeader(MemcachedDecoder.scala:73)
at
org.infinispan.server.core.AbstractProtocolDecoder.decodeHeader(AbstractProtocolDecoder.scala:92)
at
org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:69)
at
org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:45)
at
org.infinispan.server.core.transport.CustomReplayingDecoder.callDecode(CustomReplayingDecoder.java:250)
at
org.infinispan.server.core.transport.CustomReplayingDecoder.messageReceived(CustomReplayingDecoder.java:223)
at
org.infinispan.server.core.AbstractProtocolDecoder.messageReceived(AbstractProtocolDecoder.scala:360)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680){code}
Client shows:
{code}2012-04-23 16:46:28.742 INFO net.spy.memcached.MemcachedConnection: Reconnecting
due to exception on {QA sa=/127.0.0.1:11211, #Rops=0, #Wops=0, #iq=0, topRop=null,
topWop=null, toWrite=0, interested=0}
java.net.SocketException: Connection reset by peer
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:369)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:242)
at net.spy.memcached.MemcachedConnection.run(MemcachedConnection.java:833){code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:51 p.m.
[
https://issues.jboss.org/browse/ISPN-2005?page=com.atlassian.jira.plugin....
]
Galder Zamarreño commented on ISPN-2005:
----------------------------------------
In spite of not being an Infinispan issue, I'm adding some code to help diagnose
similar problems in case they happen again. So, I'll be sending a pull req in a sec.
Memcached server throwing UnknownOperationException with many
clients
---------------------------------------------------------------------
Key: ISPN-2005
URL: https://issues.jboss.org/browse/ISPN-2005
Project: Infinispan
Issue Type: Bug
Components: Cache Server
Affects Versions: 5.1.3.FINAL
Reporter: Galder Zamarreño
Assignee: Galder Zamarreño
Fix For: 5.1.4.FINAL
With native memcached out of the box, I can execute such a test without any problems:
1000 clients (threads), 400 operations per thread, 34% writes
However, running Infinispan Memcached distribution with 1000 worker threads results in
exceptions such as:
{code}2012-04-23 16:46:59,893 ERROR (MemcachedServerWorker-1-957)
[org.infinispan.server.memcached.MemcachedDecoder] ISPN005003: Exception reported
org.infinispan.server.core.UnknownOperationException: Unknown operation:
at
org.infinispan.server.memcached.RequestResolver$.toRequest(MemcachedDecoder.scala:659)
at
org.infinispan.server.memcached.MemcachedDecoder.readHeader(MemcachedDecoder.scala:73)
at
org.infinispan.server.core.AbstractProtocolDecoder.decodeHeader(AbstractProtocolDecoder.scala:92)
at
org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:69)
at
org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:45)
at
org.infinispan.server.core.transport.CustomReplayingDecoder.callDecode(CustomReplayingDecoder.java:250)
at
org.infinispan.server.core.transport.CustomReplayingDecoder.messageReceived(CustomReplayingDecoder.java:223)
at
org.infinispan.server.core.AbstractProtocolDecoder.messageReceived(AbstractProtocolDecoder.scala:360)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680){code}
Client shows:
{code}2012-04-23 16:46:28.742 INFO net.spy.memcached.MemcachedConnection: Reconnecting
due to exception on {QA sa=/127.0.0.1:11211, #Rops=0, #Wops=0, #iq=0, topRop=null,
topWop=null, toWrite=0, interested=0}
java.net.SocketException: Connection reset by peer
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:369)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:242)
at net.spy.memcached.MemcachedConnection.run(MemcachedConnection.java:833){code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
7:57 p.m.
[
https://issues.jboss.org/browse/ISPN-2005?page=com.atlassian.jira.plugin....
]
Galder Zamarreño updated ISPN-2005:
-----------------------------------
Status: Pull Request Sent (was: Coding In Progress)
Git Pull Request: https://github.com/infinispan/infinispan/pull/1094
Memcached server throwing UnknownOperationException with many
clients
---------------------------------------------------------------------
Key: ISPN-2005
URL: https://issues.jboss.org/browse/ISPN-2005
Project: Infinispan
Issue Type: Bug
Components: Cache Server
Affects Versions: 5.1.3.FINAL
Reporter: Galder Zamarreño
Assignee: Galder Zamarreño
Fix For: 5.1.4.FINAL
With native memcached out of the box, I can execute such a test without any problems:
1000 clients (threads), 400 operations per thread, 34% writes
However, running Infinispan Memcached distribution with 1000 worker threads results in
exceptions such as:
{code}2012-04-23 16:46:59,893 ERROR (MemcachedServerWorker-1-957)
[org.infinispan.server.memcached.MemcachedDecoder] ISPN005003: Exception reported
org.infinispan.server.core.UnknownOperationException: Unknown operation:
at
org.infinispan.server.memcached.RequestResolver$.toRequest(MemcachedDecoder.scala:659)
at
org.infinispan.server.memcached.MemcachedDecoder.readHeader(MemcachedDecoder.scala:73)
at
org.infinispan.server.core.AbstractProtocolDecoder.decodeHeader(AbstractProtocolDecoder.scala:92)
at
org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:69)
at
org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:45)
at
org.infinispan.server.core.transport.CustomReplayingDecoder.callDecode(CustomReplayingDecoder.java:250)
at
org.infinispan.server.core.transport.CustomReplayingDecoder.messageReceived(CustomReplayingDecoder.java:223)
at
org.infinispan.server.core.AbstractProtocolDecoder.messageReceived(AbstractProtocolDecoder.scala:360)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680){code}
Client shows:
{code}2012-04-23 16:46:28.742 INFO net.spy.memcached.MemcachedConnection: Reconnecting
due to exception on {QA sa=/127.0.0.1:11211, #Rops=0, #Wops=0, #iq=0, topRop=null,
topWop=null, toWrite=0, interested=0}
java.net.SocketException: Connection reset by peer
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:369)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:242)
at net.spy.memcached.MemcachedConnection.run(MemcachedConnection.java:833){code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
12:41 p.m.
[
https://issues.jboss.org/browse/ISPN-2005?page=com.atlassian.jira.plugin....
]
Dan Berindei updated ISPN-2005:
-------------------------------
Status: Resolved (was: Pull Request Sent)
Resolution: Done
Integrated.
Memcached server throwing UnknownOperationException with many
clients
---------------------------------------------------------------------
Key: ISPN-2005
URL: https://issues.jboss.org/browse/ISPN-2005
Project: Infinispan
Issue Type: Bug
Components: Cache Server
Affects Versions: 5.1.3.FINAL
Reporter: Galder Zamarreño
Assignee: Galder Zamarreño
Fix For: 5.1.4.FINAL
With native memcached out of the box, I can execute such a test without any problems:
1000 clients (threads), 400 operations per thread, 34% writes
However, running Infinispan Memcached distribution with 1000 worker threads results in
exceptions such as:
{code}2012-04-23 16:46:59,893 ERROR (MemcachedServerWorker-1-957)
[org.infinispan.server.memcached.MemcachedDecoder] ISPN005003: Exception reported
org.infinispan.server.core.UnknownOperationException: Unknown operation:
at
org.infinispan.server.memcached.RequestResolver$.toRequest(MemcachedDecoder.scala:659)
at
org.infinispan.server.memcached.MemcachedDecoder.readHeader(MemcachedDecoder.scala:73)
at
org.infinispan.server.core.AbstractProtocolDecoder.decodeHeader(AbstractProtocolDecoder.scala:92)
at
org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:69)
at
org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:45)
at
org.infinispan.server.core.transport.CustomReplayingDecoder.callDecode(CustomReplayingDecoder.java:250)
at
org.infinispan.server.core.transport.CustomReplayingDecoder.messageReceived(CustomReplayingDecoder.java:223)
at
org.infinispan.server.core.AbstractProtocolDecoder.messageReceived(AbstractProtocolDecoder.scala:360)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680){code}
Client shows:
{code}2012-04-23 16:46:28.742 INFO net.spy.memcached.MemcachedConnection: Reconnecting
due to exception on {QA sa=/127.0.0.1:11211, #Rops=0, #Wops=0, #iq=0, topRop=null,
topWop=null, toWrite=0, interested=0}
java.net.SocketException: Connection reset by peer
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:369)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:242)
at net.spy.memcached.MemcachedConnection.run(MemcachedConnection.java:833){code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
11:58 a.m.
[
https://issues.jboss.org/browse/ISPN-2005?page=com.atlassian.jira.plugin....
]
Galder Zamarreño commented on ISPN-2005:
----------------------------------------
Spymemcached issue: http://code.google.com/p/spymemcached/issues/detail?id=250
Memcached server throwing UnknownOperationException with many
clients
---------------------------------------------------------------------
Key: ISPN-2005
URL: https://issues.jboss.org/browse/ISPN-2005
Project: Infinispan
Issue Type: Bug
Components: Cache Server
Affects Versions: 5.1.3.FINAL
Reporter: Galder Zamarreño
Assignee: Galder Zamarreño
Fix For: 5.1.4.FINAL
With native memcached out of the box, I can execute such a test without any problems:
1000 clients (threads), 400 operations per thread, 34% writes
However, running Infinispan Memcached distribution with 1000 worker threads results in
exceptions such as:
{code}2012-04-23 16:46:59,893 ERROR (MemcachedServerWorker-1-957)
[org.infinispan.server.memcached.MemcachedDecoder] ISPN005003: Exception reported
org.infinispan.server.core.UnknownOperationException: Unknown operation:
at
org.infinispan.server.memcached.RequestResolver$.toRequest(MemcachedDecoder.scala:659)
at
org.infinispan.server.memcached.MemcachedDecoder.readHeader(MemcachedDecoder.scala:73)
at
org.infinispan.server.core.AbstractProtocolDecoder.decodeHeader(AbstractProtocolDecoder.scala:92)
at
org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:69)
at
org.infinispan.server.core.AbstractProtocolDecoder.decode(AbstractProtocolDecoder.scala:45)
at
org.infinispan.server.core.transport.CustomReplayingDecoder.callDecode(CustomReplayingDecoder.java:250)
at
org.infinispan.server.core.transport.CustomReplayingDecoder.messageReceived(CustomReplayingDecoder.java:223)
at
org.infinispan.server.core.AbstractProtocolDecoder.messageReceived(AbstractProtocolDecoder.scala:360)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:351)
at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:282)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:202)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680){code}
Client shows:
{code}2012-04-23 16:46:28.742 INFO net.spy.memcached.MemcachedConnection: Reconnecting
due to exception on {QA sa=/127.0.0.1:11211, #Rops=0, #Wops=0, #iq=0, topRop=null,
topWop=null, toWrite=0, interested=0}
java.net.SocketException: Connection reset by peer
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:369)
at net.spy.memcached.MemcachedConnection.handleIO(MemcachedConnection.java:242)
at net.spy.memcached.MemcachedConnection.run(MemcachedConnection.java:833){code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.jboss.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
4378
days inactive
4386
days old
infinispan-issues@lists.jboss.org
10 comments
3 participants
participants (3)
-
Dan Berindei (JIRA) -
Galder Zamarreño (JIRA)
-
Manik Surtani (JIRA)