12:35 p.m.
Hi all,
in a servlet environment, you sometimes have to claim back a thread for your
thread pool.
Jetty has a nice "denial of service attack" filter and a setting
(maxIdleTime) interrupting servlet threads if nothing comes out (nothing is
written in the pipe).
Well it does not play well with infinispan. Everybody knows that when you
open a stream, the best way to reclaim scarce resources is to close it in a
'finally' statement. What about applying it to infinispan locking mechanism
? ;-)
Calls to acquireLock should be guarded by a finally {releaseLock} (at least
to catch an interruptedException)
It looks very easy to implement, but since releaseLock does not seems to be
called every time in the code, I need help !
The only places where aquireLock is called are
CacheLoaderInterceptor.loadIfNeeded and
EntryFactoryImpl.wrapEntryForWriting.
And BTW, the locks never expire ... so if you interrupt a thread and the
lock is not released, you better say bye bye to the cache entry!
Cheers,
Phil
4:47 a.m.
On 16 May 2010, at 18:35, Philippe Van Dyck wrote:
Hi all,
in a servlet environment, you sometimes have to claim back a thread for your thread
pool.
Jetty has a nice "denial of service attack" filter and a setting (maxIdleTime)
interrupting servlet threads if nothing comes out (nothing is written in the pipe).
Well it does not play well with infinispan. Everybody knows that when you open a stream,
the best way to reclaim scarce resources is to close it in a 'finally' statement.
What about applying it to infinispan locking mechanism ? ;-)
Calls to acquireLock should be guarded by a finally {releaseLock} (at least to catch an
interruptedException)
This is a good point, in the event of a failure (for whatever reason) the lock may still
be acquired and should be cleaned up. Care to create a JIRA for this?
It looks very easy to implement, but since releaseLock does not seems
to be called every time in the code, I need help !
The only places where aquireLock is called are CacheLoaderInterceptor.loadIfNeeded and
EntryFactoryImpl.wrapEntryForWriting.
And BTW, the locks never expire ... so if you interrupt a thread and the lock is not
released, you better say bye bye to the cache entry!
Hmm. While I agree in principle (we could implement a max lock period in the config
file), how do you see this working in practice? Maintain timestamps of all acquired
locks, and have a maintenance thread regularly check these for locks that have expired
(and then interrupt the locking thread)?
Cheers
Manik
--
Manik Surtani
manik(a)jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org
6:37 a.m.
On Mon, May 17, 2010 at 11:47 AM, Manik Surtani <manik(a)jboss.org> wrote:
On 16 May 2010, at 18:35, Philippe Van Dyck wrote:
> Hi all,
>
> in a servlet environment, you sometimes have to claim back a thread for
your thread pool.
> Jetty has a nice "denial of service attack" filter and a setting
(maxIdleTime) interrupting servlet threads if nothing comes out (nothing is
written in the pipe).
>
> Well it does not play well with infinispan. Everybody knows that when you
open a stream, the best way to reclaim scarce resources is to close it in a
'finally' statement. What about applying it to infinispan locking mechanism
? ;-)
>
> Calls to acquireLock should be guarded by a finally {releaseLock} (at
least to catch an interruptedException)
This is a good point, in the event of a failure (for whatever reason) the
lock may still be acquired and should be cleaned up. Care to create a JIRA
for this?
Done - ISPN-444 </browse/ISPN-444>
> It looks very easy to implement, but since releaseLock does not seems to
be called every time in the code, I need help !
>
> The only places where aquireLock is called are
CacheLoaderInterceptor.loadIfNeeded and
EntryFactoryImpl.wrapEntryForWriting.
>
> And BTW, the locks never expire ... so if you interrupt a thread and the
lock is not released, you better say bye bye to the cache entry!
Hmm. While I agree in principle (we could implement a max lock period in
the config file), how do you see this working in practice? Maintain
timestamps of all acquired locks, and have a maintenance thread regularly
check these for locks that have expired (and then interrupt the locking
thread)?
I think there is a need for this kind of cleanup whatever the reason a lock
is in 'stale' mode.
But IMHO there is a lot of cleanup to do when you interrupt a thread !
What about some timeout callback attached to each lock ?
Now, to test this, you should replay test suites with a 'one millisecond'
timeout and ensure that everything is correctly cleaned up.
Great way to improve the quality of the code, but hell of a job!
Cheers,
phil
Cheers
Manik
--
Manik Surtani
manik(a)jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
6:54 a.m.
On 17 May 2010, at 12:37, Philippe Van Dyck wrote:
Now, to test this, you should replay test suites with a 'one millisecond' timeout
and ensure that everything is correctly cleaned up.
Great way to improve the quality of the code, but hell of a job!
I agree on both counts. :) Create a JIRA for this so we don't forget, but don't
expect a fix soon, at least until we have a convincing strategy on how to deal with this.
--
Manik Surtani
manik(a)jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org
7:05 a.m.
Done - ISPN-445 </browse/ISPN-445>
On Mon, May 17, 2010 at 1:54 PM, Manik Surtani <manik(a)jboss.org> wrote:
On 17 May 2010, at 12:37, Philippe Van Dyck wrote:
>
> Now, to test this, you should replay test suites with a 'one millisecond'
timeout and ensure that everything is correctly cleaned up.
> Great way to improve the quality of the code, but hell of a job!
I agree on both counts. :) Create a JIRA for this so we don't forget, but
don't expect a fix soon, at least until we have a convincing strategy on how
to deal with this.
--
Manik Surtani
manik(a)jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
9:39 a.m.
On 17 May 2010, at 12:37, Philippe Van Dyck wrote:
On Mon, May 17, 2010 at 11:47 AM, Manik Surtani <manik(a)jboss.org> wrote:
On 16 May 2010, at 18:35, Philippe Van Dyck wrote:
> Hi all,
>
> in a servlet environment, you sometimes have to claim back a thread for your thread
pool.
> Jetty has a nice "denial of service attack" filter and a setting
(maxIdleTime) interrupting servlet threads if nothing comes out (nothing is written in the
pipe).
>
> Well it does not play well with infinispan. Everybody knows that when you open a
stream, the best way to reclaim scarce resources is to close it in a 'finally'
statement. What about applying it to infinispan locking mechanism ? ;-)
>
> Calls to acquireLock should be guarded by a finally {releaseLock} (at least to catch
an interruptedException)
This is a good point, in the event of a failure (for whatever reason) the lock may still
be acquired and should be cleaned up. Care to create a JIRA for this?
Done - ISPN-444
Hmm, about this - I suppose it really isn't enough to wrap the tryLock call since a
thread could be interrupted doing a multitude of other things in addition to just
acquiring locks - e.g., waiting to write state to a cache store, network comms, etc.
So while I have checked in a fix for ISPN-444 [1], I presume what we really need is a
finally block around the InvocationContextInterceptor where all invocations come in [2],
to clean up any locks acquired within the scope of the current call.
Thoughts?
Cheers
Manik
[1] http://fisheye.jboss.org/changelog/Infinispan/trunk?cs=1794
[2]
http://fisheye.jboss.org/browse/Infinispan/trunk/core/src/main/java/org/i...
--
Manik Surtani
manik(a)jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org
10:56 a.m.
On Mon, May 17, 2010 at 4:39 PM, Manik Surtani <manik(a)jboss.org> wrote:
On 17 May 2010, at 12:37, Philippe Van Dyck wrote:
>
> On Mon, May 17, 2010 at 11:47 AM, Manik Surtani <manik(a)jboss.org> wrote:
>
> On 16 May 2010, at 18:35, Philippe Van Dyck wrote:
>
> > Hi all,
> >
> > in a servlet environment, you sometimes have to claim back a thread for
your thread pool.
> > Jetty has a nice "denial of service attack" filter and a setting
(maxIdleTime) interrupting servlet threads if nothing comes out (nothing is
written in the pipe).
> >
> > Well it does not play well with infinispan. Everybody knows that when
you open a stream, the best way to reclaim scarce resources is to close it
in a 'finally' statement. What about applying it to infinispan locking
mechanism ? ;-)
> >
> > Calls to acquireLock should be guarded by a finally {releaseLock} (at
least to catch an interruptedException)
>
> This is a good point, in the event of a failure (for whatever reason) the
lock may still be acquired and should be cleaned up. Care to create a JIRA
for this?
>
> Done - ISPN-444
Hmm, about this - I suppose it really isn't enough to wrap the tryLock call
since a thread could be interrupted doing a multitude of other things in
addition to just acquiring locks - e.g., waiting to write state to a cache
store, network comms, etc.
So while I have checked in a fix for ISPN-444 [1], I presume what we really
need is a finally block around the InvocationContextInterceptor where all
invocations come in [2], to clean up any locks acquired within the scope of
the current call.
Thoughts?
Sound like a better idea indeed!
Do you have a plan to test it ?
BTW, this it what it looks like in the logs... when jetty lose the
connection on the TCP level (connection reset by peer) it seems to interrupt
the thread... (it is happening right now and ... it is quite problematic!)
2010-05-17 14:47:08,129[Thread-13] ERROR org.eclipse.jetty.util.log -
EXCEPTION
org.eclipse.jetty.io.RuntimeIOException: org.eclipse.jetty.io.EofException
at
org.eclipse.jetty.io.UncheckedPrintWriter.close(UncheckedPrintWriter.java:139)
at
com.netzep.common.frontend.application.DoSFilter.closeConnection(DoSFilter.java:426)
at
com.netzep.common.frontend.application.DoSFilter$3.expired(DoSFilter.java:394)
at org.eclipse.jetty.util.thread.Timeout.tick(Timeout.java:136)
at
com.netzep.common.frontend.application.DoSFilter$2.run(DoSFilter.java:242)
Caused by: org.eclipse.jetty.io.EofException
at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:892)
at org.eclipse.jetty.http.HttpGenerator.complete(HttpGenerator.java:791)
at
org.eclipse.jetty.server.HttpConnection.commitResponse(HttpConnection.java:701)
at
org.eclipse.jetty.server.HttpConnection$Output.close(HttpConnection.java:1077)
at org.eclipse.jetty.server.HttpWriter.close(HttpWriter.java:80)
at
org.eclipse.jetty.io.UncheckedPrintWriter.close(UncheckedPrintWriter.java:133)
... 4 more
Caused by: java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
...
at
org.eclipse.jetty.http.ssl.SslSelectChannelEndPoint.flush(SslSelectChannelEndPoint.java:438)
at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:831)
... 9 more
2010-05-17 14:47:08,425[Qi4j - Transaction : ] ERROR
org.infinispan.interceptors.InvocationContextInterceptor - Execution
error:
org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock
after [60 seconds] on key [User/w916obtfCq3D0a7W2l5KrQAAABKg] for requestor
[GlobalTransaction:<domU-12-31-39-09-9C-54-54012>:99]! Lock held by
[GlobalTransaction:<domU-12-31-39-09-9C-54-54012>:98]
at
org.infinispan.container.EntryFactoryImpl.acquireLock(EntryFactoryImpl.java:213)
at
org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:121)
...
at
org.qi4j.entitystore.map.MapEntityStoreMixin$1$1.visitMap(MapEntityStoreMixin.java:132)
at
org.qi4j.entitystore.s3jclouds.S3jcloudsMapEntityStoreMixin$1.run(S3jcloudsMapEntityStoreMixin.java:157)
Exception in thread "Qi4j - Transaction : "
org.qi4j.spi.entitystore.EntityStoreException: Exception during transaction
- rollback true -
at
org.qi4j.entitystore.s3jclouds.S3jcloudsMapEntityStoreMixin$1.run(S3jcloudsMapEntityStoreMixin.java:190)
Caused by: org.infinispan.util.concurrent.TimeoutException: Unable to
acquire lock after [60 seconds] on key [User/w916obtfCq3D0a7W2l5KrQAAABKg]
for requestor [GlobalTransaction:<domU-12-31-39-09-9C-54-54012>:99]! Lock
held by [GlobalTransaction:<domU-12-31-39-09-9C-54-54012>:98]
at
org.infinispan.container.EntryFactoryImpl.acquireLock(EntryFactoryImpl.java:213)
at
org.infinispan.container.EntryFactoryImpl.wrapEntryForWriting(EntryFactoryImpl.java:121)
...
at
org.qi4j.entitystore.map.MapEntityStoreMixin$1$1.visitMap(MapEntityStoreMixin.java:132)
at
org.qi4j.entitystore.s3jclouds.S3jcloudsMapEntityStoreMixin$1.run(S3jcloudsMapEntityStoreMixin.java:157)
2010-05-17 14:47:09,001[Qi4j - Transaction : ] ERROR
org.infinispan.interceptors.InvocationContextInterceptor - Execution
error:
org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock
after [60 seconds] on key [User/w9oM7pOrl2_D3N0KMIbWQwAAAAng] for requestor
[GlobalTransaction:<domU-12-31-39-09-9C-54-54012>:100]! Lock held by
[GlobalTransaction:<domU-12-31-39-09-9C-54-54012>:98]
at
org.infinispan.container.EntryFactoryImpl.acquireLock(EntryFactoryImpl.java:213)
at
org.infinispan.interceptors.CacheLoaderInterceptor.loadIfNeeded(CacheLoaderInterceptor.java:119)
at
org.infinispan.interceptors.CacheLoaderInterceptor.visitPutKeyValueCommand(CacheLoaderInterceptor.java:77)
...
at
org.qi4j.entitystore.map.MapEntityStoreMixin$1$1.visitMap(MapEntityStoreMixin.java:126)
at
org.qi4j.entitystore.s3jclouds.S3jcloudsMapEntityStoreMixin$1.run(S3jcloudsMapEntityStoreMixin.java:157)
Exception in thread "Qi4j - Transaction : "
org.qi4j.spi.entitystore.EntityStoreException: Exception during transaction
- rollback true -
at
org.qi4j.entitystore.s3jclouds.S3jcloudsMapEntityStoreMixin$1.run(S3jcloudsMapEntityStoreMixin.java:190)
Caused by: org.infinispan.util.concurrent.TimeoutException: Unable to
acquire lock after [60 seconds] on key [User/w9oM7pOrl2_D3N0KMIbWQwAAAAng]
for requestor [GlobalTransaction:<domU-12-31-39-09-9C-54-54012>:100]! Lock
held by [GlobalTransaction:<domU-12-31-39-09-9C-54-54012>:98]
at
org.infinispan.container.EntryFactoryImpl.acquireLock(EntryFactoryImpl.java:213)
at
org.infinispan.interceptors.CacheLoaderInterceptor.loadIfNeeded(CacheLoaderInterceptor.java:119)
at
org.infinispan.interceptors.CacheLoaderInterceptor.visitPutKeyValueCommand(CacheLoaderInterceptor.java
...
at
org.qi4j.entitystore.map.MapEntityStoreMixin$1$1.visitMap(MapEntityStoreMixin.java:126)
at
org.qi4j.entitystore.s3jclouds.S3jcloudsMapEntityStoreMixin$1.run(S3jcloudsMapEntityStoreMixin.java:157)
cheers,
phil
Cheers
Manik
[1] http://fisheye.jboss.org/changelog/Infinispan/trunk?cs=1794
[2]
http://fisheye.jboss.org/browse/Infinispan/trunk/core/src/main/java/org/i...
--
Manik Surtani
manik(a)jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
11:51 a.m.
On 17 May 2010, at 16:56, Philippe Van Dyck wrote:
BTW, this it what it looks like in the logs... when jetty lose the connection on the TCP
level (connection reset by peer) it seems to interrupt the thread... (it is happening
right now and ... it is quite problematic!)
Try a build off trunk and see if it solves your issue?
--
Manik Surtani
manik(a)jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org
12:08 p.m.
Thanks Manik, I will try it. I took a look at at it and it seems to be
exactly what's needed.
But currently I can't use snapshots in preproduction with my maven
provisioning scripts :-( (you know the release plugin, don't you?)
cheers,
phil
On Mon, May 17, 2010 at 6:51 PM, Manik Surtani <manik(a)jboss.org> wrote:
On 17 May 2010, at 16:56, Philippe Van Dyck wrote:
>
> BTW, this it what it looks like in the logs... when jetty lose the
connection on the TCP level (connection reset by peer) it seems to interrupt
the thread... (it is happening right now and ... it is quite problematic!)
>
Try a build off trunk and see if it solves your issue?
--
Manik Surtani
manik(a)jboss.org
Lead, Infinispan
Lead, JBoss Cache
http://www.infinispan.org
http://www.jbosscache.org
_______________________________________________
infinispan-dev mailing list
infinispan-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/infinispan-dev
6:06 p.m.
Anyway, thanks for the effort.
But I cannot expect infinispan to support an "interruptible" api ;-)
In order to isolate the problem, all transactions are now running in their
own thread (not jetty's one).
I hope that it will reduce the "I am interrupted and going to kill the whole
system" problem ;-)
thanks again
phil
On Mon, May 17, 2010 at 7:08 PM, Philippe Van Dyck <pvdyck(a)gmail.com> wrote:
Thanks Manik, I will try it. I took a look at at it and it seems to
be
exactly what's needed.
But currently I can't use snapshots in preproduction with my maven
provisioning scripts :-( (you know the release plugin, don't you?)
cheers,
phil
On Mon, May 17, 2010 at 6:51 PM, Manik Surtani <manik(a)jboss.org> wrote:
>
> On 17 May 2010, at 16:56, Philippe Van Dyck wrote:
>
> >
> > BTW, this it what it looks like in the logs... when jetty lose the
> connection on the TCP level (connection reset by peer) it seems to interrupt
> the thread... (it is happening right now and ... it is quite problematic!)
> >
>
> Try a build off trunk and see if it solves your issue?
>
> --
> Manik Surtani
> manik(a)jboss.org
> Lead, Infinispan
> Lead, JBoss Cache
> http://www.infinispan.org
> http://www.jbosscache.org
>
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> infinispan-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
5619
days inactive
5620
days old
infinispan-dev@lists.jboss.org
9 comments
2 participants
participants (2)
-
Manik Surtani -
Philippe Van Dyck