12:53 p.m.
Moving to a new thread.
The big problem we run into with this is that almost every application
of a thread pool that we have needs to be highly tailored to its usage
to get the most optimal performance. So we end up with quite a few
different pools and it becomes difficult to impose a server wide limit.
There however some potential strategies we could take. Although I am
unsure as to how the overall effectiveness would be:
1. Sharing idle threads between pools
2. Force everything to go through a special blocking thread factory via
instrumentation of java.lang.Thread. Any attempt to allocate over the
max would lead to thread reclamation attempts and finally blocking until
a timeout is reached.
3. Some kind of auto-tuning weighting model. If the max total threads is
N, force all thread pools to use a percentage of N, potentially based on
establishing current config value divided by combined total.
One thing I wonder though is if cloud providers are "barking up the
wrong tree"? It seems a better limitation of an application is raw CPU
clock time and max memory usage. How they split that time into threads
doesn't really affect the scalability of the physical server, it's all
virtual process performance (who cares if someone wastes time context
switching?).
On 9/14/11 10:39 AM, Scott Stark wrote:
The other big cross cutting concern is controlling the total number
of
threads in use by the application server. When running under a
constrained environment that uses something like pam_limits module to
control how many process(==java threads) a user can have, it is
difficult to know what the server max thread usage is right now.
--
Jason T. Greene
JBoss AS Lead / EAP Platform Architect
JBoss, a division of Red Hat
2:03 p.m.
The question of whether this is the right limit is one I have raised,
especially since the limit is being imposed via a pam_limits module
which actually limits the number of processes a user is allowed. A java
thread is mapping to a process in this model, which further blurs the
effectiveness/validity.
However, this is an issue that does commonly come up with system admins
as a way of gauging max requests and therefore some measure of the max
resource requirements.
Perhaps a global limit is really not an effective approach.
Similar to how we have a global view of ports in use, perhaps just
having a global view of the thread pools in the domain model stats would
be enough. If I can see of the server thread pools with the min/max
settings, and there is a base thread pool dsl for min/max settings in
every subsystem utilizing a thread pool, one can at least understand the
thread usage, and make some choices about how to limit it.
On 9/14/11 10:53 AM, Jason T. Greene wrote:
Moving to a new thread.
The big problem we run into with this is that almost every application
of a thread pool that we have needs to be highly tailored to its usage
to get the most optimal performance. So we end up with quite a few
different pools and it becomes difficult to impose a server wide limit.
There however some potential strategies we could take. Although I am
unsure as to how the overall effectiveness would be:
1. Sharing idle threads between pools
2. Force everything to go through a special blocking thread factory via
instrumentation of java.lang.Thread. Any attempt to allocate over the
max would lead to thread reclamation attempts and finally blocking until
a timeout is reached.
3. Some kind of auto-tuning weighting model. If the max total threads is
N, force all thread pools to use a percentage of N, potentially based on
establishing current config value divided by combined total.
One thing I wonder though is if cloud providers are "barking up the
wrong tree"? It seems a better limitation of an application is raw CPU
clock time and max memory usage. How they split that time into threads
doesn't really affect the scalability of the physical server, it's all
virtual process performance (who cares if someone wastes time context
switching?).
On 9/14/11 10:39 AM, Scott Stark wrote:
> The other big cross cutting concern is controlling the total number of
> threads in use by the application server. When running under a
> constrained environment that uses something like pam_limits module to
> control how many process(==java threads) a user can have, it is
> difficult to know what the server max thread usage is right now.
>
3:43 p.m.
We should actually invert the model (especially for EJB remote invocations).
Instead of having threads blocking on pool constraints, make the pool
leading in thread management.
Carlo
On 09/14/2011 09:03 PM, Scott Stark wrote:
The question of whether this is the right limit is one I have
raised,
especially since the limit is being imposed via a pam_limits module
which actually limits the number of processes a user is allowed. A java
thread is mapping to a process in this model, which further blurs the
effectiveness/validity.
However, this is an issue that does commonly come up with system admins
as a way of gauging max requests and therefore some measure of the max
resource requirements.
Perhaps a global limit is really not an effective approach.
Similar to how we have a global view of ports in use, perhaps just
having a global view of the thread pools in the domain model stats would
be enough. If I can see of the server thread pools with the min/max
settings, and there is a base thread pool dsl for min/max settings in
every subsystem utilizing a thread pool, one can at least understand the
thread usage, and make some choices about how to limit it.
On 9/14/11 10:53 AM, Jason T. Greene wrote:
> Moving to a new thread.
>
> The big problem we run into with this is that almost every application
> of a thread pool that we have needs to be highly tailored to its usage
> to get the most optimal performance. So we end up with quite a few
> different pools and it becomes difficult to impose a server wide limit.
>
> There however some potential strategies we could take. Although I am
> unsure as to how the overall effectiveness would be:
>
> 1. Sharing idle threads between pools
> 2. Force everything to go through a special blocking thread factory via
> instrumentation of java.lang.Thread. Any attempt to allocate over the
> max would lead to thread reclamation attempts and finally blocking until
> a timeout is reached.
> 3. Some kind of auto-tuning weighting model. If the max total threads is
> N, force all thread pools to use a percentage of N, potentially based on
> establishing current config value divided by combined total.
>
> One thing I wonder though is if cloud providers are "barking up the
> wrong tree"? It seems a better limitation of an application is raw CPU
> clock time and max memory usage. How they split that time into threads
> doesn't really affect the scalability of the physical server, it's all
> virtual process performance (who cares if someone wastes time context
> switching?).
>
> On 9/14/11 10:39 AM, Scott Stark wrote:
> > The other big cross cutting concern is controlling the total number of
> > threads in use by the application server. When running under a
> > constrained environment that uses something like pam_limits module to
> > control how many process(==java threads) a user can have, it is
> > difficult to know what the server max thread usage is right now.
> >
>
_______________________________________________
jboss-as7-dev mailing list
jboss-as7-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
5:03 p.m.
Can you translate that?
If your saying pools should be consulting some thread factory(s), how
does that differ from what Jason was talking about as strategies one
could possibly use to achieve a global constraint?
On 9/14/11 1:43 PM, Carlo de Wolf wrote:
We should actually invert the model (especially for EJB remote
invocations).
Instead of having threads blocking on pool constraints, make the pool
leading in thread management.
Carlo
6:18 a.m.
If you enter a pool with a thread that can potentially block you're
already too late in the game.
During performance testing of EAP 5 one bottleneck we found was that
releasing permits on a semaphore is a single threaded operation.
https://issues.jboss.org/browse/JBPAPP-4805
Suppose we had a resource pool of 10. Then it would be much better if
there are just 10 threads working through that pool. They can be fed
with an unfair BlockingQueue of tasks. The resource itself would not
have to be guarded with a semaphore. And the unfair BlockingQueue would
maximize throughput without sacrificing response time deviation.
(https://issues.jboss.org/browse/JBPAPP-2763)
Carlo
On 09/15/2011 12:03 AM, Scott Stark wrote:
Can you translate that?
If your saying pools should be consulting some thread factory(s), how
does that differ from what Jason was talking about as strategies one
could possibly use to achieve a global constraint?
On 9/14/11 1:43 PM, Carlo de Wolf wrote:
> We should actually invert the model (especially for EJB remote
> invocations).
>
> Instead of having threads blocking on pool constraints, make the pool
> leading in thread management.
>
> Carlo
>
3:46 p.m.
So effectively, there should be one thread pool and every subsystem
using an event model/executors to feed it?
To Jason's point, is one queuing strategy sufficient for all subsystems?
On 9/15/11 4:18 AM, Carlo de Wolf wrote:
If you enter a pool with a thread that can potentially block
you're
already too late in the game.
During performance testing of EAP 5 one bottleneck we found was that
releasing permits on a semaphore is a single threaded operation.
https://issues.jboss.org/browse/JBPAPP-4805
Suppose we had a resource pool of 10. Then it would be much better if
there are just 10 threads working through that pool. They can be fed
with an unfair BlockingQueue of tasks. The resource itself would not
have to be guarded with a semaphore. And the unfair BlockingQueue
would maximize throughput without sacrificing response time deviation.
(https://issues.jboss.org/browse/JBPAPP-2763)
Carlo
3:50 a.m.
No, you would have a thread pool per resource guarded.
Jason's proposal is working to guarding the total number of threads. So
in those proposals the total thread pool is the resource to be guarded.
So the two complement one-another.
The tricky bit would be predicting which execution paths would use
certain resources. This can only be done if the invocation of a method
is disconnected from the execution, like an EJB. An EJB method uses a
predefined set of resources.
It does become tricky however once multiple resources are involved. I
need to figure out how to deal with that.
Carlo
On 09/15/2011 10:46 PM, Scott Stark wrote:
So effectively, there should be one thread pool and every subsystem
using an event model/executors to feed it?
To Jason's point, is one queuing strategy sufficient for all subsystems?
On 9/15/11 4:18 AM, Carlo de Wolf wrote:
> If you enter a pool with a thread that can potentially block you're
> already too late in the game.
>
> During performance testing of EAP 5 one bottleneck we found was that
> releasing permits on a semaphore is a single threaded operation.
> https://issues.jboss.org/browse/JBPAPP-4805
>
> Suppose we had a resource pool of 10. Then it would be much better if
> there are just 10 threads working through that pool. They can be fed
> with an unfair BlockingQueue of tasks. The resource itself would not
> have to be guarded with a semaphore. And the unfair BlockingQueue
> would maximize throughput without sacrificing response time deviation.
> (https://issues.jboss.org/browse/JBPAPP-2763)
>
> Carlo
>
3:10 a.m.
On Wed, 2011-09-14 at 12:03 -0700, Scott Stark wrote:
The question of whether this is the right limit is one I have raised,
especially since the limit is being imposed via a pam_limits module
which actually limits the number of processes a user is allowed. A java
thread is mapping to a process in this model, which further blurs the
effectiveness/validity.
However, this is an issue that does commonly come up with system admins
as a way of gauging max requests and therefore some measure of the max
resource requirements.
Perhaps a global limit is really not an effective approach.
Similar to how we have a global view of ports in use, perhaps just
having a global view of the thread pools in the domain model stats would
be enough. If I can see of the server thread pools with the min/max
settings, and there is a base thread pool dsl for min/max settings in
every subsystem utilizing a thread pool, one can at least understand the
thread usage, and make some choices about how to limit it.
I think it is a good idea if all subsystems can be configured to use a
single thread pool for all their application threads. However, I think
this limit should mean user threads, not including the threads the
server and JVM already use.
If you say, for example, that there are 50 threads max, it should mean
that the user can have up to 50 threads running his servlets, not that
he can have 25 threads used by the server + JVM (that they will never
let go), and actually only up to 25 that can do something for him.
--
Remy Maucherat <rmaucher(a)redhat.com>
Red Hat Inc
7:39 a.m.
On 15/09/2011 01:10, Remy Maucherat wrote:
On Wed, 2011-09-14 at 12:03 -0700, Scott Stark wrote:
> The question of whether this is the right limit is one I have raised,
> especially since the limit is being imposed via a pam_limits module
> which actually limits the number of processes a user is allowed. A java
> thread is mapping to a process in this model, which further blurs the
> effectiveness/validity.
>
> However, this is an issue that does commonly come up with system admins
> as a way of gauging max requests and therefore some measure of the max
> resource requirements.
>
> Perhaps a global limit is really not an effective approach.
>
> Similar to how we have a global view of ports in use, perhaps just
> having a global view of the thread pools in the domain model stats would
> be enough. If I can see of the server thread pools with the min/max
> settings, and there is a base thread pool dsl for min/max settings in
> every subsystem utilizing a thread pool, one can at least understand the
> thread usage, and make some choices about how to limit it.
I think it is a good idea if all subsystems can be configured to use a
single thread pool for all their application threads. However, I think
this limit should mean user threads, not including the threads the
server and JVM already use.
If you say, for example, that there are 50 threads max, it should mean
that the user can have up to 50 threads running his servlets, not that
he can have 25 threads used by the server + JVM (that they will never
let go), and actually only up to 25 that can do something for him.
I wonder how that would work for the OSGi subsystem since in OSGi user
code is allowed to create new java.lang.Thread objects...
Cheers,
David
1:35 a.m.
What about if we had 1 underlying thread pool that provides all the threads, and then
subsystems use a custom executor that provides a 'view' of this underlying thread
pool.
So for instance we have the main thread pool that has a max of 25 threads, we could then
give the EJB remote invocation pool is then given an executor that can use a maximum of 10
threads from the pool.
This would also allow you to give these executors a priority, so for example you could
give the web subsystem thread pool a higher priority to make sure that web requests always
get handled first.
I'm not really sure how well this would work, but I just thought I would put it out
there.
Stuart
On 15/09/2011, at 3:53 AM, Jason T. Greene wrote:
Moving to a new thread.
The big problem we run into with this is that almost every application
of a thread pool that we have needs to be highly tailored to its usage
to get the most optimal performance. So we end up with quite a few
different pools and it becomes difficult to impose a server wide limit.
There however some potential strategies we could take. Although I am
unsure as to how the overall effectiveness would be:
1. Sharing idle threads between pools
2. Force everything to go through a special blocking thread factory via
instrumentation of java.lang.Thread. Any attempt to allocate over the
max would lead to thread reclamation attempts and finally blocking until
a timeout is reached.
3. Some kind of auto-tuning weighting model. If the max total threads is
N, force all thread pools to use a percentage of N, potentially based on
establishing current config value divided by combined total.
One thing I wonder though is if cloud providers are "barking up the
wrong tree"? It seems a better limitation of an application is raw CPU
clock time and max memory usage. How they split that time into threads
doesn't really affect the scalability of the physical server, it's all
virtual process performance (who cares if someone wastes time context
switching?).
On 9/14/11 10:39 AM, Scott Stark wrote:
> The other big cross cutting concern is controlling the total number of
> threads in use by the application server. When running under a
> constrained environment that uses something like pam_limits module to
> control how many process(==java threads) a user can have, it is
> difficult to know what the server max thread usage is right now.
>
--
Jason T. Greene
JBoss AS Lead / EAP Platform Architect
JBoss, a division of Red Hat
_______________________________________________
jboss-as7-dev mailing list
jboss-as7-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
4:15 a.m.
I'll illustrate how inverting the pool complements the big thread pool.
On 09/16/2011 08:35 AM, Stuart Douglas wrote:
What about if we had 1 underlying thread pool that provides all the
threads, and then subsystems use a custom executor that provides a 'view' of this
underlying thread pool.
+1. We need this to guard the total number of threads.
So for instance we have the main thread pool that has a max of 25
threads, we could then give the EJB remote invocation pool is then given an executor that
can use a maximum of 10 threads from the pool.
If we just stick to EJB, this means that those 10 (or more) threads
become an extremely precious resource. Any time one of these threads
blocks we're wasting precious cycles. So instead of blocking for
resources to become available we should only pick up a request that can
proceed without blocking. Further more we should prioritize jobs that
require lesser work, this is called 'maximum throughput scheduling'.
Note that there are more criteria (like max wait time, queue size etc)
that need to be worked in, but that should be expressible in a formula
the drives a 'priority queue'.
This would also allow you to give these executors a priority, so for
example you could give the web subsystem thread pool a higher priority to make sure that
web requests always get handled first.
We should not put a maximum on a 'subsystem' pool, because that would
not allow it to fully utilize the total pool if the others are idle.
Better to leave a minimum on hot standby.
Carlo
I'm not really sure how well this would work, but I just thought
I would put it out there.
Stuart
On 15/09/2011, at 3:53 AM, Jason T. Greene wrote:
> Moving to a new thread.
>
> The big problem we run into with this is that almost every application
> of a thread pool that we have needs to be highly tailored to its usage
> to get the most optimal performance. So we end up with quite a few
> different pools and it becomes difficult to impose a server wide limit.
>
> There however some potential strategies we could take. Although I am
> unsure as to how the overall effectiveness would be:
>
> 1. Sharing idle threads between pools
> 2. Force everything to go through a special blocking thread factory via
> instrumentation of java.lang.Thread. Any attempt to allocate over the
> max would lead to thread reclamation attempts and finally blocking until
> a timeout is reached.
> 3. Some kind of auto-tuning weighting model. If the max total threads is
> N, force all thread pools to use a percentage of N, potentially based on
> establishing current config value divided by combined total.
>
> One thing I wonder though is if cloud providers are "barking up the
> wrong tree"? It seems a better limitation of an application is raw CPU
> clock time and max memory usage. How they split that time into threads
> doesn't really affect the scalability of the physical server, it's all
> virtual process performance (who cares if someone wastes time context
> switching?).
>
> On 9/14/11 10:39 AM, Scott Stark wrote:
>> The other big cross cutting concern is controlling the total number of
>> threads in use by the application server. When running under a
>> constrained environment that uses something like pam_limits module to
>> control how many process(==java threads) a user can have, it is
>> difficult to know what the server max thread usage is right now.
>>
> --
> Jason T. Greene
> JBoss AS Lead / EAP Platform Architect
> JBoss, a division of Red Hat
> _______________________________________________
> jboss-as7-dev mailing list
> jboss-as7-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
_______________________________________________
jboss-as7-dev mailing list
jboss-as7-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
8:19 a.m.
On 09/16/2011 05:15 AM, Carlo de Wolf wrote:
I'll illustrate how inverting the pool complements the big thread
pool.
On 09/16/2011 08:35 AM, Stuart Douglas wrote:
> What about if we had 1 underlying thread pool that provides all the threads, and then
subsystems use a custom executor that provides a 'view' of this underlying thread
pool.
+1. We need this to guard the total number of threads.
> So for instance we have the main thread pool that has a max of 25 threads, we could
then give the EJB remote invocation pool is then given an executor that can use a maximum
of 10 threads from the pool.
If we just stick to EJB, this means that those 10 (or more) threads
become an extremely precious resource. Any time one of these threads
blocks we're wasting precious cycles. So instead of blocking for
resources to become available we should only pick up a request that can
proceed without blocking. Further more we should prioritize jobs that
require lesser work, this is called 'maximum throughput scheduling'.
Note that there are more criteria (like max wait time, queue size etc)
that need to be worked in, but that should be expressible in a formula
the drives a 'priority queue'.
The application developer, should have a feel for which beans or even
bean methods are expensive and to what level. For example, the
developer may want to scale the application, in such a way that only N
of a certain bean execute on a single cloud node/standalone
installation/clustered installation. I have done this in clustered
applications, on both the web tier and for session beans. It gives the
app deployer/installer person, a way to make better use of the hardware
(because they know that the most expensive requests, will be throttled
separately).
Another simple example, imagine that there is an admin function "recalc
data", invoked in a bean that reads a certain subset of the database
(thousands of database rows), recalculating something that couldn't be
done in a stored procedure. Imagine if you will, that this function,
isn't invoked very often, but when it is. Each app server instance, can
only handle a max of 4 concurrent users executing this function, without
seriously reducing performance for other (lightweight) users of the
system. For this simple example, each app server instance can easily
handle many more lightweight user requests (e.g. 20 times the number of
CPUs), without pegging the CPU.
Without the ability to throttle the machines based on per session bean
thread limits, users can still use the max number of threads to prevent
too many requests from executing but they don't have the flexibility to
handle the above. Basically, they are forced into handling this
operationally (training users to coordinate when expensive functions are
performed).
Is there room to introduce per bean throttling or thread limits?
Perhaps this is what someone already had in mind?
> This would also allow you to give these executors a priority, so
for example you could give the web subsystem thread pool a higher priority to make sure
that web requests always get handled first.
We should not put a maximum on a 'subsystem' pool, because that would
not allow it to fully utilize the total pool if the others are idle.
Better to leave a minimum on hot standby.
Carlo
> I'm not really sure how well this would work, but I just thought I would put it
out there.
>
> Stuart
>
> On 15/09/2011, at 3:53 AM, Jason T. Greene wrote:
>
>> Moving to a new thread.
>>
>> The big problem we run into with this is that almost every application
>> of a thread pool that we have needs to be highly tailored to its usage
>> to get the most optimal performance. So we end up with quite a few
>> different pools and it becomes difficult to impose a server wide limit.
>>
>> There however some potential strategies we could take. Although I am
>> unsure as to how the overall effectiveness would be:
>>
>> 1. Sharing idle threads between pools
>> 2. Force everything to go through a special blocking thread factory via
>> instrumentation of java.lang.Thread. Any attempt to allocate over the
>> max would lead to thread reclamation attempts and finally blocking until
>> a timeout is reached.
>> 3. Some kind of auto-tuning weighting model. If the max total threads is
>> N, force all thread pools to use a percentage of N, potentially based on
>> establishing current config value divided by combined total.
>>
>> One thing I wonder though is if cloud providers are "barking up the
>> wrong tree"? It seems a better limitation of an application is raw CPU
>> clock time and max memory usage. How they split that time into threads
>> doesn't really affect the scalability of the physical server, it's all
>> virtual process performance (who cares if someone wastes time context
>> switching?).
>>
>> On 9/14/11 10:39 AM, Scott Stark wrote:
>>> The other big cross cutting concern is controlling the total number of
>>> threads in use by the application server. When running under a
>>> constrained environment that uses something like pam_limits module to
>>> control how many process(==java threads) a user can have, it is
>>> difficult to know what the server max thread usage is right now.
>>>
>> --
>> Jason T. Greene
>> JBoss AS Lead / EAP Platform Architect
>> JBoss, a division of Red Hat
>> _______________________________________________
>> jboss-as7-dev mailing list
>> jboss-as7-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
> _______________________________________________
> jboss-as7-dev mailing list
> jboss-as7-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
_______________________________________________
jboss-as7-dev mailing list
jboss-as7-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
10:36 a.m.
In pull request https://github.com/jbossas/jboss-as/pull/2407, Brian and
I have been discussing whether the (soon to be async) PU service start()
should run in a thread from a global thread pool or a private pool owned
by the PU service. There was also the suggestion that we create a
temporary thread for every PU service start.
Has there been any discussion of our general thread pooling design since
this email discussion (on irc perhaps?) that perhaps captures what we
want to do next (with regard to deployment/runtime thread pools that are
currently private and share enough attributes that they could be
combined into a global thread pool).
On 09/14/2011 01:53 PM, Jason T. Greene wrote:
Moving to a new thread.
The big problem we run into with this is that almost every application
of a thread pool that we have needs to be highly tailored to its usage
to get the most optimal performance. So we end up with quite a few
different pools and it becomes difficult to impose a server wide limit.
There however some potential strategies we could take. Although I am
unsure as to how the overall effectiveness would be:
1. Sharing idle threads between pools
2. Force everything to go through a special blocking thread factory via
instrumentation of java.lang.Thread. Any attempt to allocate over the
max would lead to thread reclamation attempts and finally blocking until
a timeout is reached.
3. Some kind of auto-tuning weighting model. If the max total threads is
N, force all thread pools to use a percentage of N, potentially based on
establishing current config value divided by combined total.
One thing I wonder though is if cloud providers are "barking up the
wrong tree"? It seems a better limitation of an application is raw CPU
clock time and max memory usage. How they split that time into threads
doesn't really affect the scalability of the physical server, it's all
virtual process performance (who cares if someone wastes time context
switching?).
On 9/14/11 10:39 AM, Scott Stark wrote:
> The other big cross cutting concern is controlling the total number of
> threads in use by the application server. When running under a
> constrained environment that uses something like pam_limits module to
> control how many process(==java threads) a user can have, it is
> difficult to know what the server max thread usage is right now.
>
10:58 a.m.
On 05/31/2012 05:36 PM, Scott Marlow wrote:
In pull request https://github.com/jbossas/jboss-as/pull/2407, Brian
and
I have been discussing whether the (soon to be async) PU service start()
should run in a thread from a global thread pool or a private pool owned
by the PU service. There was also the suggestion that we create a
temporary thread for every PU service start.
I am having the same issue while starting JMS Bridges asynchronously.
To keep things simple, I wrapped the code in a new Thread() but I am
looking to code it in a proper way.
In the messaging subsystem, each instance of HornetQ servers have a
startup pool service to get their own pool of threads to start their
services (queues, topic, connection factory).
If I want to be consistent, I could add *another* pool service to get
threads for JMS bridges. I am not sure that's a great thing to multiply
pool services just to start other services asynchronously.
(On the same topic, HornetQ has also its own thread pools can be tuned
depending on the messaging usage. At the moment, they are not shared
with AS7 but that something that I need to look at)
--
jeff
Has there been any discussion of our general thread pooling design since
this email discussion (on irc perhaps?) that perhaps captures what we
want to do next (with regard to deployment/runtime thread pools that are
currently private and share enough attributes that they could be
combined into a global thread pool).
On 09/14/2011 01:53 PM, Jason T. Greene wrote:
> Moving to a new thread.
>
> The big problem we run into with this is that almost every application
> of a thread pool that we have needs to be highly tailored to its usage
> to get the most optimal performance. So we end up with quite a few
> different pools and it becomes difficult to impose a server wide limit.
>
> There however some potential strategies we could take. Although I am
> unsure as to how the overall effectiveness would be:
>
> 1. Sharing idle threads between pools
> 2. Force everything to go through a special blocking thread factory via
> instrumentation of java.lang.Thread. Any attempt to allocate over the
> max would lead to thread reclamation attempts and finally blocking until
> a timeout is reached.
> 3. Some kind of auto-tuning weighting model. If the max total threads is
> N, force all thread pools to use a percentage of N, potentially based on
> establishing current config value divided by combined total.
>
> One thing I wonder though is if cloud providers are "barking up the
> wrong tree"? It seems a better limitation of an application is raw CPU
> clock time and max memory usage. How they split that time into threads
> doesn't really affect the scalability of the physical server, it's all
> virtual process performance (who cares if someone wastes time context
> switching?).
>
> On 9/14/11 10:39 AM, Scott Stark wrote:
> > The other big cross cutting concern is controlling the total number of
> > threads in use by the application server. When running under a
> > constrained environment that uses something like pam_limits module to
> > control how many process(==java threads) a user can have, it is
> > difficult to know what the server max thread usage is right now.
> >
>
_______________________________________________
jboss-as7-dev mailing list
jboss-as7-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
--
Jeff Mesnil
JBoss, a division of Red Hat
http://jmesnil.net/
5:46 p.m.
See related discussion in irc logs
http://echelog.com/logs/browse/jboss-as7/1338415200 Starting around
19:59:28 and ending before AS7-4934 is mentioned.
On 05/31/2012 11:58 AM, Jeff Mesnil wrote:
On 05/31/2012 05:36 PM, Scott Marlow wrote:
> In pull request https://github.com/jbossas/jboss-as/pull/2407, Brian and
> I have been discussing whether the (soon to be async) PU service start()
> should run in a thread from a global thread pool or a private pool owned
> by the PU service. There was also the suggestion that we create a
> temporary thread for every PU service start.
I am having the same issue while starting JMS Bridges asynchronously.
To keep things simple, I wrapped the code in a new Thread() but I am
looking to code it in a proper way.
In the messaging subsystem, each instance of HornetQ servers have a
startup pool service to get their own pool of threads to start their
services (queues, topic, connection factory).
If I want to be consistent, I could add *another* pool service to get
threads for JMS bridges. I am not sure that's a great thing to multiply
pool services just to start other services asynchronously.
(On the same topic, HornetQ has also its own thread pools can be tuned
depending on the messaging usage. At the moment, they are not shared
with AS7 but that something that I need to look at)
--
jeff
>
> Has there been any discussion of our general thread pooling design since
> this email discussion (on irc perhaps?) that perhaps captures what we
> want to do next (with regard to deployment/runtime thread pools that are
> currently private and share enough attributes that they could be
> combined into a global thread pool).
>
>
> On 09/14/2011 01:53 PM, Jason T. Greene wrote:
>> Moving to a new thread.
>>
>> The big problem we run into with this is that almost every application
>> of a thread pool that we have needs to be highly tailored to its usage
>> to get the most optimal performance. So we end up with quite a few
>> different pools and it becomes difficult to impose a server wide limit.
>>
>> There however some potential strategies we could take. Although I am
>> unsure as to how the overall effectiveness would be:
>>
>> 1. Sharing idle threads between pools
>> 2. Force everything to go through a special blocking thread factory via
>> instrumentation of java.lang.Thread. Any attempt to allocate over the
>> max would lead to thread reclamation attempts and finally blocking until
>> a timeout is reached.
>> 3. Some kind of auto-tuning weighting model. If the max total threads is
>> N, force all thread pools to use a percentage of N, potentially based on
>> establishing current config value divided by combined total.
>>
>> One thing I wonder though is if cloud providers are "barking up the
>> wrong tree"? It seems a better limitation of an application is raw CPU
>> clock time and max memory usage. How they split that time into threads
>> doesn't really affect the scalability of the physical server, it's all
>> virtual process performance (who cares if someone wastes time context
>> switching?).
>>
>> On 9/14/11 10:39 AM, Scott Stark wrote:
>> > The other big cross cutting concern is controlling the total number
of
>> > threads in use by the application server. When running under a
>> > constrained environment that uses something like pam_limits module
to
>> > control how many process(==java threads) a user can have, it is
>> > difficult to know what the server max thread usage is right now.
>> >
>>
>
> _______________________________________________
> jboss-as7-dev mailing list
> jboss-as7-dev(a)lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
1:30 p.m.
Thanks, Scott, for posting that.
https://github.com/jbossas/jboss-as/pull/2434 exposes the pool currently
used for parallel boot tasks as a service that extension services can
inject and use for blocking tasks during start/stop. I also tweaked a
couple subsystems to use the pool instead of creating their own.
On IRC we discussed possibly limiting the pool to EE subsystems, but
that would have introduced new dependencies between modules while not
covering some potential uses. So I just went to a more generally usable
pool.
Jeff, yes, for a later release, probably 7.3, I want to go through the
various thread pool usages again to improve their manageability, and
those HQ pools are ones we should look at.
On 5/31/12 5:46 PM, Scott Marlow wrote:
See related discussion in irc logs
http://echelog.com/logs/browse/jboss-as7/1338415200 Starting around
19:59:28 and ending before AS7-4934 is mentioned.
On 05/31/2012 11:58 AM, Jeff Mesnil wrote:
>
> On 05/31/2012 05:36 PM, Scott Marlow wrote:
>> In pull request https://github.com/jbossas/jboss-as/pull/2407, Brian and
>> I have been discussing whether the (soon to be async) PU service start()
>> should run in a thread from a global thread pool or a private pool owned
>> by the PU service. There was also the suggestion that we create a
>> temporary thread for every PU service start.
>
> I am having the same issue while starting JMS Bridges asynchronously.
> To keep things simple, I wrapped the code in a new Thread() but I am
> looking to code it in a proper way.
>
> In the messaging subsystem, each instance of HornetQ servers have a
> startup pool service to get their own pool of threads to start their
> services (queues, topic, connection factory).
> If I want to be consistent, I could add *another* pool service to get
> threads for JMS bridges. I am not sure that's a great thing to multiply
> pool services just to start other services asynchronously.
>
> (On the same topic, HornetQ has also its own thread pools can be tuned
> depending on the messaging usage. At the moment, they are not shared
> with AS7 but that something that I need to look at)
>
>
> --
> jeff
>
>>
>> Has there been any discussion of our general thread pooling design since
>> this email discussion (on irc perhaps?) that perhaps captures what we
>> want to do next (with regard to deployment/runtime thread pools that are
>> currently private and share enough attributes that they could be
>> combined into a global thread pool).
>>
>>
>> On 09/14/2011 01:53 PM, Jason T. Greene wrote:
>>> Moving to a new thread.
>>>
>>> The big problem we run into with this is that almost every application
>>> of a thread pool that we have needs to be highly tailored to its usage
>>> to get the most optimal performance. So we end up with quite a few
>>> different pools and it becomes difficult to impose a server wide limit.
>>>
>>> There however some potential strategies we could take. Although I am
>>> unsure as to how the overall effectiveness would be:
>>>
>>> 1. Sharing idle threads between pools
>>> 2. Force everything to go through a special blocking thread factory via
>>> instrumentation of java.lang.Thread. Any attempt to allocate over the
>>> max would lead to thread reclamation attempts and finally blocking until
>>> a timeout is reached.
>>> 3. Some kind of auto-tuning weighting model. If the max total threads is
>>> N, force all thread pools to use a percentage of N, potentially based on
>>> establishing current config value divided by combined total.
>>>
>>> One thing I wonder though is if cloud providers are "barking up the
>>> wrong tree"? It seems a better limitation of an application is raw CPU
>>> clock time and max memory usage. How they split that time into threads
>>> doesn't really affect the scalability of the physical server, it's
all
>>> virtual process performance (who cares if someone wastes time context
>>> switching?).
>>>
>>> On 9/14/11 10:39 AM, Scott Stark wrote:
>>> > The other big cross cutting concern is controlling the total
number of
>>> > threads in use by the application server. When running under a
>>> > constrained environment that uses something like pam_limits
module to
>>> > control how many process(==java threads) a user can have, it
is
>>> > difficult to know what the server max thread usage is right
now.
>>> >
>>>
>>
>> _______________________________________________
>> jboss-as7-dev mailing list
>> jboss-as7-dev(a)lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
>
>
_______________________________________________
jboss-as7-dev mailing list
jboss-as7-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/jboss-as7-dev
--
Brian Stansberry
Principal Software Engineer
JBoss by Red Hat
4553
days inactive
4817
days old
15 comments
9 participants
participants (9)
-
Brian Stansberry -
Carlo de Wolf -
David Bosschaert -
Jason T. Greene -
Jeff Mesnil -
Remy Maucherat -
Scott Marlow -
Scott Stark -
Stuart Douglas