[hibernate-dev] Pooled Optimiser Improvements
Scott Marlow
smarlow at redhat.com
Tue Dec 15 19:58:46 EST 2015
On 12/15/2015 05:58 PM, Scott Marlow wrote:
>
>
> On 12/15/2015 05:40 PM, Scott Marlow wrote:
>> I changed the new test methods a bit. [2] seems to be passed the tests
>> but I am not understanding how PooledThreadLocalLoOptimizer should
>> coordinate with the AccessCallback to allocate the next chunk of
>> sequence numbers.
>>
>> We seem to be able to call AccessCallback.getNextValue() to get the next
>> available sequence number but how do we reserve a block of 5000 sequence
>> ids? Am I supposed to call callback.getNextValue() an extra time to get
>> a range of values? Is there a separate database transaction that is
>> used by the AccessCallback.getNextValue() calls? I'm missing something.
>
> Thinking more about this, I assume that AccessCallback.getNextValue() is
> operating under a database transaction that we are probably ending
> before AccessCallback.getNextValue() returns. It also sounds like the
> database table is tracking the "lo" value, as mentioned in the
> PooledLoOptimizer. This implies that only the application layer knows
> what the range is. This seems like an important dependency to understand.
>
> Make sense?
http://in.relation.to/2007/04/10/new-323-hibernate-identifier-generators
seems to explain how increment_size is used. Since the user is already
configured that, will look into switching to that for
PooledThreadLocalLoOptimizer.
>
>>
>> Note that [2] also includes a test change to comment out a few lines in
>> SchemaUpdateDelimiterTest, due to the compiler error that I am seeing in
>> intellij. Will need to remember to remove that change.
>>
>> [2]
>> https://github.com/scottmarlow/hibernate-orm/commits/pooled-optimiser-hack-2
>>
>> On 12/15/2015 12:36 PM, Steve Ebersole wrote:
>>> Those tests tend to assert the increments. We seem to agree that this
>>> ThreadLocal one can skip gaps of values. I'd look there first.
>>>
>>>
>>> On Tue, Dec 15, 2015 at 11:34 AM Scott Marlow <smarlow at redhat.com
>>> <mailto:smarlow at redhat.com>> wrote:
>>>
>>> I'm trying to move the optimizer to PooledThreadLocalLoOptimizer [1].
>>> We are currently failing some new unit tests, which are cloned from
>>> existing PooledLoOptimizer tests which might be part of the problem.
>>>
>>> Scott
>>>
>>> [1]
>>> https://github.com/scottmarlow/hibernate-orm/tree/pooled-optimiser-hack
>>>
>>> On 12/14/2015 10:12 PM, Scott Marlow wrote:
>>> >
>>> >
>>> > On 12/11/2015 09:30 AM, Steve Ebersole wrote:
>>> >> It's hard to say without understanding the scenario where you
>>> are seeing
>>> >> this as a problem. I have some guesses as to what may be the
>>> problem,
>>> >> but without understanding more about why you see this as a
>>> problem in
>>> >> the first place it is hard to give you an answer. For example,
>>> I wonder
>>> >> if for environments not using multi-tenancy whether the recent
>>> changes
>>> >> for the generators to support multi-tenancy might be the
>>> culprit. If
>>> >> that is the case, and those changes are in fact the underlying
>>> cause of
>>> >> the perf issues you see then I think there is actually a better
>>> >> solution. But again, its hard to say unless we understand the
>>> reason
>>> >> this "shows up" as a perf problem for you.
>>> >
>>> > As best as I can tell from looking at the current PooledLoOptimizer,
>>> > versus the proposed change (to have a chunk of ids per thread),
>>> we went
>>> > from accessing a contented lock, to instead using per thread memory
>>> > (eliminating the contended lock on PooledLoOptimizer.generate()).
>>> >
>>> >>
>>> >> Until we hear more I think at this stage I'd vote for a separate
>>> >> optimizer. And maybe even not one that is upstream.
>>> >>
>>> >> Also I agree with Scott that I am VERY leery of not cleaning up a
>>> >> ThreadLocal.
>>> >
>>> > My mistake, as Stuart pointed out, the TL is not static, so we
>>> shouldn't
>>> > introduce any leaks.
>>> >
>>> >>
>>> >> On Fri, Dec 11, 2015 at 7:55 AM Scott Marlow <smarlow at redhat.com
>>> <mailto:smarlow at redhat.com>
>>> >> <mailto:smarlow at redhat.com <mailto:smarlow at redhat.com>>> wrote:
>>> >>
>>> >> Should this be a specialized pooled optimizer that is only
>>> used in
>>> >> environments that do not suffer from leaving the
>>> ThreadLocal around
>>> >> after the application is undeployed? In other words, the
>>> expectation is
>>> >> that classloader leaks with this pooled optimizer are
>>> expected (e.g.
>>> >> user must restart the jvm to really undeploy the application
>>> >> completely).
>>> >>
>>> >> I am thinking that there are at least three typical situations:
>>> >>
>>> >> 1. Applications are deployed in Java standalone edition.
>>> Generally,
>>> >> when the app undeploys the jvm is shutting down.
>>> >>
>>> >> 2. Applications are deployed as part of some container
>>> (e.g. an EE
>>> >> server) and the Hibernate jars are on the global
>>> classloader path (or
>>> >> something like that). On each shared container thread,
>>> there would be
>>> >> one Optimizer for all deployed applications. I wonder if
>>> instead, we
>>> >> would want one Optimizer instance per Hibernate SessionFactory
>>> >> associated with the many container threads?
>>> >>
>>> >> 3. Applications are deployed as part of some container
>>> (e.g. an EE
>>> >> server) and the Hibernate jars are deployed with the
>>> application. The
>>> >> ThreadLocals are associated with threads that are shared by
>>> different
>>> >> deployed applications. The application classloader contains the
>>> >> Hibernate classes. Each deployed application has its own
>>> Optimizer
>>> >> threadlocal. On each shared container thread, there would
>>> be one
>>> >> Optimizer per application (since each application has its
>>> Optimizer TL).
>>> >> Like (2), there would be sharing of the same Optimizer
>>> with the many
>>> >> application session factories. Should we instead have an
>>> optimizer per
>>> >> session factory?
>>> >>
>>> >> Scott
>>> >>
>>> >> On 12/10/2015 11:31 PM, Stuart Douglas wrote:
>>> >> > Hello,
>>> >> >
>>> >> > I have been working on a change to the pooled optimizer
>>> that we
>>> >> have been seeing good performance results with. Basically
>>> it hands
>>> >> out blocks of ID's to a thread local, rather than having every
>>> >> thread contend on the lock every time an ID is required.
>>> >> >
>>> >> >
>>> >>
>>> https://github.com/hibernate/hibernate-orm/compare/master...stuartwdouglas:pooled-optimiser-hack
>>> >> >
>>> >> > What would I need to do to get a change like this in? In
>>> particular:
>>> >> >
>>> >> > - Does it need to be a new type of optimizer, or is
>>> modifying the
>>> >> existing one like I have done OK?
>>> >> > - How should it be configured?
>>> >> >
>>> >> > I am happy to do up a PR for this, but I am just not
>>> really sure
>>> >> what would be required to get it to a point where it would be
>>> >> acceptable for inclusion.
>>> >> >
>>> >> > Stuart
>>> >> > _______________________________________________
>>> >> > hibernate-dev mailing list
>>> >> > hibernate-dev at lists.jboss.org
>>> <mailto:hibernate-dev at lists.jboss.org>
>>> <mailto:hibernate-dev at lists.jboss.org
>>> <mailto:hibernate-dev at lists.jboss.org>>
>>> >> > https://lists.jboss.org/mailman/listinfo/hibernate-dev
>>> >> >
>>> >> _______________________________________________
>>> >> hibernate-dev mailing list
>>> >> hibernate-dev at lists.jboss.org
>>> <mailto:hibernate-dev at lists.jboss.org>
>>> <mailto:hibernate-dev at lists.jboss.org
>>> <mailto:hibernate-dev at lists.jboss.org>>
>>> >> https://lists.jboss.org/mailman/listinfo/hibernate-dev
>>> >>
>>> > _______________________________________________
>>> > hibernate-dev mailing list
>>> > hibernate-dev at lists.jboss.org <mailto:hibernate-dev at lists.jboss.org>
>>> > https://lists.jboss.org/mailman/listinfo/hibernate-dev
>>> >
>>>
>> _______________________________________________
>> hibernate-dev mailing list
>> hibernate-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/hibernate-dev
>>
> _______________________________________________
> hibernate-dev mailing list
> hibernate-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hibernate-dev
>
More information about the hibernate-dev
mailing list