[hibernate-dev] Pooled Optimiser Improvements

Scott Marlow smarlow at redhat.com
Tue Dec 15 19:58:46 EST 2015



On 12/15/2015 05:58 PM, Scott Marlow wrote:
>
>
> On 12/15/2015 05:40 PM, Scott Marlow wrote:
>> I changed the new test methods a bit.  [2] seems to be passed the tests
>> but I am not understanding how PooledThreadLocalLoOptimizer should
>> coordinate with the AccessCallback to allocate the next chunk of
>> sequence numbers.
>>
>> We seem to be able to call AccessCallback.getNextValue() to get the next
>> available sequence number but how do we reserve a block of 5000 sequence
>> ids?  Am I supposed to call callback.getNextValue() an extra time to get
>> a range of values?  Is there a separate database transaction that is
>> used by the AccessCallback.getNextValue() calls?  I'm missing something.
>
> Thinking more about this, I assume that AccessCallback.getNextValue() is
> operating under a database transaction that we are probably ending
> before AccessCallback.getNextValue() returns.  It also sounds like the
> database table is tracking the "lo" value, as mentioned in the
> PooledLoOptimizer.  This implies that only the application layer knows
> what the range is.  This seems like an important dependency to understand.
>
> Make sense?

http://in.relation.to/2007/04/10/new-323-hibernate-identifier-generators 
seems to explain how increment_size is used.  Since the user is already 
configured that, will look into switching to that for 
PooledThreadLocalLoOptimizer.

>
>>
>> Note that [2] also includes a test change to comment out a few lines in
>> SchemaUpdateDelimiterTest, due to the compiler error that I am seeing in
>> intellij.  Will need to remember to remove that change.
>>
>> [2]
>> https://github.com/scottmarlow/hibernate-orm/commits/pooled-optimiser-hack-2
>>
>> On 12/15/2015 12:36 PM, Steve Ebersole wrote:
>>> Those tests tend to assert the increments.  We seem to agree that this
>>> ThreadLocal one can skip gaps of values.  I'd look there first.
>>>
>>>
>>> On Tue, Dec 15, 2015 at 11:34 AM Scott Marlow <smarlow at redhat.com
>>> <mailto:smarlow at redhat.com>> wrote:
>>>
>>>       I'm trying to move the optimizer to PooledThreadLocalLoOptimizer [1].
>>>       We are currently failing some new unit tests, which are cloned from
>>>       existing PooledLoOptimizer tests which might be part of the problem.
>>>
>>>       Scott
>>>
>>>       [1]
>>>       https://github.com/scottmarlow/hibernate-orm/tree/pooled-optimiser-hack
>>>
>>>       On 12/14/2015 10:12 PM, Scott Marlow wrote:
>>>        >
>>>        >
>>>        > On 12/11/2015 09:30 AM, Steve Ebersole wrote:
>>>        >> It's hard to say without understanding the scenario where you
>>>       are seeing
>>>        >> this as a problem.  I have some guesses as to what may be the
>>>       problem,
>>>        >> but without understanding more about why you see this as a
>>>       problem in
>>>        >> the first place it is hard to give you an answer.  For example,
>>>       I wonder
>>>        >> if for environments not using multi-tenancy whether the recent
>>>       changes
>>>        >> for the generators to support multi-tenancy might be the
>>>       culprit.  If
>>>        >> that is the case, and those changes are in fact the underlying
>>>       cause of
>>>        >> the perf issues you see then I think there is actually a better
>>>        >> solution.  But again, its hard to say unless we understand the
>>>       reason
>>>        >> this "shows up" as a perf problem for you.
>>>        >
>>>        > As best as I can tell from looking at the current PooledLoOptimizer,
>>>        > versus the proposed change (to have a chunk of ids per thread),
>>>       we went
>>>        > from accessing a contented lock, to instead using per thread memory
>>>        > (eliminating the contended lock on PooledLoOptimizer.generate()).
>>>        >
>>>        >>
>>>        >> Until we hear more I think at this stage I'd vote for a separate
>>>        >> optimizer.  And maybe even not one that is upstream.
>>>        >>
>>>        >> Also I agree with Scott that I am VERY leery of not cleaning up a
>>>        >> ThreadLocal.
>>>        >
>>>        > My mistake, as Stuart pointed out, the TL is not static, so we
>>>       shouldn't
>>>        > introduce any leaks.
>>>        >
>>>        >>
>>>        >> On Fri, Dec 11, 2015 at 7:55 AM Scott Marlow <smarlow at redhat.com
>>>       <mailto:smarlow at redhat.com>
>>>        >> <mailto:smarlow at redhat.com <mailto:smarlow at redhat.com>>> wrote:
>>>        >>
>>>        >>      Should this be a specialized pooled optimizer that is only
>>>       used in
>>>        >>      environments that do not suffer from leaving the
>>>       ThreadLocal around
>>>        >>      after the application is undeployed?  In other words, the
>>>       expectation is
>>>        >>      that classloader leaks with this pooled optimizer are
>>>       expected (e.g.
>>>        >>      user must restart the jvm to really undeploy the application
>>>        >>      completely).
>>>        >>
>>>        >>      I am thinking that there are at least three typical situations:
>>>        >>
>>>        >>      1.  Applications are deployed in Java standalone edition.
>>>       Generally,
>>>        >>      when the app undeploys the jvm is shutting down.
>>>        >>
>>>        >>      2. Applications are deployed as part of some container
>>>       (e.g. an EE
>>>        >>      server) and the Hibernate jars are on the global
>>>       classloader path (or
>>>        >>      something like that).  On each shared container thread,
>>>       there would be
>>>        >>      one Optimizer for all deployed applications.  I wonder if
>>>       instead, we
>>>        >>      would want one Optimizer instance per Hibernate SessionFactory
>>>        >>      associated with the many container threads?
>>>        >>
>>>        >>      3. Applications are deployed as part of some container
>>>       (e.g. an EE
>>>        >>      server) and the Hibernate jars are deployed with the
>>>       application. The
>>>        >>      ThreadLocals are associated with threads that are shared by
>>>       different
>>>        >>      deployed applications. The application classloader contains the
>>>        >>      Hibernate classes. Each deployed application has its own
>>>       Optimizer
>>>        >>      threadlocal. On each shared container thread, there would
>>>       be one
>>>        >>      Optimizer per application (since each application has its
>>>       Optimizer TL).
>>>        >>         Like (2), there would be sharing of the same Optimizer
>>>       with the many
>>>        >>      application session factories.  Should we instead have an
>>>       optimizer per
>>>        >>      session factory?
>>>        >>
>>>        >>      Scott
>>>        >>
>>>        >>      On 12/10/2015 11:31 PM, Stuart Douglas wrote:
>>>        >>       > Hello,
>>>        >>       >
>>>        >>       > I have been working on a change to the pooled optimizer
>>>       that we
>>>        >>      have been seeing good performance results with. Basically
>>>       it hands
>>>        >>      out blocks of ID's to a thread local, rather than having every
>>>        >>      thread contend on the lock every time an ID is required.
>>>        >>       >
>>>        >>       >
>>>        >>
>>>       https://github.com/hibernate/hibernate-orm/compare/master...stuartwdouglas:pooled-optimiser-hack
>>>        >>       >
>>>        >>       > What would I need to do to get a change like this in? In
>>>       particular:
>>>        >>       >
>>>        >>       > - Does it need to be a new type of optimizer, or is
>>>       modifying the
>>>        >>      existing one like I have done OK?
>>>        >>       > - How should it be configured?
>>>        >>       >
>>>        >>       > I am happy to do up a PR for this, but I am just not
>>>       really sure
>>>        >>      what would be required to get it to a point where it would be
>>>        >>      acceptable for inclusion.
>>>        >>       >
>>>        >>       > Stuart
>>>        >>       > _______________________________________________
>>>        >>       > hibernate-dev mailing list
>>>        >>       > hibernate-dev at lists.jboss.org
>>>       <mailto:hibernate-dev at lists.jboss.org>
>>>       <mailto:hibernate-dev at lists.jboss.org
>>>       <mailto:hibernate-dev at lists.jboss.org>>
>>>        >>       > https://lists.jboss.org/mailman/listinfo/hibernate-dev
>>>        >>       >
>>>        >>      _______________________________________________
>>>        >>      hibernate-dev mailing list
>>>        >> hibernate-dev at lists.jboss.org
>>>       <mailto:hibernate-dev at lists.jboss.org>
>>>       <mailto:hibernate-dev at lists.jboss.org
>>>       <mailto:hibernate-dev at lists.jboss.org>>
>>>        >> https://lists.jboss.org/mailman/listinfo/hibernate-dev
>>>        >>
>>>        > _______________________________________________
>>>        > hibernate-dev mailing list
>>>        > hibernate-dev at lists.jboss.org <mailto:hibernate-dev at lists.jboss.org>
>>>        > https://lists.jboss.org/mailman/listinfo/hibernate-dev
>>>        >
>>>
>> _______________________________________________
>> hibernate-dev mailing list
>> hibernate-dev at lists.jboss.org
>> https://lists.jboss.org/mailman/listinfo/hibernate-dev
>>
> _______________________________________________
> hibernate-dev mailing list
> hibernate-dev at lists.jboss.org
> https://lists.jboss.org/mailman/listinfo/hibernate-dev
>


More information about the hibernate-dev mailing list