On Friday we had been pair-programming and likely finished the implementation:
it looks good but we couldn't run the test.
The blocker is that Map/Reduce on Infinispan only works on DIST, and since
we can't iterate on entries we need M/R so we might need to
reconfigure Infinispan in our tests.
That's annoying as DIST will make our testsuite significantly slower,
an alternative is to have Infinispan fix this limitation first.
Sanne
On 13 March 2013 11:14, Davide D'Alto <daltodavide(a)gmail.com> wrote:
No problem.
The association is lazy but I will investigate about Hibernate.initialize
On Tue, Mar 12, 2013 at 8:01 PM, Emmanuel Bernard
<emmanuel(a)hibernate.org> wrote:
> I have not forgotten, I'm just in a middle of a Bean Validation crisis
> that delayed my look into this issue.
> Could it be BTW that the mass indexer does not ask for these objects to
> be loaded using Hibernate.initialize ? It coudl also be a bug in OGM but
> not necessarily. In particular is the association lazy or eager?
>
> Emmanuel
>
> On Mon 2013-03-11 11:00, Davide D'Alto wrote:
>> I have created a branch for OGM-228 (OGM MassIndexer) that includes
>> OGM-151 (Metamodel) and OGM-273 (load entities from tuple):
>>
https://github.com/DavideD/hibernate-ogm/tree/OGM-228
>>
>> A test I've added fails though (AssociationMassIndexerTest):
>>
https://github.com/DavideD/hibernate-ogm/blob/74549a4d264af30fa88960c30e2...
>>
>> The test uses two entitties IndexedNews and IndexedLabel, with a
>> relationship one to many from news to label.
>> The mass indexing works fine but when I retrieve the list of indexed
>> labels with the query "FROM IndexedLabel", the result contains a list
>> of proxy and the equals fails because the class of the objects in the
>> list is not IndexedLabel.
>>
>> If I first get the list of news and than for each of them I called the
>> method news.getLabels(), everything works fine.
>>
>> Any thoughts
>>
>> Thanks
>>
>> On Thu, Mar 7, 2013 at 10:15 AM, Emmanuel Bernard
>> <emmanuel(a)hibernate.org> wrote:
>> > I have no more coin for this one so I have dumped what I have so far
>> >
https://github.com/hibernate/hibernate-ogm/pull/175
>> >
>> > Emmanuel
>> >
>> > On Wed 2013-03-06 19:18, Emmanuel Bernard wrote:
>> >> I've successfully implemented OGM-151 for EntityKey which is the one
we
>> >> need to move OGM-273 forward for now.
>> >> I am trying to implement it for AssociationKey but caching here is
>> >> significantly harder as data is cross reference across associations.
>> >>
>> >> Sanne, when you worked on the profiling of OGM, do you remember
>> >> AssociationKey putting a pressure in build time or memory wise? Because
>> >> caching them per persister means some rather complex race conditions
and
>> >> more memory used permanently (as opposed to on demand).
>> >>
>> >> So I'm wondering if that's worth it. As an intermediary step, I
could
>> >> introduce AssociationKeyMetadata but build it on-demand - that one is
>> >> easier to achieve.
>> >>
>> >> Emmanuel
>> >>
>> >> On Wed 2013-03-06 15:32, Davide D'Alto wrote:
>> >> > it's ok for me
>> >> >
>> >> > Davide
>> >> >
>> >> > On Wed, Mar 6, 2013 at 3:28 PM, Emmanuel Bernard
<emmanuel(a)hibernate.org> wrote:
>> >> > > I'm planning on working on OGM-151. Fine with everyone?
>> >> > > That will likely be my last before I move back to BVAL and
close the
>> >> > > final issues there.
>> >> > >
>> >> > > Emmanuel
>> >> > >
>> >> > > On Tue 2013-03-05 19:04, Sanne Grinovero wrote:
>> >> > >> Nice!
>> >> > >> n+1 is something Hibernate Search has to deal with too,
that's why I
>> >> > >> was interested in the fetch profiles and graph loading in
JPA 2.1
>> >> > >>
>> >> > >> On 5 March 2013 17:44, Emmanuel Bernard
<emmanuel(a)hibernate.org> wrote:
>> >> > >> > I have implemented a solution that gives an entity
based on a tuple.
>> >> > >> >
https://hibernate.onjira.com/browse/OGM-273#comment-50082
>> >> > >> >
>> >> > >> > Note that it does not currently works for MongoDB,
but that's waiting
>> >> > >> > for the dedicated GridDialect method as well as
OGM-151.
>> >> > >> > Also note that I have no idea how that will work for
associations. I
>> >> > >> > suspect some nasty n+1 is happening as best. Worse
case, an exception :)
>> >> > >> >
>> >> > >> > Emmanuel
>> >> > >> >
>> >> > >> > On Tue 2013-03-05 10:30, Emmanuel Bernard wrote:
>> >> > >> >> We might hope for a stable enough contract on
Hibernate Search and
>> >> > >> >> hope that we won't break serializability
between micro or minor
>> >> > >> >> versions. That will need to be taken into account
in the test suite and
>> >> > >> >> design.
>> >> > >> >> On the OGM side though, we are not at that level
of maturity and we will
>> >> > >> >> force homogenous Hibernate OGM version across all
the cluster. The grid
>> >> > >> >> will have to go down for upgrades or enforce that
no mpa reduce job
>> >> > >> >> using OGM is used while the version roll out is
in process.
>> >> > >> >>
>> >> > >> >> Emmanuel
>> >> > >> >>
>> >> > >> >> On Mon 2013-03-04 18:30, Sanne Grinovero wrote:
>> >> > >> >> > Found an example, this is all the code it
needs to have a MassIndexer working
>> >> > >> >> > on top of Infinispan's Map/Reduce:
>> >> > >> >> >
>> >> > >> >> >
https://github.com/infinispan/infinispan/blob/master/query/src/main/java/...
>> >> > >> >> >
>> >> > >> >> > Note it's initialize method which
injects needed components; the
>> >> > >> >> > implementation is serialized across nodes.
>> >> > >> >> >
>> >> > >> >> > Sanne
>> >> > >> >> >
>> >> > >> >> > On 4 March 2013 18:26, Sanne Grinovero
<sanne(a)hibernate.org> wrote:
>> >> > >> >> > > We finished this discussion on IRC, in
case someone else was interested:
>> >> > >> >> > >
>> >> > >> >> > > <sanne> hum I forgot the first
step.. transformation from entry into entity
>> >> > >> >> > > <sanne> updated
>> >> > >> >> > > <sanne> emmanuel, the
"hidrate" step is what DavideD is bashing is
>> >> > >> >> > > head against, but let's assume he
finds a workaround and we focus on
>> >> > >> >> > > the pattern as first step?
>> >> > >> >> > > <emmanuel>
https://gist.github.com/emmanuelbernard/5084039
>> >> > >> >> > > <emmanuel> sanne: ^ that's
how I would do it if I had an Iterator from the tuple
>> >> > >> >> > > <emmanuel> assuming
pushToExecutor pushes to whatever concurrent work
>> >> > >> >> > > mechanism you planned to use on
consumes
>> >> > >> >> > > <emmanuel> Plus I am not folloing
exactly how you plan consumes(Entry)
>> >> > >> >> > > to be executed concurrently
>> >> > >> >> > > <emmanuel> is that the
GridDialect responsibility?
>> >> > >> >> > > <emmanuel> That looks like a lot
of work on the dialect's side
>> >> > >> >> > > <sanne> emmanuel, imagine the
backend is Infinispan and has some large
>> >> > >> >> > > amount of data per node, plus that each
node has its own backend
>> >> > >> >> > > IndexManager (like and ideal sharding)
>> >> > >> >> > > <emmanuel> ie pool mgt and cap +
queuing
>> >> > >> >> > > <sanne> then with your approach
the iterator needs to fetch data from
>> >> > >> >> > > all remote nodes, and then enqueue in a
local blocking queue which is
>> >> > >> >> > > returning the data to the original
owners
>> >> > >> >> > > <sanne> but if you skip that
step, you can just forward the statless
>> >> > >> >> > > consumer to each node and have it run
on data locality
>> >> > >> >> > > <emmanuel> I was thinking that if
you had the luncene index locally on
>> >> > >> >> > > each node you would ahve a different
impl of the MassIndexer anyways
>> >> > >> >> > > <emmanuel> that would simply send
a command to each local node
>> >> > >> >> > > <sanne> To answer your question:
that would be an optional GridDialect
>> >> > >> >> > > responsibility. I would endorse a
trivial first draft doing a
>> >> > >> >> > > single-threaded loop.
>> >> > >> >> > > <emmanuel> and have
GridDialect.getDataFor() returnlocal data
>> >> > >> >> > > <sanne> The "consumes"
implementation can be either implemented with a
>> >> > >> >> > > simple iterator - as in your design -
so I don't think it pushes much
>> >> > >> >> > > complexity to the GridDialect
implementor?
>> >> > >> >> > > <sanne> The benefit of the
consumer is that *optionally* it can be
>> >> > >> >> > > mapped on the Map phase, and that's
trivial if your backend supports
>> >> > >> >> > > Map/Reduce
>> >> > >> >> > > <emmanuel> sanne: I don't
follow that soory
>> >> > >> >> > > <emmanuel> how does that make it
mappable to the Map phase?
>> >> > >> >> > > <sanne> "public void
consume(Entry e) " is a degenerate (simplified)
>> >> > >> >> > > form of map.
>> >> > >> >> > > <sanne> mm infinispan IDE crashes
at the right moment.
>> >> > >> >> > > <emmanuel> I thought Map was
about *filtering*
>> >> > >> >> > > <emmanuel> not processing
>> >> > >> >> > > <sanne> you can decide to accept
100% of values (without filtering),
>> >> > >> >> > > but actually you might want to filter
on the specified tables only.
>> >> > >> >> > > <sanne> also, the return type
doesn't have to match the input type:
>> >> > >> >> > > hence you define a transformation
function, which is inherently
>> >> > >> >> > > applied in parallel on all matching
entries.
>> >> > >> >> > > <emmanuel> sanne: but then you
require the OGM code to be everywhere
>> >> > >> >> > > (ie on each node of the targetNoSQL
>> >> > >> >> > > <emmanuel> to eb able to do tuple
-> entity
>> >> > >> >> > > <emmanuel> that's not
realistic
>> >> > >> >> > > <emmanuel> assuming your
transform phase is about tuple -> entity and
>> >> > >> >> > > some HSearch ops
>> >> > >> >> > > <sanne> yes right
>> >> > >> >> > > <sanne> but isn;t it worth it?
it's optional and much more efficient,
>> >> > >> >> > > as you avoid transferring any data.
>> >> > >> >> > > <sanne> btw we often assume all
nodes in the grid are equally
>> >> > >> >> > > configured, so having same apps &
libraries deployed.
>> >> > >> >> > > <emmanuel> sanne: let me try and
summarize what I understand
>> >> > >> >> > > <emmanuel> it's more
efficient if you store the Lucene index locally
>> >> > >> >> > > with the data, and if the grid is
written in Java or at least can run
>> >> > >> >> > > code in Java including libraries and if
you distribute the OGM
>> >> > >> >> > > configuration across the whole grid
>> >> > >> >> > > <emmanuel> Otherwise, it does not
make any difference
>> >> > >> >> > > <emmanuel> Also the GridDialect
implementation need to know if you are
>> >> > >> >> > > doing this trick to only return local
data
>> >> > >> >> > > <sanne> no there are other
drawbacks which get defeated, but minor so
>> >> > >> >> > > I didn't mention them
>> >> > >> >> > > <emmanuel> am I right?
>> >> > >> >> > > <sanne> mainly, you skip the need
for the contentions point as there
>> >> > >> >> > > is no push to a shared blocking queue
>> >> > >> >> > > <sanne> no the GridDialect
doesn't need to know.
>> >> > >> >> > > <emmanuel> sanne: sure if you can
process the code on each node you
>> >> > >> >> > > avoid the shared blocking queue, at
lest until you reach the
>> >> > >> >> > > IndexManager
>> >> > >> >> > > <sanne> you'll just forward a
simple (standard) M/R task, and it will
>> >> > >> >> > > need to execute it as always.
>> >> > >> >> > > <sanne> the IndexManager is
parallel ;)
>> >> > >> >> > > <emmanuel> sanne: parallel on a
single node
>> >> > >> >> > > <sanne> yes, but no contentions
points other than the internal
>> >> > >> >> > > structure of the IW
>> >> > >> >> > > <emmanuel> I mean updating the
index for a given table is better done
>> >> > >> >> > > on a singlle node
>> >> > >> >> > > <sanne> IndexWriter
>> >> > >> >> > > <emmanuel> sorry I meant
IndexWriter
>> >> > >> >> > > <emmanuel> ah but ou mention
perfect sharding
>> >> > >> >> > > <emmanuel> you need cosmological
alignment for this shit to happen
>> >> > >> >> > > <sanne> not if we plan for it :)
>> >> > >> >> > > <sanne> you might remember the
changes to Segments in the ISPN code,
>> >> > >> >> > > to accomodate index storage consistent
with the data locality
>> >> > >> >> > > <sanne> that's expected in
6.0
>> >> > >> >> > > <emmanuel> So
gridDialect.getData(Consumer consumer, String.. tables) is wrong
>> >> > >> >> > > <emmanuel> it's more
gridDialect.getData(ConsumerImpl.class, String... tables)
>> >> > >> >> > > <emmanuel> as you ened to send
the Comsumer impl
>> >> > >> >> > > <emmanuel> not simply use it
>> >> > >> >> > > <sanne> hu, it needs a reference
to the current SearchFactory at very least
>> >> > >> >> > > <emmanuel> sanne: but you're
telling me you send the M/R task
>> >> > >> >> > > <emmanuel> so you need to send
the M/R code as well
>> >> > >> >> > > <sanne> yes but here we enter
Infinspan specific implementation
>> >> > >> >> > > <sanne> I would register the
needed components in Infinispan and use
>> >> > >> >> > > the ServiceRegistry to look them up
remotely
>> >> > >> >> > > <sanne> not to mention Infinispan
could accomodate a custom command for it
>> >> > >> >> > > <emmanuel> What I am saying is
that you don't pass the Consumer
>> >> > >> >> > > *instance* tot he grid dialect but
rather the impl, no?
>> >> > >> >> > > <sanne> the impl class
definition?
>> >> > >> >> > > <emmanuel> sanne: you tell me.
How do I send M/R code today?
>> >> > >> >> > > <emmanuel> certainly not an impl
instance
>> >> > >> >> > > <sanne> yes you do
>> >> > >> >> > > <sanne> JBMar will take care of
it, including state.
>> >> > >> >> > > <sanne> but in this case that
would be wrong of course as I don't want
>> >> > >> >> > > to serialize the whole SearchFactory so
I'd use injection and lookup,
>> >> > >> >> > > but that's a detail of Infinispan.
>> >> > >> >> > > <sanne> But this shouldn't be
MassIndexer specific right? it's good to
>> >> > >> >> > > expose a general "execute on
all" method, and I think accepting
>> >> > >> >> > > instances would make life easier for
most - even though we might need
>> >> > >> >> > > to document some limitations.
>> >> > >> >> > > <emmanuel> alright, I guess
'll have to live with a visitor pattern
>> >> > >> >> > > for a feature that has 5% chance of
happening :)
>> >> > >> >> > > <sanne> I'm going to punch
Davide
>> >> > >> >> > > <sanne> as he's yelling
"it's not a visitor" but doesn't have the guts
>> >> > >> >> > > to write it down :)
>> >> > >> >> > > <emmanuel> sanne: DavideD 's
would have nothing to do about it, that's
>> >> > >> >> > > requires a lot of config and Infinispan
machinery I'm not sure is here
>> >> > >> >> > > today
>> >> > >> >> > > <DavideD> :)
>> >> > >> >> > > <emmanuel> ah
>> >> > >> >> > > <emmanuel> I don't care how
it's called, it's one of those patterns
>> >> > >> >> > > that make the code harder to follow
>> >> > >> >> > > <DavideD> I was actually trying
to remember the name of the pattern
>> >> > >> >> > > <sanne> ok now we agree :)
>> >> > >> >> > > <emmanuel> Obfuscator pattern
family
>> >> > >> >> > > <sanne> very popular among
consultants, I don't understand why you complain :P
>> >> > >> >> > > <sanne> Anyway, let's wrap up
and broaden the horizon:
>> >> > >> >> > > <emmanuel> ok so we are left with
findin to to load a entity from a tuple
>> >> > >> >> > > <sanne> you don't think
it's useful as a general purpose method?
>> >> > >> >> > > <emmanuel> sanne: wil be for
queries
>> >> > >> >> > > <emmanuel> It's just that
it's non obvious
>> >> > >> >> > > <sanne> Exactly. Also I think
lambda methods are getting widely better known.
>> >> > >> >> > > <emmanuel> syntactically yes
>> >> > >> >> > > <emmanuel> VM wise, perf
improvements will come later
>> >> > >> >> > > <sanne> what I mean is that by
defining the SPI this way, I don't
>> >> > >> >> > > expect it to be more complex for the
GridDialect implementors, while
>> >> > >> >> > > we can reuse it for a wider scope of
needs.
>> >> > >> >> > >
>> >> > >> >> > > --Sanne
>> >> > >> >> > >
>> >> > >> >> > > On 4 March 2013 17:02, Emmanuel Bernard
<emmanuel(a)hibernate.org> wrote:
>> >> > >> >> > >>
>> >> > >> >> > >>
>> >> > >> >> > >> On 4 mars 2013, at 17:39, Sanne
Grinovero <sanne(a)hibernate.org> wrote:
>> >> > >> >> > >>
>> >> > >> >> > >>> On 4 March 2013 16:20, Emmanuel
Bernard <emmanuel(a)hibernate.org> wrote:
>> >> > >> >> > >>>> I already gave what I knew
on how to load an entity from a tuple (which
>> >> > >> >> > >>>> isn't much) but we can
try and dig together. Something I thought about
>> >> > >> >> > >>>> is that ORM probably has a
mechanism to load an entity from a resultset
>> >> > >> >> > >>>> via the query parser. And
that probably looks also like the second half
>> >> > >> >> > >>>> of OgmLoader.load. We could
look at this part and see if we can make an
>> >> > >> >> > >>>> OGM version of it. We never
had the need before as we never had query
>> >> > >> >> > >>>> support (the way SQL does
it).
>> >> > >> >> > >>>
>> >> > >> >> > >>> I would also need to study the
ORM code, but to add a high level observation,
>> >> > >> >> > >>> the methods currently defined
by the GridDialect are focusing on
>> >> > >> >> > >>> loading from well known key
instances,
>> >> > >> >> > >>> there is nothing to makes us
able to scan/inspect for all values.
>> >> > >> >> > >>>
>> >> > >> >> > >>> In other words: even if we
wanted to load keys first, we don't have definitions
>> >> > >> >> > >>> of functions from
raw->primary key instances either.
>> >> > >> >> > >>
>> >> > >> >> > >> I understand that. I'm not
denying the need for the method.
>> >> > >> >> > >>
>> >> > >> >> > >>>
>> >> > >> >> > >>>
>> >> > >> >> > >>>> On the visitor vs Iterator
approach, I still don't see how implementing
>> >> > >> >> > >>>> an Iterator on a map /
reduce backend would be harder than the visitor
>> >> > >> >> > >>>> but maybe I'm missing
something.
>> >> > >> >> > >>>>
>> >> > >> >> > >>>> class IteratorAsStream
{
>> >> > >> >> > >>>> final Query
someMapReduceQuery = ...;
>> >> > >> >> > >>>>
>> >> > >> >> > >>>> public Object next()
{
>> >> > >> >> > >>>> if
(!someMapReduceQuery.started()) {
>> >> > >> >> > >>>> // execute
and collect results in parallel
>> >> > >> >> > >>>>
someMapReduceQuery.execute();
>> >> > >> >> > >>>> }
>> >> > >> >> > >>>> Object result =
someMapReduce.getNextOrBlock();
>> >> > >> >> > >>>> return result;
>> >> > >> >> > >>>> }
>> >> > >> >> > >>>> }
>> >> > >> >> > >>>
>> >> > >> >> > >>> That could work to *load* all
entities in parallel, but I'd like to
>> >> > >> >> > >>> process the entities in
parallel as well.
>> >> > >> >> > >>> And I'd rather not force
the GridDialect implementors to write some
>> >> > >> >> > >>> Hibernate Search specific
code,
>> >> > >> >> > >>> so to break out we need some
form of "Execute X on each": a closure or a lambda.
>> >> > >> >> > >>>
>> >> > >> >> > >>
>> >> > >> >> > >> I can't see how the visitor
model helps in your processing of entities in parallel. To me both approaches are strictly
equivalent. Care to show some pseudo-code?
>> >> > >> >> _______________________________________________
>> >> > >> >> hibernate-dev mailing list
>> >> > >> >> hibernate-dev(a)lists.jboss.org
>> >> > >> >>
https://lists.jboss.org/mailman/listinfo/hibernate-dev
>> >> > > _______________________________________________
>> >> > > hibernate-dev mailing list
>> >> > > hibernate-dev(a)lists.jboss.org
>> >> > >
https://lists.jboss.org/mailman/listinfo/hibernate-dev
>> >> _______________________________________________
>> >> hibernate-dev mailing list
>> >> hibernate-dev(a)lists.jboss.org
>> >>
https://lists.jboss.org/mailman/listinfo/hibernate-dev
_______________________________________________
hibernate-dev mailing list
hibernate-dev(a)lists.jboss.org
https://lists.jboss.org/mailman/listinfo/hibernate-dev