[hibernate-dev] HSEARCH State to transfer

[Sanne Grinovero]

2011/8/4 Emmanuel Bernard <emmanuel at hibernate.org>:
>
> On 3 août 2011, at 20:06, Sanne Grinovero wrote:
>
>> 2011/8/3 Emmanuel Bernard <emmanuel at hibernate.org>:
>>> Here is a first prototype of the serializable model and an almost complete converter between List<LuceneWork> and this model.
>>>
>>> There is not test yet. It does compile but I don't know if it works.
>>
>> (translation of the above:
>> https://github.com/emmanuelbernard/hibernate-search/tree/745 )
>>
>> I only had a brief scan through the code, so I might send more comments later;
>>
>> 1) Remember all operations are implicitly scoped to a single index, so
>> for example you don't need to make a difference between
>> Optimize(classType) and OptimizeAll, they will do the same: optimize
>> the index.
>
> That's a good point. Is that your last word though? Will we want to limit messages passing through by regrouping several backends under one message?

I'm 99% confident on it; do you foresee a good reason to re-shard /
split again the index?
Only reason I would think of that is to use the Index splitter tools
because it's too big, but then again it would still contain the same
entity types and be considered as one from a logic point of view.
Could be interesting only if you think someone will deploy a chain of
them, maybe to enable sharding "on the fly" reconfiguring the master
only. Definitely some weird things could be made if you support this,
but if it makes your work simpler to ignore it for now I'd take
advantage of it. You're designing an upgradeable protocol right :P ?

>> 2) It would be great if you could avoid the "instanceof" :)
>> (see also org.hibernate.search.backend.impl.WorkVisitor )
>
> Hum, I have always found the visitor pattern difficult to understand, that's why I usually chose this imperative approach.


>> 3) I'm confused about what your intentions are about those
>> Serializable wrappers. Wrapping objects in some container implementing
>> SerializableFieldable is not going to make them serializable. I think
>> I missed, or you didn't add yet, where the conversion to byte[]
>> happens ?
>
> The model represents the data we want to pass in.
> I am now exploring how to serialize it with various libraries. It looks though that these libraries won't be able to support version based deserialization logic and reuse the model as it is. We will likely need some unstructured version of the model and a code similar to the Converter logic.

Makes sense.

> So far I'm looking at MessagePack without much success. Documentation is sparse and they don't seem to support reading the version before the rest of the message.

Since MessagePack is "JSon like", as far as I understood it should be
able to always succeed in parsing, so we would need the version number
only *after* it parsed the message to see how we can interpret it.
Though I'm inclined to think that Proto Buffers is a better fit, if
you're considering external libraries, as it helps with the problem of
adding/removing data in different releases.

Sanne