[hibernate-issues] [Hibernate-JIRA] Commented: (HSEARCH-880) Discussion on how to support backward / forward compatible serialization layer

[Sanne Grinovero (JIRA)]

    [ http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=43374#comment-43374 ] 

Sanne Grinovero commented on HSEARCH-880:
-----------------------------------------

{quote}Dynamically assigning them is scaring me as the numbers might not be unique across a cluster or upon restart. It also makes debugging very hard as these numbers would change "at random".

Also, if someone wants to change the serialization provider, it needs to stop the whole cluster making sure the master has processed all remaining messages before changing the configuration and restarting the cluster. The serialization id could prevent that.

Also as discussed in the previous comment, the serialization id helps differentiate a version number from another provider {quote}

I think we agree on the fact that dynamic assignments is scary and hard to debug: my first paragraph is exactly to make the point that in case you want them they should be dynamic, and that brings a complexity, so I think we should not support more than a single SerializationProvider (per IndexManager), making it unnecessary to have such ids. People will be able to upgrade serialization versions of the same type, but switching dynamically from one technology to another doesn't seem worth the effort.

> Discussion on how to support backward / forward compatible serialization layer
> ------------------------------------------------------------------------------
>
>                 Key: HSEARCH-880
>                 URL: http://opensource.atlassian.com/projects/hibernate/browse/HSEARCH-880
>             Project: Hibernate Search
>          Issue Type: New Feature
>          Components: serialization
>            Reporter: Emmanuel Bernard
>             Fix For: 4.0
>
>
> h1. General principles
> The serialized message needs the following elements:
> * index name: to redirect the flux to the appropriate backend
> * serialization provider id: if not present, a cluster must make sure to use the same SerializationProvider for a given IndexManager
> * protocol version: today the version is major.minor where the major increase means incompatibility at the stream level, whereas minor means compatibility but with missing features
> * stream: this is the SerializationProvider specific byte[]
> bq. Do we need a serialization provider id? In other words, do we need to be able to hot-upgrade the SerializationProvider in a cluster?
> h1. Exchanging messages in an heterogeneous cluster
> h2. Cluster with one way communication (JMS)
> In this case the master receives a message and must try and process it.
> Receives an index name + serial provider id.
> Use the serial provider id to deserialize the message.
> If message_major > node_major, the serialization provider fails
> If message_minor > node_minor, the serialization provider proceeds but some features might not be supported and the deserialization might fail.
> bq. this requires to send the Avro schema with each message which would be a huge loss to support message_minor > node_minor
> In the minor bump case:
> * some feature might not be deserialized and simply ignored. A user is aware of the list of features differences between each node.
> * the stream might not be readable by an old version after all due to the use of some new features => Exception
> If message_major or message_minor < node_major or node_minor, we use the older protocol deserializer. 
> bq. could there ever be a problem where a new HSearch Engine cannot deal with an old HSearch engine's message?
> h2. Cluster with two way communication (JGroups)
> Each time a node A needs to send a message to a node B for the first time. It sends the list of supported SerializationProvider id and for each the list of Versions supported. The first SerializationProvider id is preferred and the latest versions are preferred. 
> A version is more recent if majorA > majorB and with majorA = majorB if minorA > minorB.
> Node B receives the handshake message and returns the appropriate serialization provider id and version. Subsequent messages are exchanged with this accepted version between A and B
> bq. Is the JGroups clustering using multicast to send change messages ie does it know which node it sends the message to to do the handshake?
> bq. What happens if B goes down and back up? Does it have a "new" name that uniquely identify it?
> h1. API changes
> SerializationProvider will need the following adjustments:
> * a getSupportedVersions()
> * a getSerializer(Version)
> * a getDeserializer(Version)
> bq. could it be that Serializer / Deserializer / LuceneWorksBuilder lead to the inability to support a version n-1 (by adding of new methods or stuff like that?
>  

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira