[infinispan-dev] ProtoStream and ease of use
Emmanuel Bernard
emmanuel at hibernate.org
Tue Jul 16 04:29:11 EDT 2013
I've been thinking about it over night.
## Protofiles provided by the server
Let's assume, the Protofiles are retrieved from the server for a given
(class) name. And that if it's not there the client pushes a protofile
and register it. (Is that what you guys had in mind?
In this case, ProtoStream uses the protofile to create a binding between
a fieldname and a field id. If the convention based approach is built on
top of ProtoStream, then, we just need the fieldnames by reflection and
we are good.
If the protofile is not present on the server, then we are free to
create one from either the protostream method calls or from the
convention based approach.
## Protofiles are packaged with the application
Assuming, the previous hypothesis is not correct and that you had in mind
to impose the protofile to be present in the aplication classpath
(bundled), then we have two options
### Have the user write the protofile manually
(I believe that's ProtoStream's approach as well, the user has to write
a protofile).
If we have the protofile, then we are good, we can use a convention or a
simple annotation binding the domain model classes to their protofile
resource and we can use fieldnames and optional @PSType.
### Infer the Protofile from the convention
(that was my original idea as discussed yesterday)
If a protofile is not present, then we could infer one from the domain
model structure (plus some annotation hint). Here the problem of field
ordering is a problem if we don't store the infered protofile.
JVMs do not guarantee that the reflection provides the fields in a
deterministic order. From some reading we could use an
AnnotationProcessor to record the source ordering of fields, this is
guaranteed. We would need an annotation to mark the classes Infinispan
is interested in unfortunately
@Data
class Address {
private String street;
...
}
And the AP would react to @Data
## Conclusion
Totally inferring the schema is more complicated than I initially
imagined but assuming that the user has to write the protofile, then the
convention based approach seems to work smoothly enough.
Emmanuel
On Mon 2013-07-15 20:29, Adrian Nistor wrote:
> BaseMessage is not mandatory. It's there to 'help' you implement Message interface, which is optional anyway and trivially simple to implement; it's only a holder for the bag of unknown fields. Ignore that and you loose support for maintaining unknown fields but otherwise everything works fine.
>
> The class field ordering issue is problematic. Besides that, schema evolution might lead to removed fields so we must be able to skip a range of ids easily. I think the sane (but verbose) way to solve this is to make @PSType mandatory for all fields to be marshalled as you suggested, and PSType should have and 'index' or 'id' parameter. But that's mostly goodbye convention then.
>
> Let's also note that were' striving to be cross language here and on the server side we won't have the user's domain classes available in CP (because they might not even be written in java). So the annotations are not going to help there. We would still need the protobuf file, which could be generated based on the annotated java code as we discussed earlier today, but ... generators kill kittens :( so would try to avoid that.
>
> Since we want to be cross-language I think we should go schema-first. So maybe we could have the protobuf hand-written and the java classes (pojos) minimally annotated with just the full name of the corresponding protobuf message type. From here everything could work automagically provided that class field names match the field name in the proto file and also field types match so no value convertor needs to be specified. If they don't, then annotate.
>
> Does this seem a little better?
>
> On 07/15/2013 06:57 PM, Emmanuel Bernard wrote:
>
> >ProtoStream is like this
> >
> >https://github.com/infinispan/protostream/blob/master/core/src/test/java/org/infinispan/protostream/domain/Account.java
> >https://github.com/infinispan/protostream/blob/master/core/src/test/java/org/infinispan/protostream/domain/marshallers/AccountMarshaller.java
> >(I believe the BaseMessage superclass of Account is optional, not sure).
> >
> >A convention based approach would be like this
> >
> > package org.infinispan.protostream.domain;
> > import org.infinispan.protostream.BaseMessage;
> > /**
> > * @author ebernard at redhat.com
> > */
> > public class Account {
> > private int id;
> > private String description;
> > public int getId() {
> > return id;
> > }
> > public void setId(int id) {
> > this.id = id;
> > }
> > public String getDescription() {
> > return description;
> > }
> > public void setDescription(String description) {
> > this.description = description;
> > }
> > @Override
> > public String toString() {
> > return "Account{" +
> > "id=" + id +
> > ", description='" + description + '\'' +
> > ", unknownFieldSet='" + unknownFieldSet + '\'' +
> > '}';
> > }
> > }
> >
> >Or let's imagine that we need to make id use a specific protobuf type
> >
> > package org.infinispan.protostream.domain;
> > import org.infinispan.protostream.BaseMessage;
> > /**
> > * @author ebernard at redhat.com
> > */
> > public class Account {
> > @PSType(UINT32)
> > private int id;
> > private String description;
> > public int getId() {
> > return id;
> > }
> > public void setId(int id) {
> > this.id = id;
> > }
> > public String getDescription() {
> > return description;
> > }
> > public void setDescription(String description) {
> > this.description = description;
> > }
> > @Override
> > public String toString() {
> > return "Account{" +
> > "id=" + id +
> > ", description='" + description + '\'' +
> > ", unknownFieldSet='" + unknownFieldSet + '\'' +
> > '}';
> > }
> > }
> >
> >Note that a concern is that field ordering (in the bytecode) is not guaranteed across VMs and compilation and I believe that is an important factor of ProtoBuf. So somehow we would need a way to express field indexes wich would amke the annotation approach more verbose.
> >
> >On Mon 2013-07-15 16:04, Manik Surtani wrote:
> >>I'm sorry I missed this. Is there an example of each API somewhere?
> >>
> >>On 15 Jul 2013, at 14:01, Emmanuel Bernard <emmanuel at hibernate.org> wrote:
> >>
> >>>Mircea, Adrian and I had an IRC chat on ProtoStream and ProtoStuff.
> >>>
> >>>check out
> >>>http://transcripts.jboss.org/channel/irc.freenode.org/%23infinispan/2013/%23infinispan.2013-07-15.log.html
> >>>starting at 11:00 and finishing at 12:30
> >>>
> >>>A short summary of what has been discussed:
> >>>
> >>>- ProtoStream is a good cross-platform solution but
> >>> - complicated for the simple pure Java case
> >>> - encourages a technical superclass (EJB 2 !!!!!)
> >>>- ProtoStuff convention + annotation based approach
> >>> https://code.google.com/p/protostuff/wiki/ProtostuffRuntime is nice
> >>> for the pure Java case
> >>>- ProtoStuff is many things and has a non ProtoBuf compliant format for
> >>> cycle ref and polymorphism
> >>>- ProtoStream supports unknown fields (future version of a schema),
> >>> ProtoBuf does not
> >>>- we could build a convention based solution atop ProtoStream
> >>> - assuming UnknownFieldSet and BaseMessage are optional
> >>> - using (cross platform) conventions
> >>> - with metadata to go beyond conventions (annotation, programmatic
> >>> API, XML...)
> >>>
> >>> public long size; //uses fixed64 by default
> >>> @PSType(UINT64) long size; //override protobuf type
> >>>
> >>>- Infinispan will/could(?) have a repo of schema that can be queries
> >>>- we are talking about how the schema is resolved / generated
> >>> - what we send through the wire is independent of the Proto*
> >>>
> >>>A ProtoBuf vs ProtoStream comparison points
> >>>https://gist.github.com/mmarkus/5999646
> >>>
> >>>Emmanuel
> >>>_______________________________________________
> >>>infinispan-dev mailing list
> >>>infinispan-dev at lists.jboss.org
> >>>https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >>--
> >>Manik Surtani
> >>manik at jboss.org
> >>twitter.com/maniksurtani
> >>
> >>Platform Architect, JBoss Data Grid
> >>http://red.ht/data-grid
> >>
> >>
> >>_______________________________________________
> >>infinispan-dev mailing list
> >>infinispan-dev at lists.jboss.org
> >>https://lists.jboss.org/mailman/listinfo/infinispan-dev
> >_______________________________________________
> >infinispan-dev mailing list
> >infinispan-dev at lists.jboss.org
> >https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
More information about the infinispan-dev
mailing list