Hi, Really interesting. I'm also thinking a lot about the data synch process and I'm glad to see that your ideas are pretty much the same as me. But great challenges to come like dealing multiple edits of the same record by different clients :)

About triggering the synch process we could start a POC with the "online" event on the client side, that could spawn a webwork script taking care of the synch process. On the server side, we have to manage a sort of queue maintaining the state of the "dirty" records on which the clients can subscribe (and being noticed by push events , web sockets etc ...) , very exciting stuff to come. Seb Envoyé de mon iPhone Le Oct 10, 2012 à 17:31, Kris Borchers <kris(a)redhat.com&gt; a écrit : > Hey all, > > Just wanted to let you know what I am working on right now. It is very early but Matthias loves e-mail so I thought I would throw this out there. > > Basically, right now all I am doing is working on keeping track of the status of data in DataManager. Assuming a new dataSync setting is set to true, these are the things I am planning. > > • New record is added to store, status is set to NEW and a UUID is generated > • Record is updated, status is set to MODIFIED > • Record is removed, status is set to REMOVED (data is not actually removed, future will need to keep storage limits in mind) > • Add a new sync method that will run through the data and sync with the server. Assumes client is data of record for now during development until we can determine a strategy for informing the client how the server tracks data status and how it should be informed that this data is being synced > • This brings us back to the discussion about sending metadata to the client on first app load. That metadata could inform the client of the sync strategy > > There's probably more to this but these are my original thoughts right now. > > Kris > _______________________________________________ > aerogear-dev mailing list > aerogear-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/aerogear-dev _______________________________________________ aerogear-dev mailing list aerogear-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/aerogear-dev

I think we should also take a look at the "slowly changing dimension " theories : http://en.m.wikipedia.org/wiki/Slowly_changing_dimension# In particularly type VI Envoyé de mon iPhone Le Oct 10, 2012 à 18:38, Douglas Campos <qmx(a)qmx.me&gt; a écrit : > > On Oct 10, 2012, at 1:07 PM, Sébastien Blanc wrote: > >> Hi, >> Really interesting. I'm also thinking a lot about the data synch process and I'm glad to see that your ideas are pretty much the same as me. >> But great challenges to come like dealing multiple edits of the same record by different clients :) > One strategy that I've seen in the wild is having smaller records, (a contact entity could potentially have a timestamp for each field, to aid during the merge) > > I know it's pretty aggressive and wasteful - just food for thought > >> About triggering the synch process we could start a POC with the "online" event on the client side, that could spawn a webwork script taking care of the synch process. >> On the server side, we have to manage a sort of queue maintaining the state of the "dirty" records on which the clients can subscribe (and being noticed by push events , web sockets etc ...) , very exciting stuff to come. >> >> Seb >> Envoyé de mon iPhone >> >> Le Oct 10, 2012 à 17:31, Kris Borchers <kris(a)redhat.com&gt; a écrit : >> >>> Hey all, >>> >>> Just wanted to let you know what I am working on right now. It is very early but Matthias loves e-mail so I thought I would throw this out there. >>> >>> Basically, right now all I am doing is working on keeping track of the status of data in DataManager. Assuming a new dataSync setting is set to true, these are the things I am planning. >>> >>> • New record is added to store, status is set to NEW and a UUID is generated >>> • Record is updated, status is set to MODIFIED >>> • Record is removed, status is set to REMOVED (data is not actually removed, future will need to keep storage limits in mind) >>> • Add a new sync method that will run through the data and sync with the server. Assumes client is data of record for now during development until we can determine a strategy for informing the client how the server tracks data status and how it should be informed that this data is being synced >>> • This brings us back to the discussion about sending metadata to the client on first app load. That metadata could inform the client of the sync strategy >>> >>> There's probably more to this but these are my original thoughts right now. >>> >>> Kris >>> _______________________________________________ >>> aerogear-dev mailing list >>> aerogear-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/aerogear-dev >> _______________________________________________ >> aerogear-dev mailing list >> aerogear-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/aerogear-dev > > -- qmx > > > _______________________________________________ > aerogear-dev mailing list > aerogear-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/aerogear-dev _______________________________________________ aerogear-dev mailing list aerogear-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/aerogear-dev

On Oct 10, 2012, at 3:22 PM, Burr Sutter <bsutter(a)redhat.com&gt; wrote: > Ideally our framework would be highly configurable: > - collision detection and resolution at the row/tuple/object-level > - or at the column/property level > - sync-synchronization - block the user thread > - async-synchronization - perform on a "background" thread > - utilization push notifications to cause the client-app to walk up and request the changes > > It is relatively rare to have a collision (two users or two batch processes) updating the same record simultaneously but it is a use case that should be addressed. There was one technology I worked with in the past that took a hash of the before "image" of the record and submit that back with the deltas - relatively small over-the-wire packet and allowed easy detection to see if another data element changed. That's actually a good idea. Easy enough to stringify and hash a chunk of data and send that along with the changed record. Then the server does the same hash before an update and can report collisions. I like that a lot actually. > > On Oct 10, 2012, at 12:53 PM, Sébastien Blanc wrote: > >> I think we should also take a look at the "slowly changing dimension " theories : >> http://en.m.wikipedia.org/wiki/Slowly_changing_dimension# >> In particularly type VI >> >> Envoyé de mon iPhone >> >> Le Oct 10, 2012 à 18:38, Douglas Campos <qmx(a)qmx.me&gt; a écrit : >> >>> >>> On Oct 10, 2012, at 1:07 PM, Sébastien Blanc wrote: >>> >>>> Hi, >>>> Really interesting. I'm also thinking a lot about the data synch process and I'm glad to see that your ideas are pretty much the same as me. >>>> But great challenges to come like dealing multiple edits of the same record by different clients :) >>> One strategy that I've seen in the wild is having smaller records, (a contact entity could potentially have a timestamp for each field, to aid during the merge) >>> >>> I know it's pretty aggressive and wasteful - just food for thought >>> >>>> About triggering the synch process we could start a POC with the "online" event on the client side, that could spawn a webwork script taking care of the synch process. >>>> On the server side, we have to manage a sort of queue maintaining the state of the "dirty" records on which the clients can subscribe (and being noticed by push events , web sockets etc ...) , very exciting stuff to come. >>>> >>>> Seb >>>> Envoyé de mon iPhone >>>> >>>> Le Oct 10, 2012 à 17:31, Kris Borchers <kris(a)redhat.com&gt; a écrit : >>>> >>>>> Hey all, >>>>> >>>>> Just wanted to let you know what I am working on right now. It is very early but Matthias loves e-mail so I thought I would throw this out there. >>>>> >>>>> Basically, right now all I am doing is working on keeping track of the status of data in DataManager. Assuming a new dataSync setting is set to true, these are the things I am planning. >>>>> >>>>> • New record is added to store, status is set to NEW and a UUID is generated >>>>> • Record is updated, status is set to MODIFIED >>>>> • Record is removed, status is set to REMOVED (data is not actually removed, future will need to keep storage limits in mind) >>>>> • Add a new sync method that will run through the data and sync with the server. Assumes client is data of record for now during development until we can determine a strategy for informing the client how the server tracks data status and how it should be informed that this data is being synced >>>>> • This brings us back to the discussion about sending metadata to the client on first app load. That metadata could inform the client of the sync strategy >>>>> >>>>> There's probably more to this but these are my original thoughts right now. >>>>> >>>>> Kris >>>>> _______________________________________________ >>>>> aerogear-dev mailing list >>>>> aerogear-dev(a)lists.jboss.org >>>>> https://lists.jboss.org/mailman/listinfo/aerogear-dev >>>> _______________________________________________ >>>> aerogear-dev mailing list >>>> aerogear-dev(a)lists.jboss.org >>>> https://lists.jboss.org/mailman/listinfo/aerogear-dev >>> >>> -- qmx >>> >>> >>> _______________________________________________ >>> aerogear-dev mailing list >>> aerogear-dev(a)lists.jboss.org >>> https://lists.jboss.org/mailman/listinfo/aerogear-dev >> >> _______________________________________________ >> aerogear-dev mailing list >> aerogear-dev(a)lists.jboss.org >> https://lists.jboss.org/mailman/listinfo/aerogear-dev > > > _______________________________________________ > aerogear-dev mailing list > aerogear-dev(a)lists.jboss.org > https://lists.jboss.org/mailman/listinfo/aerogear-dev _______________________________________________ aerogear-dev mailing list aerogear-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/aerogear-dev