Re: [aerogear-dev] Use of Differential Synchronization for data sync

Perhaps we’re looking at this from different perspectives. It’s great that you guys are trying to better understand DS so that you can compare it to other techniques, including OT. That certainly needs to be done. I guess I was looking at DS from the perspective of how a data service might need to implement it, knowing that the choice of how data sync is ultimately done will be influenced in part by how the data service would implement each approach and the impact on scalability and performance. Perhaps it’s too early to provide my thoughts along those lines. On Aug 4, 2014, at 9:34 AM, Summers Pittman <supittma(a)redhat.com&gt; wrote: Sure, you don’t have to keep it in memory. But it does have to be in memory to do anything with it in an efficient way. And, yes, you certainly can build a data service that uses this technique. My point was that having multiple copies of a document being edited will reduce the scalability of the data service compared to other techniques. Yes, a document might be a JSON document that is an aggregate of multiple objects, and not just a flat map of key-value pairs. The use of aggregate data structures and denormalization are some of the ways that eventually-consistent data stores work. The goal is to reduce the scope of a set of operations to a single aggregate. Other data stores (like graph and hierarchical databases) require strong consistency and transactions because operations necessarily span multiple objects. But limiting operations to a single aggregate is also quite constraining w/r/t app development, since you can’t always denormalize all data to separate aggregates. So even if a collection (in the MongoDB sense) contains documents that are aggregates of multiple “entities” (in the Hibernate sense of the word), my point still stands that generally any given JSON document will still be smaller than a document used in a collaborative document editor. Also, I would not be surprised if the sheer number of documents in a MongoDB collection is orders of magnitude larger than the number of documents stored by a collaborative editor app. Hmm… even if you could do it this way, do you not want to be able to give feedback to the user that the changes might not have been accepted/persisted? Do you have some scenarios that describe the kinds of applications you’re considering? I’m wondering if I’m envisioning a different kind of app. I was referring to features of the data service, and especially how the data service’s implementation can satisfy the difficult non-functional requirements like scalability and performance. While the algorithm might be really simple, that doesn’t mean implementing it on the server is efficient. What I’ve read so far makes me think that it’s could very well be less efficient and scalable than other techniques used in data services. I guess I was hoping that the client can manipulate documents locally without having to coordinate that with the server. Again, I’m concerned about server scalability. I agree, it’s not a problem with the algorithm. It’s a problem insofar as it would mandate what the server has to support. Perhaps I should have said “expensive” rather than “complicated”. This is actually what a number of MBaaS offerings do, although it’s often hidden by the client SDKs. It may not be ideal because it places more work onto the client SDK, but the benefit is that a good portion of the work is done on the client, and the load on the server is reduced (and scalability increased). It’s also trivially easy for the data service to implement. I completely agree. Only that it seems beneficial that the same mechanism be used for “events" (while connected) and both online and offline data-sync.