On 03/27/2013 01:38 PM, Summers Pittman wrote: * So for offline and sync for 2.0 I’ve been doing some thinking/research and come up with some broad topics to discuss so we can start honing in on what we want it to do/look like. This isn’t a spec, this isn’t a proposal, this is just trying to narrow down what we want at a high level so we can pick things to focus on. 1. Documents vs Transactions There are two “big picture” methods of doing sync. One is a Document sync (Think like a gallery with videos and pictures). Documents are saved, the whole document is sent to the server, then the whole document is pushed out to other clients who are syncing against the same source. The other is a Transactional sync where many small atomic operations are sent to the server. The best analog I have is Google Drive/operation transforms. Obviously the server side implementations of these are hilariously divergent and I will leave the relative complexity of each as an exercise for the list. 2. Background vs Foreground sync Does the application have to be opened (foreground) for syncing to happen? As far as I know, native Web requires this (barring extensions to the browser, plugins etc). I think Cordova can in Android, but I haven’t researched it. iOS seems like a mixed bag, but generally you can only sync if your application is in the foreground (but you can use notifications and badges to communicate that there is a pending sync or new data). I understand there is CoreData + iCloud that is supposed to do something, but that seems like it is still foreground only. On Android background sync is easy. Is it OK to only have background syncing on some platforms but not others? 3. Push vs Poll? Obviously pushing updates is better for devices and users, but polling will let things work better for legacy services which may not have push support or which may be difficult to integrate into AG-Controller. Should we support both on the clients? 4. Multiple clients,multiple users, and conflicts How do we want to support multiple users and multiple clients? How should we try to do conflict resolution? What does authorization and authentication look like here? At a high level here are some options I have seen for conflict resolution: A. Last in always wins. The server explicitly trusts things in the order it gets and pushes that data out to users. B. Clients are allowed one submit at a time and must wait for the server to acknowledge the receipt. If there is a conflict the app can either a) merge the data, b)reload the latest from the server and make the user do his operation again, or c) create a new document and inform the user. C. Operation Transforms. This was meant to solve the conflict and sync problem. However it is a LOT of work 5. Offline Support Really this is more what do we want to do for coming from an “offline” mode to an “online” mode? Abstractly, operations which happen offline are the same as operations which happen online just with a REALLY REALLY laggy connection. :) We could just only viewing data when offline and requiring a connection for editing, queueing an upload, etc. 6. How much of this is the responsibility of AG-controller vs underlying services? How should the controller expose resources to clients, how should the controller send data to its underlying services, how much data should the controller be responsible itself for? Should it be easy for an Operation Transform system to integrate with AeroGear-controller? Should it be easy to write a Controller based project which polls a third party source? How would the server handle passing credentials to the third party source? Appendix Use Cases: Here are a few contrived use cases that we may want to keep in mind. 1. Legacy Bug Trackers From Hell a. It is a webapp written in COBOL, no one will ever EVER update or change the code b. It has TONS of legacy but important data c. It has TONS of users d. It only has a few transactions per day, all creating and updating bug reports e. Multiple users can edit the same report 2. Slacker Gallery a. Each User has a multiple galleries, each gallery has multiple photos b. A Gallery has only one user, but the user may be on multiple devices c. Galleries may be renamed, created, and deleted d. Photos may only be created or deleted. Photos also have meta data which may be updated, but its creation and deletion is tied to the Photo object. 3. Dropbox clone a. A folder of files may be shared among users b. There is a size limit to files and how much storage may be used per folder c. Files are not updated. If there is a new file, there is an atomic delete and create operation 4. Email client a. This is an AG-controller which accesses a mail account. b. There are mobile offline and sync enabled clients which connect to this controller. 5. Google Docs clone a. Operational Transform out the wazzoo b. What would the server need? c. What would the client need? Appendix Reference (Open Source) Products: Wave-in-a-box CouchDB Google Drive RealtimeAPI Can you guys think of more projects/examples to look at for inspiration? * Matzew mentions http://sharejs.org/ And also there is etherpad. _______________________________________________ aerogear-dev mailing listaerogear-dev@lists.jboss.orghttps://lists.jboss.org/mailman/listinfo/aerogear-dev _______________________________________________ aerogear-dev mailing list aerogear-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/aerogear-dev

Sorry for the questionable formatting before, this should be more correct now. So for offline and sync for 2.0 I’ve been doing some thinking/research and come up with some broad topics to discuss so we can start honing in on what we want it to do/look like. This isn’t a spec, this isn’t a proposal, this is just trying to narrow down what we want at a high level so we can pick things to focus on. 1. Documents vs Transactions There are two “big picture” methods of doing sync. One is a Document sync (Think like a gallery with videos and pictures). Documents are saved, the whole document is sent to the server, then the whole document is pushed out to other clients who are syncing against the same source. The other is a Transactional sync where many small atomic operations are sent to the server. The best analog I have is Google Drive/operation transforms. Obviously the server side implementations of these are hilariously divergent and I will leave the relative complexity of each as an exercise for the list. 2. Background vs Foreground sync Does the application have to be opened (foreground) for syncing to happen? As far as I know, native Web requires this (barring extensions to the browser, plugins etc). I think Cordova can in Android, but I haven’t researched it. iOS seems like a mixed bag, but generally you can only sync if your application is in the foreground (but you can use notifications and badges to communicate that there is a pending sync or new data). I understand there is CoreData + iCloud that is supposed to do something, but that seems like it is still foreground only. On Android background sync is easy. Is it OK to only have background syncing on some platforms but not others? 3. Push vs Poll? Obviously pushing updates is better for devices and users, but polling will let things work better for legacy services which may not have push support or which may be difficult to integrate into AG-Controller. Should we support both on the clients? 4. Multiple clients,multiple users, and conflicts How do we want to support multiple users and multiple clients? How should we try to do conflict resolution? What does authorization and authentication look like here? At a high level here are some options I have seen for conflict resolution: A. Last in always wins. The server explicitly trusts things in the order it gets and pushes that data out to users. B. Clients are allowed one submit at a time and must wait for the server to acknowledge the receipt. If there is a conflict the app can either a) merge the data, b)reload the latest from the server and make the user do his operation again, or c) create a new document and inform the user. C. Operation Transforms. This was meant to solve the conflict and sync problem. However it is a LOT of work 5. Offline Support Really this is more what do we want to do for coming from an “offline” mode to an “online” mode? Abstractly, operations which happen offline are the same as operations which happen online just with a REALLY REALLY laggy connection. :) We could just only viewing data when offline and requiring a connection for editing, queueing an upload, etc. 6. How much of this is the responsibility of AG-controller vs underlying services? How should the controller expose resources to clients, how should the controller send data to its underlying services, how much data should the controller be responsible itself for? Should it be easy for an Operation Transform system to integrate with AeroGear-controller? Should it be easy to write a Controller based project which polls a third party source? How would the server handle passing credentials to the third party source? Appendix Use Cases: Here are a few contrived use cases that we may want to keep in mind. 1. Legacy Bug Trackers From Hell a. It is a webapp written in COBOL, no one will ever EVER update or change the code b. It has TONS of legacy but important data c. It has TONS of users d. It only has a few transactions per day, all creating and updating bug reports e. Multiple users can edit the same report 2. Slacker Gallery a. Each User has a multiple galleries, each gallery has multiple photos b. A Gallery has only one user, but the user may be on multiple devices c. Galleries may be renamed, created, and deleted d. Photos may only be created or deleted. Photos also have meta data which may be updated, but its creation and deletion is tied to the Photo object. 3. Dropbox clone a. A folder of files may be shared among users b. There is a size limit to files and how much storage may be used per folder c. Files are not updated. If there is a new file, there is an atomic delete and create operation 4. Email client a. This is an AG-controller which accesses a mail account. b. There are mobile offline and sync enabled clients which connect to this controller. 5. Google Docs clone a. Operational Transform out the wazzoo b. What would the server need? c. What would the client need? Appendix Reference (Open Source) Products: Wave-in-a-box CouchDB Google Drive RealtimeAPI share.js etherpad Can you guys think of more projects/examples to look at for inspiration? _______________________________________________ aerogear-dev mailing list aerogear-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/aerogear-dev

On Wed, Mar 27, 2013 at 7:43 PM, Summers Pittman <supittma(a)redhat.com <mailto:supittma@redhat.com>> wrote: Sorry for the questionable formatting before, this should be more correct now. So for offline and sync for 2.0 I've been doing some thinking/research and come up with some broad topics to discuss so we can start honing in on what we want it to do/look like. This isn't a spec, this isn't a proposal, this is just trying to narrow down what we want at a high level so we can pick things to focus on. 1. Documents vs Transactions There are two "big picture" methods of doing sync. One is a Document sync (Think like a gallery with videos and pictures). Documents are saved, the whole document is sent to the server, then the whole document is pushed out to other clients who are syncing against the same source. The other is a Transactional sync where many small atomic operations are sent to the server. The best analog I have is Google Drive/operation transforms. Obviously the server side implementations of these are hilariously divergent and I will leave the relative complexity of each as an exercise for the list. IMO, because Aerogear is a library we should focus on a Transactional sync to let the users a more fine-grained control. But later, we should be able to provide some wrappers around this atomic operations to offer something close to document sync (or let the users easily define a "Document", like a view in SQL) 2. Background vs Foreground sync Does the application have to be opened (foreground) for syncing to happen? As far as I know, native Web requires this (barring extensions to the browser, plugins etc). I think Cordova can in Android, but I haven't researched it. iOS seems like a mixed bag, but generally you can only sync if your application is in the foreground (but you can use notifications and badges to communicate that there is a pending sync or new data). I understand there is CoreData + iCloud that is supposed to do something, but that seems like it is still foreground only. On Android background sync is easy. Is it OK to only have background syncing on some platforms but not others? Shouldn't we trigger a sync only when we go from an offline status to an online status ?

And sync strategy should be configurable (both on client side and server side) 3. Push vs Poll? Obviously pushing updates is better for devices and users, but polling will let things work better for legacy services which may not have push support or which may be difficult to integrate into AG-Controller. Should we support both on the clients? Yes, we should promote push but have a fallback to a pull strategy

4. Multiple clients,multiple users, and conflicts How do we want to support multiple users and multiple clients? How should we try to do conflict resolution? What does authorization and authentication look like here? At a high level here are some options I have seen for conflict resolution: A. Last in always wins. The server explicitly trusts things in the order it gets and pushes that data out to users. B. Clients are allowed one submit at a time and must wait for the server to acknowledge the receipt. If there is a conflict the app can either a) merge the data, b)reload the latest from the server and make the user do his operation again, or c) create a new document and inform the user. C. Operation Transforms. This was meant to solve the conflict and sync problem. However it is a LOT of work I would add another option : E. The Strongest always win : i.e An admin user change will always be considered with a higher weight when merging a conflict.

5. Offline Support Really this is more what do we want to do for coming from an "offline" mode to an "online" mode? Abstractly, operations which happen offline are the same as operations which happen online just with a REALLY REALLY laggy connection. :) We could just only viewing data when offline and requiring a connection for editing, queueing an upload, etc. Ideally, we should be able to provide the same functionality no matter we are online or offline. For example, when I'm in the middle of the ocean, I still want to be able to tag my Dolphins ;) But we could think about "degraded service levels" ... 6. How much of this is the responsibility of AG-controller vs underlying services? How should the controller expose resources to clients, how should the controller send data to its underlying services, how much data should the controller be responsible itself for? Should it be easy for an Operation Transform system to integrate with AeroGear-controller? Should it be easy to write a Controller based project which polls a third party source? How would the server handle passing credentials to the third party source? Appendix Use Cases: Here are a few contrived use cases that we may want to keep in mind. 1. Legacy Bug Trackers From Hell a. It is a webapp written in COBOL, no one will ever EVER update or change the code b. It has TONS of legacy but important data c. It has TONS of users d. It only has a few transactions per day, all creating and updating bug reports e. Multiple users can edit the same report 2. Slacker Gallery a. Each User has a multiple galleries, each gallery has multiple photos b. A Gallery has only one user, but the user may be on multiple devices c. Galleries may be renamed, created, and deleted d. Photos may only be created or deleted. Photos also have meta data which may be updated, but its creation and deletion is tied to the Photo object. 3. Dropbox clone a. A folder of files may be shared among users b. There is a size limit to files and how much storage may be used per folder c. Files are not updated. If there is a new file, there is an atomic delete and create operation 4. Email client a. This is an AG-controller which accesses a mail account. b. There are mobile offline and sync enabled clients which connect to this controller. 5. Google Docs clone a. Operational Transform out the wazzoo b. What would the server need? c. What would the client need? Appendix Reference (Open Source) Products: Wave-in-a-box CouchDB Google Drive RealtimeAPI share.js etherpad Can you guys think of more projects/examples to look at for inspiration? We could take a look at the Grails html5-scaffolding plugin which cover some of this use cases https://github.com/3musket33rs/html5-mobile-scaffolding _______________________________________________ aerogear-dev mailing list aerogear-dev(a)lists.jboss.org <mailto:aerogear-dev@lists.jboss.org> https://lists.jboss.org/mailman/listinfo/aerogear-dev _______________________________________________ aerogear-dev mailing list aerogear-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/aerogear-dev

On Wed, Apr 10, 2013 at 3:00 PM, Summers Pittman <supittma(a)redhat.com <mailto:supittma@redhat.com>> wrote: On 04/10/2013 05:23 AM, Sebastien Blanc wrote: > > > > On Wed, Mar 27, 2013 at 7:43 PM, Summers Pittman > <supittma(a)redhat.com <mailto:supittma@redhat.com>> wrote: > > Sorry for the questionable formatting before, this should be more > correct now. > > So for offline and sync for 2.0 I've been doing some > thinking/research > and come up with some broad topics to discuss so we can start > honing in > on what we want it to do/look like. > > This isn't a spec, this isn't a proposal, this is just trying > to narrow > down what we want at a high level so we can pick things to > focus on. > > 1. Documents vs Transactions > > There are two "big picture" methods of doing sync. One is a > Document > sync (Think like a gallery with videos and pictures). > Documents are > saved, the whole document is sent to the server, then the > whole document > is pushed out to other clients who are syncing against the > same source. > The other is a Transactional sync where many small atomic > operations are > sent to the server. The best analog I have is Google > Drive/operation > transforms. > > Obviously the server side implementations of these are > hilariously > divergent and I will leave the relative complexity of each as an > exercise for the list. > > IMO, because Aerogear is a library we should focus on a > Transactional sync to let the users a more fine-grained control. > But later, we should be able to provide some wrappers around this > atomic operations to offer something close to document sync (or > let the users easily define a "Document", like a view in SQL) > > 2. Background vs Foreground sync > > Does the application have to be opened (foreground) for > syncing to > happen? As far as I know, native Web requires this (barring > extensions > to the browser, plugins etc). I think Cordova can in Android, > but I > haven't researched it. iOS seems like a mixed bag, but > generally you can > only sync if your application is in the foreground (but you > can use > notifications and badges to communicate that there is a > pending sync or > new data). I understand there is CoreData + iCloud that is > supposed to > do something, but that seems like it is still foreground only. On > Android background sync is easy. > > Is it OK to only have background syncing on some platforms > but not others? > > > Shouldn't we trigger a sync only when we go from an offline > status to an online status ? No. The only difference (conceptually) between being online and offline is latency. If I am polling (yuck) ever 5 minutes and I am offline for two of those minutes, I should still only poll when my five minutes is up and am online. We don't want an extra sync in the middle. > And sync strategy should be configurable (both on client side and > server side) > > > 3. Push vs Poll? > > Obviously pushing updates is better for devices and users, > but polling > will let things work better for legacy services which may not > have push > support or which may be difficult to integrate into > AG-Controller. > > Should we support both on the clients? > > > Yes, we should promote push but have a fallback to a pull strategy And push could lean on a NNP Notifier perhaps? > > 4. Multiple clients,multiple users, and conflicts > > How do we want to support multiple users and multiple > clients? How > should we try to do conflict resolution? What does > authorization and > authentication look like here? > > At a high level here are some options I have seen for > conflict resolution: > > A. Last in always wins. The server explicitly trusts things > in the order > it gets and pushes that data out to users. > > B. Clients are allowed one submit at a time and must wait for > the server > to acknowledge the receipt. If there is a conflict the app > can either a) > merge the data, b)reload the latest from the server and make > the user do > his operation again, or c) create a new document and inform > the user. > > C. Operation Transforms. This was meant to solve the conflict > and sync > problem. However it is a LOT of work > > > I would add another option : > E. The Strongest always win : i.e An admin user change will > always be considered with a higher weight when merging a conflict. Does ag-controller have an idea of privileged levels right now? Yes, if combined with ag-sec : https://github.com/aerogear/aerogear-controller-demo/blob/master/src/main...

> > 5. Offline Support > > Really this is more what do we want to do for coming from an > "offline" > mode to an "online" mode? Abstractly, operations which happen > offline > are the same as operations which happen online just with a > REALLY REALLY > laggy connection. :) We could just only viewing data when > offline and > requiring a connection for editing, queueing an upload, etc. > > > Ideally, we should be able to provide the same functionality no > matter we are online or offline. For example, when I'm in the > middle of the ocean, I still want to be able to tag my Dolphins ;) > But we could think about "degraded service levels" ... > > > 6. How much of this is the responsibility of AG-controller vs > underlying > services? > > How should the controller expose resources to clients, how > should the > controller send data to its underlying services, how much > data should > the controller be responsible itself for? > > Should it be easy for an Operation Transform system to > integrate with > AeroGear-controller? > Should it be easy to write a Controller based project which > polls a > third party source? > How would the server handle passing credentials to the third > party source? > > Appendix Use Cases: > > Here are a few contrived use cases that we may want to keep > in mind. > > 1. Legacy Bug Trackers From Hell > a. It is a webapp written in COBOL, no one will ever EVER > update or > change the code > b. It has TONS of legacy but important data > c. It has TONS of users > d. It only has a few transactions per day, all creating and > updating bug > reports > e. Multiple users can edit the same report > > 2. Slacker Gallery > a. Each User has a multiple galleries, each gallery has > multiple photos > b. A Gallery has only one user, but the user may be on > multiple devices > c. Galleries may be renamed, created, and deleted > d. Photos may only be created or deleted. Photos also have > meta data > which may be updated, but its creation and deletion is tied > to the Photo > object. > > 3. Dropbox clone > a. A folder of files may be shared among users > b. There is a size limit to files and how much storage may be > used per > folder > c. Files are not updated. If there is a new file, there is an > atomic > delete and create operation > > 4. Email client > a. This is an AG-controller which accesses a mail account. > b. There are mobile offline and sync enabled clients which > connect to > this controller. > > 5. Google Docs clone > a. Operational Transform out the wazzoo > b. What would the server need? > c. What would the client need? > > Appendix Reference (Open Source) Products: > > Wave-in-a-box > CouchDB > Google Drive RealtimeAPI > share.js > etherpad > > Can you guys think of more projects/examples to look at for > inspiration? > > > We could take a look at the Grails html5-scaffolding plugin which > cover some of this use cases > https://github.com/3musket33rs/html5-mobile-scaffolding > > > _______________________________________________ > aerogear-dev mailing list > aerogear-dev(a)lists.jboss.org > <mailto:aerogear-dev@lists.jboss.org> > https://lists.jboss.org/mailman/listinfo/aerogear-dev > > > > > _______________________________________________ > aerogear-dev mailing list > aerogear-dev(a)lists.jboss.org <mailto:aerogear-dev@lists.jboss.org> > https://lists.jboss.org/mailman/listinfo/aerogear-dev _______________________________________________ aerogear-dev mailing list aerogear-dev(a)lists.jboss.org <mailto:aerogear-dev@lists.jboss.org> https://lists.jboss.org/mailman/listinfo/aerogear-dev _______________________________________________ aerogear-dev mailing list aerogear-dev(a)lists.jboss.org https://lists.jboss.org/mailman/listinfo/aerogear-dev