I recently released my first app on the Mac App Store, Toki, and I decided that talking about the inner workings of the sync mechanism I’m using would not only be interesting, but helpful for me to think about some of the problems I’m having[footnote]This is inspired by the “Vesper Sync Diary” series of blog posts by Brent Simmons[/footnote].
Toki uses an App.net sync mechanism that I’ve been thinking about and working on for a while — I thought that Toki would be a good low-profile app to test out some ideas in real-world use.
Although Toki is a production app, the sync portion is not feature-complete yet. I’m working on ironing out the basic bugs (planned to be released in 1.0.1 and 1.0.2 releases), before moving on to the more advanced features. My “end goal” for the App.net sync mechanism is an open-source multi-platform library that anyone can use in their own app to use the App.net backend to keep their objects synchronized[footnote]There already has been some work been done with a Ruby-based command-line client.[/footnote]. From now on (and until I think of a better name), I will call this sync mechanism “App.net Object Sync” or “ADNOS”.
ADNOS uses the private message / channel feature of the App.net API, using a channel for the app namespace, and a message representing each object.
There are two fundamental steps of the sync protocol (as of version 1.0.2, still in development):
- Pulling remote changes: ADNOS pulls all new messages from the channel, processing them one by one, creating or updating local records as necessary. If the record is deemed “duplicate”, it will overwrite local changes[footnote]Conflict resolution is not applicable in the context of Toki, but is high on the list of features I want to implement.[/footnote]. If the record was newly created in the local database, ADNOS will also record the App.net Message ID in the local database.
- Pushing local changes: ADNOS makes a new message post for each record that does not have a App.net Message ID recorded in the local database. When the message is successfully created, the message ID is set.
This has some limits, some inherent in the protocol, and some inherent in the App.net backend:
- Message creation is rate-limited to 20 messages per minute[footnote]This is also why Toki works with such low-resolution data.[/footnote].
- Each message is limited to 8192 bytes. Less, actually, because of serialization and metadata.
- Recreating the database requires a full playback of messages — this takes time, especially on databases with more than 200 objects.
These limits are pretty much unavoidable. I work around the rate-limiting problem by queueing sync up requests, and pausing the queue when X-RateLimit-Remaining gets dangerously low. The queue is re-started after X-RateLimit-Reset elapses. Any app that plans on using ADNOS will want to be aware of these limitations before implementing it.
And, now for the features I’m planning on implementing (in no particular order):
- Proper conflict resolution.
- Streaming support for pulling new changes.
- Updating and deleting previously synced items. This isn’t implemented yet, but should be relatively easy.
I’ll be writing more as I progress. I hope to get some sort of open-source library out there in the near future.