Categories English Open Source Projects Object Sync

I recently released my first app on the Mac App Store, Toki, and I decided that talking about the inner workings of the sync mechanism I’m using would not only be interesting, but helpful for me to think about some of the problems I’m having[footnote]This is inspired by the “Vesper Sync Diary” series of blog posts by Brent Simmons[/footnote].

Toki uses an sync mechanism that I’ve been thinking about and working on for a while — I thought that Toki would be a good low-profile app to test out some ideas in real-world use.

Although Toki is a production app, the sync portion is not feature-complete yet. I’m working on ironing out the basic bugs (planned to be released in 1.0.1 and 1.0.2 releases), before moving on to the more advanced features. My “end goal” for the sync mechanism is an open-source multi-platform library that anyone can use in their own app to use the backend to keep their objects synchronized[footnote]There already has been some work been done with a Ruby-based command-line client.[/footnote]. From now on (and until I think of a better name), I will call this sync mechanism “ Object Sync” or “ADNOS”.

ADNOS uses the private message / channel feature of the API, using a channel for the app namespace, and a message representing each object.

There are two fundamental steps of the sync protocol (as of version 1.0.2, still in development):

  1. Pulling remote changes: ADNOS pulls all new messages from the channel, processing them one by one, creating or updating local records as necessary. If the record is deemed “duplicate”, it will overwrite local changes[footnote]Conflict resolution is not applicable in the context of Toki, but is high on the list of features I want to implement.[/footnote]. If the record was newly created in the local database, ADNOS will also record the Message ID in the local database.
  2. Pushing local changes: ADNOS makes a new message post for each record that does not have a Message ID recorded in the local database. When the message is successfully created, the message ID is set.

This has some limits, some inherent in the protocol, and some inherent in the backend:

  1. Message creation is rate-limited to 20 messages per minute[footnote]This is also why Toki works with such low-resolution data.[/footnote].
  2. Each message is limited to 8192 bytes. Less, actually, because of serialization and metadata.
  3. Recreating the database requires a full playback of messages — this takes time, especially on databases with more than 200 objects.

These limits are pretty much unavoidable. I work around the rate-limiting problem by queueing sync up requests, and pausing the queue when X-RateLimit-Remaining gets dangerously low. The queue is re-started after X-RateLimit-Reset elapses. Any app that plans on using ADNOS will want to be aware of these limitations before implementing it.

And, now for the features I’m planning on implementing (in no particular order):

  • Proper conflict resolution.
  • Streaming support for pulling new changes.
  • Updating and deleting previously synced items. This isn’t implemented yet, but should be relatively easy.

I’ll be writing more as I progress. I hope to get some sort of open-source library out there in the near future.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.