Skip to content

Syncing Strategy

July 25, 2012

I have been thinking about how to implement a 1-way sync with Dropbox for Cloudpipes.
So far it seems that it cannot be done without retaining state at the client side.
If Dropbox would be returning some kind of hash that could be computed and compared the the client side, there would not be any need for keeping metadata state at the client, but it seems we have no other option.
The two parameters that can be used for syncing are the rev and the last modified date.

Now consider the case for a downstream 1-way sync:

  1. On first run, we will have to download everything, and also store the ‘rev’ for each file.
  2. Create and save a sync_table with {rev, filename}.
  3. On the next run, fetch the metadata first. Create a new table: remote_table with {new_rev, filename}
  4. Compare each files rev vs new_rev and if it has changed, download the new file. If not, just return a successful transfer.
  5. For all successful transfers, generate a new sync_table.

Now let us consider the case for upstream 1-way sync which is more complicated:

  1. On first run, upload everything.
  2. Create and save a sync_table with {local_modified_date, filename} or {filehash, filename}
  3. On the next run, generate a new table: local_table with {new_local_mod_date, filename} or {new_filehash, filename}
  4. Compare each local_modified_date with new_local_mod_date (or hash) and if they are different perform an upload. If not, just return a successful transfer.
  5. For all successful transfers, generate a new sync_table.

In theory this should work, but we need to work out error conditions in cases of no connectivity etc.

Advertisements
No comments yet

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: