MobleySync protocol

 by libimobiledevice.org

iDevice Sync Protocol Overview

Terms/Definitions:

Data Class: One of the supported data classes.
RecordEntity: RecordEntities define a record's data fields. A data class supports a specific set of RecordEntities.

Data Class > RecordEntities
com.apple.Bookmarks > com.apple.bookmarks.Bookmark, com.apple.bookmarks.Folder

Sync Types:
SDSyncTypeReset (all records are submitted to again to provide a new consistent sync state)
SDSyncTypeSlow (all records are submitted)
SDSyncTypeFast (only changes from the device are submitted)

Sync Data Classes known so far:

com.apple.Contacts
com.apple.Calendars
com.apple.Bookmarks
com.apple.Notes
com.apple.MailAccounts (device sends accounts which were setup on the device, it does not accept any new accounts/records!)

Lockdown sync preferences:

Following commands print plists which can be retrieved using lockdownd_get_value() in code:
# ideviceinfo -x -q com.apple.mobile.iTunes -k SyncDataClasses
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" " http://www.apple.com/DTDs/PropertyList-1.0.dtd ">
<plist version="1.0">
<array>
        <string>Contacts</string>
        <string>Calendars</string>
        <string>Bookmarks</string>
        <string>Mail Accounts</string>
        <string>Notes</string>
</array>
</plist>

# ideviceinfo -x -q com.apple.mobile.tethered_sync

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" " http://www.apple.com/DTDs/PropertyList-1.0.dtd ">
<plist version="1.0">
<dict>
        <key>Bookmarks</key>
        <dict>
                <key>DisableTethered</key>
                <false/>
                <key>SyncingOS</key>
                <string>iPhone</string>
        </dict>
        <key>Calendars</key>
        <dict>
                <key>DisableTethered</key>
                <false/>
                <key>SyncingOS</key>
                <string>OSX</string>
        </dict>
        <key>Contacts</key>
        <dict>
                <key>DisableTethered</key>
                <false/>
                <key>SyncingOS</key>
                <string>OSX</string>
        </dict>
        <key>Mail Accounts</key>
        <dict>
                <key>SyncingOS</key>
                <string>iPhone</string>
        </dict>
        <key>Notes</key>
        <dict>
                <key>DisableTethered</key>
                <false/>
                <key>SyncingOS</key>
                <string>OSX</string>
        </dict>
</dict>
</plist>

Notes:
The SyncingOS values seen are: iPhone|Windows|OSX
If DisableTethered is set to <integer>1</integer>, the device does always respond with SDMessageRefuseToSyncDataClassWithComputer messages.

How changes are synced:

Following applies for the records within SDMessageProcessChanges messages.

Deleting:     Send a record with it's identifier with an "___EmptyParameterString___" for the value (instead of the data fields).
Modifiying:  Send the full record with all fields (including not modified ones)
Adding:       Send the record with an identifier unknown to the device (this will cause it to send remapping for the record aswell)

Sync Process

The sync process is similar to what is described here:
http://www.opensync.org/wiki/devel/synchronizationOverview

Initial handshake/session start:
1. host -> device: SDMessageSyncDataClassWithDevice
2. device -> host: SDMessageSyncDataClassWithComputer (sync can start) or SDMessageRefuseToSyncDataClassWithComputer (refusing to sync)

Request records:
3. host -> device: SDMessageGetAllRecordsFromDevice (first or full sync) or SDMessageGetChangesFromDevice (following sync) or 

Retrieve records from device:
4. device -> host: SDMessageProcessChanges
5. host -> device: SDMessageAcknowledgeChangesFromDevice
6. repeat 4 and 5 for each entity type for the schema until
7. device-> host: SDMessageDeviceReadyToReceiveChanges (end of device -> host changes transfer)

Change sync direction / should be handled by inherited DeviceLinkService interface:
8. host -> device: DLMessagePing

Send records from host:
9. host -> device: SDMessageProcessChanges
10. device -> host: SDMessageRemapRecordIdentifiers
11. repeat 10 and 11 for each entity type for the schema until no more changes to send
12. host -> device: SDMessageFinishSessionOnDevice (end of host -> device changes transfer)

Finished session (allows to start from 1. again for syncing a different data class):
13. device -> host: SDMessageDeviceFinishedSession

Remove all records for a schema:
1. host -> device: SDMessageClearAllRecordsOnDevice (clears all records)
2. device -> host: SDMessageDeviceWillClearAllRecords

Cancel the sync session any time:
1. device -> host: SDMessageCancelSession (cancelled the session, reason is within the sent message plist)

Protocol Messages with signatures:

Both directions:

SDMessageProcessChanges
    data class (string)
    entities (dictionary)
        id (key)
        value (dictionary)
            com.apple.syncservices.RecordEntityName (key)
            entity name (string)
            data key (key)
            data value (key)
            ...
        ...
    going_to_send_further_changes (boolean)
    sync actions (dictionary) or "___EmptyParameterString___" (string)
        "SyncDeviceLinkEntityNamesKey" (key)
        [] (array)
            entity name (string)
            ...
        "SyncDeviceLinkAllRecordsOfPulledEntityTypeSentKey" (key)
        merge the entites listed in SyncDeviceLinkEntityNamesKey and report any remapping in next message (boolean)

SDMessageCancelSession
    data class (string)
    reason (string)

device -> host:

SDMessageSyncDataClassWithComputer
    data class (string)
    new device anchor (date as a string: "%Y-%m-%d %H:%M:%S %z" or "---" for initial sync)
    last known host anchor (string)
    sync type (string)
    data class version on the device (integer)
    "___EmptyParameterString___" (string)

SDMessageRefuseToSyncDataClassWithComputer
    data class (string)
    reason (string)

SDMessageDeviceWillClearAllRecords
    data class (string)

SDMessageDeviceReadyToReceiveChanges
    data class (string)

SDMessageRemapRecordIdentifiers
    data class (string)
    mapping (dictionary) or "___EmptyParameterString___" (string)
        from id (key)
        to id (string)
        ...

SDMessageDeviceFinishedSession
    data class (string)

host -> device:

DLMessagePing (this infact somehow belongs into the DeviceLinkServiceInterface)
    "Preparing to get changes for device" (string)

SDMessageSyncDataClassWithDevice
    data class (string)
    last known device anchor (date as a string: "%Y-%m-%d %H:%M:%S %z" or "---" for initial sync)
    new host anchor (date as string: "%Y-%m-%d %H-%M-%S %z")
    data class version on the computer (integer: 106)
    "___EmptyParameterString___" (string)

SDMessageClearAllRecordsOnDevice
    data class (string)
    "___EmptyParameterString___" (string)

SDMessageGetChangesFromDevice
    data class (string)

SDMessageGetAllRecordsFromDevice
    data class (string)

SDMessageAcknowledgeChangesFromDevice
    data class (string)

SDMessageFinishSessionOnDevice
    data class (string)

C API Draft - Asynchronous

#define MOBILESYNC_E_SUCCESS                0
#define MOBILESYNC_E_INVALID_ARG           -1
#define MOBILESYNC_E_PLIST_ERROR           -2
#define MOBILESYNC_E_MUX_ERROR             -3
#define MOBILESYNC_E_BAD_VERSION           -4
#define MOBILESYNC_E_SYNC_REFUSED          -5
#define MOBILESYNC_E_CANCELLED             -6

#define MOBILESYNC_E_UNKNOWN_ERROR       -256

typedef enum {
    MOBILESYNC_SYNC_TYPE_FAST,
    MOBILESYNC_SYNC_TYPE_SLOW,
    MOBILESYNC_SYNC_TYPE_RESET
} mobilesync_sync_type_t;

/* interface */
typedef uint8_t (*mobilesync_process_records_cb_t) (const char *entity_type, plist_t records, uint8_t more_changes, void *user_data);

mobilesync_error_t mobilesync_start_session(mobilesync_client_t client, const char* schema, struct timeval* last_sync_time, struct timeval* current_time, uint64_t version, mobilesync_sync_type_t* sync_type);
mobilesync_error_t mobilesync_finish_session(mobilesync_client_t client, const char* schema);

mobilesync_error_t mobilesync_get_all_records(mobilesync_client_t client, const char* schema, mobilesync_process_records_cb_t process_records, void *user_data);
mobilesync_error_t mobilesync_get_changes(mobilesync_client_t client, const char* schema, mobilesync_process_records_cb_t process_records, void *user_data);
mobilesync_error_t mobilesync_clear_all_records(mobilesync_client_t client, const char* schema);
mobilesync_error_t mobilesync_send_changes(mobilesync_client_t client, const char* schema, const char* entity_type, plist_t entity_mapping, uint8_t more_changes, plist_t *remapping);

mobilesync_error_t mobilesync_cancel(mobilesync_client_t client, const char* schema, const char* reason);

C API Draft - Synchronous

#define MOBILESYNC_E_SUCCESS                0
#define MOBILESYNC_E_INVALID_ARG           -1
#define MOBILESYNC_E_PLIST_ERROR           -2
#define MOBILESYNC_E_MUX_ERROR             -3
#define MOBILESYNC_E_BAD_VERSION           -4
#define MOBILESYNC_E_SYNC_REFUSED          -5
#define MOBILESYNC_E_CANCELLED             -6 /* how do we get the reason string here? :/ */
#define MOBILESYNC_E_NO_SESSION                 -7
#define MOBILESYNC_E_ACTIVE_SESSION           -8

#define MOBILESYNC_E_UNKNOWN_ERROR       -256

typedef enum {
    MOBILESYNC_SYNC_TYPE_FAST,
    MOBILESYNC_SYNC_TYPE_SLOW,
    MOBILESYNC_SYNC_TYPE_RESET
} mobilesync_sync_type_t;

struct mobilesync_anchors {
    char *device_anchor;
    char *host_anchor;
};
typedef struct mobilesync_anchors *mobilesync_anchors_t;

/* interface (synchronous) */
/*
We might store the data_class and if a session is running within the private client struct on start_session().
That way one is bound to supply a new data_class with a new start_session call to sync other data_classes and we can scrap redundant arguments.
*/

/* gets the sync_type and exchanges anchors from/with the device */
mobilesync_error_t mobilesync_session_start(mobilesync_client_t client, const char *data_class, uint32_t *data_class_version, mobilesync_anchors_t anchors, mobilesync_sync_type_t *sync_type);
mobilesync_error_t mobilesync_session_cancel(mobilesync_client_t client, const char *reason);
mobilesync_error_t mobilesync_session_finish(mobilesync_client_t client);

mobilesync_error_t mobilesync_get_all_records_from_device(mobilesync_client_t client);
mobilesync_error_t mobilesync_get_changes_from_device(mobilesync_client_t client);
mobilesync_error_t mobilesync_clear_all_records_on_device(mobilesync_client_t client);

/* must be useful for processing within a loop */
mobilesync_error_t mobilesync_receive_changes(mobilesync_client_t client, plist_t *entities, uint8_t *is_last_record, plist_t *actions);
mobilesync_error_t mobilesync_send_changes(mobilesync_client_t client, plist_t entities, uint8_t is_last_record, plist_t actions);

/* somehow these three reflect some kind of status response after ProcessChanges messages */
mobilesync_error_t mobilesync_device_cancelled_session(mobilesync_client_t client);
mobilesync_error_t mobilesync_device_ready_to_receive_changes(mobilesync_client_t client);
mobilesync_error_t mobilesync_acknowledge_changes_from_device(mobilesync_client_t client);
mobilesync_error_t mobilesync_remap_record_identifiers(mobilesync_client_t client, plist_t *mapping);

/* helpers */
mobilesync_error_t mobilesync_session_get_data_class(mobilesync_client_t client, char **data_class);
/* TODO: Ease parsing and creation of SDMessageProcessChanges plists */

SDMessageProcessChanges Examples:

Bookmarks received from the device:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" " http://www.apple.com/DTDs/PropertyList-1.0.dtd ">
<plist version="1.0">
<array>
        <string>SDMessageProcessChanges</string>
        <string>com.apple.Bookmarks</string>
        <dict>
                <key>E7D45694-F91C-4A48-8012-99E9944ECA27</key>
                <dict>
                        <key>parent</key>
                        <array>
                                <key>76CD94F9-B1A5-934D-8E49-E07A5F354684</key>
                        </array>
                        <key>url</key>
                        <string> http://go.microsoft.com/fwlink/?linkid=69151 </string>
                        <key>position</key>
                        <integer>2</integer>
                        <key>name</key>
                        <string>Marketplace</string>
                        <key>com.apple.syncservices.RecordEntityName</key>
                        <string>com.apple.bookmarks.Bookmark</string>
                </dict>
        </dict>
        </false>
        <string>___EmptyParameterString___</string>
</array>
</plist>

Contact received from the device:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" " http://www.apple.com/DTDs/PropertyList-1.0.dtd ">
<plist version="1.0">
<array>
        <string>SDMessageProcessChanges</string>
        <string>com.apple.Contacts</string>
        <dict>
                <key>4/15/0</key>
                <dict>
                        <key>contact</key>
                        <array>
                                <string>15</string>
                        </array>
                        <key>type</key>
                        <string>home</string>
                        <key>com.apple.syncservices.RecordEntityName</key>
                        <string>com.apple.contacts.Email Address</string>
                        <key>value</key>
                        <string>[email protected]</string>
                </dict>
        </dict>
        <true/>
        <string>___EmptyParameterString___</string>
</array>
</plist>

No data for a record entity yields such a message (note that it's also the last process changes message due to the boolean being false):

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" " http://www.apple.com/DTDs/PropertyList-1.0.dtd ">
<plist version="1.0">
<array>
        <string>SDMessageProcessChanges</string>
        <string>com.apple.Contacts</string>
        <dict>
        </dict>
        <false/>
        <string>___EmptyParameterString___</string>
</array>
</plist>