Backup your Android without root or custom recovery -- adb backup

ecently discovered a neat new way to back up apps on my Android without having to use Titanium Backup, having to unlock bootloader or root the device to take a NANDroid snapshot. The icing on the cake - I can do it from command line!

Warning: this method is only available to devices running Ice Cream Sandwich or higher and it has bugs, but it is an alternative for folks who want to make a backup beforerooting their device.

You'll need to have the latest adb installed and in you path, USB debugging enabled and your phone connected to your Mac.

The command in question has the following format:

adb backup [-f <file>] [-apk|-noapk] [-shared|-noshared] [-all] [-system|-nosystem] [<packages...>]

Here are some common uses:

1. Backup all non-system apps, their data and your shared data (i.e. SD card contents):

adb backup -f /path/to/backup-file -apk -shared -all -nosystem

2. Backup all apps, their data and shared data (note: won't work on DRM protected apps):

adb backup -f /path/to/backup-file -apk -shared -all -system

3. Backup only app data (not the APKs themselves) and your shared data:

adb backup -f /path/to/backup-file -all

4. Backup only non-system apps:

adb backup -f /path/to/backup-file -apk -noshared -nosystem

5. Backup only your shared data:

adb backup -f /path/to/backup-file -noapk -shared -nosystem

6. Restore everything from your backup:

adb restore /path/to/backup-file

During backup your phone will display a backup confirmation screen giving you an option to enter an encryption password. In my case it would only work with non-empty password, otherwise it would generate a zero-sized file.

 

Like a lot of you, I have been putting off unlocking the bootloader on my Nexus because I didn't want to have to go through the hassle of backing up everything manually and restoring individual application data; logging back into apps; saving settings; etc. I found an undocumented (at least as far as my googling was able to find) feature in the latest version of the ADB platform tools (for Android 4.0+) that allows you to create a full system backup, including app apks, their respective data, as well as the internal storage. 

Keep in mind this is experimental and not exactly publicized as a feature of ICS, so don't count on this as your only method of backup!

This guide assumes you have already installed the Android SDK, and updated the Android SDK Platform Tools to the latest version (currently Rev 10) using the SDK Manager.

1. Connect your device via USB, and open a command prompt.

2. Optionally, type the command 'adb devices' to ensure that your device is properly recognized. If you're comfortable with ADB already, just skip this.

There is a command, 'adb backup' (to be detailed shortly), that will now allow you to create a full system backup.

The command parameters format is:

Quote:

adb backup [-f <file>] [-apk|-noapk] [-shared|-noshared] [-all] [-system|nosystem] [<packages...>]

 

The most basic command you can use* is simply:

Quote:

adb backup -all

 

This will use the defaults to backup only app and device data (not the APKs themselves) to the current directory as 'backup.ab'

* This may not work for every setup. If you get an error such as "adb: cannot open file ./backup.ab", use:

Quote:

adb backup -all -f C:\backup.ab

 

Or substitute the path of your choice in place of C:\.

To explain the parameters:

Quote:

-f <file>

 

Use this to choose where the backup file will be stored, e.g. '-f /backup/mybackup.ab', which will save it at the root of your drive (C:\ for Windows, etc.) in a folder called backup, as a file named 'mybackup.ab'. I recommend using this flag to set a location manually, as with my first backup test, it said that it completed successfully, but I was unable to locate the backup file. I have no idea where it was saved, but it wasn't where it should have been located.

Quote:

-apk|-noapk

 

This flags whether or not the APKs should be included in the backup or just the apps' respective data. I personally use -apk just in case the app isn't available in the Market, so that I don't have to go hunt it down again. The default is -noapk.

Quote:

-shared|-noshared

 

This flag is used to "enable/disable backup of the device's shared storage / SD card contents; the default is noshared.", which for the Nexus I would certainly flag to -shared, but from my test, it did not restore all of the contents of my internal storage, so I recommend backing up music, pictures, video, and other internal storage items manually, just to be on the safe side. The default is -noshared.

Quote:

-all

 

This flag is just an easy way to say to backup ALL apps. The packages flag (further on) can be used to choose individual packages, but unless you're just wanting to backup a specific application, use -all for a full system backup.

Quote:

-system|-nosystem

 

This flag sets whether or not the -all flag also includes system applications or not. I used -system, but this is probably unnecessary, and I would almost guess that it is safer to use -nosystem, but use your own judgment on this. The default is -system.

Quote:

<packages...>

 

Here you can list the package names (e.g. com.google.android.apps.plus) specifically that you would like to backup. Use this only if you're looking to backup a specific application.

3. Once you've made your decision on how to perform the backup, simply type the command as you would like it; in my case, this is the command that I used:

Quote:

adb backup -apk -shared -all -f C:\backup20111230.ab

 

4. You will see a screen like the following:



5. Enter a password (if desired) for encryption of the backup file. RETAIN THIS PASSWORD FOR RESTORING LATER.

6. This process will take several minutes to complete, depending on the settings you've chosen, but when completed, you will get a toast on-screen saying 'Backup Complete', or if you miss that, you'll know once your phone returns to the home screen.

7. Now go unlock your bootloader (not going to go into the process for this guide, but you probably know how already, and if not, there are several guides a search away).

8. Once you're booted back into Android, you can choose to add your account now, or skip that for later. I skipped it for later, but I think it might make the process more smooth to sign in before the restore. YMMV. 

9. To restore, with your device connected open your command prompt again, and type:

Quote:

adb restore C:\backup20111230.ab

 

replacing 'C:\backup20111230.ab' with the location of your backup file. 

10. You will see a screen like the one below:



11. Simply type in your current encryption password (if you've set one), and the password with which the backup was encrypted (if you chose to set a password), and the restore will begin. It again will take several minutes depending on the size of the backup and the options chosen.

12. You're back to normal, short of possibly some widgets on the home screen. My wallpaper was even restored, my app folders remained just as I had them before, my alarms remained, and for most applications, I didn't even have to log back in; it kept everything. 

________________________

NOTE: I did have an issue with not all files being restored to the Internal Storage; in particular, the Gallery still displayed all the folders and files that it had cached (which it expected to be there) as only gray boxes, and would not display the images, nor would it rescan the media. I simply copied the files back to the Internal Storage directory manually, and all was well again. Again, YMMV. 

NOTE 12/31: Also to note, this will not back up SMS messages, so if you're concerned about those, you may want to look into an alternative application to back up SMS. 

NOTE 06/12: There seems to be a bug in which backup and restore operations will fail unless a desktop backup password is set under Developer Options. It will not work with a blank password.

 

Android Backup Splitter

 

This shell script utility reads an Android adb backup and generates an individual adb backup for each item on apps/ folder. That equals to generating an adb backup for each app.

shared/0 and shared/1 (sdcards) are ignored, so that means that external data and OBB files, although may be backed up, may not be restored. For doing so backup the folder Android of the sdcards.
Creating a backup with -shared flag has know issues (corruption) so is not recommended. Future releases of this script may add the option to automatically detect things from sdcards inside the adb backup file.

This script works on Cygwin also, and in general where star does.

apk-missing.html is generated when apks are missing.

Other software is needed. The bundle is included in Android Backup Extractor, with the full README as well:

Android adb backup extractor and creator

The package contains both Java and Perl tool. I myself prefer Perl over Java any day, so I extracted the Perl codes, make sure they are executable, installed the required Perl library, and run the backupdecrypt.pl against an adb backup file, and it convert it into a tar or gzipped tar file without any issue.

I even formed a one liner in Bash 3 that allow me to do adb backup directly to gzipped tar file:

adb backup -f >(backupdecrypt.pl -D -z - backup.tgz) -all

LAW WARNING
If you are working with password encrypted adb backups, you need to download these two files:
- local_policy.jar
- US_export_policy.jar
from http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html
and put them in the lib/security folder of all your Java SE 7 installations, for example:
--Windows:
C:\Program Files\Java\jdk1.7.0_60\jre\lib\security\
C:\Program Files\Java\jre7\lib\security\
C:\Program Files (x86)\Java\jdk1.7.0_60\jre\lib\security\
C:\Program Files (x86)\Java\jre7\lib\security\
--Linux:
/usr/local/jdk1.7/jre/lib/security/
--BSD:
/usr/lib/jvm/java-7-openjdk-*/jre/lib/security/
--OS X:
/Library/Java/JavaVirtualMachines/jdk1.7.0_60.jdk/Contents/Home/jre/lib/security/
Won't work if you supply them with the application, you must have them installed onto your system.

Author: Nikolay Elenkov
Packager: dragomerlin
Format: runnable jar file
IDE: Kepler Service Release 2 Build id: 20140224-0627
Java SE 1.7.0_60
Bouncy Castle version: bcprov-jdk15on-150

Application name: Android Backup Extractor
Original source code: https://github.com/nelenkov/android-backup-extractor
Website documentation: http://nelenkov.blogspot.com/2012/06/unpacking-android-backups.html
Full local backup infrastructure: https://android.googlesource.com/platform/frameworks/base/+/4a627c71ff53a4fca1f961f4b1dcc0461df18a06

Usage:
info: java -jar abe.jar info <backup.ab> [password]
unpack: java -jar abe.jar unpack <backup.ab> <backup.tar> [password]
pack: java -jar abe.jar pack <backup.tar> <backup.ab> [password]
pack-kk: java -jar abe.jar pack-kk <backup.tar> <backup.ab> [password]

*pack-kk is to create version 2 backups for android kitkat 4.4.3+.
If you don't specify a password the backup archive won't be encrypted but only compressed.
Option to create unencrypted and uncompressed backups it's not available.

Requirements for compilation:
- Bouncy Castle java release "JCE with provider and lightweight API"
http://www.bouncycastle.org/latest_releases.html
- Oracle Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files 7
http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html
- Java Standard Edition version 7
- IDE Eclipse (optional but recommended)

Description and scope:
- Android Backup Extractor is an application written in Java that can extract and create android adb backups. It doesn't edit directly.
- Android Backup Splitter is a bash script called adb-split.sh that creates individual adb backups for each app from the full one, so you can restore apps individually. Same encryption is preserved to them, if any. It also informs if any apk is missing.
- The adb backups (usually with .ab extension) can be password protected or not. If the device is encrypted this is mandatory and the password must be the same.
- The adb backups are extracted to tar format, and are created from the tar format as well.
- The files and folders inside the tar archive must be in some specific order, not necessarily alphabetical like tar does by default. Such order is listed in "Full local backup infrastructure". The easiest way is to repackage a tar archive from an already existing one; for example a subpart of an archive, and repackage it following the same order as files are listed in the original. If we don't have the original archive, we have to guess out the order.

Considerations:
- The tar archives may be extracted and created on filesystems that respect tar's content's permissions, although seems to work in any case.
- Directories inside tar archives must not contain trailing slashes. Since GNU tar doesn't have this option, the use of star or pax is necessary. For Ubuntu star is included.
- star is recommended instead of pax, which supports path length only up to 100 characters, so will fail in some cases. This usually happens with browsers history.
- Java 7 or higher is required because of SYNC_FLUSH mode for the Deflater.
- If the backup contains apk+data for an app, restore will work. If contains only the data, you must install the app first on the device and then restore. Installing the app later won't work.
- If you want a full backup use a nandroid backup with Clockworkmod or TWRP instead of ADB. Nandroid backup has the disadvantage that is not always 100% compatible between devices. ADB backup intends to be.
- Some apps include a policy where the apk is never backed up even if specified.
- Some applications or its data can't be restored on a different device, because may be set specific for the device's id or because are not compatible.
- Android debugging bridge is very slow: about 1 MBps if the backup is password encrypted, 1.6 MBps if not, so consider that backing and restoring may take a long time.
- When adding or removing files for an app you may need to edit _manifest so adb restore accepts the changes.
- ADB backups are not a replacement for a Nandroid backup. The whole /data partition is not backed up, only a part of it. Also, other partitions like /system, /preload, /cache, modem, RIL, efs, kernel or recovery are never backed up because are not user data. This is to prevent issues when restoring on a different device. It has also the biggest advantage that doesn't require root to operate, so is totally compatible with stock roms, locked bootloaders and device encryption.
- If you have the device encrypted with a password, you must use that particular one for backup creation and restore. You can't create a backup without password or a different one in that particular case, or if you try to restore will fail.
- star for cygwin for windows is attached (move it to C:\cygwin\bin\star.exe) since there is no package available. It can be compiled from the schily (ftp://ftp.berlios.de/pub/schily/).
- The best way to test if an adb backup has errors, is to convert it to tar and then check.

Example of quick backup editing:
Here is an example on how to make a custom adb backup from a complete one, for the game Grand Theft Auto III from Rockstar Games, installed on the Nexus 7.

1) Unpack the original adb backup:
$ java -jar abe.jar unpack nexus7.ab nexus7.tar <password>

2) Extract the contents of the tar archive. This may be done on a filesystem where the permissions of the files inside the tar are preserved, for example using linux, mac or bsd. Up to two folders may appear, apps and shared:
$ tar -xvf nexus7.tar

3) Make a list of all the contents of the original archive in the order they are archived:
$ tar -tf nexus7.tar > nexus7.list

4) Create a new list only with the files and folders you want, in proper order. For example for the GTA 3 (you can try savegames instead of all data):
$ cat nexus7.list | grep com.rockstar.gta3 > gta3.list

5) Create the new tar archive. The directories stored on tar mustn't contain trailing slashes, so I use star instead of tar. Pax works also if paths are up to 100 of length:
$ star -c -v -f gta3.tar -no-dirslash list=gta3.list
OR
$ cat gta3.list | pax -wd > gta3.tar

6) Create the adb backup from the tar archive. Password is optional except if the device is encrypted:
$ java -jar abe.jar pack gta3.tar gta3.ab <password>

Note: if the backup is not encrypted zlib can be used instead for both unpack and pack the ab archive:
- Quick unpacking:
$ dd if=nexus7.ab bs=24 skip=1 | openssl zlib -d > nexus7.tar
- Quick packing:
$ dd if=nexus7.ab bs=24 count=1 of=gta3.ab ; openssl zlib -in gta3.tar >> gta3.ab

Example of adb splitting to create an individual adb backup for each app:
Use ADB Backup Splitter (is a shell script). Works on Cygwin also:
sh adb-split.sh backup.ab [password if needed]
Resulting files go in app-ab folder.
Missing apk are listed in apk-missing.html

# ADB Backup Splitter:
#
# This bash script utility generates an adb backup for each item
# on apps/ folder. That equals to generating an adb backup for
# each app.
#
# shared/0 and shared/1 (sdcards) are ignored, so that means that
# external data and OBB files, although may be backed up, may not
# be restored. For doing so backup the folder Android of the sdcards.
# Creating a backup with -shared flag has know issues (corruption)
# so is not recommended. Future releases of this script may add
# the option to automatically detect things from sdcards inside
# the adb backup file.
#
# This script works on Cygwin also, and in general where star does.
#
# There may be apps in the backup that don't include the apk. In that
# particular case, the data won't be restored, so the individual backup
# is generated for testing purposes only. Installing the apk afterwards won't work.
# There is generated an html file called apk-missing.html so you can open it and
# install all applications from Play Store on any of your devices.

Little changelog:
(Initial)
(6 June 2014)
- Changed "enabled" to "allowed" and "not enabled" to "not allowed" relating to strong AES encryption because some user complained
- JDK updated to 1.7.0_60, mixed mode
- Added support for android 4.4 encrypted backups
- Removed bcprov-jdk15on documentation
- Updated Bouncy Castle source (JCE with provider and lightweight API) from 1.48 to 1.50
- Updated adb-split.sh to add "${SCRIPT_DIR}/" before abe.jar
- Eclipse updated from 20130225-0426 to 20140224-0627
(15 June 2014)
- Updated program to create version 2 backups for android 4.4.3+.
(30 June 2014)
- Added backup version 3 for android L. No real code changes.

 

Android手机透过《Simple ADB Backup》不用Root也能完整备份

Android 手机如果要完整备份，懂的人第一想法就是Root + 钛备份，不过并不是每个人都想要Root，也不是每个人都懂的把手机Root（就算给你完整教学文也一样），但现在开始要完整备份已经不需要Root，XDA论坛已经有神人开发了一套好用的备份工具《 Simple ADB Backup 》，让我们可以直接备份手机上全部的资料，而且真的不用Root，使用方式也很简单，缺点是只能在同一支手机上还原，相对的如果你觉得手机觉得顿了想要回复原厂值时就派上用场了。

XDA页面： 请点我 
适用手机系统：Android 4.0+ 
商用作业系统：Windows/Linux 
下载： XDA载点 （如果载点失效，请回XDA原始页面查看是否有新版） 
注：建议看完所有内容后再试，而所有的操作，除了还原之后，操作失败并不会影响手机任何资料。

注：此工具不一定可以相容全部的手机，操作前请谨慎考量。

How to extract, create or edit android adb backups

What is an android adb backup?
An adb backup is a file with ab extension, generated by android's backup manager when we request it via adb shell. This allows you to backup some data of the phone, but is not a replacement of a clockworkmod backup:

- Java 7 or higher is required because of SYNC_FLUSH mode for the Deflater.
- If the backup contains apk+data for an app, restore will work. If contains only the data, you must install the app first on the device and then restore. Installing the app later won't work.
- Some apps include a policy where the apk is never backed up even if specified.
- Inside an ab file is a tar file, which contains files and folders in a certain order. You have you respect that order, which is not necessarily alphabetical like tar does by default. Such order is listed in "Full local backup infrastructure".
- Inside the tar file, directories must not have trailing slash, for that reason pax, star or equivalent has to be used.
- There are some bugs present in the android source code.
- 'adb backup' or 'adb restore' are pretty slow, between 1 and 2.6 MBps.
- The adb backups (usually with .ab extension) can be password protected or not. If the device is encrypted this is a must and has to be the same used for the device.
- Star is recommended instead of pax, which supports path length only up to 100 characters, so will fail in some cases. This usually happens with browsers history.
- ADB backups are not a replacement for a Nandroid backup. The whole /data partition is not backed up, only a part of it. Also, other partitions like /system, /preload, /cache, modem, RIL, efs, kernel or recovery are never backed up because are not user data. This is to prevent issue when restoring on a different device. It has also the biggest 
- If you have the device encrypted with a password, you must use that particular one for backup creation and restore. You can't create a backup without password or a different one in that particular case, or if you try to restore will fail.
- star for cygwin for windows is attached (move it to C:\cygwin\bin\star.exe) since there is no package available. It can be compiled from the schily (ftp://ftp.berlios.de/pub/schily/).
- The best way to test if an adb backup has errors, is to convert it to tar and then check.advantage that doesn't require root to operate, so is totally compatible with stock roms, locked bootloaders and device encryption.
- There's also a bash script called adb-split.sh that creates individual adb backups for each recognized app from the full one, so you can restore apps individually. Same encryption is preserved to them, if any.

You can use java or perl, although with perl can be more complicated because requires downloading some modules from cpan and some ssl headers.

To know more information about types of android backups:
[Guide] Types of Android backups

Software needed
The entire process can be done from Windows, but is better to use a unix-like operating system, like Linux, OS X or BSD, since we should extract the files on a filesystem that preservers file permissions and owners and repack the tar archive. Furthermore some tools like star are easier to get for linux. In such cases, using Virtualbox or VMWare Workstation is highly recommended. Using Windows should work in any case whatsoever.

Cygwin
Preferably 32 bits: has more packages and is less prone to errors. Furthermore, the star for cygwin attached is 32 bits.

Java 7 or higher

• Java 7 or OpenJDK 7 (more builds here)

• Oracle Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files 7 if you are going to work with password encrypted backups. 
  You need to install the files local_policy.jar and US_export_policy.jar under jre's lib/security folder, for example:
  - For Windows:
  • C:\Program Files\Java\jdk1.7.0_09\jre\lib\security\
  • C:\Program Files\Java\jre7\lib\security\
  • C:\Program Files (x86)\Java\jdk1.7.0_07\jre\lib\security\
  • C:\Program Files (x86)\Java\jre7\lib\security\
  - For Linux or BSD:
  /usr/local/jdk1.7/jre/lib/security/ 
  /usr/lib/jvm/java-7-openjdk-*/jre/lib/security/
  /usr/local/openjdk7/jre/lib/security/
  - For OS X:
  • /Library/Java/JavaVirtualMachines/jdk1.7.0_09.jdk/Contents/Home/jre/lib/security/

Perl

• Perl is available for several operating systems
• libssl or openssl headers. If you are using Linux or Cygwin is much more easier
• cpan modules required by use functions

pax or star
pax is an archiving utility that compresses in ustar by default. In this case is extremely useful because stores directories without trailing slash. Is available for all operating systems, and in windows can be installed via cygwin's setup.exe or download a native version like gnuwin32 (sometimes available via bsdtar with pax option). pax doesn't work when paths are more than 100 characters lenght, so I recommend star instead.

star allows storing directories without trailing slash. You can get the ubuntu version 1.5 here. For Windows you can use Cygwin's version included.

Android Backup Extractor
Android Backup Extractor is the java application that does all the job. It includes the perl scripts.


How it works, better with an example
I will use an example to demonstrate how it works, with java version. If you like to use perl, just grab the perl scripts and is nearly the same.
This is extracted from the readme file, and will extract a whole adb backup and repack only the data for the game Grand Theft Auto III for android.

1) Convert the original adb backup to tar format: 

Code:

java -jar abe.jar unpack nexus7.ab nexus7.tar <password>

2) Extract the contents of the tar archive. This should be done on a filesystem where the permissions of the files inside the tar are preserved, for example using linux, mac or bsd. Up to two folders may appear, apps and shared:

Code:

tar -xvf nexus7.tar

3) Make a list of all the contents of the original archive in the order they are archived: 

Code:

tar -tf nexus7.tar > nexus7.list

4) Create a new list only with the files and folders you want, in proper order. For example for the GTA 3 (you can try savegames instead of all data): 

Code:

cat nexus7.list | grep com.rockstar.gta3 > gta3.list

5) Create the new tar archive. The directories stored on tar shouldn't contain trailing slashes, so I use pax or star instead of tar. Pax works also if paths are up to 100 of lenght: 

Code:

cat gta.list | pax -wd > gta3.tar

OR

Code:

star -c -v -f gta3.tar -no-dirslash list=gta3.list

6) Create the adb backup from the tar archive. Password is optional: 

Code:

java -jar abe.jar pack gta3.tar gta3.ab <password>

Note: if the backup is not encrypted zlib can be used instead for both unpack and pack the ab archive: 
- Quick unpacking:

Code:

dd if=nexus7.ab bs=24 skip=1 | openssl zlib -d > nexus7.tar

- Quick packing: 

Code:

dd if=nexus7.ab bs=24 count=1 of=gta3.ab ; openssl zlib -in gta3.tar >> gta3.ab

Split an Android backup
Use Android Backup Splitter (is a shell script). Works on Cygwin too:
sh adb-split.sh backup.ab [password if needed]
Resulting files go in app-ab folder

# ADB Backup Splitter:
#
# This bash script utility generates an adb backup for each item
# on apps/ folder. That equals to generating an adb backup for
# each app.
#
# shared/0 and shared/1 (sdcards) are ignored, so that means that
# external data and OBB files, although may be backed up, may not
# be restored. For doing so backup the folder Android of the sdcards.
# Creating a backup with -shared flag has know issues (corruption)
# so is not recommended. Future releases of this script may add
# the option to automatically detect things from sdcards inside
# the adb backup file.
#
# This script works on Cygwin also, and in general where star does. 
#
# There may be apps in the backup that don't include the apk. In that
# particular case, the data won't be restored, so the individual backup
# is generated for testing purposes only. Installing the apk afterwards won't work.
# There is generated an html file called apk-missing.html so you can open it and
# install all applications from Play Store on any of your devices.

Links
Android Backup Splitter (@ Sourceforge)
Android Backup Extractor (@ Sourceforge)
Android Backup Extractor (@ GitHub)
Documentation at Blogspot
Full local backup infrastructure
[Guide] Types of Android backups
Backing Up Android Games
Titanium Backup Decrypter
Perl scripts to encrypt/decrypt adb backup files
[GUIDE] Full Phone Backup without Unlock or Root
[GUIDE] Internal Memory Data Recovery - Yes We Can!
[Q] [adb] backup not working for Temple Run

Related adb backup and restore bugs:
Issue 55860: adb backup skips the apk for certain apps and doesn't prompt for download when restoring
Issue 48126: device adb connecting to localhost adbd cannot execute commands anymore since 4.2.2 (error device offline)
Issue 3254: adb shell doesn't return exit code of program
Issue 53065: Debugging via ADB stopping the onClick action on ImageView widget
Issue 53668: ADB : An existing connection was forcibly closed by the remote host
Issue 54781: adb install: single quotes in filenames cause quoting issues
Issue 55173: (Game) Offroad Legends does not detect obb when restored via adb restore
Issue 55178: (Game) Carmageddon Promo does not detect obb when restored via adb restore
Issue 28303: adb backup doesn't respect -noshared flag
Issue 32830: adb restore errors not displayed on device
Issue 34311: Galaxy Nexus gets stuck when restoring adb backup
Issue 25780: BackupManager causes reboot when BackupAgent missing
Issue 40691: ab backup and restore Ski Safari does not restore saved data
Issue 16286: Restoration of phone not working properly
Issue 39014: Nexus 7 adb restore freezes when restoring udk.android.reader

 

Is there a way to look inside and modify an adb backup created file?

I started working on this. I'm posting my results so far here as a "community wiki" answer for two reasons: first, if someone else wants to join in, there's a place to talk; second, if I get pulled away from this project, there'll be hints for someone else to start working.

 

The backup logic on the host is entirely contained withinhttps://github.com/android/platform_system_core/blob/master/adb/commandline.c, in the function named backup. The function is very simple: it validates the command line options, sends the command mostly as-is to the adb daemon on the phone, and writes the phone's output to the file. There isn't even error-checking: if, for example, you refuse the backup on the phone, adb just writes out an empty file.

On the phone, the backup logic starts in service_to_fd() inhttps://github.com/android/platform_system_core/blob/master/adb/services.c. The function identifies that the command from the host is "backup", and passes the unparsed command to backup_service() inhttps://github.com/android/platform_system_core/blob/master/adb/backup_service.c. That, in turn, is a thin wrapper around /system/bin/bu, which is a trivial shell script to launch com.android.commands.bu.Backup as then main-class of a new Android app process. That calls ServiceManager.getService("backup") to get the backup service as an IBackupManager, and calls IBackupManager.fullBackup(), passing it the still-unused file descriptor (very indirectly) connected to the backup.ab file on the host.

Control passes to fullBackup() in com.android.server.BackupManagerService, which pops up the GUI asking the user to confirm/reject the backup. When the user do so, acknowledgeFullBackupOrRestore() (same file) is called. If the user approved the request, acknowledgeFullBackupOrRestore() figures out if the backup is encrypted, and passes a message to BackupHandler (same file.) BackupHandler then instantiates and kicks off a PerformFullBackupTask (same file, line 2228 as of time of writing)

We finally start generating output there, in PerformFullBackupTask.run(), between line 2367 andline 2453.

First, run() writes a header, which consists of either 4 or 9 ASCII lines:

 

1. "ANDROID BACKUP"
2. the backup format version: currently always "1"
3. either "0" if the backup is uncompressed or "1" if it is
4. the encryption method: currently either "none" or "AES-256"
5. (if encrypted), the "user password salt" encoded in hex, all caps
6. (if encrypted), the "master key checksum salt" encoded in hex, all caps
7. (if encrypted), the "number of PBKDF2 rounds used" as a decimal number
8. (if encrypted), the "IV of the user key" encoded in hex, all caps
9. (if encrypted), the "master IV + key blob, encrypted by the user key" encoded in hex, all caps

 

The actual backup data follows, either as (depending on compression and encryption) tar, deflate(tar), encrypt(tar), or encrypt(deflate(tar)).

 

TODO: write up the code path that generates the tar output -- you can simply use tar as long as entries are in the proper order (see below).

Tar archive format

App data is stored under the app/ directory, starting with a _manifest file, the APK (if requested) in a/, app files in f/, databases in db/ and shared preferences in sp/. If you requested external storage backup (using the -shared option), there will also be a shared/ directory in the archive containing external storage files.

$ tar tvf mybackup.tar

-rw------- 1000/1000      1019 2012-06-04 16:44 apps/org.myapp/_manifest

-rw-r--r-- 1000/1000   1412208 2012-06-02 23:53 apps/org.myapp/a/org.myapp-1.apk

-rw-rw---- 10091/10091     231 2012-06-02 23:41 apps/org.myapp/f/share_history.xml

-rw-rw---- 10091/10091       0 2012-06-02 23:41 apps/org.myapp/db/myapp.db-journal

-rw-rw---- 10091/10091    5120 2012-06-02 23:41 apps/org.myapp/db/myapp.db

-rw-rw---- 10091/10091    1110 2012-06-03 01:29 apps/org.myapp/sp/org.myapp_preferences.xml

Encryption details

1. An AES 256 key is derived from the backup encryption password using 10000 rounds of PBKDF2 with a randomly generated 512 bit salt.
2. An AES 256 master key is randomly generated
3. A master key 'checksum' is generated by running the master key through 10000 rounds of PBKDF2 with a new randomly generated 512 bit salt.
4. A random backup encryption IV is generated.
5. The IV, master key, and checksum are concatenated and encrypted with the key derived in 1. The resulting blob is saved in the header as a hex string.
6. The actual backup data is encrypted with the master key and appended to end of the file.

Sample pack/unpack code implementation (produces/uses) tar archives:

https://github.com/nelenkov/android-backup-extractor

Some more details here:

 http://nelenkov.blogspot.com/2012/06/unpacking-android-backups.html

Perl scripts for packing/unpacking and fixing broken archives:

http://forum.xda-developers.com/showthread.php?p=27840175#post27840175