A simple dead-lock issue of android app

描述了Android app中一个简单的死锁及其解决, 此处仅涉及两个线程; 更复杂点的死锁可能涉及多个线程, 形成一个环.

[SYMPTOM]
----- pid 850 at yyyy-mm-dd hh:mm:ss -----
Cmd line: com.android.phone

DALVIK THREADS:
(mutexes: tll=0 tsl=0 tscl=0 ghl=0)
"main" prio=5 tid=1 MONITOR
  | group="main" sCount=1 dsCount=0 obj=0x40ae9610 self=0x52ae18
  | sysTid=850 nice=0 sched=0/0 cgrp=default handle=1074324872
  | schedstat=( 0 0 0 ) utm=2724 stm=417 core=0
  at com.android.phone.BluetoothHandsfree.stopVoiceRecognition(BluetoothHandsfree.java:~3095)
  - waiting to lock <0x418e45a8> (a com.android.phone.BluetoothHandsfree) held by tid=28 (HandsfreeScoSocketConnectThread)
  at com.android.phone.BluetoothHeadsetService$6.stopVoiceRecognition(BluetoothHeadsetService.java:687)
  at android.bluetooth.IBluetoothHeadset$Stub.onTransact(IBluetoothHeadset.java:170)
  at android.os.Binder.execTransact(Binder.java:338)
  at android.os.BinderProxy.transact(Native Method)
  at android.media.IAudioService$Stub$Proxy.setMode(IAudioService.java:781)
  at android.media.AudioManager.setMode(AudioManager.java:1211)
  at com.android.internal.telephony.CallManager.setAudioMode(CallManager.java:612)
  at com.android.phone.PhoneUtils.setAudioMode(PhoneUtils.java:2071)
  at com.android.phone.CallNotifier.onPhoneStateChanged(CallNotifier.java:848)
  at com.android.phone.CallNotifier.handleMessage(CallNotifier.java:273)
  at android.os.Handler.dispatchMessage(Handler.java:99)
  at android.os.Looper.loop(Looper.java:214)
  at android.app.ActivityThread.main(ActivityThread.java:4473)
  at java.lang.reflect.Method.invokeNative(Native Method)
  at java.lang.reflect.Method.invoke(Method.java:511)
  at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:787)
  at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:554)
  at dalvik.system.NativeStart.main(Native Method)

"Binder Thread #6" prio=5 tid=36 NATIVE
  | group="main" sCount=1 dsCount=0 obj=0x420c1ef0 self=0x8960b0
  | sysTid=8913 nice=0 sched=0/0 cgrp=default handle=18216864
  | schedstat=( 0 0 0 ) utm=0 stm=0 core=0
  at dalvik.system.NativeStart.run(Native Method)

"HandsfreeScoSocketConnectThread" prio=5 tid=28 MONITOR
  | group="main" sCount=1 dsCount=0 obj=0x419ff9d8 self=0x1160590
  | sysTid=8866 nice=0 sched=0/0 cgrp=default handle=10042840
  | schedstat=( 0 0 0 ) utm=0 stm=0 core=0
  at com.android.phone.BluetoothHeadsetService$6.setAudioState(BluetoothHeadsetService.java:~880)
  - waiting to lock <0x418e46f0> (a com.android.phone.BluetoothHeadsetService) held by tid=1 (main)
  at android.bluetooth.BluetoothHeadset.setAudioState(BluetoothHeadset.java:706)
  at com.android.phone.BluetoothHandsfree.setAudioState(BluetoothHandsfree.java:1596)
  at com.android.phone.BluetoothHandsfree.access$700(BluetoothHandsfree.java:76)
  at com.android.phone.BluetoothHandsfree$ScoSocketConnectThread.connectSco(BluetoothHandsfree.java:435)
  at com.android.phone.BluetoothHandsfree$ScoSocketConnectThread.run(BluetoothHandsfree.java:424)

"Thread-224" prio=5 tid=34 NATIVE
  | group="main" sCount=1 dsCount=0 obj=0x421f37d8 self=0x8971b0
  | sysTid=8841 nice=0 sched=0/0 cgrp=default handle=10825016
  | schedstat=( 0 0 0 ) utm=0 stm=0 core=0
  at android.os.MessageQueue.nativePollOnce(Native Method)
  at android.os.MessageQueue.next(MessageQueue.java:118)
  at android.os.Looper.loop(Looper.java:188)
  at com.android.phone.XxxxTonePlayer$ToneGeneratorEventThread.run(XxxxTonePlayer.java:221)

"HeadsetBase Event Thread" prio=5 tid=27 NATIVE
  | group="main" sCount=1 dsCount=0 obj=0x423ba790 self=0xd311a8
  | sysTid=7062 nice=0 sched=0/0 cgrp=default handle=7988320
  | schedstat=( 0 0 0 ) utm=16 stm=4 core=0
  at android.bluetooth.HeadsetBase.readNative(Native Method)
  at android.bluetooth.HeadsetBase.access$100(HeadsetBase.java:34)
  at android.bluetooth.HeadsetBase$1.run(HeadsetBase.java:161)

[Analysis]
From the logcat, dead-lock can be found.
"main" prio=5 tid=1 MONITOR
at com.android.phone.BluetoothHandsfree.stopVoiceRecognition(BluetoothHandsfree.java:~3095)
- waiting to lock <0x418e45a8> (a com.android.phone.BluetoothHandsfree) held by tid=28 (HandsfreeScoSocketConnectThread)
at com.android.phone.BluetoothHeadsetService$6.stopVoiceRecognition(BluetoothHeadsetService.java:687)

"HandsfreeScoSocketConnectThread" prio=5 tid=28 MONITOR
at com.android.phone.BluetoothHeadsetService$6.setAudioState(BluetoothHeadsetService.java:~880)
- waiting to lock <0x418e46f0> (a com.android.phone.BluetoothHeadsetService) held by tid=1 (main)
at android.bluetooth.BluetoothHeadset.setAudioState(BluetoothHeadset.java:706)

The execution flow of the two threads:
1. Main thread locks BluetoothHeadsetService in BluetoothHeadsetService#stopVoiceRecognition().
2. HandsfreeScoSocketConnectThread locks BluetoothHandsfree in ScoSocketConnectThread#connectSco().
3. HandsfreeScoSocketConnectThread tries to lock BluetoothHeadsetService in BluetoothHeadsetService#setAudioState(), but cannot lock because it's already locked at above step1.
4. Main thread tries to lock BluetoothHandsfree in BluetoothHandsfree#stopVoiceRecognition(), but cannot lock because it's already locked at above step2.

        public boolean BluetoothHeadsetService::startVoiceRecognition(BluetoothDevice device) {
            enforceCallingOrSelfPermission(BLUETOOTH_PERM, "Need BLUETOOTH permission");
            synchronized (BluetoothHeadsetService.this){
                if (device == null ||
                    mRemoteHeadsets.get(device) == null ||
                    mRemoteHeadsets.get(device).mState != BluetoothProfile.STATE_CONNECTED) {
                    return false;
                }
                return mBtHandsfree.startVoiceRecognition();
            }
        }

        public boolean BluetoothHeadsetService::stopVoiceRecognition(BluetoothDevice device) {
            enforceCallingOrSelfPermission(BLUETOOTH_PERM, "Need BLUETOOTH permission");
            synchronized (BluetoothHeadsetService.this){
                if (device == null ||
                    mRemoteHeadsets.get(device) == null ||
                    mRemoteHeadsets.get(device).mState != BluetoothProfile.STATE_CONNECTED) {
                    return false;
                }

                return mBtHandsfree.stopVoiceRecognition();
            }
        }

        public boolean BluetoothHeadsetService::setAudioState(BluetoothDevice device, int state) {
            synchronized (BluetoothHeadsetService.this) {
                int prevState = mRemoteHeadsets.get(device).mAudioState;
                mRemoteHeadsets.get(device).mAudioState = state;
                if (state == BluetoothHeadset.STATE_AUDIO_CONNECTED) {
                    mAudioConnectedDevice = device;
                } else if (state == BluetoothHeadset.STATE_AUDIO_DISCONNECTED) {
                    mAudioConnectedDevice = null;
                }
                Intent intent = new Intent(BluetoothHeadset.ACTION_AUDIO_STATE_CHANGED);
                intent.putExtra(BluetoothHeadset.EXTRA_STATE, state);
                intent.putExtra(BluetoothHeadset.EXTRA_PREVIOUS_STATE, prevState);
                intent.putExtra(BluetoothDevice.EXTRA_DEVICE, device);
                sendBroadcast(intent, android.Manifest.permission.BLUETOOTH);
                if (PhoneApp.DBG) log("AudioStateIntent: " + device + " State: " + state
                  + " PrevState: " + prevState);
                return true;
            }
        }

        public int BluetoothHeadsetService::getAudioState(BluetoothDevice device) {
            synchronized (BluetoothHeadsetService.this){
                BluetoothRemoteHeadset headset = mRemoteHeadsets.get(device);
                if (headset == null) return BluetoothHeadset.STATE_AUDIO_DISCONNECTED;

                return headset.mAudioState;
           }
       }

        private void BluetoothHandsfree::ScoSocketConnectThread::connectSco() {
            synchronized (BluetoothHandsfree.this) {
                if (!Thread.currentThread().interrupted() &&
                    isHeadsetConnected() &&
                    (mAudioPossible || allowAudioAnytime()) &&
                    mConnectedSco == null) {
                    Log.i(TAG, "Routing audio for incoming SCO connection");
                    mConnectedSco = mIncomingSco;
                    mAudioManager.setBluetoothScoOn(true);
                    setAudioState(BluetoothHeadset.STATE_AUDIO_CONNECTED, mHeadset.getRemoteDevice());

                    if (mSignalScoCloseThread == null) {
                        mSignalScoCloseThread = new SignalScoCloseThread();
                        mSignalScoCloseThread.setName("SignalScoCloseThread");
                        mSignalScoCloseThread.start();
                    }
                } else {
                    Log.i(TAG, "Rejecting incoming SCO connection");
                    try {
                        mIncomingSco.close();
                    }catch (IOException e) {
                        Log.e(TAG, "Error when closing incoming Sco socket");
                    }
                    mIncomingSco = null;
                }
            }
        }

    private void BluetoothHandsfree::setAudioState(int state, BluetoothDevice device) {
        if (VDBG) log("setAudioState(" + state + ")");
        if (mBluetoothHeadset == null) {
            mAdapter.getProfileProxy(mContext, mProfileListener, BluetoothProfile.HEADSET);
            mPendingAudioState = true;
            mAudioState = state;
            return;
        }
        mBluetoothHeadset.setAudioState(device, state);
    }

It is a very bad design.

Check carefully whether the following synchronized block can be removed.
Removing the synchronized block in BluetoothHeadsetService::set/getAudioState, and using ConcurrentHashMap instead of the HashMap only decrease the occurence of race condition.

Change the synchronized blocks with same lock sequence to avoid dead lock.
Change BluetoothHeadsetService::startVoiceRecognition/stopVoiceRecognition as follows.
Though it is ugly, it works.
        public boolean BluetoothHeadsetService::startVoiceRecognition(BluetoothDevice device) {
            enforceCallingOrSelfPermission(BLUETOOTH_PERM, "Need BLUETOOTH permission");
            synchronized (mBtHandsfree) {
            synchronized (BluetoothHeadsetService.this) {
                if (device == null ||
                    mRemoteHeadsets.get(device) == null ||
                    mRemoteHeadsets.get(device).mState != BluetoothProfile.STATE_CONNECTED) {
                    return false;
                }
                return mBtHandsfree.startVoiceRecognition();
            }
            }
        }

        public boolean BluetoothHeadsetService::stopVoiceRecognition(BluetoothDevice device) {
            enforceCallingOrSelfPermission(BLUETOOTH_PERM, "Need BLUETOOTH permission");
            synchronized (mBtHandsfree) {
            synchronized (BluetoothHeadsetService.this) {
                if (device == null ||
                    mRemoteHeadsets.get(device) == null ||
                    mRemoteHeadsets.get(device).mState != BluetoothProfile.STATE_CONNECTED) {
                    return false;
                }

                return mBtHandsfree.stopVoiceRecognition();
            }
            }
        }

A principle is to use one instance-scope lock ONLY to protect data of itself. And use a outside lock to protect the combination.
So, change BluetoothHeadsetService::startVoiceRecognition/stopVoiceRecognition and BluetoothHandsfree::ScoSocketConnectThread::connectSco as follows, others as the same.
        public boolean BluetoothHeadsetService::startVoiceRecognition(BluetoothDevice device) {
            enforceCallingOrSelfPermission(BLUETOOTH_PERM, "Need BLUETOOTH permission");
            synchronized (gGlobalObject) {
            synchronized (BluetoothHeadsetService.this){
                if (device == null ||
                    mRemoteHeadsets.get(device) == null ||
                    mRemoteHeadsets.get(device).mState != BluetoothProfile.STATE_CONNECTED) {
                    return false;
                }
            }
            return mBtHandsfree.startVoiceRecognition();
            }
        }

        public boolean BluetoothHeadsetService::stopVoiceRecognition(BluetoothDevice device) {
            enforceCallingOrSelfPermission(BLUETOOTH_PERM, "Need BLUETOOTH permission");
            synchronized (gGlobalObject) {
            synchronized (BluetoothHeadsetService.this){
                if (device == null ||
                    mRemoteHeadsets.get(device) == null ||
                    mRemoteHeadsets.get(device).mState != BluetoothProfile.STATE_CONNECTED) {
                    return false;
                }
            }
            return mBtHandsfree.stopVoiceRecognition();
            }
        }

        private void BluetoothHandsfree::ScoSocketConnectThread::connectSco() {
            synchronized (gGlobalObject) {
            synchronized (BluetoothHandsfree.this) {
                if (!Thread.currentThread().interrupted() &&
                    isHeadsetConnected() &&
                    (mAudioPossible || allowAudioAnytime()) &&
                    mConnectedSco == null) {
                    Log.i(TAG, "Routing audio for incoming SCO connection");
                    mConnectedSco = mIncomingSco;
                    mAudioManager.setBluetoothScoOn(true);
                    setAudioState(BluetoothHeadset.STATE_AUDIO_CONNECTED, mHeadset.getRemoteDevice());

                    if (mSignalScoCloseThread == null) {
                        mSignalScoCloseThread = new SignalScoCloseThread();
                        mSignalScoCloseThread.setName("SignalScoCloseThread");
                        mSignalScoCloseThread.start();
                    }
                } else {
                    Log.i(TAG, "Rejecting incoming SCO connection");
                    try {
                        mIncomingSco.close();
                    }catch (IOException e) {
                        Log.e(TAG, "Error when closing incoming Sco socket");
                    }
                    mIncomingSco = null;
                }
            }
            }
        }
       
Using the big synchronized block is very low efficient here.
It is only to illustrate the solution principle here.
It is better to use object lock/unlock pair to protect what should be protected precisely.