keystore文件里PrivateKeyEntry错误配置导致TLS连接失败
某银行客户的cluster里同时安装了Spectrum Symphony + Spectrum Conductor，属于multihomed模式。这种安装和配置是支持的，详情可以参考IBM文档。
出于安全要求，他们在tier 2和tier 3启用了TLS，详情参考IBM文档。结果是，在tier 3一切顺利，访问网页没有问题；但是在tier 2却遇到了问题，报错如下。
"Failed to retrieve the Spark applications. Connection refused. Ensure that either the required IBM Spectrum Conductor services are running (ascd and REST) or SSL is configured properly."
因为只有tier 2才有问题而tier 3没问题问题，而且tier 2和tier 3的certificate都放在相同的keystore里，所以我们有理由怀疑可能tier 3的certificate配置出错了。当然，脑子里先要有关于certificate的相关知识，不然可能也不会怀疑到这。SSL certificate相关知识可以参考我的这篇"一文读懂HTTP, HTTPS, SSL和TLS"讲解。
于是，我们可以通过下面的步骤来测试certificate的配置。
openssl s_client -CAfile /path/to/target/keystore/file -connect target_FQDN:target_port
针对tier 3上，测试得到的结果如下，连接状态是CONNECTED，certificate chain和certificate都可以返回来，没问题。
$openssl s_client -CAfile /opt/sym/certificates/truststore.pem -connect bens3-a1.svr.us.jpm.net:8643
CONNECTED(00000003)
depth=2 DC = NET, DC = JPMCHASE, DC = EXCHAD, CN = JPMCROOTCA
verify return:1
depth=1 DC = net, DC = jpmchase, DC = exchad, CN = PSIN0P551
verify return:1
depth=0 C = US, ST = NJ, L = Jersey City, O = JPMorg, OU = Compute Backbone, CN = bens3-a1.svr.us.jpm.net
verify return:1
---
Certificate chain
0 s:/C=US/ST=NJ/L=Jersey City/O=JPMorg /OU=Compute Backbone/CN=bens3-a1.svr.us.jpm.net
i:/DC=net/DC=jpmchase/DC=exchad/CN=PSIN0P551
1 s:/DC=net/DC=jpmchase/DC=exchad/CN=PSIN0P551
i:/DC=NET/DC=JPMCHASE/DC=EXCHAD/CN=JPMCROOTCA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIHszCCBZugAwIBAgITRQAC1Y89tfV7k9/q/gABAALVjzANBgkqhkiG9w0BAQsF
ADBbMRMwEQYKCZImiZPyLGQBGRYDbmV0MRgwFgYKCZImiZPyLGQBGRYIanBtY2hh
c2UxFjAUBgoJkiaJk/IsZAEZFgZleGNoYWQxEjAQBgNVBAMTCVBTSU4wUDU1MTAe
Fw0xOTEwMDIxMjU0MDZaFw0yMTEwMDExMjU0MDZaMIGNMQswCQYDVQQGEwJVUzEL
MAkGA1UECBMCTkoxFDASBgNVBAcTC0plcnNleSBDaXR5MRcwFQYDVQQKEw5KUE1v
cmdhbiBDaGFzZTEZMBcGA1UECxMQQ29tcHV0ZSBCYWNrYm9uZTEnMCUGA1UEAxMe
Y2JiZW5zMy1hMS5zdnIudXMuanBtY2hhc2UubmV0MIIBIjANBgkqhkiG9w0BAQEF
AAOCAQ8AMIIBCgKCAQEAo/khQh8MHdTkTuKa7eO7Qigx9UuqRlZ+lMQImtZxhiEQ
g9vpEhZk193G9IRuV8lVHbV6fMe6WYCuSGP0V+ZF1OVe5XtmFnWNNW5FS8WyApk3
hcSeWeeI6QDArMutidpya30a21UUv+ZxoOdnEDwAvMjoWBS6caJPiRnKQ77TXl+J
HHVv2Q6SDCSQiwuLxRZzD+c637bJXvvw0Tt1YKwcijp0DBwGmZotdvONulEJNvtM
J7Pn8bhgWoVC7UkM1TY6M4xikJgFHh+AlT0+Z+tYfGMbu7aPUBjO61f2Qq9KSouT
n6di8ule0c9hntat+JS1bDHz9Czd0IcmIfNpGeS8cwIDAQABo4IDOzCCAzcwKQYD
VR0RBCIwIIIeY2JiZW5zMy1hMS5zdnIudXMuanBtY2hhc2UubmV0MB0GA1UdDgQW
BBTi/DxO6VfcEMq8ZqpNiDDpPeaQ7jAfBgNVHSMEGDAWgBQy3mGo4/el4t5HICuq
......
cyxDTj1TZXJ2aWNlcyxDTj1Db25maWd1cmF0aW9uLERDPWV4Y2hhZCxEQz1qcG1j
aGFzZSxEQz1uZXQ/Y2VydGlmaWNhdGVSZXZvY2F0aW9uTGlzdD9iYXNlP29iamVj
dENsYXNzPWNSTERpc3RyaWJ1dGlvblBvaW50MIIBLgYIKwYBBQUHAQEEggEgMIIB
HDA5BggrBgEFBQcwAoYtaHR0cDovL2FkY3MuanBtY2hhc2UubmV0L2NybC9QU0lO
MFA1NTEoMSkuY3J0MCkGCCsGAQUFBzABhh1odHRwOi8vYWRjcy5qcG1jaGFzZS5u
ZXQvb2NzcDCBswYIKwYBBQUHMAKGgaZsZGFwOi8vL0NOPVBTSU4wUDU1MSxDTj1B
SUEsQ049UHVibGljJTIwS2V5JTIwU2VydmljZXMsQ049U2VydmljZXMsQ049Q29u
ZmlndXJhdGlvbixEQz1leGNoYWQsREM9anBtY2hhc2UsREM9bmV0P2NBQ2VydGlm
aWNhdGU/YmFzZT9vYmplY3RDbGFzcz1jZXJ0aWZpY2F0aW9uQXV0aG9yaXR5MA4G
A1UdDwEB/wQEAwIFoDA8BgkrBgEEAYI3FQcELzAtBiUrBgEEAYI3FQiBg5o1g/PQ
QIKBkwyC3ZYCk506RIOUsA+Etq87AgFkAgEKMB0GA1UdJQQWMBQGCCsGAQUFBwMB
BggrBgEFBQcDAjAnBgkrBgEEAYI3FQoEGjAYMAoGCCsGAQUFBwMBMAoGCCsGAQUF
BwMCMA0GCSqGSIb3DQEBCwUAA4ICAQASNIP+nc1/TAYpIzY45C+c69pFlv0QupDq
ovOs9uPz/4oiGfwLaXmVVYmmUZIdlH8QaR4v/AYGkbYnej9BAHX7/NynevTT908v
VjFMb0GNkGC+KgCOaEeLv5fR9/x2xoVFOyztjysHnDjvi1A5VcyTqRiZynwOzrMZ
jtLS/jtI/65K7yDTYQDLATuUWmi3xcl0QyV11bxgDeU6ggOu1w/SyiFPPng9mWEA
UfE8yIWiXTrEZlKo00tV8L5x6vizq4sBQTxbuOuDJbqTCJKkZUv+GQuvWuwcPcFi
xRZboWVOaZ6v9i3HOv1Yd7mCjkT67rC2lzqPgxpZAD2ew9/LTmtTQYRc7iWUUBPb
9PRIIuf8sLp/9Lt06loVGe5saFvxG/ooGSfe2JwLvQUIg9HKhZNFaIvLdu6V/dXq
DLzYdEfhF7KuM2TzwIRETSahMadk6+z17OUlzu87aWPVBr7YRmBtupBC1J1QaFH6
tbmh5+56gAmSSvNt5l6yVGgZB0ooklTJYwkc9lH7NYzunzksaXPbVvjJEDUl+e6w
z2XIripgZRZfnOiGHrNPjuPuUGP2gPFfm7NViGUoOY11GzTzU2l2xFzSMlngvIwR
sq1waInp1NDkr0ue08l27NnwBurqmiXfP9KQsu7gpaj8RAXiq8afQpReCHV9Ra3X
Oj+YAovtzA==
-----END CERTIFICATE-----
subject=/C=US/ST=NJ/L=Jersey City/O=JPMorg/OU=Compute Backbone/CN=bens3-a1.svr.us.jpm.net
issuer=/DC=net/DC=jpmchase/DC=exchad/CN=PSIN0P551
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 4767 bytes and written 415 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 5DF8DBBCAA37AC5D809C6831174368C0545E3E06A0E8BE2F6450F03C96DCA198
Session-ID-ctx:
Master-Key: 582ABE9363DE36147A845750A7199639CF8CC88D7C3C50EE3B3C7941EE9713F120DF8558504F41CECB6838C5B6E32C47
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1576590268
Timeout : 300 (sec)
Verify return code: 0 (ok)
---针对tier 2，测试的结果如下，得到sslv3 alert handshake failure的错误，无法返回server端的certificate chain和certificate。这更一步说明tier 3的证书配置有问题。
[root@bens3-a1 ibm]# openssl s_client -CAfile /opt/sym/certificates/truststore.pem -connect bens3-ca001.svr.us.jpm.net:6091
CONNECTED(00000003)
139871883896720:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure:s23_clnt.c:769:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 289 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1576590140
Timeout : 300 (sec)
Verify return code: 0 (ok)
---下一步，我们就需要用下面的命令来读取keystore文件，看看里面的具体内容就可以一目了然。
keytool -list -keystore your_keystore_file
下面是客户那边keytool -list的部分结果。
[root@bens3-a1 egoadmin]# keytool -list -keystore psk2-eng2-tier2and3.jks
Enter keystore password:
Keystore type: jks
Keystore provider: SUN
Your keystore contains 4 entries
rootca, Dec 6, 2019, trustedCertEntry,
Certificate fingerprint (SHA1): 1A:58:C1:67:02:09:45:31:0F:25:E9:90:B9:94:CD:59:C8:F2:6B:A5
psk2-eng2-tier3, Dec 18, 2019, trustedCertEntry,
Certificate fingerprint (SHA1): 38:18:DB:6C:CE:C6:45:EF:99:B9:65:A9:2A:3F:E9:71:18:2D8:B9:C6
psk2-eng2-tier2, Dec 18, 2019, trustedCertEntry,
Certificate fingerprint (SHA1): F5:30:ED:72:B2:43:DE:06:82:40:48:C2:FA:38:B5:6A:E9:74:F2:6A
intermediate, Dec 6, 2019, trustedCertEntry,
Certificate fingerprint (SHA1): 35:1E:74:B2:98:01:21:1C:5E:16:58:95:B6:34:20:B4:F7:9C:26:FD可以看到，keystore文件里的内容没有PrivateKeyEntry而全都是trustedCertEntry，这是不完整的，难怪会“no peer certificate available”。关于keystore文件，在这篇"一文读懂HTTP, HTTPS, SSL和TLS"讲解过了，PrivateKeyEntry是必须包括在里头的。
所以解决方案就是，客户重新制作证书生成keystore文件，问题就消失了，新的keystore部分内容如下，既有PrivateKeyEntry也有trustedCertEntry。
[root@bens3-a1 egoadmin]# keytool -list -keystore psk2-eng2-tier2and3.jks
Enter keystore password:
Keystore type: jks
Keystore provider: SUN
Your keystore contains 4 entries
rootca, Dec 6, 2019, trustedCertEntry,
Certificate fingerprint (SHA1): 1A:58:C1:67:02:09:45:31:0F:25:E9:90:B9:94:CD:59:C8:F2:6B:A5
psk-eng2-tier3, Dec 18, 2019, PrivateKeyEntry,
Certificate fingerprint (SHA1): 98:A8:DB:6C:CE:C6:35:EF:99:B9:D5:A9:2A:74:E9:71:18:E8:B9:C6
psk-eng2-tier2, Dec 18, 2019, PrivateKeyEntry,
Certificate fingerprint (SHA1): F2:30:EF:32:B2:43:DE:76:82:40:48:C9:FA:38:C5:6A:E0:74:92:8C
intermediate, Dec 6, 2019, trustedCertEntry,
Certificate fingerprint (SHA1): 35:1E:74:B2:98:01:21:1C:5E:16:58:95:B6:34:20:B4:F7:9C:26:FD