RocketMQ 初次尝试受挫篇【ssl错误】

环境

RocketMQ：4.1.0
操作系统：win7
服务器：centos6.5
工具：CRT

错误

//以上日志省略。。。
2017-12-15 13:56:43 INFO main - Using JDK SSL provider
2017-12-15 13:56:43 ERROR main - Failed to create SSLContext for server
java.security.cert.CertificateException: No provider succeeded to generate a self-signed certificate. See debug log for the root cause.
        at io.netty.handler.ssl.util.SelfSignedCertificate.(SelfSignedCertificate.java:157) ~[netty-all-4.0.42.Final.jar:4.0.42.Final]
        at io.netty.handler.ssl.util.SelfSignedCertificate.(SelfSignedCertificate.java:110) ~[netty-all-4.0.42.Final.jar:4.0.42.Final]
        at io.netty.handler.ssl.util.SelfSignedCertificate.(SelfSignedCertificate.java:88) ~[netty-all-4.0.42.Final.jar:4.0.42.Final]
        at io.netty.handler.ssl.util.SelfSignedCertificate.(SelfSignedCertificate.java:79) ~[netty-all-4.0.42.Final.jar:4.0.42.Final]
        at org.apache.rocketmq.remoting.netty.TlsHelper.buildSslContext(TlsHelper.java:133) ~[rocketmq-remoting-4.2.0-SNAPSHOT.jar:4.2.0-SNAPSHOT]
        at org.apache.rocketmq.remoting.netty.NettyRemotingServer.(NettyRemotingServer.java:147) ~[rocketmq-remoting-4.2.0-SNAPSHOT.jar:4.2.0-SNAPSHOT]
        at org.apache.rocketmq.namesrv.NamesrvController.initialize(NamesrvController.java:75) [rocketmq-namesrv-4.2.0-SNAPSHOT.jar:4.2.0-SNAPSHOT]
        at org.apache.rocketmq.namesrv.NamesrvStartup.main0(NamesrvStartup.java:108) [rocketmq-namesrv-4.2.0-SNAPSHOT.jar:4.2.0-SNAPSHOT]
        at org.apache.rocketmq.namesrv.NamesrvStartup.main(NamesrvStartup.java:46) [rocketmq-namesrv-4.2.0-SNAPSHOT.jar:4.2.0-SNAPSHOT]
2017-12-15 13:56:43 INFO NettyEventExecutor - NettyEventExecutor service started
2017-12-15 13:56:43 INFO main - The Name Server boot success. serializeType=JSON

情况

昨天了解到了我们国产的消息中间件RocketMQ,阿里巴巴出的，非常兴奋；立马去官网了解。

跟着官网的步骤走：

  > git clone -b develop https://github.com/apache/rocketmq.git
  > cd rocketmq
  > mvn -Prelease-all -DskipTests clean install -U
  > cd distribution/target/apache-rocketmq
  > # nohup sh bin/mqnamesrv &
  > nohup sh bin/mqnamesrv
  > tail -f ~/logs/rocketmqlogs/namesrv.log

执行完上面的步骤时，起不来；看下错误，jdk版本不对。
说明下：

我公司由于历史的原因，jdk1.8基本不会使用，一直停留在jdk1.7

所以我只要在自己的家目录中下载一个jdk1.8。
把启动脚本runserver.sh的java环境修改下：

export JAVA_HOME=/home/yutao/jdk1.8.0_121
export CLASSPATH=$JAVA_HOME/lib:$JAVA_HOME/jre/lib
error_exit ()
{
    echo "ERROR: $1 !!"
    exit 1
}

#[ ! -e "$JAVA_HOME/bin/java" ] && JAVA_HOME=$HOME/jdk/java
#[ ! -e "$JAVA_HOME/bin/java" ] && JAVA_HOME=/usr/java
#[ ! -e "$JAVA_HOME/bin/java" ] && error_exit "Please set the JAVA_HOME variable in your environment, We need java(x64)!"

#export JAVA_HOME
export JAVA="$JAVA_HOME/bin/java"
export BASE_DIR=$(dirname $0)/..
export CLASSPATH=.:${BASE_DIR}/conf:${CLASSPATH}
echo "$CLASSPATH"

之后，启动没问题，但是报了个错误，就是上面那个（我再复制一遍）。

2017-12-15 13:56:43 INFO main - Using JDK SSL provider
2017-12-15 13:56:43 ERROR main - Failed to create SSLContext for server
java.security.cert.CertificateException: No provider succeeded to generate a self-signed certificate. See debug log for the root cause.
        at io.netty.handler.ssl.util.SelfSignedCertificate.(SelfSignedCertificate.java:157) ~[netty-all-4.0.42.Final.jar:4.0.42.Final]
        at io.netty.handler.ssl.util.SelfSignedCertificate.(SelfSignedCertificate.java:110) ~[netty-all-4.0.42.Final.jar:4.0.42.Final]
        at io.netty.handler.ssl.util.SelfSignedCertificate.(SelfSignedCertificate.java:88) ~[netty-all-4.0.42.Final.jar:4.0.42.Final]
        at io.netty.handler.ssl.util.SelfSignedCertificate.(SelfSignedCertificate.java:79) ~[netty-all-4.0.42.Final.jar:4.0.42.Final]
        at org.apache.rocketmq.remoting.netty.TlsHelper.buildSslContext(TlsHelper.java:133) ~[rocketmq-remoting-4.2.0-SNAPSHOT.jar:4.2.0-SNAPSHOT]
        at org.apache.rocketmq.remoting.netty.NettyRemotingServer.(NettyRemotingServer.java:147) ~[rocketmq-remoting-4.2.0-SNAPSHOT.jar:4.2.0-SNAPSHOT]
        at org.apache.rocketmq.namesrv.NamesrvController.initialize(NamesrvController.java:75) [rocketmq-namesrv-4.2.0-SNAPSHOT.jar:4.2.0-SNAPSHOT]
        at org.apache.rocketmq.namesrv.NamesrvStartup.main0(NamesrvStartup.java:108) [rocketmq-namesrv-4.2.0-SNAPSHOT.jar:4.2.0-SNAPSHOT]
        at org.apache.rocketmq.namesrv.NamesrvStartup.main(NamesrvStartup.java:46) [rocketmq-namesrv-4.2.0-SNAPSHOT.jar:4.2.0-SNAPSHOT]
2017-12-15 13:56:43 INFO NettyEventExecutor - NettyEventExecutor service started
2017-12-15 13:56:43 INFO main - The Name Server boot success. serializeType=JSON

这个错误困扰了我一天，我甚至还去钉钉上问，RocketMQ的开发人冯嘉。
（我一开始以为大佬都是不会理我这种小白的，没想到居然回了我）。
他给的解释，是权限不对。

这个我也猜到了，以为我使用root账号执行时，不报错，但是使用我自己的账号就是不行。

猜测

脚本没有改全

因为bin目录下有很多脚本，我只改了一个，所以呢！我全部都改了一遍，就是把java环境指定了1.8。
历史原因：系统配置文件（/etc/profile）里的不能改（线上已经有程序了，jdk1.8并不兼容1.7）。

之后启动下，还是不行；
无奈，晚上在家虚拟机中执行时，完全使用自己的账号，一次就OK啦！
这让我更加肯定是权限问题。

网上资料本来就少，去官网留言还得用英文；真是欲哭无泪。。。

---

后来我Google的时候，查到了netty的源码：

https://netty.io/4.0/xref/io/netty/handler/ssl/util/SelfSignedCertificate.html

public SelfSignedCertificate(String fqdn, SecureRandom random, int bits, Date notBefore, Date notAfter)
             throws CertificateException {
         // Generate an RSA key pair.
         final KeyPair keypair;
         try {
             KeyPairGenerator keyGen = KeyPairGenerator.getInstance("RSA");
             keyGen.initialize(bits, random);
             keypair = keyGen.generateKeyPair();
         } catch (NoSuchAlgorithmException e) {
             // Should not reach here because every Java implementation must have RSA key pair generator.
             throw new Error(e);
         }

         String[] paths;
         try {
             // Try the OpenJDK's proprietary implementation.
             paths = OpenJdkSelfSignedCertGenerator.generate(fqdn, keypair, random, notBefore, notAfter);
         } catch (Throwable t) {
             logger.debug("Failed to generate a self-signed X.509 certificate using sun.security.x509:", t);
             try {
                 // Try Bouncy Castle if the current JVM didn't have sun.security.x509.
                paths = BouncyCastleSelfSignedCertGenerator.generate(fqdn, keypair, random, notBefore, notAfter);
             } catch (Throwable t2) {
                 logger.debug("Failed to generate a self-signed X.509 certificate using Bouncy Castle:", t2);
                 throw new CertificateException(
                         "No provider succeeded to generate a self-signed certificate. " +
                                 "See debug log for the root cause.", t2);
                 // TODO: consider using Java 7 addSuppressed to append t
             }
         }

         certificate = new File(paths[0]);
         privateKey = new File(paths[1]);
         key = keypair.getPrivate();
         FileInputStream certificateInput = null;
         try {
             certificateInput = new FileInputStream(certificate);
             cert = (X509Certificate) CertificateFactory.getInstance("X509").generateCertificate(certificateInput);
         } catch (Exception e) {
             throw new CertificateEncodingException(e);
         } finally {
             if (certificateInput != null) {
                 try {
                     certificateInput.close();
                 } catch (IOException e) {
                     logger.warn("Failed to close a file: " + certificate, e);
                 }
             }
         }
     }

单纯的看，没发现什么特别的，只知道是要生成ssl证书。

后来我Google的时候，查了一个类似的错误：
Input is starting although generating TLS cert fails

里面提到确保有tmpDir的写权限。这让我想起了之前我在指定elasticsearch启动脚本的jdk版本时也有过类似的问题。但是一路追查源码，才知道需要有/tmp目录的写权限。

抱着试试的心态，把/tmp的权限改为777。（我公司通过软连接，定位到了/home/.tmp这个目录目前只有root才有权限）。

再次启动，成功啦！没有报错啦！

我靠，心累啦！为什么netty会往里面写东西呢？看看了：

-rw-rw-r-- 1 yutao yutao  914 Dec 15 14:09 keyutil_example.com_4665592468277863911.key
-rw-rw-r-- 1 yutao yutao  634 Dec 15 14:09 keyutil_example.com_9153033687312290589.crt

就是生成了ssl的证书。

好吧！

查看源码

对于为什么要往/tmp文件加里写ssl证书，我得查查源码。

主要是看到：

// Try Bouncy Castle if the current JVM didn't have sun.security.x509.
154                 paths = BouncyCastleSelfSignedCertGenerator.generate(fqdn, keypair, random, notBefore, notAfter);

而这个方法BouncyCastleSelfSignedCertGenerator.generate(fqdn, keypair, random, notBefore, notAfter)：

final class BouncyCastleSelfSignedCertGenerator
{
    private static final Provider PROVIDER;

    static String[] generate(final String fqdn, final KeyPair keypair, final SecureRandom random, final Date notBefore, final Date notAfter) throws Exception {
        final PrivateKey key = keypair.getPrivate();
        final X500Name owner = new X500Name("CN=" + fqdn);
        final X509v3CertificateBuilder builder = (X509v3CertificateBuilder)new JcaX509v3CertificateBuilder(owner, new BigInteger(64, random), notBefore, notAfter, owner, keypair.getPublic());
        final ContentSigner signer = new JcaContentSignerBuilder("SHA256WithRSAEncryption").build(key);
        final X509CertificateHolder certHolder = builder.build(signer);
        final X509Certificate cert = new JcaX509CertificateConverter().setProvider(BouncyCastleSelfSignedCertGenerator.PROVIDER).getCertificate(certHolder);
        cert.verify(keypair.getPublic());
        //看到这里
        return SelfSignedCertificate.newSelfSignedCertificate(fqdn, key, cert);
    }

    static {
        PROVIDER = (Provider)new BouncyCastleProvider();
    }
}

我们再看到SelfSignedCertificate.newSelfSignedCertificate(fqdn, key, cert)，其源码是：

static String[] newSelfSignedCertificate(final String fqdn, final PrivateKey key, final X509Certificate cert) throws IOException, CertificateEncodingException {
        ByteBuf wrappedBuf = Unpooled.wrappedBuffer(key.getEncoded());
        String keyText;
        try {
            final ByteBuf encodedBuf = Base64.encode(wrappedBuf, true);
            try {
                keyText = "-----BEGIN PRIVATE KEY-----\n" + encodedBuf.toString(CharsetUtil.US_ASCII) + "\n-----END PRIVATE KEY-----\n";
            }
            finally {
                encodedBuf.release();
            }
        }
        finally {
            wrappedBuf.release();
        }
        final File keyFile = File.createTempFile("keyutil_" + fqdn + '_', ".key");
        keyFile.deleteOnExit();
        OutputStream keyOut = new FileOutputStream(keyFile);
        try {
            keyOut.write(keyText.getBytes(CharsetUtil.US_ASCII));
            keyOut.close();
            keyOut = null;
        }
        finally {
            if (keyOut != null) {
                safeClose(keyFile, keyOut);
                safeDelete(keyFile);
            }
        }
        wrappedBuf = Unpooled.wrappedBuffer(cert.getEncoded());
        String certText;
        try {
            final ByteBuf encodedBuf = Base64.encode(wrappedBuf, true);
            try {
                certText = "-----BEGIN CERTIFICATE-----\n" + encodedBuf.toString(CharsetUtil.US_ASCII) + "\n-----END CERTIFICATE-----\n";
            }
            finally {
                encodedBuf.release();
            }
        }
        finally {
            wrappedBuf.release();
        }
        final File certFile = File.createTempFile("keyutil_" + fqdn + '_', ".crt");
        certFile.deleteOnExit();
        OutputStream certOut = new FileOutputStream(certFile);
        try {
            certOut.write(certText.getBytes(CharsetUtil.US_ASCII));
            certOut.close();
            certOut = null;
        }
        finally {
            if (certOut != null) {
                safeClose(certFile, certOut);
                safeDelete(certFile);
                safeDelete(keyFile);
            }
        }
        return new String[] { certFile.getPath(), keyFile.getPath() };
    }

然后我们再看到这一句：

final File keyFile = File.createTempFile("keyutil_" + fqdn + '_', ".key");

里面的File.createTempFile()是创建一个临时文件，在哪里创建呢？
对于Linux而已，默认就是/tmp目录中，所以普通用户默认是没有写权限的。导致出现了今天这个错误。（我家里虚拟机之所以可以，是因为我改过它的权限）。

这个情况和elasticsearch真的非常类似。
大家明明都知道不建议以root用户来启动项目，而偏偏又要往/tmp这个文件临时写东西，关键是这个还不能通过配置来改变。只能。。。：
要么修改权限，要么以root身份来执行。

这不是什么大问题，但是很浪费时间。。。

参考地址：

https://github.com/Graylog2/graylog2-server/issues/2054

https://netty.io/4.0/xref/io/netty/handler/ssl/util/SelfSignedCertificate.html