Java 利用Linux Openssl 库计算文件校验码

在传统Java编程中, 我们可以另用ProcessBuilder类来创建系统进程或者Runtime类来取得JVM的接口对象运行我们指定的系统命令, 并且可以通过读入器来获取系统的out以及err输出.这里有一个简单的例子, 没有用过的同学可以参考一下. 

http://wuhongyu.iteye.com/blog/461477 

由于前段时间有这样的一个需求, 尽可能快的算出Hadoop系统里的MD码, 我曾经想过另用Java的Digest类来读取文件, 然后工作交给Digest类来完成, 可没想到的是Java的这个类算MD校验码，效率及其的低下，我在XP上算50M的文件, 居然花费8秒钟的时间.直接放弃. 这个Demo我也顺便推荐一下: 

http://www.javablogging.com/?s=Java+Digest+SHA1+MD5 

后来google该问题的解决方案, 有一个比较有意思的贴子Fast MD5 Implementation in Java, 呵呵, 由于时间关系, 没来的急看,分享一下： 

http://www.twmacinta.com/myjava/fast_md5.php 

Fast MD5.. 总给我一个不太舒服的感觉, 最后还是决定放弃使用. 由于Hadoop运行于Linux 环境, 回过神来我想到了Linux上的openssl库. 大伙都知道openssl 里其中就有计算数据的安全通信命令. $cat XXX | openssl sha1. 

这里有一个问题, 由于hadoop的文件是存在于集群的, 而openssl 是接受本地文件作为参数的, 我可以把文件从HDFS里读出, 然后再构造文件供openssl 计算. 不过这种方式太囧. 
我的解决方法是另用Runtime获取运行对象, 直接执行openssl sha1命令, 其实当系统执行该命令的时候会等待输入的. 此时我可以通过我的文件输入流把数据打印到等待输入的输出流上。 

 // TODO this is a job. not a filedescriptor
     protected void handle(FileDescriptor job) throws IOException {
         if (job == null || job.path == null) {
             log.warn("Invalid job: " + job);
         }

         // wait for GC.
         Configuration conf = new Configuration();
         Path path = new Path(job.path);
         FileSystem fs = path.getFileSystem(conf);

         if (!fs.exists(path) || !fs.isFile(path)) {
             log.warn("*HADLEJOB file does not exist for path: " + path);
             return;
         }

         InputStream is = fs.open(path);
         // Run a command
         String cmd = "openssl sha1";
         boolean completed = false;

         Process process = null; // Sub process used to execute the command
         int exitCode;
         process = Runtime.getRuntime().exec(cmd);

         BufferedOutputStream os = new BufferedOutputStream(process.getOutputStream(), 4096);
         byte buf[] = new byte[4096];
         while (true) {
             int bytesRead = is.read(buf);
             if (bytesRead == -1) {
                 break;
             }
             os.write(buf, 0, bytesRead);
         }
         os.flush();
         os.close();

         final BufferedReader errReader =
             new BufferedReader(new InputStreamReader(process.getErrorStream()));

         BufferedReader inReader =
             new BufferedReader(new InputStreamReader(process.getInputStream()));

         final StringBuffer errMsg = new StringBuffer();
         // read error and input streams as this would free up the buffers
         // free the error stream buffer
         Thread errThread = new Thread(){

             public void run() {
                 try {
                     String line = errReader.readLine();
                     while ((line != null) && !isInterrupted()) {
                         errMsg.append(line);
                         errMsg.append(System.getProperty("line.separator"));
                         line = errReader.readLine();
                     }
                 } catch (IOException ioe) {
                     log.warn("Error reading the error stream", ioe);
                 }
             }

         };
         try {
             errThread.start();
         } catch (IllegalStateException ise) {
         }

         try {
             String line = inReader.readLine();
             if (line == null) {
                 throw new IOException("Exception a line not the end of stream");
             }
             job.set_sha1(line.trim());
             // clear the input stream buffer
             line = inReader.readLine();
             while (line != null) {
                 line = inReader.readLine();
             }

             // wait for the process to finish and check the exit code
             exitCode = process.waitFor();

             try {
                 errThread.join();
             } catch (InterruptedException ie) {
                 log.warn("Interrupted while reading the error stream", ie);
             }
             completed = true;
             if (exitCode != 0) {
                 throw new IOException(exitCode + errMsg.toString());
             }
         } catch (InterruptedException ie) {
             throw new IOException(ie.toString());
         } finally {
             // close the input stream
             try {
                 inReader.close();
             } catch (IOException e) {
                 log.warn("Error whilke closing the input stream", e);
             }
             if (!completed) {
                 errThread.interrupt();
             }
             try {
                 errReader.close();
             }catch(IOException ioe){
                 log.warn("Error while closing the error stream", ioe);
             }
             process.destroy();
         }
     }

该执行方法运行于线程池中, 下一个贴子我将记录真正应用中线程池的实现, 提供线程池的大小设置, shrink等实现.