java自定义hive sparksql thriftServer连接池

由于公司需要做hive和sparksql查询功能，且都通过thrift server进行查询，那么如何有效的创建链接并缓存链接就是比较关键的步骤了。

thrift connection类似于jdbc connection只是引入的driver class不同罢了。由于公司集群开了kerbrose，所以需要使用代理用户进行通信，每一次登录的用户都会有一个clusterName(代理用户的名称，集群名)。 所以每一个connection就可能都不一样了。  这里需要一个动态的连接池来保存已经创建和正在使用的链接，并需要有效的释放链接的机制。

废话不多说，直接上关键代码：

1.基础类  ObjectPool 

public abstract class ObjectPool {

    static Logger logger = org.slf4j.LoggerFactory.getLogger(ObjectPool.class);
    private static long expirationTime;

    private static HashMap> locked;
    private static HashMap> unlocked;

    public ObjectPool() {
        if (locked == null) {
            locked = new HashMap<>();
        }
        if (unlocked == null) {
            unlocked = new HashMap<>();
        }
        expirationTime = 30 * 60 * 1000; // 30 minute 过期
    }

    protected abstract ConnectionBean create();

    public abstract boolean validate(ConnectionBean o);

    public abstract void expire(ConnectionBean o);

    public ConnectionBean get(String clusterName) {
        synchronized (locked) {
            String key = Thread.currentThread().getName() + clusterName;
            logger.info("【POOL】 lock the LOCKED map, the clusterName is {}", clusterName);
            long now = System.currentTimeMillis();
            ConcurrentLinkedQueue beans;
            if (!unlocked.isEmpty()) {
                beans = unlocked.get(clusterName);
                if (beans != null) {
                    while (!beans.isEmpty()) {
                        // 获取头元素，并在资源队列中删除头元素
                        ConnectionBean bean = beans.poll();
                        // 如果头元素的时间过期了，那么关闭连接
                        if (now - bean.getUpdateTime() > expirationTime) {
                            logger.info("【POOL】 the connection is out of time ,bean time is {}", bean.getUpdateTime());
                            // 释放
                            expire(bean);
                            bean = null;
                        } else {
                            if (validate(bean)) {
                                logger.info("【POOL】 get the connection from poll and the clusterName is {}",
                                        clusterName);
                                bean.setUpdateTime(now);
                                // 放入锁定的队列中并返回 锁定队列需要
                                locked.get(clusterName).put(key, bean);
                                return bean;
                            } else {
                                // 如果链接已经关闭
                                unlocked.remove(clusterName);
                                expire(bean);
                                bean = null;
                            }
                        }
                    }
                }
            }
            // 由于unlock可能为空，所以初始化对应的clusterName
            unlocked.put(clusterName, new ConcurrentLinkedQueue());
            // 如果没有链接则新建一个操作
            ConnectionBean bean = create();
            logger.info("【POOL】 the pool could not provide a connection, so create it,clusterName is {}",
                    clusterName);
            if (locked.get(clusterName) == null) {
                logger.info("【POOL】 the clusterName in pool is null, create a new Map in LOCKED, clusterName is {}",
                        clusterName);
                locked.put(clusterName, new ConcurrentHashMap());
            }
            locked.get(clusterName).put(key, bean);
            return bean;

        }
    }


    public void release(String clusterName) {
        synchronized (locked) {
            String key = Thread.currentThread().getName() + clusterName;
            ConcurrentHashMap connectionBeans = locked.get(clusterName);
            ConnectionBean bean = connectionBeans.get(key);
            connectionBeans.remove(key);
            bean.setUpdateTime(System.currentTimeMillis());
            unlocked.get(clusterName).add(bean);
            System.out.println("......................................................" + Thread.currentThread().getName());
        }
    }
}

在这个连接池中 包含了locked 和 unlocked 两个连接队列和concurrentHashMap（释放链接主要为了对应key的释放，不使用队列出列）。

使用get 获取指定的connection，首先 如果连接池中有，且存活时间没有 超过 expirationTime 默认30min 则会为用户使用，但是超过30min则会回收。当获取到链接会，会和当前ThreadName以及集群名称作为key值存储至locked map中。

2.  ConnectionBeanPool 这个对象继承了抽象类 并实现相应操作

public class ConnectionBeanPool extends ObjectPool {

    private String url;  // 链接url
    private String usr;  // 账户名
    private String pwd;  // 密码

    public ConnectionBeanPool(String driver, String url, String usr, String pwd) {
        super();
        try {
            Class.forName(driver).newInstance();
        } catch (Exception e) {
            e.printStackTrace();
        }
        this.url = url;
        this.usr = usr;
        this.pwd = pwd;
    }

    @Override
    protected ConnectionBean create() {
        try {
            ConnectionBean connectionBean = new ConnectionBean();
            Connection connection = DriverManager.getConnection(url, usr, pwd);
            if (connection == null) {
                System.out.print("null connection");
            }
            connectionBean.setConnection(connection);
            connectionBean.setUpdateTime(new Date().getTime());
            return connectionBean;
        } catch (SQLException e) {
            e.printStackTrace();
            return null;
        }
    }

    @Override
    public void expire(ConnectionBean o) {
        try {
            o.getConnection().close();
        } catch (SQLException e) {
            e.printStackTrace();
        }
    }

    @Override
    public boolean validate(ConnectionBean o) {
        try {
            return (!(o.getConnection()).isClosed());
        } catch (SQLException e) {
            e.printStackTrace();
            return false;
        }
    }
}

3. bean

public class ConnectionBean {

    private Connection connection; // 链接信息
    private long updateTime;       // 更新时间

    public Connection getConnection() {
        return connection;
    }

    public void setConnection(Connection connection) {
        this.connection = connection;
    }

    public long getUpdateTime() {
        return updateTime;
    }

    public void setUpdateTime(long updateTime) {
        this.updateTime = updateTime;
    }
}

目前还没对链接数量进行限定。未来需要设置一个最大连接量，如果超过则让请求等待或者返回错误信息。