Xmemcached使用【二】

Xmemcached的Wiki：http://code.google.com/p/xmemcached/wiki/User_Guide_zh#与Hibernate-memcached集成

一、软件环境

     1、memcached-1.4.5

     2、xmemcached-1.4.1

二、简单示例

    1、设置节点权重

        如果需要编程设置，通过下面代码：

 MemcachedClientBuilder builder = new    

 XMemcachedClientBuilder(AddrUtil.getAddresses("localhost:12000 localhost:12001"),new int[]{1,3});

 MemcachedClient memcachedClient=builder.build();

传入一个int数组，里面的元素就是节点对应的权重值，比如这里设置"localhost:12000"节点的权重为1，而"localhost:12001"的权重为3。注意，xmemcached的权重是通过复制连接的多个引用来实现的，比如权重为3，那么就复制3个同一个连接的引用放在集合中让MemcachedSessionLocator查找。

 

改变节点权重，可以通过setServerWeight方法：

public interface XMemcachedClientMBean{

             ....

         /**

         * Set a memcached server's weight

         *

         * @param server

         * @param weight

         */

        public void setServerWeight(String server, int weight);

   }

    2、使用二进制协议

       Memcached 1.4开始正式启用二进制协议，xmemcached 1.2开始支持二进制协议，启用这一特性也非常简单，设置相应的CommandFactory即可：

    MemcachedClientBuilder builder = new    XMemcachedClientBuilder(AddrUtil.getAddresses("localhost:12000 localhost:12001"),new int[]{1,3});

  builder.setCommandFactory(new BinaryCommandFactory());//use binary protocol

  MemcachedClient memcachedClient=builder.build();

    3、动态添加/删除节点

       MemcachedClient client=new XMemcachedClient(AddrUtil.getAddresses("server1:11211 server2:11211"));

   //Add two new memcached nodes

   client.addServer("server3:11211 server4:11211");

   //Remove memcached servers

   client.removeServer("server1:11211 server2:11211");

    4、NIO连接池

        Xmemcached是基于java nio的client实现，默认对一个memcached节点只有一个连接，这在通常情况下已经有非常优异的表现。但是在典型的高并发环境下,nio的  单连接也会遇到性能瓶颈。因此XMemcached支持设置nio的连接池，允许建立多个连接到同一个memcached节点，但是请注意，这些连接之间是不同步的，因此你的应用需要自己保证数据更新的同步，启用连接池可以通过下面代码：

MemcachedClientBuilder builder = new    XMemcachedClientBuilder(AddrUtil.getAddresses("localhost:12000"));

builder.setConnectionPoolSize(5);

    5、Failure模式和standby节点

从1.3版本开始，xmemcached支持failure模式。所谓failure模式是指，当一个memcached节点down掉的时候，发往这个节点的请求将直接失败，而不是发送给下一个有效的memcached节点。具体可以看memcached的文档。默认不启用failure模式，启用failure模式可以通过下列代码：MemcachedClientBuilder builder=……； builder.setFailureMode(true);

不仅如此，xmemcached还支持主辅模式，你可以设置一个memcached的节点的备份节点，当主节点down掉的情况下，会将本来应该发往主节点的请求转发给standby备份节点。使用备份节点的前提是启用failure模式。备份节点设置如下：MemcachedClient builder=new XmemcachedClientBuilder(AddrUtil.getAddressMap("localhost:11211,localhost:11212 host2:11211,host2:11212"));

 

    6、与Kestrel交互

       Kestrel是twitter开源的一个scala写的简单高效MQ，它支持 memcached文本协议，但是并不完全兼容，例如它不支持flag，导致很多利用flag做序列化的客户端无法正常运作。因此Xmemcached特意提供了KestrelCommandFactory?用于支持Kestrel。使用KestrelCommandFactory?即可拥有如下好处：默认关闭get优化，因为kestrel不支持bulk get；支持kestrel的阻塞获取和可靠获取；允许向kestrel存储任意java序列化类型。设置KestrelCommandFactory:

MemcachedClientBuilder builder = new    XMemcachedClientBuilder(AddrUtil.getAddresses("localhost:12000 localhost:12001"),new int[]{1,3});

builder.setCommandFactory(new KestrelCommandFactory());

MemcachedClient memcachedClient=builder.build();

 

关于最后一点需要补充说明，由于kestrel不支持flag，因此xmemcached在存储的数据之前加了4个字节的flag，如果你的全部应用都使用xmemcached，那么没有问题，如果使用其他clients，会有兼容性的问题，因此Xmemcached还允许关闭这个功能，通过client.setPrimitiveAsString(true);设置为true后，原生类型都将存储为字符串，而序列化类型将无法存储了。

 

    7、与tokyotyrant交互

       通过使用TokyoTyrantTranscoder就可以跟TokyoTyrant进行交互，但是由于TokyoTyrant对memcached文本协议的flag,exptime不支持，因此内部TokyoTyrantTranscoder加了4个字节的flag在value前面，如果你的全部应用都使用xmemcached，那么没有问题，如果使用其他clients，会有兼容性的问题，这一点与跟kestrel交互相同。

MemcachedClientBuilder builder = new XMemcachedClientBuilder(AddrUtil.getAddresses("localhost:12000 localhost:12001"),new int[]{1,3});

builder.setTranscoder(new TokyoTyrantTranscoder());

MemcachedClient memcachedClient=builder.build();

    8、数据压缩

        memcached存储大数据的效率是比较低的，当数据比较大的时候xmemcached会帮你压缩在存储，取出来的时候自动解压并反序列化，这个大小阀值默认是16K，可以通过Transcoder接口的setCompressionThreshold(1.2.1引入)方法修改阀值，比如设置为1K：memcachedClient.getTranscoder()).setCompressionThreshold(1024);

    9、packZeros

         XMemcached的序列化转换器在序列化数值类型的时候有个特殊处理，如果前面N个字节都是0，那么将会去除这些0，缩减后的数据将更小，例如数字3序列化是0x0003，那么前面3个0将去除掉成一个字节0x3。反序列化的时候将自动在前面根据数值类型补0。这一特性默认是开启的，如果考虑到与其他client兼容的话需要关闭此特性可以通过：memcachedClient.getTranscoder()).setPackZeros(false);

    10、sanitizeKey    

         在官方客户端有提供一个sanitizeKeys选项，当选择用URL当key的时候，MemcachedClient会自动将URL encode再存储。默认是关闭的，想启用可以通过：   

         memcachedClient.setSanitizeKeys(true);

 

配置选项参数表:

属性名	值
servers	memcached节点列表，形如“主节点1:port,备份节点1:port 主节点2:port,备份节点2:port“的字符串，可以不设置备份节点，主备节点逗号隔开，不同分组空格隔开。
weights	与servers对应的节点的权重
authInfoMap	授权验证信息，仅在xmemcached 1.2.5及以上版本有效
connectionPoolSize	nio连接池大小，默认为1
commandFactory	协议工厂，net.rubyeye.xmemcached.command.BinaryCommandFactory,TextCommandFactory(默认),KestrelCommandFactory
sessionLocator	分布策略，一致性哈希net.rubyeye.xmemcached.impl.KetamaMemcachedSessionLocator或者ArraySessionLocator(默认)

 

transcoder	序列化转换器，默认使用net.rubyeye.xmemcached.transcoders.SerializingTranscoder，更多选项参见javadoc
bufferAllocator	IoBuffer分配器，默认为net.rubyeye.xmemcached.buffer.SimpleBufferAllocator，可选CachedBufferAllocator(不推荐)
failureMode	是否启用failure模式，true为启用，默认不启用

 完整代码示例

memcached.properties

#127.0.0.1:11211 127.0.0.1:11212 
addressList=127.0.0.1:11211 127.0.0.1:11212 127.0.0.1:11213 127.0.0.1:11214
weights=1,2,3,4
#1-30
connectionPoolSize=15
failureMode=true
#----------------------------------------------------------------------------------------------------------------#
#连接池大小即客户端个数
memcached.connectionPoolSize=5
memcached.failureMode=true
#server1
memcached.server1.host=127.0.0.1
memcached.server1.port=11211
memcached.server1.weight=4
#server2
memcached.server2.host=127.0.0.1
memcached.server2.port=11212
memcached.server2.weight=3				
#server3
memcached.server3.host=127.0.0.1
memcached.server3.port=11213
memcached.server3.weight=2					
#server4
memcached.server4.host=127.0.0.1
memcached.server4.port=11214
memcached.server4.weight=1		 

 

如果配置主备，需要修改 addressList

addressList=127.0.0.1:11211,127.0.0.1:11212 127.0.0.1:11213,127.0.0.1:11214

 

XmemcachedUtil.java

package com.wy.util;

import java.io.IOException;
import java.util.Properties;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.google.code.yanf4j.util.ResourcesUtils;

import net.rubyeye.xmemcached.MemcachedClient;
import net.rubyeye.xmemcached.MemcachedClientBuilder;
import net.rubyeye.xmemcached.XMemcachedClientBuilder;
import net.rubyeye.xmemcached.auth.AuthInfo;
import net.rubyeye.xmemcached.command.BinaryCommandFactory;
import net.rubyeye.xmemcached.impl.ElectionMemcachedSessionLocator;
import net.rubyeye.xmemcached.utils.AddrUtil;

/**
 * Xmemcached工具类
 * 
 * @author wy
 * @serial 2013-08-12
 */
public class XmemcachedUtil {
	private static Logger log = LoggerFactory.getLogger(XmemcachedUtil.class);
	private static MemcachedClientBuilder builder = null;
	private static MemcachedClient memcacheddClient = null;
	private static Properties prop = null;
	
	static {
		try {
			prop = ResourcesUtils.getResourceAsProperties("memcached.properties");
		} catch (IOException e) {
			e.printStackTrace();
			log.error("请配置memcached服务地址!");
		}
		if(prop != null){
			String addrs = prop.getProperty("addressList");
			String weight = prop.getProperty("weights");
			if(weight != null){
				String[] weightsArray = weight.split(",");
				int len = weightsArray.length;
				int[] weights = new int[len];
				for(int i=0;i<len;i++){
					weights[i] = Integer.parseInt(weightsArray[i]);
				}
				//builder = new XMemcachedClientBuilder(AddrUtil.getAddresses(addrs), weights);
				//standby 主备模式
				builder = new XMemcachedClientBuilder(AddrUtil.getAddressMap(addrs), weights);
			}else{
				builder = new XMemcachedClientBuilder(AddrUtil.getAddressMap(addrs));
			}
		}else{
			log.error("请配置memcached服务地址!");
		}
		
		//设置连接池大小，即客户端个数
		//xmemcached是基于nio的，通常不要设置连接池，一个连接可以支撑绝大多数应用。
		//除非你的应用并发规模特别大，比如我压测的时候测试100个线程或者300个线程并发的时候，连接池才能显示出优势。
		builder.setConnectionPoolSize(Integer.valueOf(prop.getProperty("connectionPoolSize")));
		//宕机报警
		builder.setFailureMode(Boolean.valueOf(prop.getProperty("failureMode")));
		//使用二进制协议，默认使用的TextCommandFactory即文本协议
		builder.setCommandFactory(new BinaryCommandFactory());
		
		//1.Standard Hash, hash(key) mod server_count (余数分布)
		//默认使用余数分布
		//2.Consistent Hash(一致性哈希)
		//builder.setSessionLocator(new KetamaMemcachedSessionLocator());
		//3.Election Hash(选举散列)
		builder.setSessionLocator(new ElectionMemcachedSessionLocator());		
		//SASL验证, 客户端授权验证授权验证,仅支持二进制协议
		builder.addAuthInfo(AddrUtil.getOneAddress("localhost:11213"), AuthInfo.typical("wy", "wy"));
	}
	
	/**
	 * 获取MemcachedClient
	 * 
	 * @return
	 */
	public static MemcachedClient getMemcachedClient(){
		try {
			if(memcacheddClient == null){
				memcacheddClient = builder.build();
				//当选择用URL当key的时候，MemcachedClient会自动将URL encode再存储。默认是关闭的
				memcacheddClient.setSanitizeKeys(true);
				//memcached存储大数据的效率是比较低的，当数据比较大的时候xmemcached会帮你压缩在存储，取出来的时候自动解压并反序列化，这个大小阀值默认是16K
				memcacheddClient.getTranscoder().setCompressionThreshold(32768);
				return memcacheddClient;
			}
		} catch (IOException e) {
			e.printStackTrace();
			log.error("获取MemcachedClient失败!");
		}
		return null;
	}
	
	/**
	 * 关闭MemcachedClient
	 * 
	 * @param memcachedClient
	 */
	public static void close(MemcachedClient memcachedClient){
		if(memcachedClient != null && !memcachedClient.isShutdown()){
			try {
				memcachedClient.shutdown();
			} catch (IOException e) {
				e.printStackTrace();
			}
		}
	}
}

 XmemcachedTest.java

package com.wy;

import java.net.URL;
import java.util.concurrent.TimeoutException;

import net.rubyeye.xmemcached.GetsResponse;
import net.rubyeye.xmemcached.KeyIterator;
import net.rubyeye.xmemcached.MemcachedClient;
import net.rubyeye.xmemcached.exception.MemcachedException;
import net.rubyeye.xmemcached.utils.AddrUtil;

import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import com.wy.util.XmemcachedUtil;

/**
 * 
 * 
 * @author wy
 *
 */
public class XmemcachedTest {
	private static Logger log = LoggerFactory.getLogger(XmemcachedTest.class);
	private MemcachedClient memcachedClient = null;

	@Before
	public void setUp() throws Exception {
		long start1 = System.currentTimeMillis();
		memcachedClient = XmemcachedUtil.getMemcachedClient();
		long end1 = System.currentTimeMillis();
		System.out.println("创建MemcachedClient消耗时间:"+(end1 - start1));
    }

    @After
    public void tearDown() throws Exception {
    	XmemcachedUtil.close(memcachedClient);
    }
    
    @Test
	public void main() {
		try {
			memcachedClient.flushAll();
			long start2 = System.currentTimeMillis();
			memcachedClient.set("wy", 3600, "wy test");
			long end2 = System.currentTimeMillis();
			System.out.println("向memcached中存值消耗时间:"+(end2 - start2));
			
			long start3 = System.currentTimeMillis();
			System.out.println(memcachedClient.get("wy").toString());
			long end3 = System.currentTimeMillis();
			System.out.println("从memcached中取值消耗时间:"+(end3 - start3));
			
			System.out.println(memcachedClient.add("wy123", 3600, "123") +" "+ memcachedClient.get("wy123").toString());
			
			memcachedClient.append("wy", "_123");
			System.out.println(memcachedClient.get("wy").toString());
			
			memcachedClient.replace("wy", 3600, "wy_123");
			System.out.println(memcachedClient.get("wy"));
			
			/** 
             * Memcached是通过cas协议实现原子更新，所谓原子更新就是compare and set， 
             * 原理类似乐观锁，每次请求存储某个数据同时要附带一个cas值， memcached比对这个cas值与当前存储数据的cas值是否相等， 
             * 如果相等就让新的数据覆盖老的数据，如果不相等就认为更新失败， 这在并发环境下特别有用 
             */
			GetsResponse<Integer> result = memcachedClient.get("wy");
			long cas = result.getCas();
			//将wy的值修改为cas
			if(!memcachedClient.cas("wy", 3600, "cas", cas)){
				System.err.println("cas error");
			}
			
			memcachedClient.set("http://www.baidu.com", 3600, "http://www.baidu.com");
			
			KeyIterator iterator = memcachedClient.getKeyIterator(AddrUtil.getOneAddress("127.0.0.1:11213"));
			while(iterator.hasNext()){
				 String key = iterator.next();
				 System.out.println("key = "+key);
			}
			
			System.out.println("value = " + memcachedClient.get("wy"));
			System.out.println("value = " + memcachedClient.get("http://www.baidu.com"));
			
//			memcachedClient.delete("wy");
//			System.out.println(memcachedClient.get("wy").toString());
			
		} catch (TimeoutException e) {
			e.printStackTrace();
		} catch (InterruptedException e) {
			e.printStackTrace();
		} catch (MemcachedException e) {
			e.printStackTrace();
		}
	}
    
}

 

 

三、与Spring框架集成    

    applicationContext.xml

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context"
	xmlns:p="http://www.springframework.org/schema/p"
	xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
		http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context.xsd">
	
	<context:property-placeholder location="memcached.properties" />
	
	<bean id="memcachedClientBuilder" class="net.rubyeye.xmemcached.XMemcachedClientBuilder"
		p:connectionPoolSize="${memcached.connectionPoolSize}" p:failureMode="${memcached.failureMode}">
		<!-- XMemcachedClientBuilder have two arguments.First is server list,and second is weights array. -->
		<constructor-arg>
			<list>
				<bean class="java.net.InetSocketAddress">
					<constructor-arg>
						<value>${memcached.server1.host}</value>
					</constructor-arg>
					<constructor-arg>
						<value>${memcached.server1.port}</value>
					</constructor-arg>
				</bean>
				<bean class="java.net.InetSocketAddress">
					<constructor-arg>
						<value>${memcached.server2.host}</value>
					</constructor-arg>
					<constructor-arg>
						<value>${memcached.server2.port}</value>
					</constructor-arg>
				</bean>
				<bean class="java.net.InetSocketAddress">
					<constructor-arg>
						<value>${memcached.server3.host}</value>
					</constructor-arg>
					<constructor-arg>
						<value>${memcached.server3.port}</value>
					</constructor-arg>
				</bean>
				<bean class="java.net.InetSocketAddress">
					<constructor-arg>
						<value>${memcached.server4.host}</value>
					</constructor-arg>
					<constructor-arg>
						<value>${memcached.server4.port}</value>
					</constructor-arg>
				</bean>
			</list>
		</constructor-arg>
		<!-- server's weights -->
		<constructor-arg>
			<list>
				<value>${memcached.server1.weight}</value>
				<value>${memcached.server2.weight}</value>
				<value>${memcached.server3.weight}</value>
				<value>${memcached.server4.weight}</value>
			</list>
		</constructor-arg>
		<!-- 协议 -->
		<property name="commandFactory">
			<bean class="net.rubyeye.xmemcached.command.TextCommandFactory" />
		</property>
		<!-- 分布策略  -->
		<property name="sessionLocator">
			<bean class="net.rubyeye.xmemcached.impl.KetamaMemcachedSessionLocator" />
		</property>
		<!-- 序列化转换器  -->
		<property name="transcoder">
			<bean class="net.rubyeye.xmemcached.transcoders.SerializingTranscoder" />
		</property>
	</bean>
	<!-- Use factory bean to build memcached client -->
	<bean id="memcachedClient" factory-bean="memcachedClientBuilder"
		factory-method="build" destroy-method="shutdown" />
		
</beans>

 MemcachedSpringTest.java

package com.wy;

import static junit.framework.Assert.*;

import java.util.concurrent.TimeoutException;

import net.rubyeye.xmemcached.MemcachedClient;
import net.rubyeye.xmemcached.exception.MemcachedException;

import org.junit.Before;
import org.junit.Test;
import org.springframework.context.ApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;

public class MemcachedSpringTest {

	private ApplicationContext app;
	private MemcachedClient memcachedClient;

	@Before
	public void init() {
		app = new ClassPathXmlApplicationContext("applicationContext.xml");
		memcachedClient = (MemcachedClient) app.getBean("memcachedClient");
	}

	@Test
	public void test() {
		try {
			// 设置/获取
			memcachedClient.set("wy", 36000, "set/get");
			assertEquals("set/get", memcachedClient.get("wy"));

			// 替换
			memcachedClient.replace("wy", 36000, "replace");
			assertEquals("replace", memcachedClient.get("wy"));

			// 移除
			memcachedClient.delete("wy");
			assertNull(memcachedClient.get("wy"));
			
		} catch (TimeoutException e) {
			e.printStackTrace();
		} catch (InterruptedException e) {
			e.printStackTrace();
		} catch (MemcachedException e) {
			e.printStackTrace();
		}
	}
}

 

 

四、SASL验证

   

 参考http://lguan.iteye.com/blog/1279537