手写分布式存储系统v0.1版本

引言

这是手写分布式存储系统v0.1版本，只有一个目标就是支持通过tcp接收数据并落地到磁盘文件(单机模式)，那接下来就开始吧

一、设计

实现一个系统，设计是最过瘾的过程没有之一，类似你搭积木前在脑海设计构建一副大致的“雏形”，只有有了这个东西之后才能够指导最终实现的方向以及确保不会偏离的太差。这里我对v0.1的预期是如下的，只要客户端能够通过tcp将数据请求到Linux机器的端口，咱们的v0.1版本就能够监听到并且将数据落地到磁盘，只需要实现这个功能就可以了。

二、代码实现

这个功能中会跟网络和写磁盘打交道，那直接用Netty现成的包就好了，至于写磁盘的话用JDK原生自带的就够了。大致抽象出两个对应的接口以及实现，如下

public interface NetService {
    void start();
    void stop();
}

public class NetServiceImpl implements NetService{

    private static final Logger LOG = LoggerFactory.getLogger(NetServiceImpl.class);

    private EventLoopGroup bossGroup = null;
    private EventLoopGroup workerGroup = null;

    public void start() {
        //bossGroup就是parentGroup，是负责处理TCP/IP连接的
        EventLoopGroup bossGroup = new NioEventLoopGroup(1);
        //workerGroup就是childGroup,是负责处理Channel(通道)
        EventLoopGroup workerGroup = new NioEventLoopGroup(30);

        try {
            ServerBootstrap bootstrap = new ServerBootstrap()
                    .group(bossGroup, workerGroup)
                    .channel(NioServerSocketChannel.class)
                    //初始化服务端可连接队列,指定了队列的大小128
                    .option(ChannelOption.SO_BACKLOG, 128)
                    //通过NoDelay禁用Nagle,使消息立即发出去，不用等待到一定的数据量才发出去
                    .option(ChannelOption.TCP_NODELAY, true)
                    //保持长连接
                    .childOption(ChannelOption.SO_KEEPALIVE, true)
                    .handler(new LoggingHandler(LogLevel.INFO))
                    .childHandler(new ServerInitializer());
            ChannelFuture future = bootstrap.bind(8888).sync();
            future.channel().closeFuture().sync();
        } catch (Exception e){

        }finally {
            bossGroup.shutdownGracefully();
            workerGroup.shutdownGracefully();
        }
    }

    public void stop() {
        bossGroup.shutdownGracefully();
        workerGroup.shutdownGracefully();
    }
}

再写下数据存储相关的接口和类如下

public interface DataStorage<T> {
    void save(T t) throws Exception;
}

public class LocalDataStorageImpl implements DataStorage<String>{

    private static MappedByteBuffer mappedByteBuffer;

    private static Integer _1Gb = 1024*1024*1024;

    private static Integer _1MB = 1024*1024;

    public LocalDataStorageImpl() {
        try {
            FileChannel fileChannel = new RandomAccessFile("./testWrite", "rw").getChannel();
            mappedByteBuffer = fileChannel.map(FileChannel.MapMode.READ_WRITE, 0, _1MB);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    @Override
    public void save(String data) throws Exception{
        System.out.println("start writeDataToFile data is :"+data);
        mappedByteBuffer.put(data.getBytes());
        System.out.println("writeDataToFile end!");
    }
}

再实现Netty相关的逻辑

@ChannelHandler.Sharable
public class ServerHandler extends SimpleChannelInboundHandler<String> {

    private static final Logger LOG = LoggerFactory.getLogger(ServerHandler.class);

    private DataStorage dataStorage = new LocalDataStorageImpl();


    @Override
    public void channelActive(ChannelHandlerContext channelHandlerContext) throws Exception {
        channelHandlerContext.write("Welcome to sherlock home!");
        channelHandlerContext.write("It is "+ new Date()+"\n");
        channelHandlerContext.flush();
    }

    @Override
    public void channelRead0(ChannelHandlerContext ctx, String request) throws Exception {
        LOG.info("========readdata, request is {}=========", request);
        //异步通过专门的EventLoop线程池进行处理

        dataStorage.save(request);
        String response;
        boolean close = false;
        if (request.isEmpty()) {
            response = "Please type something.\r\n";
        } else if ("bye".equals(request.toLowerCase())) {
            response = "Have a good day!\r\n";
            close = true;
        } else {
            response = "Did you say '" + request + "'?\r\n";
        }

        ChannelFuture future = ctx.write(response);

        if (close) {
            future.addListener(ChannelFutureListener.CLOSE);
        }
    }

    @Override
    public void channelReadComplete(ChannelHandlerContext ctx) {
        ctx.flush();
    }

    @Override
    public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
        cause.printStackTrace();
        ctx.close();
    }
}

public class ServerInitializer extends ChannelInitializer<SocketChannel> {

    private static final Logger LOG = LoggerFactory.getLogger(ServerInitializer.class);

    private static final StringDecoder DECODER = new StringDecoder();
    private static final StringEncoder ENCODER = new StringEncoder();

    private static final ServerHandler SERVER_HANDLER = new ServerHandler();


    @Override
    public void initChannel(SocketChannel socketChannel) throws Exception {
        ChannelPipeline pipeline = socketChannel.pipeline();

        pipeline.addLast(new DelimiterBasedFrameDecoder(8192, Delimiters.lineDelimiter()));
        pipeline.addLast(DECODER);
        pipeline.addLast(ENCODER);

        pipeline.addLast(SERVER_HANDLER);
    }
}

最后，咱们再来实现主函数逻辑

public class Main {
    public static void main(String[] args) {
        NetService netService = new NetServiceImpl();
        netService.start();
    }
}

基本上就差不多了，代码优化往后放放，现在嘛，能跑就行☺️

三、运行调试

启动服务后，咱们通过下列指令往接口插入数据

(echo 'hello'; sleep 2) | telnet 127.0.0.1 8888
(echo 'sherlock'; sleep 2) | telnet 127.0.0.1 8888
(echo 'thanks'; sleep 2) | telnet 127.0.0.1 8888
(echo 'are you ok?'; sleep 2) | telnet 127.0.0.1 8888

通过下面控制台的信息能够看到接收到完整的数据了，说明v0.1版本通过socket端口读取数据的链路是正常的

再看看本地磁盘文件，通过打印出来能够看到数据是已经落到磁盘的

四、小结

以上就是实现的整个过程，代码不可谓不粗糙，不过咱们讲究的就是一个莽，快速闭环看到效果才是最重要的，至于优化嘛，放到后面的版本慢慢优化~