这个版本是通过截取桌面图片来显示远程桌面,有一个很明显的问题就是图片大小,我尝试过一些方式,但是效果都不太好,所以准备放弃这种方式进行最终实现,具体说明下我使用了那些方式去缩小图片大小...
1. 直接使用压缩工具Thumbnails来压缩宽高和质量,这样确实能压缩图片大小,客户端压缩成一半宽高,控制端再放大两倍,这样确实可行,但是图片模糊,有点像向日葵软件的免费版,最模糊的那种状态
2. 将图片进行灰化,就是去除彩色,这样也确实可以,但是结果也就是没有了彩色....,看久了受不了
3. 想通过BufferedImage取两张图片的差值,将差值发送到控制端,控制端再去填充差值,也确实可以,但是界面会闪烁,没有深究原因
所以综上所述,想放弃这种方案,换ffmpeg取帧和差值帧传输,之前研究过ffmpeg,这个是处理音视频的,使用起来也比较简单,ffmpeg在处理视频流的时候,或者对于视频直播来说的最优方案就是取图片差值传输(有什么关键帧,差值帧,太久之前搞的ffmpeg了,名词忘了,但是意思差不多吧),那么这样不需要我去处理图片差值和回填像素,肯定比我自己处理要好... 同样是很久之前我也用ffmpeg来写过桌面控制,通过它来取帧压缩后传输,效果还不错,没有什么延迟和闪烁。就用它继续搞一下吧...
服务器端说明:只有一个类,做的事很简单,就是启动一个长链接服务器。但是这个依托了我写的一个IM框架(基于netty),至于这个框架源码我放在码云,需要的自己下载吧,我再单独开一遍博客说明下这个框架... https://gitee.com/yuanmaxinxi/villa_im_sdk.git
1. 登录 我的控制端登录者为controler,被控端为client 都需要注册到长链接服务器
2. 转发消息 转发图片数据和鼠标,键盘等操作
public class StartServer {
public static void main(String[] args) {
try {
Server.getInstance().startupAll(7001,7002,7003);
System.out.println("服务器启动成功!");
} catch (Exception e) {
throw new RuntimeException("IM启动失败.");
}
}
}
客户端说明:思路就是截取桌面图片,发生给服务器,服务器转发给控制端 下面是核心类
package com.villa.view;
import com.villa.io.IMClient;
import com.villa.io.model.MyEvent;
import com.villa.io.model.Protocol;
import com.villa.util.ImgUtil;
import com.villa.util.Util;
import net.coobird.thumbnailator.Thumbnails;
import javax.imageio.ImageIO;
import java.awt.*;
import java.awt.event.InputEvent;
import java.awt.event.KeyEvent;
import java.awt.event.MouseEvent;
import java.awt.image.BufferedImage;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.util.Base64;
/**
* @作者 微笑い一刀
* @bbs_url https://blog.csdn.net/u012169821
*/
public class IndexView {
private static IMClient client;
private static String from = "client";
private static String to = "control";
private static BufferedImage old_image;//上一张图片
private static BufferedImage new_image;//当前图片
public static void main(String[] args) {
//连接im服务器
client = IMClient.getInstance("127.0.0.1", 7001);
//登录
login();
//获取本机截图
screenSend();
}
public static void login(){
Protocol protocol = new Protocol();
protocol.setFrom(from);
client.sendMsg(protocol);
}
/**
* 获取本机截图并发送
*/
public static void screenSend(){
new Thread(()->{
while (true){
long time = System.currentTimeMillis();
try {
Thread.sleep(50);
} catch (InterruptedException e) {
e.printStackTrace();
}
try {
new_image = Thumbnails.of(ImgUtil.getScreenImg()).scale(1).outputQuality(0.1).asBufferedImage();
} catch (IOException e) {
e.printStackTrace();
}
send_img(new_image);
}
}).start();
}
public static void send_img(BufferedImage img){
new Thread(()->{
ByteArrayOutputStream out = null;
try {
out = new ByteArrayOutputStream();
ImageIO.write(img,"jpeg",out);
byte[] data = out.toByteArray();
Protocol protocol = new Protocol(1,from,to, Base64.getEncoder().encodeToString(Util.compress(data)),
-1,null);
client.sendMsg(protocol);
} catch (IOException e) {
e.printStackTrace();
}finally {
if(out!=null){
try {
out.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
}).start();
}
public static void handlerEvent(MyEvent event) {
int mousebuttonmask = -100; //鼠标按键
Robot robot = null;
try {
robot = new Robot();
} catch (AWTException e) {
e.printStackTrace();
}
switch (event.getId()) {
case MouseEvent.MOUSE_MOVED: //鼠标移动
robot.mouseMove(event.getX(), event.getY());
break;
case MouseEvent.MOUSE_PRESSED: //鼠标键按下
robot.mouseMove(event.getX(), event.getY());
mousebuttonmask = getMouseClick(event.getBtn());
if (mousebuttonmask != -100)
robot.mousePress(mousebuttonmask);
break;
case MouseEvent.MOUSE_RELEASED: //鼠标键松开
robot.mouseMove(event.getX(), event.getY());
mousebuttonmask = getMouseClick(event.getBtn());//取得鼠标按键
if (mousebuttonmask != -100)
robot.mouseRelease(mousebuttonmask);
break;
case MouseEvent.MOUSE_WHEEL: //鼠标滚动
robot.mouseWheel(event.getWheel());
break;
case MouseEvent.MOUSE_DRAGGED: //鼠标拖拽
robot.mouseMove(event.getX(), event.getY());
break;
case KeyEvent.KEY_PRESSED: //按键
robot.keyPress(event.getKeyCode());
break;
case KeyEvent.KEY_RELEASED: //松键
robot.keyRelease(event.getKeyCode());
break;
default:
break;
}
}
private static int getMouseClick(int button){ //取得鼠标按键
if (button == MouseEvent.BUTTON1) //左键 ,中间键为BUTTON2
return InputEvent.BUTTON1_MASK;
if (button == MouseEvent.BUTTON3) //右键
return InputEvent.BUTTON3_MASK;
return -100;
}
}
这个是netty接收消息的处理类,就是将控制端的鼠标键盘数据进行处理 鼠标键盘对应的一个长链接类型是type-100 可以理解为100代表这次数据是鼠标键盘控制数据
package com.villa.io.handler;
import com.alibaba.fastjson.JSON;
import com.villa.io.model.MyEvent;
import com.villa.io.model.Protocol;
import com.villa.view.IndexView;
import io.netty.channel.ChannelHandlerContext;
import io.netty.channel.SimpleChannelInboundHandler;
/**
* @作者 微笑い一刀
* @bbs_url https://blog.csdn.net/u012169821
*/
public class SimpleClientHandler extends SimpleChannelInboundHandler {
protected void channelRead0(ChannelHandlerContext channelHandlerContext, String msg) throws Exception {
Protocol protocol = JSON.parseObject(msg, Protocol.class);
if(protocol.getType()==100){
IndexView.handlerEvent(JSON.parseObject(protocol.getd(),MyEvent.class));
}
}
}
控制端说明:控制端就是将接收到的图片显示到窗口 事情也比较简单
package com.villa.view;
import com.alibaba.fastjson.JSON;
import com.villa.io.IMClient;
import com.villa.io.model.Protocol;
import javax.swing.*;
import java.awt.*;
/**
* @作者 微笑い一刀
* @bbs_url https://blog.csdn.net/u012169821
*/
public class IndexView extends JFrame {
private static IMClient client;
private static Robot robot;
private static String from = "control";
private static String to = "client";
static {
try {
robot = new Robot();
} catch (AWTException e) {
e.printStackTrace();
}
}
public static void main(String[] args) {
//连接im服务器
client = IMClient.getInstance("192.168.1.14", 7001);
//登录
login();
}
public static void login(){
Protocol protocol = new Protocol();
protocol.setFrom(from);
client.sendMsg(protocol);
}
//发送控制数据的方法
public static void sendEvent(String dataContent){
new Thread(()->{
Protocol protocol = new Protocol(100, from, to, dataContent, -1, null);
client.sendMsg(protocol);
}).start();
}
}
接收消息的处理类:
package com.villa.io.handler;
import com.alibaba.fastjson.JSON;
import com.villa.io.model.MyEvent;
import com.villa.io.model.Protocol;
import com.villa.util.ImgUtil;
import com.villa.util.Util;
import com.villa.view.IndexView;
import io.netty.channel.ChannelHandlerContext;
import io.netty.channel.SimpleChannelInboundHandler;
import net.coobird.thumbnailator.Thumbnails;
import javax.imageio.ImageIO;
import javax.swing.*;
import java.awt.*;
import java.awt.event.*;
import java.awt.image.BufferedImage;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.ObjectOutputStream;
import java.nio.charset.StandardCharsets;
import java.util.Base64;
/**
* @作者 微笑い一刀
* @bbs_url https://blog.csdn.net/u012169821
*/
public class SimpleClientHandler extends SimpleChannelInboundHandler {
private JLabel jLabel;
private JFrame jFrame;
private static BufferedImage old_image;//上一张图片
private static BufferedImage new_image;//当前图片
public SimpleClientHandler() {
jFrame = new JFrame();
jLabel = new JLabel();
jFrame.add(jLabel);
//全屏
jFrame.setUndecorated(false);
jFrame.setDefaultCloseOperation(JFrame.DO_NOTHING_ON_CLOSE);
jFrame.setVisible(true);
//添加监听器
setListener(jFrame);
}
protected void channelRead0(ChannelHandlerContext channelHandlerContext, String msg){
long time = System.currentTimeMillis();
Protocol protocol = JSON.parseObject(msg, Protocol.class);
if(protocol.getType()!=1){
return;
}
ByteArrayInputStream in = null;
try{
byte[] bytes = JSON.parseObject(msg, Protocol.class).getData().getBytes(StandardCharsets.UTF_8);
System.out.println(bytes.length/1024);
in = new ByteArrayInputStream(Util.uncompress(Base64.getDecoder().decode(bytes)));
BufferedImage image = ImageIO.read(in);
// if(old_image==null){
viewImg(image);
// }else{
// //还原图片
// new_image = ImgUtil.setDiffRGB(old_image,image);
// viewImg(new_image);
// image = new_image;
// }
// old_image = image;
}catch (Exception e){
e.printStackTrace();
}finally {
if(in!=null){
try {
in.close();
} catch (IOException ex) {
ex.printStackTrace();
}
}
}
System.out.println("耗费时间:"+(System.currentTimeMillis()-time));
}
private void viewImg(BufferedImage img){
new Thread(()->{
BufferedImage bigImg = null;
try {
bigImg = Thumbnails.of(img).scale(1).outputQuality(1f).asBufferedImage();
} catch (IOException e) {
e.printStackTrace();
}
if(jFrame.getSize().width!=bigImg.getWidth()||jFrame.getSize().height!=bigImg.getHeight()+26){
System.out.println("设置大小");
jFrame.setSize(bigImg.getWidth(), bigImg.getHeight()+26);
}
jLabel.setIcon(new ImageIcon(bigImg));
}).start();
}
public void setListener(JFrame frame) {
//panel设置监听器
frame.addKeyListener(new KeyAdapter() {
public void keyPressed(KeyEvent e) {
sendEventObject(e);
}
public void keyReleased(KeyEvent e) {
sendEventObject(e);
}
public void keyTyped(KeyEvent e) {
sendEventObject(e);
}
});
frame.addMouseWheelListener(new MouseWheelListener() {
public void mouseWheelMoved(MouseWheelEvent e) {
sendEventObject(e);
}
});
frame.addMouseMotionListener(new MouseMotionListener() {
public void mouseDragged(MouseEvent e) {
sendEventObject(e);
}
public void mouseMoved(MouseEvent e) {
sendEventObject(e);
}
});
frame.addMouseListener(new MouseListener() {
public void mouseClicked(MouseEvent e) {
sendEventObject(e);
}
public void mouseEntered(MouseEvent e) {
sendEventObject(e);
}
public void mouseExited(MouseEvent e) {
sendEventObject(e);
}
public void mousePressed(MouseEvent e) {
sendEventObject(e);
}
public void mouseReleased(MouseEvent e) {
sendEventObject(e);
}
});
}
private void sendEventObject(InputEvent event) {
MyEvent myEvent = new MyEvent();
myEvent.setId(event.getID());
if(event instanceof MouseEvent){
MouseEvent mouseEvent = (MouseEvent) event;
myEvent.setX(mouseEvent.getX());
myEvent.setY(mouseEvent.getY()-26);
myEvent.setBtn(mouseEvent.getButton());
}else if(event instanceof MouseWheelEvent){
MouseWheelEvent mouseWheelEvent = (MouseWheelEvent) event;
myEvent.setX(mouseWheelEvent.getX());
myEvent.setY(mouseWheelEvent.getY()-26);
}else if(event instanceof KeyEvent){
KeyEvent keyEvent = (KeyEvent)event;
myEvent.setKeyCode(keyEvent.getKeyCode());
}
IndexView.sendEvent(JSON.toJSONString(myEvent));
}
}
一些其他的类:
这个类封装了鼠标和键盘的数据
package com.villa.io.model;
/**
* @作者 微笑い一刀
* @bbs_url https://blog.csdn.net/u012169821
*/
public class MyEvent {
private int id;
private int x;
private int y;
private int btn;
private int wheel;
private int keyCode;
public int getX() {
return x;
}
public void setX(int x) {
this.x = x;
}
public int getY() {
return y;
}
public void setY(int y) {
this.y = y;
}
public int getBtn() {
return btn;
}
public void setBtn(int btn) {
this.btn = btn;
}
public int getWheel() {
return wheel;
}
public void setWheel(int wheel) {
this.wheel = wheel;
}
public int getId() {
return id;
}
public void setId(int id) {
this.id = id;
}
public int getKeyCode() {
return keyCode;
}
public void setKeyCode(int keyCode) {
this.keyCode = keyCode;
}
}
这个是封装的与IM服务器就交互的类
package com.villa.io.model;
/**
* 交互的消息协议对象
* @作者 微笑い一刀
* @bbs_url https://blog.csdn.net/u012169821
*/
public class Protocol {
//用于请求分发的指令
private int type;
//消息从哪里来 泛指账号类唯一标识 比如userId等
private String from;
//消息到哪里去
private String to;
//消息的实际内容 可以是json字符串 可以是普通字符串
private String data;
//ack 1-ack应答包 -1 代表此消息包不是ack应答包 而是携带了消息内容
private int ack;
/**
* 普通消息必传
* 消息唯一编号 注意这个只是客户端生成的一个唯一值,uuid或者时间戳+六位随机数都可以
* 并不代表消息唯一主键 也不会存进数据库 只是用来做消息补偿
*/
private String id;
public Protocol(){
}
/**
* 实例化一个ack应答包
* @param type 对应类型的应答包
*/
public Protocol(int type) {
this.type = type;
this.ack = 1;
}
public Protocol(int type, String from, String to, String data, int ack, String id) {
this.type = type;
this.from = from;
this.to = to;
this.data = data;
this.ack = ack;
this.id = id;
}
public int getType() {
return type;
}
public void setType(int type) {
this.type = type;
}
public String getFrom() {
return from;
}
public void setFrom(String from) {
this.from = from;
}
public String getTo() {
return to;
}
public void setTo(String to) {
this.to = to;
}
public String getData() {
return data;
}
public void setData(String data) {
this.data = data;
}
public int getAck() {
return ack;
}
public void setAck(int ack) {
this.ack = ack;
}
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
}
封装了 netty客户端
package com.villa.io;
import com.alibaba.fastjson.JSON;
import com.villa.io.handler.SimpleClientHandler;
import com.villa.io.model.Protocol;
import io.netty.bootstrap.Bootstrap;
import io.netty.channel.*;
import io.netty.channel.nio.NioEventLoopGroup;
import io.netty.channel.socket.nio.NioSocketChannel;
import io.netty.handler.codec.LengthFieldBasedFrameDecoder;
import io.netty.handler.codec.LengthFieldPrepender;
import io.netty.handler.codec.string.StringDecoder;
import io.netty.handler.codec.string.StringEncoder;
/**
* @作者 微笑い一刀
* @bbs_url https://blog.csdn.net/u012169821
*/
public class IMClient {
private static IMClient imClient = new IMClient();
private static Channel channel;
public static int TCP_FRAME_FIXED_HEADER_LENGTH = 4;
public static int TCP_FRAME_MAX_BODY_LENGTH = 1024*1024*10;
private IMClient(){}
public static IMClient getInstance(String ip,int port){
// 首先,netty通过ServerBootstrap启动服务端
Bootstrap client = new Bootstrap();
//第1步 定义线程组,处理读写和链接事件,没有了accept事件
EventLoopGroup group = new NioEventLoopGroup(1);
client.group(group);
client.channel(NioSocketChannel.class);
client.option(ChannelOption.SO_KEEPALIVE, true);
client.option(ChannelOption.TCP_NODELAY, true);
client.option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 5000);
//第2步 给NIoSocketChannel初始化handler, 处理读写事件
client.handler(new ChannelInitializer() { //通道是NioSocketChannel
protected void initChannel(NioSocketChannel ch) throws Exception {
ch.pipeline().addLast("frameDecoder", new LengthFieldBasedFrameDecoder(TCP_FRAME_MAX_BODY_LENGTH+TCP_FRAME_FIXED_HEADER_LENGTH, 0, TCP_FRAME_FIXED_HEADER_LENGTH, 0, TCP_FRAME_FIXED_HEADER_LENGTH));
ch.pipeline().addLast("frameEncoder", new LengthFieldPrepender(TCP_FRAME_FIXED_HEADER_LENGTH));
//找到他的管道 增加他的handler
ch.pipeline().addLast(new StringEncoder());
ch.pipeline().addLast(new StringDecoder());
ch.pipeline().addLast(new SimpleClientHandler());
}
});
//连接服务器
try {
ChannelFuture channelFuture = client.connect(ip, port).sync();
channel = channelFuture.channel();
} catch (InterruptedException e) {
e.printStackTrace();
}
return imClient;
}
/**
* 发送消息
*/
public void sendMsg(Protocol protocol){
channel.writeAndFlush(JSON.toJSONString(protocol));
}
}
两个工具类,但是有些方法没有使用
package com.villa.util;
import javax.imageio.ImageIO;
import java.awt.*;
import java.awt.image.BufferedImage;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.util.concurrent.BrokenBarrierException;
import java.util.concurrent.CyclicBarrier;
public class ImgUtil {
public static void main(String[] args) {
while(true){
long time = System.currentTimeMillis();
//先取一张
BufferedImage oldImg = getScreenImg();
//睡眠3秒
try {
Thread.sleep(3000);
} catch (InterruptedException e) {
e.printStackTrace();
}
//再取一张
BufferedImage newImg = getScreenImg();
//获取差值
System.out.println(bufImg2Byte(getDiffRGB(oldImg,newImg)).length/1024+"kb");
System.out.println(System.currentTimeMillis() - time - 3000);
}
}
/**
* 获取屏幕截图
*/
public static BufferedImage getScreenImg(){
try {
Robot robot = new Robot();
Dimension size = Toolkit.getDefaultToolkit().getScreenSize();
return robot.createScreenCapture(new Rectangle(0, 0, (int) size.getWidth(), (int) size.getHeight()));
} catch (AWTException e) {
e.printStackTrace();
}
return null;
}
/**
* BufferedImage转byte数组
* @param img
* @return
*/
public static byte[] bufImg2Byte(BufferedImage img){
ByteArrayOutputStream out = new ByteArrayOutputStream();
try {
ImageIO.write(img, "jpg", out);
byte[] b = out.toByteArray();
out.write(b);
return b;
} catch (IOException e) {
e.printStackTrace();
}finally {
if(out!=null){
try {
out.close();
} catch (IOException e) {
e.printStackTrace();
}
}
}
return new byte[]{};
}
/**
* 获取全屏截图并转为byte数组
*/
public static byte[] getScreenByte(){
BufferedImage screenImg = getScreenImg();
return bufImg2Byte(screenImg);
}
/**
* 获取图片差值
* 相同的像素 设置为0 不相同的像素设置为新的rgb值
*/
public static BufferedImage getDiffRGB(BufferedImage oldImg, BufferedImage newImg){
if(oldImg.getWidth()!=newImg.getWidth()||oldImg.getHeight()!=newImg.getHeight()){
return newImg;
}
int threadCount = 10;
final CyclicBarrier barrier=new CyclicBarrier(threadCount);
for(int i=1;i<=threadCount;i++) {
final int tempI = i;
new Thread(()->{
int curMax = newImg.getWidth() / threadCount;
for (int x = 0*curMax; x < curMax*tempI; x++){
for (int y = 0; y < newImg.getHeight(); y++) {
//取出两张图片当前的rgb值
int oldRGB = oldImg.getRGB(x,y);
int newRGB = newImg.getRGB(x,y);
//如果相等 取
if(oldRGB==newRGB){
newImg.setRGB(x,y,0);
}
}
}
try {
barrier.await();
} catch (InterruptedException e) {
e.printStackTrace();
} catch (BrokenBarrierException e) {
e.printStackTrace();
}
}).start();
}
try {
barrier.await();
} catch (InterruptedException e) {
e.printStackTrace();
} catch (BrokenBarrierException e) {
e.printStackTrace();
}
return newImg;
}
/**
* 获取图片差值
* 相同的像素 设置为0 不相同的像素设置为新的rgb值
*/
public static BufferedImage setDiffRGB(BufferedImage oldImg, BufferedImage newImg){
if(oldImg.getWidth()!=newImg.getWidth()||oldImg.getHeight()!=newImg.getHeight()){
return newImg;
}
int threadCount = 10;
final CyclicBarrier barrier=new CyclicBarrier(threadCount);
for(int i=1;i<=threadCount;i++) {
final int tempI = i;
new Thread(()->{
int curMax = newImg.getWidth() / threadCount;
for (int x = 0*curMax; x < curMax*tempI; x++){
for (int y = 0; y < newImg.getHeight(); y++) {
//取出两张图片当前的rgb值
int newRGB = newImg.getRGB(x,y);
//如果相等 取
if(newRGB!=0){
oldImg.setRGB(x,y,newRGB);
}
}
}
try {
barrier.await();
} catch (InterruptedException e) {
e.printStackTrace();
} catch (BrokenBarrierException e) {
e.printStackTrace();
}
}).start();
}
try {
barrier.await();
} catch (InterruptedException e) {
e.printStackTrace();
} catch (BrokenBarrierException e) {
e.printStackTrace();
}
//返回一张新的图片 直接修改引用会引起图片闪烁
return oldImg.getSubimage(0,0,oldImg.getWidth(),oldImg.getHeight());
}
}
package com.villa.util;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPOutputStream;
/**
* @作者 微笑い一刀
* @bbs_url https://blog.csdn.net/u012169821
*/
public class Util {
public static byte[] uncompress(byte[] bytes) {
ByteArrayOutputStream out = new ByteArrayOutputStream();
ByteArrayInputStream in = new ByteArrayInputStream(bytes);
try {
GZIPInputStream ungzip = new GZIPInputStream(in);
byte[] buffer = new byte[256];
int n;
while ((n = ungzip.read(buffer)) >= 0) {
out.write(buffer, 0, n);
}
} catch (IOException e) {
e.printStackTrace();
}
return out.toByteArray();
}
public static byte[] compress(byte[] str) {
ByteArrayOutputStream out = new ByteArrayOutputStream();
GZIPOutputStream gzip;
try {
gzip = new GZIPOutputStream(out);
gzip.write(str);
gzip.close();
} catch (IOException e) {
e.printStackTrace();
}
return out.toByteArray();
}
}