关于增量更新云平台数据的设计思路

1、使用mysqldump进行增量文件备份

附一下以前完成的C#版代码示例：

 /// <summary>
        /// 功能：导出数据
        /// 作者：黄海
        /// 时间：2015-09-24
        /// </summary>
        /// <param name="host"></param>
        /// <param name="port"></param>
        /// <param name="user"></param>
        /// <param name="passWord"></param>
        /// <param name="dataBaseName"></param>
        /// <param name="tableName"></param>
        /// <param name="condition"></param>
        /// <param name="dirName"></param>
        private static void ExportData(String host, String port, String user, String passWord, String dataBaseName, String tableName, String condition, String dirName)
        {
            var fileName = Application.StartupPath + "\\mysqldump.exe";
            //  var arguments = " -h " + host + " -P" + port + " -u" + user + " -p" + passWord + " --compact -t " + dataBaseName + " " + tableName + " --where=\"" + condition + "\" > \"" + dirName + "\\" + tableName + ".sql\"";
            var arguments = " -h " + host + " -P" + port + " -u" + user + " -p" + passWord + "  --no-create-info --compact --skip-extended-insert  " + dataBaseName + " " + tableName + " --where=\"" + condition + "\" > \"" + dirName + "\\" + tableName + ".sql\"";

            var psi = new ProcessStartInfo("cmd.exe")
            {
                Arguments = "/c  (\"" + fileName + "\" " + arguments + ")",
                WorkingDirectory = Application.StartupPath,
                RedirectStandardOutput = true,
                UseShellExecute = false,
                CreateNoWindow = true
            };
            var p = Process.Start(psi);
            var output = p.StandardOutput.ReadToEnd();
            p.WaitForExit();
        }

 

2、如何使用java 调用mysqldump?  
http://cnhnslhs-163-com.iteye.com/blog/1774508
http://blog.csdn.net/icejadelove/article/details/4966317

大约的代码示例是这样的：

public static void backup() {
try {
     Runtime rt = Runtime.getRuntime();

     Process child = rt.exec("mysqldump -u用户名 -p密码 -R -c --set-charset=utf8 数据库名");

     InputStream in = child.getInputStream();

     InputStreamReader xx = new InputStreamReader(in, "utf8");

     String inStr;
     StringBuffer sb = new StringBuffer("");
     String outStr;

     BufferedReader br = new BufferedReader(xx);
     while ((inStr = br.readLine()) != null) {
           sb.append(inStr + "\r\n");
     }
     outStr = sb.toString();


     FileOutputStream fout = new FileOutputStream("备份的文件路径名");
     OutputStreamWriter writer = new OutputStreamWriter(fout, "utf8");
     writer.write(outStr);
     writer.flush();

     in.close();
     xx.close();
     br.close();
     writer.close();
     fout.close();
     } catch (Exception e) {
         e.printStackTrace();
     }

}

还原:
public static void load() {
try {
     String fPath = "备份的文件路径名";
     Runtime rt = Runtime.getRuntime();

     Process child = rt.exec("mysql -u用户名 -p密码 数据库名");
     OutputStream out = child.getOutputStream();
     String inStr;
     StringBuffer sb = new StringBuffer("");
     String outStr;
     BufferedReader br = new BufferedReader(new InputStreamReader(new FileInputStream(fPath), "utf8"));
     while ((inStr = br.readLine()) != null) {
         sb.append(inStr + "\r\n");
     }
     outStr = sb.toString();

     OutputStreamWriter writer = new OutputStreamWriter(out, "utf8");
     writer.write(outStr);
     writer.flush();

     out.close();
     br.close();
     writer.close();
     } catch (Exception e) {
         e.printStackTrace();
     }
} 

 

3、生成的是一个文件，返回的是OK和文件路径，然后C#去下载这个文件并执行import动作。

 private static void ImportData(String host, String port, String user, String passWord, String dataBaseName, String dirName)
        {
            var di = new DirectoryInfo(dirName);
            foreach (var o in di.GetFiles())
            {
                if (o.Extension == ".sql")
                {
                    var fileName = Application.StartupPath + "\\mysql.exe";
                    var arguments = " -h " + host + " -P" + port + " -u" + user + " -p" + passWord + " " + dataBaseName + " < \"" + o.FullName + "\" --default-character-set=utf8";
                    var psi = new ProcessStartInfo("cmd.exe")
                    {
                        Arguments = "/c  (\"" + fileName + "\" " + arguments + ")",
                        WorkingDirectory = Application.StartupPath,
                        RedirectStandardOutput = true,
                        UseShellExecute = false,
                        CreateNoWindow = true
                    };
                    var p = Process.Start(psi);
                    var output = p.StandardOutput.ReadToEnd();
                    p.WaitForExit();        
                }
            }
        }

 

4、下载成功后向服务器发起一个指令，让其删除掉这个文件，防止磁盘占用太多。

5、根据TS的范围进行重新生成SSDB的预热信息，放弃执行索引更新，让每天的全量更新起作用。

6、时间：建议每天晚上12点以后，每天6点以前进行

即： 0,1,2,3,4,5共6个小时。 6*60＝360分钟，建议处理程序

abcdefg".GetHashCode().ToString("D10") 产生一个10的数字，然后除以300取余数，就是从0点开始加N分种后的运行时间，防止多个客户端同时向服务器发起
请求，当然，除了自动功能外，还应该提供一个手工更新功能供用户使用。