MyBatis批量插入百万级数据插入（二）
最近公司要做一个项目，生成报告时每个报告有一个编码，不能重复。解决方案订为提前在数据库插入一百万条数据，每次随机取一个，然后更改状态，标记为已使用。
第一步先在数据库插入百万条不重复的数据。
代码如下
Controller
@RequestMapping("/insetData")
@ResponseBody
public String insetData(){
service.insetData();
return null;
}
ServiceImpl
/**
* 数据库插入一百万条数据
*
* @return
*/
@Override
public void insetData() {
Set set = new HashSet();
RainbowCodeModel model = new RainbowCodeModel();
while (set.size() < 1000000) {
String cardNumber = getCard();//调用生成随机数的方法
set.add(cardNumber);
}
List a = new ArrayList(set);
List list = new ArrayList();
for (int i = 0; i < a.size(); i++) {
model.setRainbowCode(a.get(i));
list.add(model);
}
mapper.insetData(list);
}
//生成随机数
public static String getCard() {
Random rand = new Random();//生成随机数
String cardNnumer = "";
for (int a = 0; a < 12; a++) {
cardNnumer += rand.nextInt(10);//生成12位数字
}
return "2" + cardNnumer;
} mapper.xml
INSERT INTO tb_ebs_rainbow_code (rainbow_code)
values
(#{item.rainbowCode,jdbcType=VARCHAR})
执行方法 异常
### Cause: com.mysql.jdbc.PacketTooBigException: Packet for query is too large (41000066 > 4194304). You can change this value on the server by setting the max_allowed_packet' variable.
; SQL []; Packet for query is too large (41000066 > 4194304). You can change this value on the server by setting the max_allowed_packet' variable.; nested exception is com.mysql.jdbc.PacketTooBigException: Packet for query is too large (41000066 > 4194304). You can change this value on the server by setting the max_allowed_packet' variable.] with root cause
com.mysql.jdbc.PacketTooBigException: Packet for query is too large (41000066 > 4194304). You can change this value on the server by setting the max_allowed_packet' variable.
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3910)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2596)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2776)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2838)
at
com.mysql.jdbc.PacketTooBigException: Packet for query is too large 异常：
原因： 查询出的数据包过大，默认情况下mysql 的字段容量不够装，所以抛出此异常
解决办法：https://blog.csdn.net/pingweicheng/article/details/80601788
数据库添加唯一索引
1.添加PRIMARY KEY（主键索引）
mysql>ALTER TABLE `table_name` ADD PRIMARY KEY ( `column` )
2.添加UNIQUE(唯一索引)
mysql>ALTER TABLE `table_name` ADD UNIQUE (
`column`
)
3.添加INDEX(普通索引)
mysql>ALTER TABLE `table_name` ADD INDEX index_name ( `column` )
4.添加FULLTEXT(全文索引)
mysql>ALTER TABLE `table_name` ADD FULLTEXT ( `column`)
5.添加多列索引
mysql>ALTER TABLE `table_name` ADD INDEX index_name ( `column1`, `column2`, `column3` )
执行方法又有异常
### Cause: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '('2648000453884')
('2577773118325')
' at line 6
; bad SQL grammar []; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '('2648000453884')
('2577773118325')
' at line 6] with root cause
com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '('2648000453884')
('2577773118325')
' at line 6
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
at com.mysql.jdbc.Util.getInstance(Util.java:384)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1052)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4232)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4164)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2615)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2776)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2838)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2082)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
mapper.xml 修改 #() 为 $()
mybatis中#和$符号的区别
https://www.cnblogs.com/qianf/p/9534126.html
INSERT INTO tb_ebs_rainbow_code (rainbow_code)
values
(${item.rainbowCode,jdbcType=VARCHAR})
报异常
org.apache.ibatis.binding.BindingException: Parameter 'VARCHAR' not found. Available parameters are [list]
at org.apache.ibatis.session.defaults.DefaultSqlSession$StrictMap.get(DefaultSqlSession.java:261)
at org.apache.ibatis.scripting.xmltags.DynamicContext$ContextAccessor.getProperty(DynamicContext.java:123)
at org.apache.ibatis.ognl.OgnlRuntime.getProperty(OgnlRuntime.java:1657)
at org.apache.ibatis.ognl.ASTProperty.getValueBody(ASTProperty.java:92)
at org.apache.ibatis.ognl.SimpleNode.evaluateGetValueBody(SimpleNode.java:170)
at org.apache.ibatis.ognl.SimpleNode.getValue(SimpleNode.java:210)
at org.apache.ibatis.ognl.ASTAssign.getValueBody(ASTAssign.java:49)
at org.apache.ibatis.ognl.SimpleNode.evaluateGetValueBody(SimpleNode.java:170)
at org.apache.ibatis.ognl.SimpleNode.getValue(SimpleNode.java:210)
at org.apache.ibatis.ognl.ASTSequence.getValueBody(ASTSequence.java:55)
at org.apache.ibatis.ognl.SimpleNode.evaluateGetValueBody(SimpleNode.java:170)
at org.apache.ibatis.ognl.SimpleNode.getValue(SimpleNode.java:210)
at org.apache.ibatis.ognl.Ognl.getValue(Ognl.java:333)
at org.apache.ibatis.ognl.Ognl.getValue(Ognl.java:413)
at org.apache.ibatis.ognl.Ognl.getValue(Ognl.java:395)
at org.apache.ibatis.scripting.xmltags.OgnlCache.getValue(OgnlCache.java:48)
at org.apache.ibatis.scripting.xmltags.TextSqlNode$BindingTokenParser.handleToken(TextSqlNode.java:64)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
最后 代码修改为
INSERT INTO tb_ebs_rainbow_code (rainbow_code)
values
(
#{item.rainbowCode}
)
数据插入成功。