beego-orm查询使用All查不到结果问题

这是一个现网问题

问题现象

客户页面记录展示不全,只展示了1000条,最新的不展示。

项目组一兄弟定位为,数据库查询到的记录被get请求返回时给截掉了(由于最大size限制)
走给我审核时,第一直觉告诉我定位的原因肯定不对,因为如果是由于get请求大小限制,不可能刚好截断保留整整1000条记录,而且如果是按字节大小截断,response应该会报错吧?

验证猜想

抱着怀疑的态度,在网上搜了一下GET返回时有什么限制,见如下链接

Limit on the length of the data that a webserver can return in response to a GET request

There are no limits on the amount of data returned on a HTTP response from Jetty.
You could stream data back to the client until shortly before the heat death of the universe.
Technically speaking, you can have a HTTP Response with no Content-Length
specified, which can be returned using either the Chunked Transfer-Encoding, or just a raw stream of bytes with a Connection: close
indicating when the data is complete (done being sent) by a close of the underlying connection. Both of which are essentially limit-less.
If you use a HTTP Response with Content-Length, be aware that Content-Length is, in practice, a 32-bit number, but more modern browsers support the 64-bit versions.

大体意思就是GET在response返回时没有大小限制,说明定位原因不对。

根因推导
  1. 不是GET返回截断的问题,且返回不全,肯定是后台截断了,后台能从哪里截断?初步判断为数据库查询只返回了1000条记录
  2. 进一步搜索日志,发现果然是数据库返回了1000条记录
  3. 接着查看代码,发现代码在查数据库时并未使用limit限制,进一步猜测调用的beego orm的All(container interface{}, cols ...string) (int64, error)有默认过滤规则
  4. 网上搜索关于beego orm All方法的资料并查看源码,发现All方法果然有默认1000的限制返回数

All / Values / ValuesList / ValuesFlat 受到 Limit 的限制,默认最大行数为 1000

源码:

func (o *querySet) All(container interface{}, cols ...string) (int64, error) {
    return o.orm.alias.DbBaser.ReadBatch(o.orm.db, o, o.mi, o.cond, container, o.orm.alias.TZ, cols)
}

func (d *dbBase) ReadBatch(q dbQuerier, qs *querySet, mi *modelInfo, cond *Condition, container interface{}, tz *time.Location, cols []string) (int64, error) {
...
    limit := tables.getLimitSQL(mi, offset, rlimit)
...
    query := fmt.Sprintf("%s %s FROM %s%s%s T0 %s%s%s%s%s", sqlSelect, sels, Q, mi.table, Q, join, where, groupBy, orderBy, limit)
...
}

func (t *dbTables) getLimitSQL(mi *modelInfo, offset int64, limit int64) (limits string) {
    if limit == 0 {
        limit = int64(DefaultRowsLimit)
    }
}

DefaultRowsLimit = 1000

Reference:
https://stackoverflow.com/questions/16172524/limit-on-the-length-of-the-data-that-a-webserver-can-return-in-response-to-a-get
http://zoomq.qiniudn.com/ZQScrapBook/ZqFLOSS/data/20131230002213/index.html

你可能感兴趣的:(beego-orm查询使用All查不到结果问题)