Json格式化输出工具jq使用学习
jq是一款用于处理JSON数据的命令行工具。它可以帮助你将复杂的JSON数据格式化、查询、筛选、编辑等操作，以便更好地处理和分析数据。
一、安装
1.1、yum 包安装
$ yum install jq
No match for argument: jq
Error: Unable to find a match: jq
机器配置yum源中没有找到包
1.2、使用rpm包安装
社区下载包安装：
常用linux 社区包地址：https://centos.pkgs.org/
$ wget http://www6.atomicorp.com/channels/atomic/centos/7/x86_64/RPMS/jq-1.6-2.el7.x86_64.rpm
$ rpm -ivh jq-1.6-2.el7.x86_64.rpm
warning: jq-1.6-2.el7.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 4520afa9: NOKEY
error: Failed dependencies:
libonig.so.5()(64bit) is needed by jq-1.6-2.el7.x86_64
安装报错，缺少依赖，安装依赖依然失败：
$ yum localinstall oniguruma-devel-6.8.2-2.el7.x86_64.rpm
Last metadata expiration check: 0:07:38 ago on Tue 27 Jun 2023 10:15:12 AM CST.
Error:
Problem: conflicting requests
- nothing provides libonig.so.5()(64bit) needed by oniguruma-devel-6.8.2-2.el7.x86_64
使用atomic-release 工具安装：
$ wget http://www6.atomicorp.com/channels/atomic/centos/7/x86_64/RPMS/atomic-release-1.0-23.el7.art.noarch.rpm
$ rpm -Uvh atomic-release-1.0-23.el7.art.noarch.rpm
warning: atomic-release-1.0-23.el7.art.noarch.rpm: Header V4 RSA/SHA256 Signature, key ID 4520afa9: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:atomic-release-1.0-23.el7.art ################################# [100%]
# 安装atomic-release实际上是在本地新建atomic.repo yum源文件
$ ll /etc/yum.repos.d/ | grep atomic
-rw-r--r-- 1 root root 743 Jan 5 2022 atomic.repo
$ yum install jq
CentOS / Red Hat Enterprise Linux 7 - atomic 80 kB/s | 205 kB 00:02
Last metadata expiration check: 0:00:00 ago on Tue 27 Jun 2023 10:28:38 AM CST.
Dependencies resolved.
============================================================================================================================================================================
Package Arch Version Repository Size
============================================================================================================================================================================
Installing:
jq x86_64 1.6-2.el7 atomic 167 k
Installing dependencies:
oniguruma x86_64 6.8.2-1.el7 atomic 181 k
Transaction Summary
============================================================================================================================================================================
Install 2 Packages
Total download size: 348 k
Installed size: 1.0 M
Is this ok [y/N]: y
Downloading Packages:
(1/2): oniguruma-6.8.2-1.el7.x86_64.rpm 90 kB/s | 181 kB 00:02
(2/2): jq-1.6-2.el7.x86_64.rpm 83 kB/s | 167 kB 00:02
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 102 kB/s | 348 kB 00:03
warning: /var/cache/dnf/atomic-e5cca8bbd0dce12d/packages/jq-1.6-2.el7.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 4520afa9: NOKEY
Importing GPG key 0x4520AFA9:
Userid : "Atomicorp (Atomicorp Official Signing Key) "
Fingerprint: 1818 66DF 9DAC A40E 5B42 9B08 FFBD 5D0A 4520 AFA9
From : https://www.atomicorp.com/RPM-GPG-KEY.atomicorp.txt
Is this ok [y/N]: y
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : oniguruma-6.8.2-1.el7.x86_64 1/2
Running scriptlet: oniguruma-6.8.2-1.el7.x86_64 1/2
Installing : jq-1.6-2.el7.x86_64 2/2
Running scriptlet: jq-1.6-2.el7.x86_64 2/2
Verifying : jq-1.6-2.el7.x86_64 1/2
Verifying : oniguruma-6.8.2-1.el7.x86_64 2/2
New leaves:
jq.x86_64
Installed:
jq.x86_64 1.6-2.el7 oniguruma.x86_64 6.8.2-1.el7
Complete!
二、jq的应用
2.1、格式化json数据
使用jq可以将复杂的JSON数据格式化为更加易读的格式
如{"foo": 0} 使用jq格式化后显示为：
$ echo '{"foo": 0}' | jq
{
"foo": 0
}
对文件操作：
$ cat test.json
{"gender":"man","name":"zhangsan","age":"10","score":{"math":"85","English":"86","Chinese":"87"}}
$ jq . test.json
{
"gender": "man",
"name": "zhangsan",
"age": "10",
"score": {
"math": "85",
"English": "86",
"Chinese": "87"
}
}
# 也可以
$ cat test.json | jq
{
"gender": "man",
"name": "zhangsan",
"age": "10",
"score": {
"math": "85",
"English": "86",
"Chinese": "87"
}
}
2.2、查询JSON数据
使用jq可以查询JSON数据中的某个字段或值。
$ cat test.json | jq '.age'
"10"
# 查看math
$ cat test.json | jq '.score.math'
"85"
或者使用：
$ jq '.age' test.json
"10"
# 查看math
$ jq '.score.math' test.json
"85"
多组数据的查询,数据如下：
$ cat test.json
[{"gender":"man","name":"zhangsan","age":10,"score":{"math":85,"English":86,"Chinese":87}},{"gender":"man","name":"lisi","age":9,"score":{"math":80,"English":81,"Chinese":82}},{"gender":"man","name":"wangwu","age":11,"score":{"math":85,"English":84,"Chinese":87}},{"gender":"female","name":"lili","age":11,"score":{"math":85,"English":88,"Chinese":90}}]
$ cat test.json | jq .
[
{
"gender": "man",
"name": "zhangsan",
"age": 10,
"score": {
"math": 85,
"English": 86,
"Chinese": 87
}
},
{
"gender": "man",
"name": "lisi",
"age": 9,
"score": {
"math": 80,
"English": 81,
"Chinese": 82
}
},
{
"gender": "man",
"name": "wangwu",
"age": 11,
"score": {
"math": 85,
"English": 84,
"Chinese": 87
}
},
{
"gender": "female",
"name": "lili",
"age": 11,
"score": {
"math": 85,
"English": 88,
"Chinese": 90
}
}
]
$ cat test.json | jq '.[0]'
{
"gender": "man",
"name": "zhangsan",
"age": 10,
"score": {
"math": 85,
"English": 86,
"Chinese": 87
}
}
$ cat test.json | jq '.[1]'
{
"gender": "man",
"name": "lisi",
"age": 9,
"score": {
"math": 80,
"English": 81,
"Chinese": 82
}
}
2.3、筛选JSON数据
使用jq可以根据某些条件筛选JSON数据中的部分内容
$ cat test.json | jq '.[] | select(.age > 10)'
{
"gender": "man",
"name": "wangwu",
"age": 11,
"score": {
"math": 85,
"English": 84,
"Chinese": 87
}
}
{
"gender": "female",
"name": "lili",
"age": 11,
"score": {
"math": 85,
"English": 88,
"Chinese": 90
}
}
$ cat test.json | jq '.[] | select(.score.math < 85)'
{
"gender": "man",
"name": "lisi",
"age": 9,
"score": {
"math": 80,
"English": 81,
"Chinese": 82
}
}
$ cat test.json | jq '.[] | select(.score.math < 80)'
2.4、编辑JSON数据
使用jq可以编辑JSON数据中的某些字段或值。
$ cat test.json | jq '.[0].age'
10
$ cat test.json | jq '.[0].age += 1'
[
{
"gender": "man",
"name": "zhangsan",
"age": 11,
"score": {
"math": 85,
"English": 86,
"Chinese": 87
}
},
{
"gender": "man",
"name": "lisi",
"age": 9,
"score": {
"math": 80,
"English": 81,
"Chinese": 82
}
},
{
"gender": "man",
"name": "wangwu",
"age": 11,
"score": {
"math": 85,
"English": 84,
"Chinese": 87
}
},
{
"gender": "female",
"name": "lili",
"age": 11,
"score": {
"math": 85,
"English": 88,
"Chinese": 90
}
}
]
$ cat test.json
[{"gender":"man","name":"zhangsan","age":10,"score":{"math":85,"English":86,"Chinese":87}},{"gender":"man","name":"lisi","age":9,"score":{"math":80,"English":81,"Chinese":82}},{"gender":"man","name":"wangwu","age":11,"score":{"math":85,"English":84,"Chinese":87}},{"gender":"female","name":"lili","age":11,"score":{"math":85,"English":88,"Chinese":90}}]
需要注意的是，原文件并没有被需改
三、其他用法
3.1、管道符 |
使用|运算符，我们可以结合两个过滤器。它的工作原理与Unix系统管道符类似。左边的过滤器的输出传递到右边的过滤器。
3.2、keys函数
keys函数获取JSON对象中的所有键（key）。keys函数返回一个列表，其中包含JSON对象中所有键的名称，多组数据获取到的是数据索引，如下：
$ cat test.json | jq '. | keys'
[
0,
1,
2,
3
]
获取某一组数据keys
$ cat test.json | jq '.[0] | keys'
[
"age",
"gender",
"name",
"score"
]
$ cat test.json | jq '.[0].score | keys'
[
"Chinese",
"English",
"math"
]
3.3、length函数
返回一个数组或字符串的长度。
$ cat test.json | jq '. | length'
4
$ cat test.json | jq '.[0] | length'
4
$ cat test.json | jq '.[0].score | length'
3
$ cat test.json | jq '.[0].name | length'
8
3.4、map函数
对一个数组中的每个元素执行一个函数，并返回一个新的数组。
$ cat test.json | jq '. | map(keys)'
[
[
"age",
"gender",
"name",
"score"
],
[
"age",
"gender",
"name",
"score"
],
[
"age",
"gender",
"name",
"score"
],
[
"age",
"gender",
"name",
"score"
]
]
$ cat test.json | jq '. | map(length)'
[
4,
4,
4,
4
]
3.5、select函数
对一个数组中的每个元素进行筛选，并返回一个新的数组。
参考2.3
3.6、if-then-else语句
if-then-else语句来根据条件执行不同的操作。语法如下：
if CONDITION then EXPRESSION1 else EXPRESSION2 end
CONDITION是一个条件表达式，可以是一个比较、逻辑或其他类型的表达式。如果CONDITION为真，则执行EXPRESSION1，否则执行EXPRESSION2。EXPRESSION1和EXPRESSION2可以是任何jq表达式，包括函数调用、变量引用、数学操作等。
$ cat test.json | jq '.[0].name= if .[0].age >5 then "first" else "second" end'
[
{
"gender": "man",
"name": "first",
"age": 10,
"score": {
"math": 85,
"English": 86,
"Chinese": 87
}
},
{
"gender": "man",
"name": "lisi",
"age": 9,
"score": {
"math": 80,
"English": 81,
"Chinese": 82
}
},
{
"gender": "man",
"name": "wangwu",
"age": 11,
"score": {
"math": 85,
"English": 84,
"Chinese": 87
}
},
{
"gender": "female",
"name": "lili",
"age": 11,
"score": {
"math": 85,
"English": 88,
"Chinese": 90
}
}
]
3.7、join函数
将一个数组中的所有元素拼接成一个字符串。oin函数只能用于字符串数组，不能用于其他类型的数组。
test01.json
{
"name": "Alice",
"hobbies": ["reading", "singing", "dancing"]
}
cat test01.json | jq '.hobbies | join(",")'
"reading,singing,dancing"
3.8、split函数
将一个字符串分割成多个子字符串，并将它们存储在一个数组中。
$ cat test.json | jq '.[0].name | split("")'
[
"z",
"h",
"a",
"n",
"g",
"s",
"a",
"n"
]
$ cat test.json | jq '.[0].name | split("n")'
[
"zha",
"gsa",
""
]
3.9、max和min函数
返回一个数组或一组数字中的最大值或最小值
$ cat test.json | jq '. | map(.score.math) | max'
85
$ cat test.json | jq '. | map(.score.math) | min'
80
$ cat test.json | jq '. | map(.age) | max'
11
3.10、flatten函数
将多维数组转换为一维数组
test02.json
$ cat test02.json
[
[1, 2, 3],
[4, 5, 6],
[7, 8, 9]
]
$ cat test02.json | jq '. | flatten'
[
1,
2,
3,
4,
5,
6,
7,
8,
9
]
3.11、sort函数
对一个数组中的元素进行排序
$ cat test.json | jq '. | map(.age) | sort'
[
9,
10,
11,
11
]
$ cat test.json | jq '. | map(.score.Chinese) | sort'
[
82,
87,
87,
90
]
3.12、group_by函数
根据一个键对一个数组中的元素进行分组
# 依据age分组
$ cat test.json | jq 'group_by(.age)'
[
[
{
"gender": "man",
"name": "lisi",
"age": 9,
"score": {
"math": 80,
"English": 81,
"Chinese": 82
}
}
],
[
{
"gender": "man",
"name": "zhangsan",
"age": 10,
"score": {
"math": 85,
"English": 86,
"Chinese": 87
}
}
],
[
{
"gender": "man",
"name": "wangwu",
"age": 11,
"score": {
"math": 85,
"English": 84,
"Chinese": 87
}
},
{
"gender": "female",
"name": "lili",
"age": 11,
"score": {
"math": 85,
"English": 88,
"Chinese": 90
}
}
]
]
3.13、unique函数
用于删除输入数据中的重复元素
test03.json
$ cat test03.json
[1, 2, 3, 1, 2, 4, 5, 3]
$ cat test03.json | jq '. | unique'
[
1,
2,
3,
4,
5
]
3.14、to_entries和from_entries函数
将一个对象转换为一个包含键值对的数组，或将一个包含键值对的数组转换为一个对象。
$ cat test.json | jq '.[0] |to_entries'
[
{
"key": "gender",
"value": "man"
},
{
"key": "name",
"value": "zhangsan"
},
{
"key": "age",
"value": 10
},
{
"key": "score",
"value": {
"math": 85,
"English": 86,
"Chinese": 87
}
}
]
test04.json
$ cat test04.json
[
{
"key": "gender",
"value": "man"
},
{
"key": "name",
"value": "zhangsan"
},
{
"key": "age",
"value": 10
},
{
"key": "score",
"value": {
"math": 85,
"English": 86,
"Chinese": 87
}
}
]
$ cat test04.json | jq '. |from_entries'
{
"gender": "man",
"name": "zhangsan",
"age": 10,
"score": {
"math": 85,
"English": 86,
"Chinese": 87
}
}
3.15、contains函数
检查一个数组中是否包含一个特定的元素
$ cat test.json | jq '.[0].name | contains("z")'
true
$ cat test.json | jq '.[0].name | contains("l")'
false
3.16、startswith和endswith函数
检查一个字符串是否以特定的前缀或后缀开头或结束
$ cat test.json | jq '.[0].name | startswith("zh")'
true
$ cat test.json | jq '.[0].name | startswith("li")'
false
$ cat test.json | jq '.[0].name | endswith("san")'
true
$ cat test.json | jq '.[0].name | endswith("hua")'
false
参考文档
1、https://mp.weixin.qq.com/s/U1VwJGFVkrUyv3jMegapCg
2、https://mp.weixin.qq.com/s/45-ztTx2scbNY5u7NQzeIA
3、https://www.cnblogs.com/FunTester/p/14713957.html
4、https://blog.csdn.net/w727655308/article/details/121207345