使用parquet-tools工具查看parquet文件

cdh默认安装了。我安装6.2下面对应的路径是/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/parquet-tools

如果找不到使用以下命令查看一下具体位置

find / -name parquet-tools

直接运行

/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/bin/parquet-tools
可以看到帮助信息。看了一下主要有8个选项；

• cat命令 打印所有记录（与linux命令同）

parquet-tools cat:
Prints the content of a Parquet file. The output contains only the data, no
metadata is displayed
usage: parquet-tools cat [option...] 
where option is one of:
       --debug     Enable debug output
    -h,--help      Show this help string
    -j,--json      Show records in JSON format.
       --no-color  Disable color output even if supported
where  is the parquet file to print to stdout

• head 打印前面几条记录的数据（默认5条）（与linux命令同）

parquet-tools head:
Prints the first n record of the Parquet file
usage: parquet-tools head [option...] 
where option is one of:
       --debug          Enable debug output
    -h,--help           Show this help string
    -n,--records   The number of records to show (default: 5)
       --no-color       Disable color output even if supported
where  is the parquet file to print to stdout

• schema 打印parquet的schema

parquet-tools schema:
Prints the schema of Parquet file(s)
usage: parquet-tools schema [option...] 
where option is one of:
    -d,--detailed      Show detailed information about the schema.
       --debug         Enable debug output
    -h,--help          Show this help string
       --no-color      Disable color output even if supported
    -o,--originalType  Print logical types in OriginalType representation.
where  is the parquet file containing the schema to show

• meta 输出元数据信息（可以看到文件是否压缩，压缩的方式）

parquet-tools meta:
Prints the metadata of Parquet file(s)
usage: parquet-tools meta [option...] 
where option is one of:
       --debug         Enable debug output
    -h,--help          Show this help string
       --no-color      Disable color output even if supported
    -o,--originalType  Print logical types in OriginalType representation.
where  is the parquet file to print to stdout

• dump

parquet-tools dump:
Prints the content and metadata of a Parquet file
usage: parquet-tools dump [option...] 
where option is one of:
    -c,--column   Dump only the given column, can be specified more than
                       once
    -d,--disable-data  Do not dump column data
       --debug         Enable debug output
    -h,--help          Show this help string
    -m,--disable-meta  Do not dump row group and page metadata
    -n,--disable-crop  Do not crop the output based on console width
       --no-color      Disable color output even if supported
where  is the parquet file to print to stdout

• merge 看介绍应该是合并多个文件使用（我没有具体操作过）

parquet-tools merge:
Merges multiple Parquet files into one. The command doesn't merge row groups,
just places one after the other. When used to merge many small files, the
resulting file will still contain small row groups, which usually leads to bad
query performance.
usage: parquet-tools merge [option...]  [ ...] 

where option is one of:
       --debug     Enable debug output
    -h,--help      Show this help string
       --no-color  Disable color output even if supported
where  is the source parquet files/directory to be merged
   
 is the destination parquet file

• rowcount 打印记录数

parquet-tools rowcount:
Prints the count of rows in Parquet file(s)
usage: parquet-tools rowcount [option...] 
where option is one of:
    -d,--detailed  Detailed rowcount of each matching file
       --debug     Enable debug output
    -h,--help      Show this help string
       --no-color  Disable color output even if supported
where  is the parquet file to count rows to stdout

• size 打印文件的大小

parquet-tools size:
Prints the size of Parquet file(s)
usage: parquet-tools size [option...] 
where option is one of:
    -d,--detailed      Detailed size of each matching file
       --debug         Enable debug output
    -h,--help          Show this help string
       --no-color      Disable color output even if supported
    -p,--pretty        Pretty size
    -u,--uncompressed  Uncompressed size