python 自动下载ERA5 netCDF4格式数据 INFO Request is queued
2024-05-21 11:18:46,271 INFO Welcome to the CDS
2024-05-21 11:18:46,289 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-pressure-levels
2024-05-21 11:18:46,512 INFO Request is completed
2024-05-21 11:18:46,543 INFO Downloading https://download-0013-clone.copernicus-climate.eu/cache-compute-0013/cache/data2/adaptor.mars.internal-1716235630.3098733-25379-2-ba28b4a9-58c9-426e-b468-9288b408f697.nc to D:\ERA5\vertical_velovity\925\2024040821.nc (5K)
0%| | 0.00/5.04k [00:00 2024-05-21 11:18:47,772 INFO Welcome to the CDS
2024-05-21 11:18:47,780 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-pressure-levels
2024-05-21 11:18:47,978 INFO Request is queued
从日志提示信息显示看，请求pending，接口服务器存在问题，相当于去窗口处理事情，窗口等待。
存在的问题：代码不报错，一直pending，对于自动下载数据时间成本较大，
解决思路：自动跳过排队等待时间长的时刻，并记录到缺测文件夹
解决方案：
在D:\python\Lib\site-packages\cdsapi，目录下找到安装包的源代码，修改request请求响应时间找到timeout，retry_max 参数，源码中设置的为60，500.经过实验将其改成0.4，5在响应时间与排队等待时间会相对较优。（注意，响应时间变小以后会造成访问网站频次增加，对网站造成压力，可能会有封号的风险）
代码：
import cdsapi
import os
c = cdsapi.Client()
#参数设置，均为字符串列表
year = '2024'
month = '04'
day = ['08'] #日期列表
time = ['21:00', '22:00', '23:00'] #hourly列表
variable = ['vertical_velovity'] #变量列表
pressure_level =['925'] #高度层列表
#数据下载
for v in variable:
for l in pressure_level:
fo_path = 'D:\\ERA5\\'+v+'\\'+l #文件存储路径设置
folder = os.path.exists(fo_path)
if not folder:
os.makedirs(fo_path)
for d in day:
for t in time:
try:
fi_path = fo_path+'\\'+year+month+d+t[0:2]+'.nc' #文件命名
c.retrieve(
'reanalysis-era5-pressure-levels',
{
'product_type': 'reanalysis',
'format': 'netcdf',
'variable': v,
'pressure_level':l ,
'year': year,
'month': month,
'day':d,
'time':t,
'area': [
35, 90, 0,
140,
],
'grid': [1,1],
},
fi_path)
except Exception as e: #响应时间过长，缺失文件记录
print(e)
fo_path = 'D:\\ERA5\\缺失'
folder = os.path.exists(fo_path)
if not folder:
os.makedirs(fo_path)
file_name = v+l+year+month+d+t[0:2]+'.nc'
fi_path = os.path.join(fo_path,file_name)
with open(fi_path,'w') as f:
print('文件缺失：',fi_path)
f.close()
效果：
