python 自动下载ERA5 netCDF4格式数据 INFO Request is queued

2024-05-21 11:18:46,271 INFO Welcome to the CDS
2024-05-21 11:18:46,289 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-pressure-levels
2024-05-21 11:18:46,512 INFO Request is completed
2024-05-21 11:18:46,543 INFO Downloading https://download-0013-clone.copernicus-climate.eu/cache-compute-0013/cache/data2/adaptor.mars.internal-1716235630.3098733-25379-2-ba28b4a9-58c9-426e-b468-9288b408f697.nc to D:\ERA5\vertical_velovity\925\2024040821.nc (5K)
  0%|          | 0.00/5.04k [00:00 2024-05-21 11:18:47,772 INFO Welcome to the CDS
2024-05-21 11:18:47,780 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-pressure-levels
2024-05-21 11:18:47,978 INFO Request is queued

从日志提示信息显示看,请求pending,接口服务器存在问题,相当于去窗口处理事情,窗口等待。

存在的问题:代码不报错,一直pending,对于自动下载数据时间成本较大,

解决思路:自动跳过排队等待时间长的时刻,并记录到缺测文件夹

解决方案: 

在D:\python\Lib\site-packages\cdsapi,目录下找到安装包的源代码,修改request请求响应时间找到timeout,retry_max 参数,源码中设置的为60,500.经过实验将其改成0.4,5在响应时间与排队等待时间会相对较优。(注意,响应时间变小以后会造成访问网站频次增加,对网站造成压力,可能会有封号的风险)

代码:

import cdsapi
import os

c = cdsapi.Client()

#参数设置,均为字符串列表
year = '2024'
month = '04'
day = ['08']                                 #日期列表
time = ['21:00', '22:00', '23:00']           #hourly列表
variable = ['vertical_velovity']             #变量列表
pressure_level =['925']                      #高度层列表 

#数据下载
for v in variable:
    for l in pressure_level:
        fo_path = 'D:\\ERA5\\'+v+'\\'+l      #文件存储路径设置
        folder = os.path.exists(fo_path)
        if not folder:
            os.makedirs(fo_path)
        for d in day:
            for t in time:
                try:
                    fi_path = fo_path+'\\'+year+month+d+t[0:2]+'.nc'    #文件命名
                    c.retrieve(
                    'reanalysis-era5-pressure-levels',
                    {
                        'product_type': 'reanalysis',
                        'format': 'netcdf',
                        'variable': v,
                        'pressure_level':l ,
                        'year': year,
                        'month': month,
                        'day':d, 
                        'time':t,
                        'area': [
                            35, 90, 0,
                            140,
                        ],
                        'grid': [1,1],
                    },
                    fi_path)
                except Exception as e: #响应时间过长,缺失文件记录
                    print(e)
                    fo_path = 'D:\\ERA5\\缺失'
                    folder = os.path.exists(fo_path)
                    if not folder:
                        os.makedirs(fo_path)
                    file_name = v+l+year+month+d+t[0:2]+'.nc'
                    fi_path = os.path.join(fo_path,file_name)
                    with open(fi_path,'w') as f:
                        print('文件缺失:',fi_path)
                    f.close()
                    

效果:

python 自动下载ERA5 netCDF4格式数据 INFO Request is queued_第1张图片

python 自动下载ERA5 netCDF4格式数据 INFO Request is queued_第2张图片

你可能感兴趣的:(python)