D4RL的踩坑记录

D4RL 的env.get_normalized_score

在调用d4rl的库函数时用了env.get_normalized_score
这里遇到的问题是
env has no attribute get_normalized_score

后记

D4RL的用法指南
D4RL本质上是一个数据库，用于离线的强化学习
D4RL的env和gym的环境并不同
简单来说，gym提供的是在线的交互式env，而D4RL则是离线的env
最直观的来看名字就不同

#gym
env = gym.make('Hopper')
#D4RL
env = gym.make('hopper-medium-v0')

分析

首先的分析的问题是env写法的问题

到D4RL官网看一看
Normalizing Scores
You can use the env.get_normalized_score(returns) function to compute a normalized score for an episode, where returns is the undiscounted total sum of rewards accumulated during an episode.
我在网上找到了一个用法，似乎没有问题

分析一下是不是D4RL的安装问题

protobuf

按照官方的下载路线之后得到了以下的报错

可以看到问题是protobuf
尝试的方法1
pip uninstall protobuf
pip install protobuf==3.20.1
打开根目录下的/d4rl后发现，下载的文件可能有问题
计划重新下载安装
rm -rf files

d4rl 的下载方案

git clone https://github.com/rail-berkeley/d4rl.git
cd d4rl
pip install -e .

遇到的报错为

fatal: unable to access ‘https://github.com/aravindr93/mjrl/’: GnuTLS recv error (-110): The TLS connection was non-properly terminated.
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with ‘git status’
and retry with 'git restore --source=HEAD
尝试的解决方案为
问题成功解决了
重新安装成功了

运行安装测试

import gym
import d4rl # Import required to register environments

# Create the environment
env = gym.make('maze2d-umaze-v1')

# d4rl abides by the OpenAI gym interface
env.reset()
env.step(env.action_space.sample())

# Each task is associated with a dataset
# dataset contains observations, actions, rewards, terminals, and infos
dataset = env.get_dataset()
print(dataset['observations']) # An N x dim_observation Numpy array of observations

# Alternatively, use d4rl.qlearning_dataset which
# also adds next_observations.

测试的结果

运行到env = gym.make(‘maze2d-umaze-v1’)遇到的问题是

  logger.warn(
pybullet build time: May 20 2022 19:44:17
 env = gym.make('maze2d-umaze-v1')
Traceback (most recent call last):
  File "", line 1, in <module>
  File "/opt/conda/lib/python3.8/site-packages/gym/envs/registration.py", line 592, in make
    env = env_creator(**_kwargs)
  File "/d4rl/d4rl/pointmaze/maze_model.py", line 175, in __init__
    mujoco_env.MujocoEnv.__init__(self, model_path=f.name, frame_skip=1)
  File "/opt/conda/lib/python3.8/site-packages/gym/envs/mujoco/mujoco_env.py", line 97, in __init__
    observation, _reward, done, _info = self.step(action)
  File "/d4rl/d4rl/pointmaze/maze_model.py", line 192, in step
    self.clip_velocity()
  File "/d4rl/d4rl/pointmaze/maze_model.py", line 222, in clip_velocity
    qvel = np.clip(self.sim.data.qvel, -5.0, 5.0)
AttributeError: 'MazeEnv' object has no attribute 'sim'
  

import d4rl 的 warning

```python
>>> import d4rl
Warning: Flow failed to import. Set the environment variable D4RL_SUPPRESS_IMPORT_ERROR=1 to suppress this message.
No module named 'flow.envs'
/opt/conda/lib/python3.8/site-packages/glfw/__init__.py:906: GLFWError: (65544) b'X11: The DISPLAY environment variable is missing'
  warnings.warn(message, GLFWError)
/opt/conda/lib/python3.8/site-packages/flatbuffers/compat.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
/opt/conda/lib/python3.8/site-packages/keras/utils/image_utils.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
  'nearest': pil_image.NEAREST,
/opt/conda/lib/python3.8/site-packages/keras/utils/image_utils.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
  'bilinear': pil_image.BILINEAR,
/opt/conda/lib/python3.8/site-packages/keras/utils/image_utils.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
  'bicubic': pil_image.BICUBIC,
/opt/conda/lib/python3.8/site-packages/keras/utils/image_utils.py:39: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.
  'hamming': pil_image.HAMMING,
/opt/conda/lib/python3.8/site-packages/keras/utils/image_utils.py:40: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.
  'box': pil_image.BOX,
/opt/conda/lib/python3.8/site-packages/keras/utils/image_utils.py:41: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
  'lanczos': pil_image.LANCZOS,
Warning: CARLA failed to import. Set the environment variable D4RL_SUPPRESS_IMPORT_ERROR=1 to suppress this message.
No module named 'agents.navigation'
/opt/conda/lib/python3.8/site-packages/gym/envs/registration.py:415: UserWarning: WARN: The `registry.env_specs` property along with `EnvSpecTree` is deprecated. Please use `registry` directly as a dictionary instead.
  logger.warn(
pybullet build time: May 20 2022 19:44:17

似乎报错里面又keras的问题
报错里面似乎有flow和Carla的问题,但是从这张图上看,maze2d似乎和这两个环境的关系并不大

mujoco似乎没有问题
import mujoco
import mujoco_py

这里面绕了一个大圈子，包括测试和重新安装

其实本质要解决的核心问题是
‘MazeEnv’ object has no attribute ‘sim’
我分别在
google stackoverflow bing上搜索，并没有任何有用的结果
突然看到github一个不起眼的地方issue

I managed to fix this error @ivanvoid by modifying the gym version:

pip uninstall gym
pip install gym==0.23.1

由于原始的版本是gym 0.24.1
所以可能是版本过高不支持
问题解决了！！！！！！

事后来看的话，首先要对问题有针对性地解决，抓住重点

确实不容易一下子定位到gym，但是如果能一开始就搜索到那一条，问题很酷啊就会解决