AItrust

强化学习算法 Sarsa 解迷宫游戏，代码逐条详解

本文内容源自百度强化学习 7 日入门课程学习整理
感谢百度 PARL 团队李科浇老师的课程讲解

强化学习算法 Sarsa 解迷宫游戏

文章目录

一、安装依赖库
二、导入依赖库
三、智能体 Agent 的算法：Sarsa
四、训练和测试语句
五、创建环境，实例化Agent，启动训练和测试
五、结果分析

一、安装依赖库

安装强化学习算法中环境库 Gym

pip install gym

二、导入依赖库

import gym
import numpy as np
import time # 用于延时程序，方便渲染画面

三、智能体 Agent 的算法：Sarsa

智能体 Agent 是和环境 environment 交互的主体
- 包含了观察当前状态
- 根据当前状态作出动作选择
- 根据选择后的结果更新 Q 值表
predict() 方法：输入观察值 observation（或者说状态state），输出 “预测” 动作 action （最优动作）
- 观察当前状态下，所有可以采用的 action 对应的 Q 值
- 在其中选取最大的，组成一个列表
- 该列表对应可能选取的最优动作列表
- 在最优动作列表中随机选取一个动作
sample() 方法：在 predict() 方法基础上使用 ε-greedy 增加探索，输出 “实际” 动作 action
- 采用 epsilon greedy 算法
- 90% 概率选择最优动作
- 10% 概率选择随机动作
learn() 方法：输入训练数据，完成一轮Q表格的更新
- 更新的是之前状态 obs 下采取动作 action 后的 Q 值
- 如果游戏结束，则 reward 为新的 Q 值
- 如果游戏没有结束，则 reward 和下一步的 Q 值结合产生新的 Q 值
- 同时用学习速率 lr 做更新约束

class SarsaAgent(object):
    def __init__(self, obs_n, act_n, learning_rate=0.01, gamma=0.9, e_greed=0.1):
        self.act_n = act_n      # 动作维度，有几个动作可选
        self.lr = learning_rate # 学习率
        self.gamma = gamma      # 后面的 Q 值对前面的影响
        self.epsilon = e_greed  # 按一定概率随机选动作
        self.Q = np.zeros((obs_n, act_n))

    # 根据输入观察值，采样输出的动作值（带 10% 的探索）
    def sample(self, obs):
        if (np.random.uniform(0, 1) < 1 - self.epsilon): # 这里是 90% 可能性
            action = self.predict(obs) # 执行最优动作
        else: # 10% 的概率
            action = np.random.choice(self.act_n) # 执行随机动作
        return action

    # 根据输入观察值，预测输出的动作值
    def predict(self, obs):
        Q_list = self.Q[obs, :] # 获取当前状态下，作出所有动作，对应的 Q 值列表
        maxQ = np.max(Q_list) # 求列表中的最大值
        action_list = np.where(Q_list == maxQ)[0] # 最大 Q 值对应的动作即最优动作
        action = np.random.choice(action_list) # 随机选择一个最优动作
        return action

    # 学习方法，也就是更新Q-table的方法
    def learn(self, obs, action, reward, next_obs, next_action, done):
        """ on-policy
            obs: 交互前的obs, s_t
            action: 本次交互选择的action, a_t
            reward: 本次动作获得的奖励r
            next_obs: 本次交互后的obs, s_t+1
            next_action: 根据当前Q表格, 针对next_obs会选择的动作, a_t+1
            done: episode是否结束
        """
        predict_Q = self.Q[obs,action] # 交互前的状态下，选择的动作所对应 Q 值
        if (done): # 游戏结束
            target_Q = reward # 新的 Q 值为 reward
        else: # 游戏没有结束
            target_Q = reward + self.gamma * self.Q[next_obs, next_action]
            # 用 reward 和 交互后状态下，选择的下一个动作对应的 Q 值，综合得到新的 Q 值
        self.Q[obs,action] += self.lr * (target_Q - predict_Q) # 使用 lr 做修正更新的幅度

    # 保存Q表格数据到文件
    def save(self):
        npy_file = './q_table.npy'
        np.save(npy_file, self.Q)
        print(npy_file + ' saved.')
    
    # 从文件中读取数据到Q表格中
    def restore(self, npy_file='./q_table.npy'):
        self.Q = np.load(npy_file)
        print(npy_file + ' loaded.')

四、训练和测试语句

每一局游戏，记录下步数 total_steps 和总奖励 total_reward

每一步都更新 Q 值表

def run_episode(env, agent, render=False):
    total_steps = 0 # 记录每个episode走了多少step
    total_reward = 0 # 记录每一局游戏的总奖励

    obs = env.reset() # 重置环境, 重新开一局（即开始新的一个episode）
    action = agent.sample(obs) # 根据算法选择一个动作

    while True:
        next_obs, reward, done, _ = env.step(action) # 与环境进行一个交互，执行动作
        next_action = agent.sample(next_obs) # 根据算法选择下一个动作
        # 训练 Sarsa 算法
        agent.learn(obs, action, reward, next_obs, next_action, done)
        # obs 执行动作前的状态，action 执行的动作，得到预测的 Q0
        # reward 执行动作后的奖励，next_obs 执行动作后的状态，next_action 选择的下一个动作，得到更新的 Q0
        # done 判断游戏是否结束
        

        action = next_action # 迭代新的动作
        obs = next_obs  # 存储上一个观察值，迭代新的状态
        total_reward += reward # 累计奖励
        total_steps += 1 # 计算step数
        if render: # 判断是否需要渲染图形显示
            env.render() #渲染新的一帧图形
        if done: # 游戏结束
            break # 跳出循环，即结束本局游戏
    return total_reward, total_steps # 返回总的奖励和总的步数

def test_episode(env, agent):
    total_reward = 0 # 记录总的奖励
    obs = env.reset() # 重置环境，obs 初始观察值，即初始状态
    while True:
        action = agent.predict(obs) # greedy，每次选择最优动作
        next_obs, reward, done, _ = env.step(action) # 交互后，获取新的状态，奖励，游戏是否结束
        total_reward += reward # 累计奖励
        obs = next_obs # 迭代更新状态
        time.sleep(0.5) # 休眠，以便于我们观察渲染的图形
        env.render() # 渲染图形显示
        if done: # 游戏结束
            break # 跳出循环
    return total_reward # 返回最终累计奖励

五、创建环境，实例化Agent，启动训练和测试

使用 Gym 库创建我们需要的环境

实例化 SarsaAgent 类，创建一个 Agent 对象，同时设定超参数

训练 500 局游戏，查看每一局游戏的结果

训练结束后进行测试

# 使用gym创建迷宫环境，设置is_slippery为False降低环境难度
env = gym.make("FrozenLake-v0", is_slippery=False)  # 0 left, 1 down, 2 right, 3 up
# 使用 make 方法创建需要的环境

# 创建一个agent实例，输入超参数
agent = SarsaAgent(
        obs_n=env.observation_space.n, # 16 个状态代表这个环境中 4*4 一共 16 个格子
        act_n=env.action_space.n, # 4 种动作选择：0 left, 1 down, 2 right, 3 up
        learning_rate=0.1, # 学习速率
        gamma=0.9, # 下一步的影响率
        e_greed=0.1) # 随机选择概率


# 训练500个episode，打印每个episode的分数
for episode in range(500):
    ep_reward, ep_steps = run_episode(env, agent, False)
    print('Episode %s: steps = %s , reward = %.1f' % (episode, ep_steps, ep_reward))

# 全部训练结束，查看算法效果
test_reward = test_episode(env, agent)
print('test reward = %.1f' % (test_reward))

运行结果：

Episode 0: steps = 6 , reward = 0.0
Episode 1: steps = 17 , reward = 0.0
Episode 2: steps = 9 , reward = 0.0
Episode 3: steps = 2 , reward = 0.0
Episode 4: steps = 8 , reward = 0.0
Episode 5: steps = 8 , reward = 0.0
Episode 6: steps = 14 , reward = 0.0
Episode 7: steps = 7 , reward = 0.0
Episode 8: steps = 7 , reward = 0.0
Episode 9: steps = 2 , reward = 0.0
Episode 10: steps = 3 , reward = 0.0
Episode 11: steps = 8 , reward = 0.0
Episode 12: steps = 3 , reward = 0.0
Episode 13: steps = 8 , reward = 0.0
Episode 14: steps = 6 , reward = 0.0
Episode 15: steps = 5 , reward = 0.0
Episode 16: steps = 5 , reward = 0.0
Episode 17: steps = 7 , reward = 0.0
Episode 18: steps = 2 , reward = 0.0
Episode 19: steps = 7 , reward = 0.0
Episode 20: steps = 2 , reward = 0.0
Episode 21: steps = 7 , reward = 0.0
Episode 22: steps = 6 , reward = 0.0
Episode 23: steps = 3 , reward = 0.0
Episode 24: steps = 4 , reward = 0.0
Episode 25: steps = 4 , reward = 0.0
Episode 26: steps = 17 , reward = 0.0
Episode 27: steps = 11 , reward = 0.0
Episode 28: steps = 4 , reward = 0.0
Episode 29: steps = 9 , reward = 0.0
Episode 30: steps = 3 , reward = 0.0
Episode 31: steps = 11 , reward = 0.0
Episode 32: steps = 7 , reward = 0.0
Episode 33: steps = 3 , reward = 0.0
Episode 34: steps = 16 , reward = 0.0
Episode 35: steps = 10 , reward = 0.0
Episode 36: steps = 2 , reward = 0.0
Episode 37: steps = 9 , reward = 0.0
Episode 38: steps = 9 , reward = 0.0
Episode 39: steps = 19 , reward = 1.0
Episode 40: steps = 6 , reward = 0.0
Episode 41: steps = 6 , reward = 0.0
Episode 42: steps = 7 , reward = 0.0
Episode 43: steps = 4 , reward = 0.0
Episode 44: steps = 4 , reward = 0.0
Episode 45: steps = 5 , reward = 0.0
Episode 46: steps = 4 , reward = 0.0
Episode 47: steps = 22 , reward = 1.0
Episode 48: steps = 2 , reward = 0.0
Episode 49: steps = 2 , reward = 0.0
Episode 50: steps = 2 , reward = 0.0
Episode 51: steps = 17 , reward = 0.0
Episode 52: steps = 14 , reward = 0.0
Episode 53: steps = 6 , reward = 0.0
Episode 54: steps = 8 , reward = 0.0
Episode 55: steps = 18 , reward = 0.0
Episode 56: steps = 5 , reward = 0.0
Episode 57: steps = 2 , reward = 0.0
Episode 58: steps = 8 , reward = 0.0
Episode 59: steps = 4 , reward = 0.0
Episode 60: steps = 10 , reward = 0.0
Episode 61: steps = 2 , reward = 0.0
Episode 62: steps = 11 , reward = 0.0
Episode 63: steps = 21 , reward = 0.0
Episode 64: steps = 4 , reward = 0.0
Episode 65: steps = 2 , reward = 0.0
Episode 66: steps = 3 , reward = 0.0
Episode 67: steps = 3 , reward = 0.0
Episode 68: steps = 18 , reward = 1.0
Episode 69: steps = 6 , reward = 0.0
Episode 70: steps = 8 , reward = 0.0
Episode 71: steps = 8 , reward = 0.0
Episode 72: steps = 4 , reward = 0.0
Episode 73: steps = 13 , reward = 0.0
Episode 74: steps = 3 , reward = 0.0
Episode 75: steps = 7 , reward = 0.0
Episode 76: steps = 8 , reward = 0.0
Episode 77: steps = 3 , reward = 0.0
Episode 78: steps = 7 , reward = 0.0
Episode 79: steps = 8 , reward = 0.0
Episode 80: steps = 7 , reward = 0.0
Episode 81: steps = 10 , reward = 1.0
Episode 82: steps = 6 , reward = 1.0
Episode 83: steps = 9 , reward = 1.0
Episode 84: steps = 6 , reward = 0.0
Episode 85: steps = 6 , reward = 1.0
Episode 86: steps = 3 , reward = 0.0
Episode 87: steps = 7 , reward = 1.0
Episode 88: steps = 6 , reward = 1.0
Episode 89: steps = 7 , reward = 1.0
Episode 90: steps = 6 , reward = 1.0
Episode 91: steps = 6 , reward = 1.0
Episode 92: steps = 10 , reward = 1.0
Episode 93: steps = 6 , reward = 1.0
Episode 94: steps = 8 , reward = 1.0
Episode 95: steps = 6 , reward = 1.0
Episode 96: steps = 7 , reward = 1.0
Episode 97: steps = 6 , reward = 1.0
Episode 98: steps = 6 , reward = 1.0
Episode 99: steps = 8 , reward = 1.0
Episode 100: steps = 6 , reward = 1.0
Episode 101: steps = 8 , reward = 1.0
Episode 102: steps = 6 , reward = 1.0
Episode 103: steps = 6 , reward = 1.0
Episode 104: steps = 6 , reward = 1.0
Episode 105: steps = 8 , reward = 1.0
Episode 106: steps = 6 , reward = 1.0
Episode 107: steps = 6 , reward = 1.0
Episode 108: steps = 6 , reward = 1.0
Episode 109: steps = 6 , reward = 1.0
Episode 110: steps = 4 , reward = 0.0
Episode 111: steps = 6 , reward = 1.0
Episode 112: steps = 6 , reward = 1.0
Episode 113: steps = 6 , reward = 1.0
Episode 114: steps = 6 , reward = 1.0
Episode 115: steps = 7 , reward = 1.0
Episode 116: steps = 7 , reward = 1.0
Episode 117: steps = 10 , reward = 1.0
Episode 118: steps = 5 , reward = 0.0
Episode 119: steps = 6 , reward = 1.0
Episode 120: steps = 3 , reward = 0.0
Episode 121: steps = 6 , reward = 1.0
Episode 122: steps = 6 , reward = 1.0
Episode 123: steps = 9 , reward = 1.0
Episode 124: steps = 6 , reward = 1.0
Episode 125: steps = 5 , reward = 0.0
Episode 126: steps = 6 , reward = 1.0
Episode 127: steps = 6 , reward = 1.0
Episode 128: steps = 8 , reward = 1.0
Episode 129: steps = 6 , reward = 1.0
Episode 130: steps = 6 , reward = 1.0
Episode 131: steps = 8 , reward = 1.0
Episode 132: steps = 8 , reward = 1.0
Episode 133: steps = 6 , reward = 1.0
Episode 134: steps = 6 , reward = 1.0
Episode 135: steps = 6 , reward = 1.0
Episode 136: steps = 6 , reward = 1.0
Episode 137: steps = 6 , reward = 1.0
Episode 138: steps = 6 , reward = 1.0
Episode 139: steps = 4 , reward = 0.0
Episode 140: steps = 6 , reward = 1.0
Episode 141: steps = 6 , reward = 1.0
Episode 142: steps = 6 , reward = 1.0
Episode 143: steps = 9 , reward = 1.0
Episode 144: steps = 6 , reward = 1.0
Episode 145: steps = 6 , reward = 1.0
Episode 146: steps = 6 , reward = 1.0
Episode 147: steps = 7 , reward = 1.0
Episode 148: steps = 7 , reward = 1.0
Episode 149: steps = 6 , reward = 1.0
Episode 150: steps = 6 , reward = 1.0
Episode 151: steps = 6 , reward = 1.0
Episode 152: steps = 7 , reward = 1.0
Episode 153: steps = 6 , reward = 1.0
Episode 154: steps = 6 , reward = 1.0
Episode 155: steps = 7 , reward = 1.0
Episode 156: steps = 7 , reward = 1.0
Episode 157: steps = 7 , reward = 1.0
Episode 158: steps = 6 , reward = 1.0
Episode 159: steps = 6 , reward = 1.0
Episode 160: steps = 6 , reward = 1.0
Episode 161: steps = 4 , reward = 0.0
Episode 162: steps = 6 , reward = 1.0
Episode 163: steps = 5 , reward = 0.0
Episode 164: steps = 6 , reward = 1.0
Episode 165: steps = 6 , reward = 1.0
Episode 166: steps = 6 , reward = 1.0
Episode 167: steps = 6 , reward = 1.0
Episode 168: steps = 9 , reward = 1.0
Episode 169: steps = 6 , reward = 1.0
Episode 170: steps = 8 , reward = 1.0
Episode 171: steps = 6 , reward = 1.0
Episode 172: steps = 6 , reward = 1.0
Episode 173: steps = 6 , reward = 1.0
Episode 174: steps = 6 , reward = 1.0
Episode 175: steps = 6 , reward = 1.0
Episode 176: steps = 6 , reward = 1.0
Episode 177: steps = 6 , reward = 1.0
Episode 178: steps = 8 , reward = 1.0
Episode 179: steps = 6 , reward = 1.0
Episode 180: steps = 6 , reward = 1.0
Episode 181: steps = 3 , reward = 0.0
Episode 182: steps = 6 , reward = 1.0
Episode 183: steps = 6 , reward = 1.0
Episode 184: steps = 6 , reward = 1.0
Episode 185: steps = 8 , reward = 1.0
Episode 186: steps = 10 , reward = 1.0
Episode 187: steps = 8 , reward = 1.0
Episode 188: steps = 6 , reward = 1.0
Episode 189: steps = 6 , reward = 1.0
Episode 190: steps = 6 , reward = 1.0
Episode 191: steps = 6 , reward = 1.0
Episode 192: steps = 7 , reward = 1.0
Episode 193: steps = 6 , reward = 1.0
Episode 194: steps = 6 , reward = 1.0
Episode 195: steps = 8 , reward = 1.0
Episode 196: steps = 6 , reward = 1.0
Episode 197: steps = 4 , reward = 0.0
Episode 198: steps = 5 , reward = 0.0
Episode 199: steps = 6 , reward = 1.0
Episode 200: steps = 6 , reward = 1.0
Episode 201: steps = 6 , reward = 1.0
Episode 202: steps = 4 , reward = 0.0
Episode 203: steps = 8 , reward = 1.0
Episode 204: steps = 8 , reward = 1.0
Episode 205: steps = 7 , reward = 1.0
Episode 206: steps = 6 , reward = 1.0
Episode 207: steps = 6 , reward = 1.0
Episode 208: steps = 6 , reward = 1.0
Episode 209: steps = 8 , reward = 1.0
Episode 210: steps = 7 , reward = 1.0
Episode 211: steps = 6 , reward = 1.0
Episode 212: steps = 6 , reward = 1.0
Episode 213: steps = 10 , reward = 1.0
Episode 214: steps = 6 , reward = 1.0
Episode 215: steps = 6 , reward = 1.0
Episode 216: steps = 6 , reward = 1.0
Episode 217: steps = 6 , reward = 1.0
Episode 218: steps = 6 , reward = 1.0
Episode 219: steps = 6 , reward = 1.0
Episode 220: steps = 6 , reward = 1.0
Episode 221: steps = 7 , reward = 1.0
Episode 222: steps = 6 , reward = 1.0
Episode 223: steps = 6 , reward = 1.0
Episode 224: steps = 6 , reward = 1.0
Episode 225: steps = 6 , reward = 1.0
Episode 226: steps = 6 , reward = 1.0
Episode 227: steps = 6 , reward = 1.0
Episode 228: steps = 7 , reward = 1.0
Episode 229: steps = 6 , reward = 1.0
Episode 230: steps = 6 , reward = 1.0
Episode 231: steps = 10 , reward = 1.0
Episode 232: steps = 6 , reward = 1.0
Episode 233: steps = 6 , reward = 1.0
Episode 234: steps = 6 , reward = 1.0
Episode 235: steps = 8 , reward = 1.0
Episode 236: steps = 6 , reward = 1.0
Episode 237: steps = 6 , reward = 1.0
Episode 238: steps = 6 , reward = 1.0
Episode 239: steps = 8 , reward = 1.0
Episode 240: steps = 6 , reward = 1.0
Episode 241: steps = 6 , reward = 1.0
Episode 242: steps = 8 , reward = 1.0
Episode 243: steps = 2 , reward = 0.0
Episode 244: steps = 6 , reward = 1.0
Episode 245: steps = 6 , reward = 1.0
Episode 246: steps = 6 , reward = 1.0
Episode 247: steps = 6 , reward = 1.0
Episode 248: steps = 6 , reward = 1.0
Episode 249: steps = 6 , reward = 1.0
Episode 250: steps = 7 , reward = 1.0
Episode 251: steps = 6 , reward = 1.0
Episode 252: steps = 2 , reward = 0.0
Episode 253: steps = 6 , reward = 1.0
Episode 254: steps = 6 , reward = 1.0
Episode 255: steps = 6 , reward = 1.0
Episode 256: steps = 8 , reward = 1.0
Episode 257: steps = 6 , reward = 1.0
Episode 258: steps = 6 , reward = 1.0
Episode 259: steps = 7 , reward = 1.0
Episode 260: steps = 6 , reward = 1.0
Episode 261: steps = 6 , reward = 1.0
Episode 262: steps = 7 , reward = 1.0
Episode 263: steps = 6 , reward = 1.0
Episode 264: steps = 6 , reward = 1.0
Episode 265: steps = 6 , reward = 1.0
Episode 266: steps = 6 , reward = 1.0
Episode 267: steps = 7 , reward = 1.0
Episode 268: steps = 6 , reward = 1.0
Episode 269: steps = 6 , reward = 1.0
Episode 270: steps = 6 , reward = 1.0
Episode 271: steps = 6 , reward = 1.0
Episode 272: steps = 6 , reward = 1.0
Episode 273: steps = 7 , reward = 1.0
Episode 274: steps = 3 , reward = 0.0
Episode 275: steps = 8 , reward = 1.0
Episode 276: steps = 7 , reward = 1.0
Episode 277: steps = 4 , reward = 0.0
Episode 278: steps = 6 , reward = 1.0
Episode 279: steps = 4 , reward = 0.0
Episode 280: steps = 7 , reward = 1.0
Episode 281: steps = 6 , reward = 1.0
Episode 282: steps = 6 , reward = 1.0
Episode 283: steps = 6 , reward = 1.0
Episode 284: steps = 6 , reward = 1.0
Episode 285: steps = 7 , reward = 1.0
Episode 286: steps = 8 , reward = 1.0
Episode 287: steps = 6 , reward = 1.0
Episode 288: steps = 5 , reward = 0.0
Episode 289: steps = 8 , reward = 1.0
Episode 290: steps = 7 , reward = 1.0
Episode 291: steps = 8 , reward = 1.0
Episode 292: steps = 4 , reward = 0.0
Episode 293: steps = 6 , reward = 1.0
Episode 294: steps = 9 , reward = 1.0
Episode 295: steps = 6 , reward = 1.0
Episode 296: steps = 6 , reward = 1.0
Episode 297: steps = 6 , reward = 0.0
Episode 298: steps = 6 , reward = 1.0
Episode 299: steps = 6 , reward = 1.0
Episode 300: steps = 6 , reward = 1.0
Episode 301: steps = 5 , reward = 0.0
Episode 302: steps = 6 , reward = 1.0
Episode 303: steps = 7 , reward = 1.0
Episode 304: steps = 6 , reward = 1.0
Episode 305: steps = 8 , reward = 1.0
Episode 306: steps = 6 , reward = 1.0
Episode 307: steps = 6 , reward = 1.0
Episode 308: steps = 6 , reward = 1.0
Episode 309: steps = 6 , reward = 1.0
Episode 310: steps = 4 , reward = 0.0
Episode 311: steps = 7 , reward = 1.0
Episode 312: steps = 8 , reward = 1.0
Episode 313: steps = 7 , reward = 1.0
Episode 314: steps = 6 , reward = 1.0
Episode 315: steps = 6 , reward = 1.0
Episode 316: steps = 7 , reward = 1.0
Episode 317: steps = 6 , reward = 1.0
Episode 318: steps = 6 , reward = 1.0
Episode 319: steps = 6 , reward = 1.0
Episode 320: steps = 6 , reward = 1.0
Episode 321: steps = 6 , reward = 1.0
Episode 322: steps = 7 , reward = 1.0
Episode 323: steps = 6 , reward = 1.0
Episode 324: steps = 6 , reward = 1.0
Episode 325: steps = 6 , reward = 1.0
Episode 326: steps = 6 , reward = 1.0
Episode 327: steps = 6 , reward = 1.0
Episode 328: steps = 6 , reward = 1.0
Episode 329: steps = 6 , reward = 1.0
Episode 330: steps = 6 , reward = 1.0
Episode 331: steps = 6 , reward = 1.0
Episode 332: steps = 6 , reward = 1.0
Episode 333: steps = 6 , reward = 1.0
Episode 334: steps = 3 , reward = 0.0
Episode 335: steps = 6 , reward = 1.0
Episode 336: steps = 6 , reward = 1.0
Episode 337: steps = 4 , reward = 0.0
Episode 338: steps = 6 , reward = 1.0
Episode 339: steps = 8 , reward = 1.0
Episode 340: steps = 6 , reward = 1.0
Episode 341: steps = 6 , reward = 1.0
Episode 342: steps = 6 , reward = 1.0
Episode 343: steps = 6 , reward = 1.0
Episode 344: steps = 6 , reward = 1.0
Episode 345: steps = 6 , reward = 1.0
Episode 346: steps = 6 , reward = 1.0
Episode 347: steps = 6 , reward = 1.0
Episode 348: steps = 6 , reward = 1.0
Episode 349: steps = 6 , reward = 1.0
Episode 350: steps = 6 , reward = 1.0
Episode 351: steps = 7 , reward = 1.0
Episode 352: steps = 6 , reward = 1.0
Episode 353: steps = 10 , reward = 1.0
Episode 354: steps = 3 , reward = 0.0
Episode 355: steps = 7 , reward = 1.0
Episode 356: steps = 7 , reward = 1.0
Episode 357: steps = 6 , reward = 1.0
Episode 358: steps = 2 , reward = 0.0
Episode 359: steps = 6 , reward = 1.0
Episode 360: steps = 6 , reward = 1.0
Episode 361: steps = 6 , reward = 1.0
Episode 362: steps = 7 , reward = 1.0
Episode 363: steps = 8 , reward = 1.0
Episode 364: steps = 6 , reward = 1.0
Episode 365: steps = 2 , reward = 0.0
Episode 366: steps = 6 , reward = 1.0
Episode 367: steps = 5 , reward = 0.0
Episode 368: steps = 6 , reward = 1.0
Episode 369: steps = 6 , reward = 1.0
Episode 370: steps = 6 , reward = 1.0
Episode 371: steps = 6 , reward = 1.0
Episode 372: steps = 6 , reward = 1.0
Episode 373: steps = 6 , reward = 1.0
Episode 374: steps = 8 , reward = 1.0
Episode 375: steps = 9 , reward = 1.0
Episode 376: steps = 6 , reward = 0.0
Episode 377: steps = 6 , reward = 1.0
Episode 378: steps = 6 , reward = 1.0
Episode 379: steps = 8 , reward = 1.0
Episode 380: steps = 6 , reward = 1.0
Episode 381: steps = 6 , reward = 1.0
Episode 382: steps = 6 , reward = 1.0
Episode 383: steps = 6 , reward = 1.0
Episode 384: steps = 6 , reward = 1.0
Episode 385: steps = 6 , reward = 1.0
Episode 386: steps = 8 , reward = 1.0
Episode 387: steps = 6 , reward = 1.0
Episode 388: steps = 6 , reward = 1.0
Episode 389: steps = 2 , reward = 0.0
Episode 390: steps = 6 , reward = 1.0
Episode 391: steps = 6 , reward = 1.0
Episode 392: steps = 6 , reward = 1.0
Episode 393: steps = 6 , reward = 1.0
Episode 394: steps = 7 , reward = 1.0
Episode 395: steps = 6 , reward = 1.0
Episode 396: steps = 6 , reward = 1.0
Episode 397: steps = 6 , reward = 1.0
Episode 398: steps = 6 , reward = 1.0
Episode 399: steps = 7 , reward = 1.0
Episode 400: steps = 6 , reward = 1.0
Episode 401: steps = 6 , reward = 1.0
Episode 402: steps = 6 , reward = 1.0
Episode 403: steps = 6 , reward = 1.0
Episode 404: steps = 8 , reward = 1.0
Episode 405: steps = 6 , reward = 1.0
Episode 406: steps = 6 , reward = 1.0
Episode 407: steps = 6 , reward = 1.0
Episode 408: steps = 6 , reward = 1.0
Episode 409: steps = 6 , reward = 1.0
Episode 410: steps = 6 , reward = 1.0
Episode 411: steps = 6 , reward = 1.0
Episode 412: steps = 6 , reward = 1.0
Episode 413: steps = 6 , reward = 1.0
Episode 414: steps = 6 , reward = 1.0
Episode 415: steps = 9 , reward = 1.0
Episode 416: steps = 6 , reward = 1.0
Episode 417: steps = 4 , reward = 0.0
Episode 418: steps = 6 , reward = 1.0
Episode 419: steps = 6 , reward = 1.0
Episode 420: steps = 7 , reward = 1.0
Episode 421: steps = 6 , reward = 1.0
Episode 422: steps = 6 , reward = 1.0
Episode 423: steps = 10 , reward = 1.0
Episode 424: steps = 6 , reward = 1.0
Episode 425: steps = 6 , reward = 1.0
Episode 426: steps = 8 , reward = 1.0
Episode 427: steps = 6 , reward = 1.0
Episode 428: steps = 9 , reward = 1.0
Episode 429: steps = 6 , reward = 1.0
Episode 430: steps = 4 , reward = 0.0
Episode 431: steps = 6 , reward = 1.0
Episode 432: steps = 6 , reward = 1.0
Episode 433: steps = 6 , reward = 1.0
Episode 434: steps = 6 , reward = 1.0
Episode 435: steps = 8 , reward = 1.0
Episode 436: steps = 6 , reward = 1.0
Episode 437: steps = 6 , reward = 1.0
Episode 438: steps = 6 , reward = 1.0
Episode 439: steps = 8 , reward = 1.0
Episode 440: steps = 2 , reward = 0.0
Episode 441: steps = 6 , reward = 1.0
Episode 442: steps = 10 , reward = 1.0
Episode 443: steps = 6 , reward = 1.0
Episode 444: steps = 6 , reward = 1.0
Episode 445: steps = 8 , reward = 1.0
Episode 446: steps = 6 , reward = 1.0
Episode 447: steps = 6 , reward = 1.0
Episode 448: steps = 5 , reward = 0.0
Episode 449: steps = 6 , reward = 1.0
Episode 450: steps = 8 , reward = 1.0
Episode 451: steps = 6 , reward = 1.0
Episode 452: steps = 8 , reward = 1.0
Episode 453: steps = 8 , reward = 1.0
Episode 454: steps = 7 , reward = 1.0
Episode 455: steps = 5 , reward = 0.0
Episode 456: steps = 6 , reward = 1.0
Episode 457: steps = 6 , reward = 1.0
Episode 458: steps = 8 , reward = 1.0
Episode 459: steps = 8 , reward = 1.0
Episode 460: steps = 10 , reward = 1.0
Episode 461: steps = 8 , reward = 1.0
Episode 462: steps = 7 , reward = 1.0
Episode 463: steps = 7 , reward = 1.0
Episode 464: steps = 6 , reward = 1.0
Episode 465: steps = 6 , reward = 1.0
Episode 466: steps = 6 , reward = 1.0
Episode 467: steps = 6 , reward = 1.0
Episode 468: steps = 6 , reward = 1.0
Episode 469: steps = 6 , reward = 1.0
Episode 470: steps = 3 , reward = 0.0
Episode 471: steps = 7 , reward = 1.0
Episode 472: steps = 6 , reward = 1.0
Episode 473: steps = 6 , reward = 1.0
Episode 474: steps = 7 , reward = 1.0
Episode 475: steps = 6 , reward = 1.0
Episode 476: steps = 8 , reward = 1.0
Episode 477: steps = 6 , reward = 1.0
Episode 478: steps = 6 , reward = 1.0
Episode 479: steps = 6 , reward = 1.0
Episode 480: steps = 6 , reward = 1.0
Episode 481: steps = 6 , reward = 1.0
Episode 482: steps = 6 , reward = 1.0
Episode 483: steps = 6 , reward = 1.0
Episode 484: steps = 5 , reward = 0.0
Episode 485: steps = 6 , reward = 1.0
Episode 486: steps = 9 , reward = 1.0
Episode 487: steps = 7 , reward = 1.0
Episode 488: steps = 6 , reward = 1.0
Episode 489: steps = 6 , reward = 1.0
Episode 490: steps = 6 , reward = 1.0
Episode 491: steps = 6 , reward = 1.0
Episode 492: steps = 9 , reward = 1.0
Episode 493: steps = 6 , reward = 1.0
Episode 494: steps = 6 , reward = 1.0
Episode 495: steps = 9 , reward = 1.0
Episode 496: steps = 6 , reward = 1.0
Episode 497: steps = 6 , reward = 1.0
Episode 498: steps = 6 , reward = 1.0
Episode 499: steps = 7 , reward = 1.0
  (Down)
SFFF
FHFH
FFFH
HFFG
  (Down)
SFFF
FHFH
FFFH
HFFG
  (Right)
SFFF
FHFH
FFFH
HFFG
  (Down)
SFFF
FHFH
FFFH
HFFG
  (Right)
SFFF
FHFH
FFFH
HFFG
  (Right)
SFFF
FHFH
FFFH
HFFG
test reward = 1.0

五、结果分析

我们可以查看下最终训练完成的 Q 表：

print(agent.Q)

运行结果：

[[0.27140285 0.4364344  0.09145568 0.15201279]
 [0.26813138 0.         0.         0.00945424]
 [0.         0.         0.         0.        ]
 [0.         0.         0.         0.        ]
 [0.26636559 0.51632351 0.         0.13684245]
 [0.         0.         0.         0.        ]
 [0.         0.         0.         0.        ]
 [0.         0.         0.         0.        ]
 [0.33346755 0.         0.68004322 0.31572772]
 [0.26970648 0.77477987 0.35436455 0.        ]
 [0.04662094 0.73217092 0.         0.        ]
 [0.         0.         0.         0.        ]
 [0.         0.         0.         0.        ]
 [0.         0.39939922 0.88159607 0.11581402]
 [0.4472322  0.72976712 1.         0.40947544]
 [0.         0.         0.         0.        ]]

16 个格子对应的情况：

SFFF
FHFH
FFFH
HFFG

其中 S 代表起点，F 代表平地，H 代表陷阱（掉进去游戏结束），G 代表终点（到达则获胜）

每个格子的排序序号：

0  1  2  3
4  5  6  7
8  9  10 11
12 13 14 15

所以测试开始后，首先在第 0 格，这个时候的 4 个动作对应的 Q 值是：

[0.27140285 0.4364344  0.09145568 0.15201279]

这 4 个 Q 值对应：0 left，1 down，2 right，3 up

所以最大值 0.4364344 对应的是 1，即动作为往下走一格

这个时候到达了第 4 个格子：

[0.26636559 0.51632351 0.         0.13684245]

选择 1，动作：down，到达第 8 个格子：

[0.33346755 0.         0.68004322 0.31572772]

选择 2，动作：right，到达第 9 个格子：

[0.26970648 0.77477987 0.35436455 0.        ]

选择 1，动作：down，到达第 13 个格子：

[0.         0.39939922 0.88159607 0.11581402]

选择 2，动作 right，到达第 14 个格子：

[0.4472322  0.72976712 1.         0.40947544]

选择 2，动作 right，到达第 15 个格子：终点！

你可能感兴趣的:(强化学习,算法,强化学习,人工智能,百度)

机器学习与深度学习间关系与区别 ℒℴѵℯ心·动ꦿ໊ོ꫞ 人工智能学习深度学习 python
一、机器学习概述定义机器学习（MachineLearning,ML）是一种通过数据驱动的方法，利用统计学和计算算法来训练模型，使计算机能够从数据中学习并自动进行预测或决策。机器学习通过分析大量数据样本，识别其中的模式和规律，从而对新的数据进行判断。其核心在于通过训练过程，让模型不断优化和提升其预测准确性。主要类型1.监督学习（SupervisedLearning）监督学习是指在训练数据集中包含输入
网易严选官方旗舰店，优质商品，卓越服务高省_飞智666600
网易严选官方旗舰店是网易旗下的一家电商平台，以提供优质商品和卓越服务而闻名。作为一名SEO优化师，我将为您详细介绍网易严选官方旗舰店，并重点强调其特点和优势。大家好！我是高省APP最大团队&联合创始人飞智导师。相较于其他返利app，高省APP的佣金更高，模式更好，最重要的是，终端用户不会流失！高省APP佣金更高，模式更好，终端用户不流失。【高省】是一个自用省钱佣金高，分享推广赚钱多的平台，百度有几
Goolge earth studio 进阶4——路径修改与平滑陟彼高冈yu Google earth studio 进阶教程旅游
如果我们希望在大约中途时获得更多的城市鸟瞰视角。可以将相机拖动到这里并创建一个新的关键帧。camera_target_clip_7EarthStudio会自动平滑我们的路径，所以当我们通过这个关键帧时，不是一个生硬的角度，而是一个平滑的曲线。camera_target_clip_8路径上有贝塞尔控制手柄，允许我们调整路径的形状。右键单击，我们可以选择“平滑路径”，这是默认的自动平滑算法，或者我们可
将cmd中命令输出保存为txt文本文件落难Coder Windows cmd window
最近深度学习本地的训练中我们常常要在命令行中运行自己的代码，无可厚非，我们有必要保存我们的炼丹结果，但是复制命令行输出到txt是非常麻烦的，其实Windows下的命令行为我们提供了相应的操作。其基本的调用格式就是：运行指令>输出到的文件名称或者具体保存路径测试下，我打开cmd并且ping一下百度：pingwww.baidu.com>./data.txt看下相同目录下data.txt的输出：如果你再
基于社交网络算法优化的二维最大熵图像分割智能算法研学社（Jack旭）智能优化算法应用图像分割算法 php 开发语言
智能优化算法应用：基于社交网络优化的二维最大熵图像阈值分割-附代码文章目录智能优化算法应用：基于社交网络优化的二维最大熵图像阈值分割-附代码1.前言2.二维最大熵阈值分割原理3.基于社交网络优化的多阈值分割4.算法结果：5.参考文献：6.Matlab代码摘要：本文介绍基于最大熵的图像分割，并且应用社交网络算法进行阈值寻优。1.前言阅读此文章前，请阅读《图像分割：直方图区域划分及信息统计介绍》htt
直返最高等级与直返APP：无需邀请码的返利新体验古楼
随着互联网的普及和电商的兴起，直返模式逐渐成为一种流行的商业模式。在这种模式下，消费者通过购买产品或服务，获得一定的返利，并可以分享给更多的人。其中，直返最高等级和直返APP是直返模式中的重要概念和工具。本文将详细介绍直返最高等级的概念、直返APP的使用以及与邀请码的关系。【高省】APP（高佣金领导者）是一个自用省钱佣金高，分享推广赚钱多的平台，百度有几百万篇报道，运行三年，稳定可靠。高省APP，
探索OpenAI和LangChain的适配器集成：轻松切换模型提供商 nseejrukjhad langchain easyui 前端 python
#探索OpenAI和LangChain的适配器集成：轻松切换模型提供商##引言在人工智能和自然语言处理的世界中，OpenAI的模型提供了强大的能力。然而，随着技术的发展，许多人开始探索其他模型以满足特定需求。LangChain作为一个强大的工具，集成了多种模型提供商，通过提供适配器，简化了不同模型之间的转换。本篇文章将介绍如何使用LangChain的适配器与OpenAI集成，以便轻松切换模型提供商
深入理解 MultiQueryRetriever：提升向量数据库检索效果的强大工具 nseejrukjhad 数据库 python
深入理解MultiQueryRetriever：提升向量数据库检索效果的强大工具引言在人工智能和自然语言处理领域，高效准确的信息检索一直是一个关键挑战。传统的基于距离的向量数据库检索方法虽然广泛应用，但仍存在一些局限性。本文将介绍一种创新的解决方案：MultiQueryRetriever，它通过自动生成多个查询视角来增强检索效果，提高结果的相关性和多样性。MultiQueryRetriever的工
121. 买卖股票的最佳时机薄荷糖的味道_fb40
给定一个数组，它的第i个元素是一支给定股票第i天的价格。如果你最多只允许完成一笔交易（即买入和卖出一支股票），设计一个算法来计算你所能获取的最大利润。注意你不能在买入股票前卖出股票。示例1:输入:[7,1,5,3,6,4]输出:5解释:在第2天（股票价格=1）的时候买入，在第5天（股票价格=6）的时候卖出，最大利润=6-1=5。注意利润不能是7-1=6,因为卖出价格需要大于买入价格。示例2:输入:
人工智能时代，程序员如何保持核心竞争力？ jmoych 人工智能
随着AIGC（如chatgpt、midjourney、claude等）大语言模型接二连三的涌现，AI辅助编程工具日益普及，程序员的工作方式正在发生深刻变革。有人担心AI可能取代部分编程工作，也有人认为AI是提高效率的得力助手。面对这一趋势,程序员应该如何应对?是专注于某个领域深耕细作，还是广泛学习以适应快速变化的技术环境?又或者，我们是否应该将重点转向AI无法轻易替代的软技能？让我们一起探讨程序员
每日算法&面试题，大厂特训二十八天——第二十天（树）肥学 ⚡算法题⚡面试题每日精进 java 算法数据结构
目录标题导读算法特训二十八天面试题点击直接资料领取导读肥友们为了更好的去帮助新同学适应算法和面试题，最近我们开始进行专项突击一步一步来。上一期我们完成了动态规划二十一天现在我们进行下一项对各类算法进行二十八天的一个小总结。还在等什么快来一起肥学进行二十八天挑战吧！！特别介绍小白练手专栏，适合刚入手的新人欢迎订阅编程小白进阶python有趣练手项目里面包括了像《机器人尬聊》《恶搞程序》这样的有趣文章
2022现在哪个打车软件比较好用又便宜实惠的打车软件合集高省APP珊珊
这是一个信息高速传播的社会。信息可以通过手机，微信，自媒体，抖音等方式进行传播。但同时这也是一个交通四通发达的社会。高省APP，是2022年推出的平台，0投资，0风险、高省APP佣金更高，模式更好，终端用户不流失。【高省】是一个自用省钱佣金高，分享推广赚钱多的平台，百度有几百万篇报道，也期待你的加入。珊珊导师，高省邀请码777777，注册送2皇冠会员，送万元推广大礼包，教你如何1年做到百万团队。高
pyecharts——绘制柱形图折线图 2224070247 信息可视化 python java 数据可视化
一、pyecharts概述自2013年6月百度EFE(ExcellentFrontEnd）数据可视化团队研发的ECharts1.0发布到GitHub网站以来，ECharts一直备受业界权威的关注并获得广泛好评，成为目前成熟且流行的数据可视化图表工具，被应用到诸多数据可视化的开发领域。Python作为数据分析领域最受欢迎的语言，也加入ECharts的使用行列，并研发出方便Python开发者使用的数据
如何成为段子手欣雅阅读
我是一个尬聊大师，与朋友聊天经常把话题聊死，留我一个人在群里，望着自己打下的最后一句话无语凝噎。看到风趣幽默的朋友与人聊天，很是艳羡，觉得自己何时才能成为这样的段子手呢？一、段子是什么？“段子”一词在百度百科上的解释：本是相声中的一个艺术术语，指的是相声作品中一节或一段艺术内容。我的理解：段子就是一些搞笑的故事或者笑话。二、为什么要会说段子？不知道大家有没有这样的朋友，本来很无趣的聚会，只要有他参
回溯算法-重新安排行程 chirou_ 算法数据结构图论 c++图搜索
leetcode332.重新安排行程这题我还没自己ac过，只能现在凭着刚学完的热乎劲把我对题解的理解记下来。本题我认为对数据结构的考察比较多，用什么数据结构去存数据，去读取数据，都是很重要的。classSolution{private:unordered_map>targets;boolbacktracking(intticketNum,vector&result){//1.确定参数和返回值//2
Faiss：高效相似性搜索与聚类的利器网络·魚大数据 faiss
Faiss是一个针对大规模向量集合的相似性搜索库，由FacebookAIResearch开发。它提供了一系列高效的算法和数据结构，用于加速向量之间的相似性搜索，特别是在大规模数据集上。本文将介绍Faiss的原理、核心功能以及如何在实际项目中使用它。Faiss原理：近似最近邻搜索：Faiss的核心功能之一是近似最近邻搜索，它能够高效地在大规模数据集中找到与给定查询向量最相似的向量。这种搜索是近似的，
数字里的世界17期：2021年全球10大顶级数据中心，中国移动榜首张三叨
你知道吗？2016年，全球的数据中心共计用电4160亿千瓦时，比整个英国的发电量还多40％！前言每天，我们都会创造超过250万TB的数据。并且随着物联网（IOT）的不断普及，这一数据将持续增长。如此庞大的数据被存储在被称为“数据中心”的专用设施中。虽然最早的数据中心建于20世纪40年代，但直到1997-2000年的互联网泡沫期间才逐渐成为主流。当前人类的技术，比如人工智能和机器学习，已经将我们推向
insert into select 主键自增_mybatis拦截器实现主键自动生成 weixin_39521651 insert into select 主键自增 mybatis delete返回值 mybatis insert返回主键 mybatis insert返回对象 mybatis plus insert返回主键 mybatis plus 插入生成id
前言前阵子和朋友聊天，他说他们项目有个需求，要实现主键自动生成，不想每次新增的时候，都手动设置主键。于是我就问他，那你们数据库表设置主键自动递增不就得了。他的回答是他们项目目前的id都是采用雪花算法来生成，因此为了项目稳定性，不会切换id的生成方式。朋友问我有没有什么实现思路，他们公司的orm框架是mybatis，我就建议他说，不然让你老大把mybatis切换成mybatis-plus。mybat
k均值聚类算法考试例题_k均值算法(k均值聚类算法计算题) 寻找你83497 k均值聚类算法考试例题
?算法：第一步：选K个初始聚类中心，z1(1),z2(1)，…，zK(1)，其中括号内的序号为寻找聚类中心的迭代运算的次序号。聚类中心的向量值可任意设定，例如可选开始的K个.k均值聚类：---------一种硬聚类算法，隶属度只有两个取值0或1，提出的基本根据是“类内误差平方和最小化”准则；模糊的c均值聚类算法：--------一种模糊聚类算法，是.K均值聚类算法是先随机选取K个对象作为初始的聚类
Python实现简单的机器学习算法 master_chenchengg python python 办公效率 python开发 IT
Python实现简单的机器学习算法开篇：初探机器学习的奇妙之旅搭建环境：一切从安装开始必备工具箱第一步：安装Anaconda和JupyterNotebook小贴士：如何配置Python环境变量算法初体验：从零开始的Python机器学习线性回归：让数据说话数据准备：从哪里找数据编码实战：Python实现线性回归模型评估：如何判断模型好坏逻辑回归：从分类开始理论入门：什么是逻辑回归代码实现：使用skl
推荐算法_隐语义-梯度下降 _feivirus_ 算法机器学习和数学推荐算法机器学习隐语义
importnumpyasnp1.模型实现"""inputrate_matrix:M行N列的评分矩阵，值为P*Q.P:初始化用户特征矩阵M*K.Q:初始化物品特征矩阵K*N.latent_feature_cnt:隐特征的向量个数max_iteration:最大迭代次数alpha:步长lamda:正则化系数output分解之后的P和Q"""defLFM_grad_desc(rate_matrix,l
K近邻算法_分类鸢尾花数据集 _feivirus_ 算法机器学习和数学分类机器学习 K近邻
importnumpyasnpimportpandasaspdfromsklearn.datasetsimportload_irisfromsklearn.model_selectionimporttrain_test_splitfromsklearn.metricsimportaccuracy_score1.数据预处理iris=load_iris()df=pd.DataFrame(data=ir
数据结构 | 栈和队列 TT-Kun 数据结构与算法数据结构栈队列 C语言
文章目录栈和队列1.栈：后进先出（LIFO）的数据结构1.1概念与结构1.2栈的实现2.队列：先进先出（FIFO）的数据结构2.1概念与结构2.2队列的实现3.栈和队列算法题3.1有效的括号3.2用队列实现栈3.3用栈实现队列3.4设计循环队列结论栈和队列在计算机科学中，栈和队列是两种基本且重要的数据结构，它们在处理数据存储和访问顺序方面有着独特的规则和应用。本文将详细介绍栈和队列的概念、结构、实
[Python] 数据结构详解及代码 AIAdvocate 算法 python 数据结构链表
今日内容大纲介绍数据结构介绍列表链表1.数据结构和算法简介程序大白话翻译,程序=数据结构+算法数据结构指的是存储,组织数据的方式.算法指的是为了解决实际业务问题而思考思路和方法,就叫:算法.2.算法的5大特性介绍算法具有独立性算法是解决问题的思路和方式,最重要的是思维,而不是语言,其(算法)可以通过多种语言进行演绎.5大特性有输入,需要传入1或者多个参数有输出,需要返回1个或者多个结果有穷性,执行
手机上有什么兼职可以做？网上兼职一单一结手机就可以做？优惠券高省
建议上班族和全职宝妈把空闲时间拿出来一点做做副业，什么也不耽搁还能多一笔收入！推荐大家一定要试一试！！！只要有手机就可以做，下面小编就为大家推荐用手机就可以做的三类网上兼职工作。一，高省APP高省APP佣金更高，模式更好，终端用户不流失。【高省】是一个自用省钱佣金高，分享推广赚钱多的平台，百度有几百万篇报道，也期待你的加入。万方导师高省邀请码005500，注册送双皇冠会员，送万元推广大礼包，教你如
Python算法L5：贪心算法小熊同学哦 Python算法算法 python 贪心算法
Python贪心算法简介目录Python贪心算法简介贪心算法的基本步骤贪心算法的适用场景经典贪心算法问题1.**零钱兑换问题**2.**区间调度问题**3.**背包问题**贪心算法的优缺点优点：缺点：结语贪心算法（GreedyAlgorithm）是一种在每一步选择中都采取当前最优或最优解的算法。它的核心思想是，在保证每一步局部最优的情况下，希望通过贪心选择达到全局最优解。虽然贪心算法并不总能得到全
【RabbitMQ 项目】服务端：数据管理模块之绑定管理月夜星辉雪 rabbitmq 分布式
文章目录一.编写思路二.代码实践一.编写思路定义绑定信息类交换机名称队列名称绑定关键字：交换机的路由交换算法中会用到没有是否持久化的标志，因为绑定是否持久化取决于交换机和队列是否持久化，只有它们都持久化时绑定才需要持久化。绑定就好像一根绳子，两端连接着交换机和队列，当一方不存在，它就没有存在的必要了定义绑定持久化类构造函数：如果数据库文件不存在则创建，打开数据库，创建binding_table插入
JAVA学习笔记之23种设计模式学习 victorfreedom Java技术设计模式 android java 常用设计模式
博主最近买了《设计模式》这本书来学习，无奈这本书是以C++语言为基础进行说明，整个学习流程下来效率不是很高，虽然有的设计模式通俗易懂，但感觉还是没有充分的掌握了所有的设计模式。于是博主百度了一番，发现有大神写过了这方面的问题，于是博主迅速拿来学习。一、设计模式的分类总体来说设计模式分为三大类：创建型模式，共五种：工厂方法模式、抽象工厂模式、单例模式、建造者模式、原型模式。结构型模式，共七种：适配器
非对称加密算法原理与应用2——RSA私钥加密文件私语茶馆云部署与开发架构及产品灵感记录 RSA2048 私钥加密
作者：私语茶馆1.相关章节（1）非对称加密算法原理与应用1——秘钥的生成-CSDN博客第一章节讲述的是创建秘钥对，并将公钥和私钥导出为文件格式存储。本章节继续讲如何利用私钥加密内容，包括从密钥库或文件中读取私钥，并用RSA算法加密文件和String。2.私钥加密的概述本文主要基于第一章节的RSA2048bit的非对称加密算法讲述如何利用私钥加密文件。这种加密后的文件，只能由该私钥对应的公钥来解密。
粒子群优化 (PSO) 在三维正弦波函数中的应用 subject625Ruben 机器学习人工智能 matlab 算法
在这篇博客中，我们将展示如何使用粒子群优化（PSO）算法求解三维正弦波函数，并通过增加正弦波扰动，使优化过程更加复杂和有趣。本文将介绍目标函数的定义、PSO参数设置以及算法执行的详细过程，并展示搜索空间中的动态过程和收敛曲线。1.目标函数定义我们使用的目标函数是一个三维正弦波函数，定义如下：objectiveFunc=@(x)sin(sqrt(x(1).^2+x(2).^2))+0.5*sin(5
[星球大战]阿纳金的背叛 comsci
本来杰迪圣殿的长老是不同意让阿纳金接受训练的......... 但是由于政治原因,长老会妥协了...这给邪恶的力量带来了机会所以......现代的地球联邦接受了这个教训...绝对不让某些年轻人进入学院
看懂它，你就可以任性的玩耍了！ aijuans JavaScript
javascript作为前端开发的标配技能，如果不掌握好它的三大特点：1.原型 2.作用域 3. 闭包 ,又怎么可以说你学好了这门语言呢？如果标配的技能都没有撑握好，怎么可以任性的玩耍呢？怎么验证自己学好了以上三个基本点呢，我找到一段不错的代码，稍加改动，如果能够读懂它，那么你就可以任性了。 function jClass(b
Java常用工具包 Jodd Kai_Ge java jodd
Jodd 是一个开源的 Java 工具集，包含一些实用的工具类和小型框架。简单，却很强大！写道 Jodd = Tools + IoC + MVC + DB + AOP + TX + JSON + HTML < 1.5 Mb Jodd 被分成众多模块，按需选择，其中工具类模块有： jodd-core &nb
SpringMvc下载 120153216 springMVC
@RequestMapping(value = WebUrlConstant.DOWNLOAD) public void download(HttpServletRequest request,HttpServletResponse response,String fileName) { OutputStream os = null; InputStream is = null;
Python 标准异常总结 2002wmj python
Python标准异常总结 AssertionError 断言语句（assert）失败 AttributeError 尝试访问未知的对象属性 EOFError 用户输入文件末尾标志EOF（Ctrl+d） FloatingPointError 浮点计算错误 GeneratorExit generator.close()方法被调用的时候 ImportError 导入模块失
SQL函数返回临时表结构的数据用于查询 357029540 SQL Server
这两天在做一个查询的SQL，这个SQL的一个条件是通过游标实现另外两张表查询出一个多条数据，这些数据都是INT类型，然后用IN条件进行查询，并且查询这两张表需要通过外部传入参数才能查询出所需数据，于是想到了用SQL函数返回值，并且也这样做了，由于是返回多条数据，所以把查询出来的INT类型值都拼接为了字符串，这时就遇到问题了，在查询SQL中因为条件是INT值，SQL函数的CAST和CONVERST都
java 时间格式化 | 比较大小| 时区个人笔记 7454103 java eclipse tomcat c MyEclipse
个人总结！不当之处多多包含！引用 1.0 如何设置 tomcat 的时区：位置：(catalina.bat---JAVA_OPTS 下面加上) set JAVA_OPT
时间获取Clander的用法 adminjun Clander 时间
/** * 得到几天前的时间 * @param d * @param day * @return */ public static Date getDateBefore(Date d,int day){ Calend
JVM初探与设置 aijuans java
JVM是Java Virtual Machine（Java虚拟机）的缩写，JVM是一种用于计算设备的规范，它是一个虚构出来的计算机，是通过在实际的计算机上仿真模拟各种计算机功能来实现的。Java虚拟机包括一套字节码指令集、一组寄存器、一个栈、一个垃圾回收堆和一个存储方法域。 JVM屏蔽了与具体操作系统平台相关的信息，使Java程序只需生成在Java虚拟机上运行的目标代码（字节码）,就可以在多种平台
SQL中ON和WHERE的区别 avords
SQL中ON和WHERE的区别数据库在通过连接两张或多张表来返回记录时，都会生成一张中间的临时表，然后再将这张临时表返回给用户。 www.2cto.com 在使用left jion时，on和where条件的区别如下： 1、 on条件是在生成临时表时使用的条件，它不管on中的条件是否为真，都会返回左边表中的记录。
说说自信 houxinyou 工作生活
自信的来源分为两种,一种是源于实力,一种源于头脑.实力是一个综合的评定,有自身的能力,能利用的资源等.比如我想去月亮上,要身体素质过硬,还要有飞船等等一系列的东西.这些都属于实力的一部分.而头脑不同,只要你头脑够简单就可以了!同样要上月亮上,你想,我一跳,1米,我多跳几下,跳个几年,应该就到了!什么?你说我会往下掉?你笨呀你!找个东西踩一下不就行了吗? 无论工作还
WEBLOGIC事务超时设置 bijian1013 weblogic jta 事务超时
系统中统计数据，由于调用统计过程，执行时间超过了weblogic设置的时间，提示如下错误：统计数据出错! 原因：The transaction is no longer active - status: 'Rolling Back. [Reason=weblogic.transaction.internal
两年已过去，再看该如何快速融入新团队 bingyingao java 互联网融入架构新团队
偶得的空闲，翻到了两年前的帖子该如何快速融入一个新团队，有所感触，就记下来，为下一个两年后的今天做参考。时隔两年半之后的今天，再来看当初的这个博客，别有一番滋味。而我已经于今年三月份离开了当初所在的团队，加入另外的一个项目组，2011年的这篇博客之后的时光，我很好的融入了那个团队，而直到现在和同事们关系都特别好。大家在短短一年半的时间离一起经历了一
【Spark七十七】Spark分析Nginx和Apache的access.log bit1129 apache
Spark分析Nginx和Apache的access.log，第一个问题是要对Nginx和Apache的access.log文件进行按行解析，按行解析就的方法是正则表达式： Nginx的access.log解析正则表达式 val PATTERN = """([^ ]*) ([^ ]*) ([^ ]*) (\\[.*\\]) (\&q
Erlang patch bookjovi erlang
Totally five patchs committed to erlang otp, just small patchs. IMO, erlang really is a interesting programming language, I really like its concurrency feature. but the functional programming style
log4j日志路径中加入日期 bro_feng java log4j
要用log4j使用记录日志，日志路径有每日的日期，文件大小5M新增文件。实现方式 log4j: <appender name="serviceLog" class="org.apache.log4j.RollingFileAppender"> <param name="Encoding" v
读《研磨设计模式》-代码笔记-桥接模式 bylijinnan java 设计模式
声明：本文只为方便我个人查阅和理解，详细的分析以及源代码请移步原作者的博客http://chjavach.iteye.com/ /** * 个人觉得关于桥接模式的例子，蜡笔和毛笔这个例子是最贴切的：http://www.cnblogs.com/zhenyulu/articles/67016.html * 笔和颜色是可分离的，蜡笔把两者耦合在一起了：一支蜡笔只有一种
windows7下SVN和Eclipse插件安装 chenyu19891124 eclipse插件
今天花了一天时间弄SVN和Eclipse插件的安装，今天弄好了。svn插件和Eclipse整合有两种方式，一种是直接下载插件包，二种是通过Eclipse在线更新。由于之前Eclipse版本和svn插件版本有差别，始终是没装上。最后在网上找到了适合的版本。所用的环境系统：windows7JDK：1.7svn插件包版本：1.8.16Eclipse：3.7.2工具下载地址：Eclipse下在地址：htt
[转帖]工作流引擎设计思路 comsci 设计模式工作应用服务器 workflow 企业应用
作为国内的同行，我非常希望在流程设计方面和大家交流，刚发现篇好文(那么好的文章，现在才发现，可惜)，关于流程设计的一些原理，个人觉得本文站得高，看得远，比俺的文章有深度，转载如下 ================================================================================= 自开博以来不断有朋友来探讨工作流引擎该如何
Linux 查看内存，CPU及硬盘大小的方法 daizj linux cpu 内存硬盘大小
一、查看CPU信息的命令 [root@R4 ~]# cat /proc/cpuinfo |grep "model name" && cat /proc/cpuinfo |grep "physical id" model name : Intel(R) Xeon(R) CPU X5450 @ 3.00GHz model name :
linux 踢出在线用户 dongwei_6688 linux
两个步骤： 1.用w命令找到要踢出的用户，比如下面： [root@localhost ~]# w 18:16:55 up 39 days, 8:27, 3 users, load average: 0.03, 0.03, 0.00 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
放手吧,就像不曾拥有过一样 dcj3sjt126com
内容提要：静悠悠编著的《放手吧就像不曾拥有过一样》集结“全球华语世界最舒缓心灵”的精华故事，触碰生命最深层次的感动，献给全世界亿万读者。《放手吧就像不曾拥有过一样》的作者衷心地祝愿每一位读者都给自己一个重新出发的理由，将那些令你痛苦的、扛起的、背负的，一并都放下吧！把憔悴的面容换做一种清淡的微笑，把沉重的步伐调节成春天五线谱上的音符，让自己踏着轻快的节奏，在人生的海面上悠然漂荡，享受宁静与
php二进制安全的含义 dcj3sjt126com PHP
PHP里，有string的概念。 string里，每个字符的大小为byte（与PHP相比，Java的每个字符为Character，是UTF8字符，C语言的每个字符可以在编译时选择）。 byte里，有ASCII代码的字符，例如ABC，123，abc，也有一些特殊字符，例如回车，退格之类的。特殊字符很多是不能显示的。或者说，他们的显示方式没有标准，例如编码65到哪儿都是字母A，编码97到哪儿都是字符
Linux下禁用T440s，X240的一体化触摸板(touchpad) gashero linux ThinkPad 触摸板
自打1月买了Thinkpad T440s就一直很火大，其中最让人恼火的莫过于触摸板。 Thinkpad的经典就包括用了小红点(TrackPoint)。但是小红点只能定位，还是需要鼠标的左右键的。但是自打T440s等开始启用了一体化触摸板，不再有实体的按键了。问题是要是好用也行。实际使用中，触摸板一堆问题，比如定位有抖动，以及按键时会有飘逸。这就导致了单击经常就
graph_dfs hcx2013 Graph
package edu.xidian.graph; class MyStack { private final int SIZE = 20; private int[] st; private int top; public MyStack() { st = new int[SIZE]; top = -1; } public void push(i
Spring4.1新特性——Spring核心部分及其他 jinnianshilongnian spring 4.1
目录 Spring4.1新特性——综述 Spring4.1新特性——Spring核心部分及其他 Spring4.1新特性——Spring缓存框架增强 Spring4.1新特性——异步调用和事件机制的异常处理 Spring4.1新特性——数据库集成测试脚本初始化 Spring4.1新特性——Spring MVC增强 Spring4.1新特性——页面自动化测试框架Spring MVC T
配置HiveServer2的安全策略之自定义用户名密码验证 liyonghui160com
具体从网上看 http://doc.mapr.com/display/MapR/Using+HiveServer2#UsingHiveServer2-ConfiguringCustomAuthentication LDAP Authentication using OpenLDAP Setting
一位30多的程序员生涯经验总结 pda158 编程工作生活咨询
1.客户在接触到产品之后，才会真正明白自己的需求。　　这是我在我的第一份工作上面学来的。只有当我们给客户展示产品的时候，他们才会意识到哪些是必须的。给出一个功能性原型设计远远比一张长长的文字表格要好。 2.只要有充足的时间，所有安全防御系统都将失败。　　安全防御现如今是全世界都在关注的大课题、大挑战。我们必须时时刻刻积极完善它，因为黑客只要有一次成功，就可以彻底打败你。 3.
分布式web服务架构的演变自由的奴隶 linux Web 应用服务器互联网
最开始，由于某些想法，于是在互联网上搭建了一个网站，这个时候甚至有可能主机都是租借的，但由于这篇文章我们只关注架构的演变历程，因此就假设这个时候已经是托管了一台主机，并且有一定的带宽了，这个时候由于网站具备了一定的特色，吸引了部分人访问，逐渐你发现系统的压力越来越高，响应速度越来越慢，而这个时候比较明显的是数据库和应用互相影响，应用出问题了，数据库也很容易出现问题，而数据库出问题的时候，应用也容易
初探Druid连接池之二——慢SQL日志记录 xingsan_zhang 日志连接池 druid 慢SQL
由于工作原因，这里先不说连接数据库部分的配置，后面会补上，直接进入慢SQL日志记录。 1.applicationContext.xml中增加如下配置： <bean abstract="true" id="mysql_database" class="com.alibaba.druid.pool.DruidDataSourc