LangChain手记 Evalutation评估

整理并翻译自DeepLearning.AI×LangChain的官方课程:Evaluation(源代码可见)

基于LLM的应用如何做评估是一个难点,本节介绍了一些思路和工具。

“从传统开发转换到基于prompt的开发,开发使用LLM的应用,整个工作流的评估方式需要重新考虑,本节会介绍很多激动人心的概念。”

Evaluation 评估

构建一个上节课介绍过的QA chain:
LangChain手记 Evalutation评估_第1张图片
不同之处仅在于加了一个参数:chain_type_kwargs,内部指定了一个doc的分隔符。

首先可以看一下数据示例:
LangChain手记 Evalutation评估_第2张图片

Hard-Code example 手动编写的用例

最容易想到的评价方法是手动构建评价数据,然后观察LLM的输出是否和评价数据中已经给定的答案一致,手动构建评价数据永远逃不过成本问题。

LangChain手记 Evalutation评估_第3张图片

LLM-Generated example LLM生成用例

可以考虑使用LLM生成代替人工编写用例,下面介绍了一个生成QA用例的QAGenerationChain
LangChain手记 Evalutation评估_第4张图片
LangChain手记 Evalutation评估_第5张图片
可以把人工编写的用例和生成的用例组合用来做评估,测试一下第一个query,得到如下回复:
LangChain手记 Evalutation评估_第6张图片

Manual Evaluation 人工评估

LangChain提供了debug模式,可以像下面这样开启:
在这里插入图片描述
再次测试第一个query,LangChain会打印整个过程中的信息:
LangChain手记 Evalutation评估_第7张图片
LangChain手记 Evalutation评估_第8张图片
LangChain手记 Evalutation评估_第9张图片
LangChain手记 Evalutation评估_第10张图片
通过设置debug标志位为False关闭debug模式:
在这里插入图片描述

LLM assisted evaluation LLM辅助评估

基于现阶段LLM已经具备比较强的能力,可以使用LLM来辅助做评估

在前面构建的所有用例生成结果:
LangChain手记 Evalutation评估_第11张图片
一共有7条用例,所以跑了7次。

LangChain提供了QAEvalChain来进行QA场景的评估,使用方式如下:
LangChain手记 Evalutation评估_第12张图片
LangChain手记 Evalutation评估_第13张图片
下面我们来看一下模型输出和评估Chain评估的结果:
LangChain手记 Evalutation评估_第14张图片

Example 0:
Question: Do the Cozy Comfort Pullover Set have side pockets?
Real Answer: Yes
Predicted Answer: The Cozy Comfort Pullover Set, Stripe does have side pockets.
Predicted Grade: CORRECT

Example 1:
Question: What collection is the Ultra-Lofty 850 Stretch Down Hooded Jacket from?
Real Answer: The DownTek collection
Predicted Answer: The Ultra-Lofty 850 Stretch Down Hooded Jacket is from the DownTek collection.
Predicted Grade: CORRECT

Example 2:
Question: What is the weight of each pair of Women's Campside Oxfords?
Real Answer: The approximate weight of each pair of Women's Campside Oxfords is 1 lb. 1 oz.
Predicted Answer: The weight of each pair of Women's Campside Oxfords is approximately 1 lb. 1 oz.
Predicted Grade: CORRECT

Example 3:
Question: What are the dimensions of the small and medium Recycled Waterhog Dog Mat?
Real Answer: The dimensions of the small Recycled Waterhog Dog Mat are 18" x 28" and the dimensions of the medium Recycled Waterhog Dog Mat are 22.5" x 34.5".
Predicted Answer: The small Recycled Waterhog Dog Mat has dimensions of 18" x 28" and the medium size has dimensions of 22.5" x 34.5".
Predicted Grade: CORRECT

Example 4:
Question: What are some features of the Infant and Toddler Girls' Coastal Chill Swimsuit?
Real Answer: The swimsuit features bright colors, ruffles, and exclusive whimsical prints. It is made of four-way-stretch and chlorine-resistant fabric, ensuring that it keeps its shape and resists snags. The swimsuit is also UPF 50+ rated, providing the highest rated sun protection possible by blocking 98% of the sun's harmful rays. The crossover no-slip straps and fully lined bottom ensure a secure fit and maximum coverage. Finally, it can be machine washed and line dried for best results.
Predicted Answer: The Infant and Toddler Girls' Coastal Chill Swimsuit is a two-piece swimsuit with bright colors, ruffles, and exclusive whimsical prints. It is made of four-way-stretch and chlorine-resistant fabric that keeps its shape and resists snags. The swimsuit has UPF 50+ rated fabric that provides the highest rated sun protection possible, blocking 98% of the sun's harmful rays. The crossover no-slip straps and fully lined bottom ensure a secure fit and maximum coverage. It is machine washable and should be line dried for best results.
Predicted Grade: CORRECT

Example 5:
Question: What is the fabric composition of the Refresh Swimwear V-Neck Tankini Contrasts?
Real Answer: The body of the Refresh Swimwear V-Neck Tankini Contrasts is made of 82% recycled nylon and 18% Lycra® spandex, while the lining is made of 90% recycled nylon and 10% Lycra® spandex.
Predicted Answer: The Refresh Swimwear V-Neck Tankini Contrasts is made of 82% recycled nylon with 18% Lycra® spandex for the body and 90% recycled nylon with 10% Lycra® spandex for the lining.
Predicted Grade: CORRECT

Example 6:
Question: What is the fabric composition of the EcoFlex 3L Storm Pants?
Real Answer: The EcoFlex 3L Storm Pants are made of 100% nylon, exclusive of trim.
Predicted Answer: The fabric composition of the EcoFlex 3L Storm Pants is 100% nylon, exclusive of trim.
Predicted Grade: CORRECT
​```

视频接下来介绍了为什么要使用LLM来做评估:
![在这里插入图片描述](https://img-blog.csdnimg.cn/73ac80581ea243d981b0db3ede2d5d8a.png)
在一个自然语言生成场景下(比如前面介绍的QA),模型的输出可以是任意字符,因而无法通过字符完全匹配(是否相等)、字符部分匹配(是否含有子串)、正则(更复杂的匹配方式)来判定输出是否正确。以上图为例,真实答案“Yes”和模型的输出“The Cozy Comfort Pullover Set, Stripe does have side pockets.”是完全不同的字符,无法通过字符匹配来判定相等,但是具备语义理解能力的LLM能够判定它们在语义上相等,这是传统字符匹配做不到的。
### LangChain 可视化评估工具
LangChain提供了可视化的评估工具`LangChainPlus`(可能需要额外安装和配置),该工具会自动记录在python notebook上的运行历史。
![在这里插入图片描述](https://img-blog.csdnimg.cn/89a584e6f74843a9af67e719ff185cbb.png)
可以点击可视化查看调用链,也可以点击节点查看当前节点chain的详细信息,包含输入、输出、时延、额外新信息(运行环境)等,如下图:
![在这里插入图片描述](https://img-blog.csdnimg.cn/1bc61a5378934a248155957d17724f73.png)
点击LLM Chain节点可以查看模型输入:包含SYSTREM、HUMAN、模型输出、模型输出元信息等内容。
![在这里插入图片描述](https://img-blog.csdnimg.cn/da19b50c29d740cab5c498f25e688722.png)
![在这里插入图片描述](https://img-blog.csdnimg.cn/a9034d980ba54ddbb6ae8a136b2fe937.png)
右上角提供了一个【to Dataset】按钮,点击可以将当前的输入输出作为一个pair构建数据集,操作方式如下:
![在这里插入图片描述](https://img-blog.csdnimg.cn/aac46bc18f6e4862bf6227e9ded7fb2c.png)
如果当前没有数据集,需要点击【Create dataset】创建一个:
![在这里插入图片描述](https://img-blog.csdnimg.cn/26e015fa2877407a90d03822d723bf7f.png)
创建数据集:
![在这里插入图片描述](https://img-blog.csdnimg.cn/96c7b5798c68423a8427cd1376d9cf57.png)
将当前QA Chain的输入输出加入到刚刚创建的数据集内:
![在这里插入图片描述](https://img-blog.csdnimg.cn/827cf6901a9640478cc0b9888fa5f00d.png)

你可能感兴趣的:(大语言模型,langchain,人工智能)