第4-4章：使用 LangSmith 评估医疗RAG 系统-大模型应用实践高级课程

在本笔记本中，我们将探索如何使用 LangSmith 跟踪 LangChain 代理中发生的不同调用。我们将使用课程 LangChain 部分中使用的熟悉的代理，其中创建了一个包含医疗信息的 RAG 系统。

因此，我们不仅会观察代理的踪迹，还会检查检索器的踪迹。此外，我们将检查发送到矢量数据库的查询和返回的结果。

1.安装库和加载数据集 #

!pip install -q langchain==0.3.0
!pip install -q langchain-openai==0.2.0
!pip install -q langchainhub==0.1.21
!pip install -q datasets==3.0.0
!pip install -q chromadb==0.5.5
!pip install -q langchain-community==0.3.0

我们将从 Hugging Face 数据集库中下载数据集。这是一个包含疾病信息的数据集。

from datasets import load_dataset

data = load_dataset("keivalya/MedQuad-MedicalQnADataset", split='train')

     

data = data.to_pandas()
data.head(10)

#uncoment this line if you want to limit the size of the data.
data = data[0:100]

如您所见，数据集中的医疗信息井然有序，对于像我这样不是该领域专家的人来说，它似乎非常有价值。这些信息可以作为任何普通医学书籍的有用补充，以支持初级保健医生。

加载 langchain 库以加载文档。

from langchain.document_loaders import DataFrameLoader
from langchain.vectorstores import Chroma
     

文档位于答案列中，其他列是元数据。

df_loader = DataFrameLoader(data, page_content_column="Answer")

df_document = df_loader.load()
display(df_document[:2])

[Document(metadata={'qtype': 'susceptibility', 'Question': 'Who is at risk for Lymphocytic Choriomeningitis (LCM)? ?'}, page_content='LCMV infections can occur after exposure to fresh urine, droppings, saliva, or nesting materials from infected rodents.  Transmission may also occur when these materials are directly introduced into broken skin, the nose, the eyes, or the mouth, or presumably, via the bite of an infected rodent. Person-to-person transmission has not been reported, with the exception of vertical transmission from infected mother to fetus, and rarely, through organ transplantation.'),
 Document(metadata={'qtype': 'symptoms', 'Question': 'What are the symptoms of Lymphocytic Choriomeningitis (LCM) ?'}, page_content='LCMV is most commonly recognized as causing neurological disease, as its name implies, though infection without symptoms or mild febrile illnesses are more common clinical manifestations. \n                \nFor infected persons who do become ill, onset of symptoms usually occurs 8-13 days after exposure to the virus as part of a biphasic febrile illness. This initial phase, which may last as long as a week, typically begins with any or all of the following symptoms: fever, malaise, lack of appetite, muscle aches, headache, nausea, and vomiting. Other symptoms appearing less frequently include sore throat, cough, joint pain, chest pain, testicular pain, and parotid (salivary gland) pain. \n                \nFollowing a few days of recovery, a second phase of illness may occur. Symptoms may consist of meningitis (fever, headache, stiff neck, etc.), encephalitis (drowsiness, confusion, sensory disturbances, and/or motor abnormalities, such as paralysis), or meningoencephalitis (inflammation of both the brain and meninges). LCMV has also been known to cause acute hydrocephalus (increased fluid on the brain), which often requires surgical shunting to relieve increased intracranial pressure. In rare instances, infection results in myelitis (inflammation of the spinal cord) and presents with symptoms such as muscle weakness, paralysis, or changes in body sensation. An association between LCMV infection and myocarditis (inflammation of the heart muscles) has been suggested. \n                \nPrevious observations show that most patients who develop aseptic meningitis or encephalitis due to LCMV survive. No chronic infection has been described in humans, and after the acute phase of illness, the virus is cleared from the body. However, as in all infections of the central nervous system, particularly encephalitis, temporary or permanent neurological damage is possible. Nerve deafness and arthritis have been reported. \n                \nWomen who become infected with LCMV during pregnancy may pass the infection on to the fetus. Infections occurring during the first trimester may result in fetal death and pregnancy termination, while in the second and third trimesters, birth defects can develop. Infants infected In utero can have many serious and permanent birth defects, including vision problems, mental retardation, and hydrocephaly (water on the brain). Pregnant women may recall a flu-like illness during pregnancy, or may not recall any illness. \n                \nLCM is usually not fatal. In general, mortality is less than 1%.')]

我们可以对文档进行分块。我们想要将文档分割成多大的尺寸是一个设计决定。尺寸越大，提示就会越大，模型的响应过程也会越慢。

我们还需要考虑最大提示尺寸，并确保文档不超过该尺寸。

from langchain.text_splitter import CharacterTextSplitter
     

text_splitter = CharacterTextSplitter(chunk_size=1250,
                                      separator="\n",
                                      chunk_overlap=100)
texts = text_splitter.split_documents(df_document)

我们看到这些警告是因为它无法执行所需大小的分区。这是因为它等待分页符来划分文本，并在可能的情况下这样做。

first_doc = texts[1]
print(first_doc.page_content)
     

LCMV is most commonly recognized as causing neurological disease, as its name implies, though infection without symptoms or mild febrile illnesses are more common clinical manifestations. 
                
For infected persons who do become ill, onset of symptoms usually occurs 8-13 days after exposure to the virus as part of a biphasic febrile illness. This initial phase, which may last as long as a week, typically begins with any or all of the following symptoms: fever, malaise, lack of appetite, muscle aches, headache, nausea, and vomiting. Other symptoms appearing less frequently include sore throat, cough, joint pain, chest pain, testicular pain, and parotid (salivary gland) pain.

2.初始化嵌入模型和向量数据库 #

from getpass import getpass
import os
if not 'OPENAI_API_KEY' in os.environ:
  os.environ["OPENAI_API_KEY"] = getpass("OpenAI API Key: ")

从 LangSmith 面板中的个人->设置区域获取您的 LangChain API 密钥。

if not 'LANGCHAIN_API_KEY' in os.environ:
  os.environ["LANGCHAIN_API_KEY"] = getpass("LangChain API Key: ")
     
LangChain API Key: ··········

os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_ENDPOINT"]="https://api.smith.langchain.com"
os.environ["LANGCHAIN_PROJECT"]="langsmith_test3"

我们从 OpenAI 加载 text-embedding-ada-002 模型。

from langchain_openai import OpenAIEmbeddings

model_name = 'text-embedding-ada-002'

embed = OpenAIEmbeddings(
    model=model_name
)
     

此单元格的执行可能需要 3 到 5 分钟。如果您希望速度更快，可以减少数据集中的记录数。

directory_cdb = '/content/drive/MyDrive/chromadb'
chroma_db = Chroma.from_documents(
    df_document, embed, persist_directory=directory_cdb
)

我们将创建三个对象:

语言模型，可以是 OpenAI 中的任何一个，最常见的是 GPT-3.5。
内存，负责保存提示和所有必要的历史记录。
检索，用于获取存储在 ChromaDB 中的信息。

model="gpt-4o"
#model="gpt-3.5-turbo"
     

from langchain.chat_models import ChatOpenAI
from langchain_openai import OpenAI
from langchain.chains.conversation.memory import ConversationBufferWindowMemory
from langchain.chains import RetrievalQA

llm=OpenAI(temperature=0.0)

conversational_memory = ConversationBufferWindowMemory(
    memory_key='chat_history',
    k=4, #Number of messages stored in memory
    return_messages=True #Must return the messages in the response.
)

qa = RetrievalQA.from_chain_type(
    llm=llm,
    #chain_type="stuff",
    retriever=chroma_db.as_retriever()
)

我们可以尝试单独的检索，看看它返回的信息是否相关。

qa.run("What is the main symptom of LCM?")

' The main symptom of LCM is a biphasic febrile illness, which includes symptoms such as fever, malaise, lack of appetite, muscle aches, headache, nausea, and vomiting.'

在下图中，您可以观察到对模型的调用，其中显示了完整的提示和响应。

3.创建Agent #

from langchain.agents import Tool

#Defining the list of tool objects to be used by LangChain.
tools = [
    Tool(
        name='Medical KB',
        func=qa.run,
        description=(
            """use this tool when answering medical knowledge queries to get
            more information about the topic"""
        )
    )
]
     

from langchain.agents import create_react_agent
from langchain import hub

prompt = hub.pull("hwchase17/react-chat")
agent = create_react_agent(
    tools=tools,
    llm=llm,
    prompt=prompt,
)
     
/usr/local/lib/python3.10/dist-packages/langsmith/client.py:5301: LangChainBetaWarning: The function `loads` is in beta. It is actively being worked on, so the API may change.
  prompt = loads(json.dumps(prompt_object.manifest))

# Create an agent executor by passing in the agent and tools
from langchain.agents import AgentExecutor
agent_executor2 = AgentExecutor(agent=agent,
                               tools=tools,
                               verbose=True,
                               memory=conversational_memory,
                               max_iterations=30,
                               max_execution_time=600,
                               handle_parsing_errors=True
                               )

4.使用对话Agent #

要进行查询，我们只需直接致电代理即可。

首先，我将尝试一个与医疗领域无关的请求。

agent_executor2.invoke({"input": "What is 2 multiplied by 2?"})

Thought: Do I need to use a tool? No
Final Answer: 2 multiplied by 2 is 4.

> Finished chain.
{'input': 'What is 2 multiplied by 2?',
 'chat_history': [],
 'output': '2 multiplied by 2 is 4.'}

对 Agent 的初始调用仅通过对 OpenAI 的一次调用进行。LangSmith 中提供的一条信息是整个提示。我将把它复制到图片下方，以便您查看。

Assistant is a large language model trained by OpenAI.

Assistant is designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, Assistant is able to generate human-like text based on the input it receives, allowing it to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand.

Assistant is constantly learning and improving, and its capabilities are constantly evolving. It is able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. Additionally, Assistant is able to generate its own text based on the input it receives, allowing it to engage in discussions and provide explanations and descriptions on a wide range of topics.

Overall, Assistant is a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether you need help with a specific question or just want to have a conversation about a particular topic, Assistant is here to assist.

TOOLS:
------

Assistant has access to the following tools:

Medical KB: use this tool when answering medical knowledge queries to get
            more information about the topic

To use a tool, please use the following format:
Thought: Do I need to use a tool? Yes
Action: the action to take, should be one of [Medical KB]
Action Input: the input to the action
Observation: the result of the action


When you have a response to say to the Human, or if you do not need to use a tool, you MUST use the format:


Thought: Do I need to use a tool? No
Final Answer: [your response here]


Begin!

Previous conversation history:
[]

New input: What is 2 multiplied by 2?

完美，模型无需访问配置的知识数据库即可做出响应。

现在我将尝试回答一个与健康无关的问题。

agent_executor2.memory.clear()
     

agent_executor2.invoke({"input": """I have a patient that can have Botulism,
how can I confirm the diagnosis?"""})
     

Thought: Do I need to use a tool? Yes
Action: Medical KB
Action Input: Botulism Botulism is a rare but serious paralytic illness caused by a nerve toxin produced by certain bacteria. It can be contracted through contaminated food, wounds, or ingestion of bacterial spores. Symptoms include muscle paralysis, difficulty swallowing, and respiratory failure. Treatment includes antitoxin, supportive care, and removal of contaminated food or wound.Do I need to use a tool? No
Final Answer: To confirm the diagnosis, you should perform a physical exam and order laboratory tests, such as a stool or blood test, to detect the presence of the bacteria or its toxin. You may also need to consult with a specialist, such as an infectious disease doctor, for further evaluation and treatment.

> Finished chain.
{'input': 'I have a patient that can have Botulism,\nhow can I confirm the diagnosis?',
 'chat_history': [],
 'output': 'To confirm the diagnosis, you should perform a physical exam and order laboratory tests, such as a stool or blood test, to detect the presence of the bacteria or its toxin. You may also need to consult with a specialist, such as an infectious disease doctor, for further evaluation and treatment.'}

完美，对我们来说最重要的是它已经能够识别应该去医疗数据库搜索有关症状的信息。

agent_executor2.invoke({"input": "Is this an important illness?"})

Thought: Do I need to use a tool? Yes
Action: Medical KB
Action Input: Botulism Botulism is a rare but serious paralytic illness caused by a nerve toxin produced by certain bacteria. It can be contracted through contaminated food, wounds, or ingestion of bacterial spores. Symptoms include muscle paralysis, difficulty swallowing, and respiratory failure. Treatment includes antitoxin, supportive care, and removal of the source of the toxin.Do I need to use a tool? No
Final Answer: Yes, botulism is an important illness that requires prompt diagnosis and treatment. If you suspect a patient may have botulism, it is important to consult with a specialist and perform necessary tests to confirm the diagnosis.

> Finished chain.
{'input': 'Is this an important illness?',
 'chat_history': [HumanMessage(content='I have a patient that can have Botulism,\nhow can I confirm the diagnosis?', additional_kwargs={}, response_metadata={}),
  AIMessage(content='To confirm the diagnosis, you should perform a physical exam and order laboratory tests, such as a stool or blood test, to detect the presence of the bacteria or its toxin. You may also need to consult with a specialist, such as an infectious disease doctor, for further evaluation and treatment.', additional_kwargs={}, response_metadata={})],
 'output': 'Yes, botulism is an important illness that requires prompt diagnosis and treatment. If you suspect a patient may have botulism, it is important to consult with a specialist and perform necessary tests to confirm the diagnosis.'}

在图片的左侧，您可以看到代理进行的各种调用，这些调用花费了 2.5 秒的时间执行，消耗了 2000 个 token。

我将尝试描述每次调用中发生的情况。

OpenAI：完整的提示，包括来自 Hugging Face 的模板，用户的问题被传递给 OpenAI 模型。它通过以下步骤做出响应。它的答案是：

想法：我需要使用工具吗？

是 行动：医疗知识库

行动输入：肉毒杆菌中毒

医疗知识库：代理使用配置的工具，只传递一个单词作为参数：“肉毒杆菌中毒”。

医疗知识库.检索器：检索器返回从矢量数据库中提取的四个文档。

医疗知识库.OpenAI：提示由矢量数据库中的信息构建。尽管我不会包含粘贴，但我设法识别出返回的四个文档实际上是两个，但重复了。在实际项目中，这可以帮助我检测到一些问题，也许我的数据集中有重复项，或者我加载了两次。无论如何，我消耗的令牌比必要的多得多。该模型返回一个根据提示中包含的信息创建的响应。

OpenAI：在对模型的最后一次调用中，它决定收到的响应是用户需要的。因此，它将其标记为正确并将其返回给用户。

而且记忆功能运行良好。考虑到模型知道之前的问题和答案，我们可以保持对话。

5.结论 #

LangSmith 是一个非常有用的工具，用于跟踪和存储调用 LangChain 时生成的所有信息。

实验取得了小小的成功。Vectorial 数据库已配置并填充了数据集中的信息。已经创建了一个 LangChain 代理，它只能在必要时从数据库中检索信息。别忘了我们的 ChatBot 有记忆。

而且您拥有存储在 LangSmith 中的项目中的所有信息！！！！