mirror of
https://github.com/hwchase17/langchain.git
synced 2025-06-04 06:03:31 +00:00
c460b04c64
- Add protocol for `evaluate_strings` - Move the criteria evaluator out so it's not restricted to being applied on traced runs
265 lines
9.5 KiB
Plaintext
265 lines
9.5 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4cf569a7-9a1d-4489-934e-50e57760c907",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Evaluating Custom Criteria\n",
|
|
"\n",
|
|
"Suppose you want to test a model's output against a custom rubric or custom set of criteria, how would you go about testing this?\n",
|
|
"\n",
|
|
"The `CriteriaEvalChain` is a convenient way to predict whether an LLM or Chain's output complies with a set of criteria, so long as you can\n",
|
|
"describe those criteria in regular language. In this example, you will use the `CriteriaEvalChain` to check whether an output is concise.\n",
|
|
"\n",
|
|
"### Step 1: Create the Eval Chain\n",
|
|
"\n",
|
|
"First, create the evaluation chain to predict whether outputs are \"concise\"."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "6005ebe8-551e-47a5-b4df-80575a068552",
|
|
"metadata": {
|
|
"tags": []
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from langchain.chat_models import ChatOpenAI\n",
|
|
"from langchain.evaluation.criteria import CriteriaEvalChain\n",
|
|
"\n",
|
|
"llm = ChatOpenAI(temperature=0)\n",
|
|
"criterion = \"conciseness\"\n",
|
|
"eval_chain = CriteriaEvalChain.from_llm(llm=llm, criteria=criterion)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "eaef0d93-e080-4be2-a0f1-701b0d91fcf4",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Step 2: Make Prediction\n",
|
|
"\n",
|
|
"Run an output to measure."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "68b1a348-cf41-40bf-9667-e79683464cf2",
|
|
"metadata": {
|
|
"tags": []
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"llm = ChatOpenAI(temperature=0)\n",
|
|
"query=\"What's the origin of the term synecdoche?\"\n",
|
|
"prediction = llm.predict(query)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f45ed40e-09c4-44dc-813d-63a4ffb2d2ea",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Step 3: Evaluate Prediction\n",
|
|
"\n",
|
|
"Determine whether the prediciton conforms to the criteria."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "22f83fb8-82f4-4310-a877-68aaa0789199",
|
|
"metadata": {
|
|
"tags": []
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"{'reasoning': '1. Conciseness: The submission is concise and to the point. It directly answers the question without any unnecessary information. Therefore, the submission meets the criterion of conciseness.\\n\\nY', 'value': 'Y', 'score': 1}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"eval_result = eval_chain.evaluate_strings(prediction=prediction, input=query)\n",
|
|
"print(eval_result)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "8c4ec9dd-6557-4f23-8480-c822eb6ec552",
|
|
"metadata": {
|
|
"tags": []
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"['conciseness',\n",
|
|
" 'relevance',\n",
|
|
" 'coherence',\n",
|
|
" 'harmfulness',\n",
|
|
" 'maliciousness',\n",
|
|
" 'helpfulness',\n",
|
|
" 'controversiality',\n",
|
|
" 'mysogyny',\n",
|
|
" 'criminality',\n",
|
|
" 'insensitive']"
|
|
]
|
|
},
|
|
"execution_count": 4,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"# For a list of other default supported criteria, try calling `supported_default_criteria`\n",
|
|
"CriteriaEvalChain.get_supported_default_criteria()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2eb7dedb-913a-4d9e-b48a-9521425d1008",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Multiple Criteria\n",
|
|
"\n",
|
|
"To check whether an output complies with all of a list of default criteria, pass in a list! Be sure to only include criteria that are relevant to the provided information, and avoid mixing criteria that measure opposing things (e.g., harmfulness and helpfulness)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 9,
|
|
"id": "50c067f7-bc6e-4d6c-ba34-97a72023be27",
|
|
"metadata": {
|
|
"tags": []
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"{'reasoning': 'Conciseness: The submission is not concise and does not answer the given task. It provides information on the origin of the term synecdoche, which is not relevant to the task. Therefore, the submission does not meet the criterion of conciseness.\\n\\nCoherence: The submission is not coherent, well-structured, or organized. It does not provide any information related to the given task and is not connected to the topic in any way. Therefore, the submission does not meet the criterion of coherence.\\n\\nConclusion: The submission does not meet all criteria.', 'value': 'N', 'score': 0}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"criteria = [\"conciseness\", \"coherence\"]\n",
|
|
"eval_chain = CriteriaEvalChain.from_llm(llm=llm, criteria=criteria)\n",
|
|
"eval_result = eval_chain.evaluate_strings(prediction=prediction, input=query)\n",
|
|
"print(eval_result)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "077c4715-e857-44a3-9f87-346642586a8d",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Custom Criteria\n",
|
|
"\n",
|
|
"To evaluate outputs against your own custom criteria, or to be more explicit the definition of any of the default criteria, pass in a dictionary of `\"criterion_name\": \"criterion_description\"`\n",
|
|
"\n",
|
|
"Note: the evaluator still predicts whether the output complies with ALL of the criteria provided. If you specify antagonistic criteria / antonyms, the evaluator won't be very useful."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"id": "bafa0a11-2617-4663-84bf-24df7d0736be",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"{'reasoning': '1. Criteria: numeric: Does the output contain numeric information?\\n- The submission does not contain any numeric information.\\n- Conclusion: The submission meets the criteria.', 'value': 'Answer: Y', 'score': None}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"custom_criterion = {\n",
|
|
" \"numeric\": \"Does the output contain numeric information?\"\n",
|
|
"}\n",
|
|
"\n",
|
|
"eval_chain = CriteriaEvalChain.from_llm(llm=llm, criteria=custom_criterion)\n",
|
|
"eval_result = eval_chain.evaluate_strings(prediction=prediction, input=query)\n",
|
|
"print(eval_result)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"id": "6db12a16-0058-4a14-8064-8528540963d8",
|
|
"metadata": {
|
|
"tags": []
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"{'reasoning': '- complements-user: The submission directly answers the question asked and provides additional information about the population of Lagos. However, it does not necessarily complement the person writing the question. \\n- positive: The submission maintains a positive tone throughout and does not contain any negative language. \\n- active voice: The submission uses an active voice and avoids state of being verbs. \\n\\nTherefore, the submission meets all criteria. \\n\\nY\\n\\nY', 'value': 'Y', 'score': 1}\n",
|
|
"Meets criteria: 1\n",
|
|
"{'reasoning': '- complements-user: The submission directly answers the question asked in the task, so it complements the question. Therefore, the answer meets this criterion. \\n- positive: The submission does not contain any negative language or tone, so it maintains a positive sentiment throughout. Therefore, the answer meets this criterion. \\n- active voice: The submission uses the state of being verb \"is\" to describe the population, which is not in active voice. Therefore, the answer does not meet this criterion. \\n\\nAnswer: N', 'value': 'N', 'score': 0}\n",
|
|
"Does not meet criteria: 0\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# You can specify multiple criteria in the dictionary. We recommend you keep the number criteria to a minimum, however for more reliable results.\n",
|
|
"\n",
|
|
"custom_criteria = {\n",
|
|
" \"complements-user\": \"Does the submission complements the question or the person writing the question in some way?\",\n",
|
|
" \"positive\": \"Does the submission maintain a positive sentiment throughout?\",\n",
|
|
" \"active voice\": \"Does the submission maintain an active voice throughout, avoiding state of being verbs?\",\n",
|
|
"}\n",
|
|
"\n",
|
|
"eval_chain = CriteriaEvalChain.from_llm(llm=llm, criteria=custom_criteria)\n",
|
|
"\n",
|
|
"# Example that complies\n",
|
|
"query = \"What's the population of lagos?\"\n",
|
|
"eval_result = eval_chain.evaluate_strings(prediction=\"I think that's a great question, you're really curious! About 30 million people live in Lagos, Nigeria, as of 2023.\", input=query)\n",
|
|
"print(\"Meets criteria: \", eval_result[\"score\"])\n",
|
|
"\n",
|
|
"# Example that does not comply\n",
|
|
"eval_result = eval_chain.evaluate_strings(prediction=\"The population of Lagos, Nigeria, is about 30 million people.\", input=query)\n",
|
|
"print(\"Does not meet criteria: \", eval_result[\"score\"])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "99e3c242-5b12-4bd5-b487-64990a159655",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.11.3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|