Wfh/json schema evaluation (#12389)

Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com>

docs/docs/guides/evaluation/string/json.ipynb

@@ -221,7 +221,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": 8,
    "id": "7a8f3ec5-1cde-4b0e-80cd-ac0ac290d375",
    "metadata": {},
    "outputs": [
@@ -261,10 +261,101 @@
     "print(result)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "6b15d18e-9b97-434f-905c-70acd4c35aea",
+   "metadata": {},
+   "source": [
+    "## JsonSchemaEvaluator\n",
+    "\n",
+    "The `JsonSchemaEvaluator` validates a JSON prediction against a provided JSON schema. If the prediction conforms to the schema, it returns a score of True (indicating no errors). Otherwise, it returns a score of 0 (indicating an error).\n",
+    "\n",
+    "### Overview:\n",
+    "- **Requires Input?**: Yes\n",
+    "- **Requires Reference?**: Yes (A JSON schema)\n",
+    "- **Score**: True (No errors) or False (Error occurred)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "85afcf33-d2f4-406e-9d8f-15dc0a4772f2",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{'score': True}\n"
+     ]
+    }
+   ],
+   "source": [
+    "from langchain.evaluation import JsonSchemaEvaluator\n",
+    "\n",
+    "evaluator = JsonSchemaEvaluator()\n",
+    "# Equivalently\n",
+    "# evaluator = load_evaluator(\"json_schema_validation\")\n",
+    "\n",
+    "result = evaluator.evaluate_strings(\n",
+    "    prediction='{\"name\": \"John\", \"age\": 30}',\n",
+    "    reference={\n",
+    "        \"type\": \"object\",\n",
+    "        \"properties\": {\"name\": {\"type\": \"string\"}, \"age\": {\"type\": \"integer\"}},\n",
+    "    },\n",
+    ")\n",
+    "print(result)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "id": "bb5b89f6-0c87-4335-9091-55fd67a0565f",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{'score': True}\n"
+     ]
+    }
+   ],
+   "source": [
+    "result = evaluator.evaluate_strings(\n",
+    "    prediction='{\"name\": \"John\", \"age\": 30}',\n",
+    "    reference='{\"type\": \"object\", \"properties\": {\"name\": {\"type\": \"string\"}, \"age\": {\"type\": \"integer\"}}}',\n",
+    ")\n",
+    "print(result)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "id": "ff914d24-36bc-482a-a9ba-259cd0dd2a52",
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{'score': False, 'reasoning': \"<ValidationError: '30 is less than the minimum of 66'>\"}\n"
+     ]
+    }
+   ],
+   "source": [
+    "result = evaluator.evaluate_strings(\n",
+    "    prediction='{\"name\": \"John\", \"age\": 30}',\n",
+    "    reference='{\"type\": \"object\", \"properties\": {\"name\": {\"type\": \"string\"},'\n",
+    "    '\"age\": {\"type\": \"integer\", \"minimum\": 66}}}',\n",
+    ")\n",
+    "print(result)"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "85afcf33-d2f4-406e-9d8f-15dc0a4772f2",
+   "id": "b073f12d-4603-481c-8081-fab1af6bfcfe",
    "metadata": {},
    "outputs": [],
    "source": []