langchain/docs/docs/integrations/callbacks/fiddler.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0cebf93b",
   "metadata": {},
   "source": [
    "# Fiddler\n",
    "\n",
    ">[Fiddler](https://www.fiddler.ai/) is the pioneer in enterprise Generative and Predictive system ops, offering a unified platform that enables Data Science, MLOps, Risk, Compliance, Analytics, and other LOB teams to monitor, explain, analyze, and improve ML deployments at enterprise scale. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38d746c2",
   "metadata": {},
   "source": [
    "## 1. Installation and Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0151955",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install langchain langchain-community langchain-openai fiddler-client"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5662f2e5-d510-4eef-b44b-fa929e5b4ad4",
   "metadata": {},
   "source": [
    "## 2. Fiddler connection details "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64fac323",
   "metadata": {},
   "source": [
    "*Before you can add information about your model with Fiddler*\n",
    "\n",
    "1. The URL you're using to connect to Fiddler\n",
    "2. Your organization ID\n",
    "3. Your authorization token\n",
    "\n",
    "These can be found by navigating to the *Settings* page of your Fiddler environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6f8b73e-d350-40f0-b7a4-fb1e68a65a22",
   "metadata": {},
   "outputs": [],
   "source": [
    "URL = \"\"  # Your Fiddler instance URL, Make sure to include the full URL (including https://). For example: https://demo.fiddler.ai\n",
    "ORG_NAME = \"\"\n",
    "AUTH_TOKEN = \"\"  # Your Fiddler instance auth token\n",
    "\n",
    "# Fiddler project and model names, used for model registration\n",
    "PROJECT_NAME = \"\"\n",
    "MODEL_NAME = \"\"  # Model name in Fiddler"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0645805a",
   "metadata": {},
   "source": [
    "## 3. Create a fiddler callback handler instance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13de4f9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.callbacks.fiddler_callback import FiddlerCallbackHandler\n",
    "\n",
    "fiddler_handler = FiddlerCallbackHandler(\n",
    "    url=URL,\n",
    "    org=ORG_NAME,\n",
    "    project=PROJECT_NAME,\n",
    "    model=MODEL_NAME,\n",
    "    api_key=AUTH_TOKEN,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2276368e-f1dc-46be-afe3-18796e7a66f2",
   "metadata": {},
   "source": [
    "## Example 1 : Basic Chain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9de0fd1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.output_parsers import StrOutputParser\n",
    "from langchain_openai import OpenAI\n",
    "\n",
    "# Note : Make sure openai API key is set in the environment variable OPENAI_API_KEY\n",
    "llm = OpenAI(temperature=0, streaming=True, callbacks=[fiddler_handler])\n",
    "output_parser = StrOutputParser()\n",
    "\n",
    "chain = llm | output_parser\n",
    "\n",
    "# Invoke the chain. Invocation will be logged to Fiddler, and metrics automatically generated\n",
    "chain.invoke(\"How far is moon from earth?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "309bde0b-e1ce-446c-98ac-3690c26a2676",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Few more invocations\n",
    "chain.invoke(\"What is the temperature on Mars?\")\n",
    "chain.invoke(\"How much is 2 + 200000?\")\n",
    "chain.invoke(\"Which movie won the oscars this year?\")\n",
    "chain.invoke(\"Can you write me a poem about insomnia?\")\n",
    "chain.invoke(\"How are you doing today?\")\n",
    "chain.invoke(\"What is the meaning of life?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48fa4782-c867-4510-9430-4ffa3de3b5eb",
   "metadata": {},
   "source": [
    "## Example 2 : Chain with prompt templates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2aa2c220-8946-4844-8d3c-8f69d744d13f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.prompts import (\n",
    "    ChatPromptTemplate,\n",
    "    FewShotChatMessagePromptTemplate,\n",
    ")\n",
    "\n",
    "examples = [\n",
    "    {\"input\": \"2+2\", \"output\": \"4\"},\n",
    "    {\"input\": \"2+3\", \"output\": \"5\"},\n",
    "]\n",
    "\n",
    "example_prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\"human\", \"{input}\"),\n",
    "        (\"ai\", \"{output}\"),\n",
    "    ]\n",
    ")\n",
    "\n",
    "few_shot_prompt = FewShotChatMessagePromptTemplate(\n",
    "    example_prompt=example_prompt,\n",
    "    examples=examples,\n",
    ")\n",
    "\n",
    "final_prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\"system\", \"You are a wondrous wizard of math.\"),\n",
    "        few_shot_prompt,\n",
    "        (\"human\", \"{input}\"),\n",
    "    ]\n",
    ")\n",
    "\n",
    "# Note : Make sure openai API key is set in the environment variable OPENAI_API_KEY\n",
    "llm = OpenAI(temperature=0, streaming=True, callbacks=[fiddler_handler])\n",
    "\n",
    "chain = final_prompt | llm\n",
    "\n",
    "# Invoke the chain. Invocation will be logged to Fiddler, and metrics automatically generated\n",
    "chain.invoke({\"input\": \"What's the square of a triangle?\"})"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}