mirror of
https://github.com/hwchase17/langchain.git
synced 2025-09-09 06:53:59 +00:00
Synthetic Data generation (#9472)
--------- Co-authored-by: William Fu-Hinthorn <13333726+hinthornw@users.noreply.github.com> Co-authored-by: Bagatur <baskaryan@gmail.com>
This commit is contained in:
PaperMoose
GitHub
@@ -1,46 +1,229 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "aa3571cc",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Data generation\n",
|
||||
"# Synthetic Data generation\n",
|
||||
"\n",
|
||||
"[](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/extras/use_cases/data_generation.ipynb)\n",
|
||||
"\n",
|
||||
"## Use case\n",
|
||||
"\n",
|
||||
"Creating synthethic language data can be beneficial for multiple reasons:\n",
|
||||
"- providing data augmentation\n",
|
||||
"- obtaining domain-specific examples\n",
|
||||
"- increasing data diversity\n",
|
||||
"- enabling quick iteration and experimentation\n",
|
||||
"Synthetic data is artificially generated data, rather than data collected from real-world events. It's used to simulate real data without compromising privacy or encountering real-world limitations. \n",
|
||||
"\n",
|
||||
"Benefits of Synthetic Data:\n",
|
||||
"\n",
|
||||
"1. **Privacy and Security**: No real personal data at risk of breaches.\n",
|
||||
"2. **Data Augmentation**: Expands datasets for machine learning.\n",
|
||||
"3. **Flexibility**: Create specific or rare scenarios.\n",
|
||||
"4. **Cost-effective**: Often cheaper than real-world data collection.\n",
|
||||
"5. **Regulatory Compliance**: Helps navigate strict data protection laws.\n",
|
||||
"6. **Model Robustness**: Can lead to better generalizing AI models.\n",
|
||||
"7. **Rapid Prototyping**: Enables quick testing without real data.\n",
|
||||
"8. **Controlled Experimentation**: Simulate specific conditions.\n",
|
||||
"9. **Access to Data**: Alternative when real data isn't available.\n",
|
||||
"\n",
|
||||
"Note: Despite the benefits, synthetic data should be used carefully, as it may not always capture real-world complexities.\n",
|
||||
"\n",
|
||||
"## Quickstart\n",
|
||||
"\n",
|
||||
"Let's see a very straightforward example of how we can use OpenAI functions for creating synthetic data in LangChain."
|
||||
"In this notebook, we'll dive deep into generating synthetic medical billing records using the langchain library. This tool is particularly useful when you want to develop or test algorithms but don't want to use real patient data due to privacy concerns or data availability issues."
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "bca57012",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Setup\n",
|
||||
"First, you'll need to have the langchain library installed, along with its dependencies. Since we're using the OpenAI generator chain, we'll install that as well. Since this is an experimental lib, we'll need to include `langchain_experimental` in our installs. We'll then import the necessary modules."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "7ae36b66",
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
},
|
||||
"id": "a0377478",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"!pip install langchain openai \n",
|
||||
"\n",
|
||||
"!pip install -U langchain langchain_experimental openai\n",
|
||||
"# Set env var OPENAI_API_KEY or load from a .env file:\n",
|
||||
"# import dotenv\n",
|
||||
"# dotenv.load_dotenv()"
|
||||
"# dotenv.load_dotenv()\n",
|
||||
"\n",
|
||||
"from langchain.prompts import FewShotPromptTemplate, PromptTemplate\n",
|
||||
"from langchain.chat_models import ChatOpenAI\n",
|
||||
"from langchain.pydantic_v1 import BaseModel\n",
|
||||
"from langchain_experimental.tabular_synthetic_data.base import SyntheticDataGenerator\n",
|
||||
"from langchain_experimental.tabular_synthetic_data.openai import create_openai_data_generator, OPENAI_TEMPLATE\n",
|
||||
"from langchain_experimental.tabular_synthetic_data.prompts import SYNTHETIC_FEW_SHOT_SUFFIX, SYNTHETIC_FEW_SHOT_PREFIX\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "a5a0917b",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 1. Define Your Data Model\n",
|
||||
"Every dataset has a structure or a \"schema\". The MedicalBilling class below serves as our schema for the synthetic data. By defining this, we're informing our synthetic data generator about the shape and nature of data we expect."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"execution_count": null,
|
||||
"id": "291bad6e",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"class MedicalBilling(BaseModel):\n",
|
||||
" patient_id: int\n",
|
||||
" patient_name: str\n",
|
||||
" diagnosis_code: str\n",
|
||||
" procedure_code: str\n",
|
||||
" total_charge: float\n",
|
||||
" insurance_claim_amount: float\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "2059ca63",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"For instance, every record will have a `patient_id` that's an integer, a `patient_name` that's a string, and so on.\n",
|
||||
"\n",
|
||||
"## 2. Sample Data\n",
|
||||
"To guide the synthetic data generator, it's useful to provide it with a few real-world-like examples. These examples serve as a \"seed\" - they're representative of the kind of data you want, and the generator will use them to create more data that looks similar.\n",
|
||||
"\n",
|
||||
"Here are some fictional medical billing records:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "b989b792",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"examples = [\n",
|
||||
" {\"example\": \"\"\"Patient ID: 123456, Patient Name: John Doe, Diagnosis Code: \n",
|
||||
" J20.9, Procedure Code: 99203, Total Charge: $500, Insurance Claim Amount: $350\"\"\"},\n",
|
||||
" {\"example\": \"\"\"Patient ID: 789012, Patient Name: Johnson Smith, Diagnosis \n",
|
||||
" Code: M54.5, Procedure Code: 99213, Total Charge: $150, Insurance Claim Amount: $120\"\"\"},\n",
|
||||
" {\"example\": \"\"\"Patient ID: 345678, Patient Name: Emily Stone, Diagnosis Code: \n",
|
||||
" E11.9, Procedure Code: 99214, Total Charge: $300, Insurance Claim Amount: $250\"\"\"},\n",
|
||||
"]\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "57e28809",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 3. Craft a Prompt Template\n",
|
||||
"The generator doesn't magically know how to create our data; we need to guide it. We do this by creating a prompt template. This template helps instruct the underlying language model on how to produce synthetic data in the desired format."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "ea6e042e",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"OPENAI_TEMPLATE = PromptTemplate(input_variables=[\"example\"], template=\"{example}\")\n",
|
||||
"\n",
|
||||
"prompt_template = FewShotPromptTemplate(\n",
|
||||
" prefix=SYNTHETIC_FEW_SHOT_PREFIX,\n",
|
||||
" examples=examples,\n",
|
||||
" suffix=SYNTHETIC_FEW_SHOT_SUFFIX,\n",
|
||||
" input_variables=[\"subject\", \"extra\"],\n",
|
||||
" example_prompt=OPENAI_TEMPLATE,\n",
|
||||
")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "fa6da3cb",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The `FewShotPromptTemplate` includes:\n",
|
||||
"\n",
|
||||
"- `prefix` and `suffix`: These likely contain guiding context or instructions.\n",
|
||||
"- `examples`: The sample data we defined earlier.\n",
|
||||
"- `input_variables`: These variables (\"subject\", \"extra\") are placeholders you can dynamically fill later. For instance, \"subject\" might be filled with \"medical_billing\" to guide the model further.\n",
|
||||
"- `example_prompt`: This prompt template is the format we want each example row to take in our prompt.\n",
|
||||
"\n",
|
||||
"## 4. Creating the Data Generator\n",
|
||||
"With the schema and the prompt ready, the next step is to create the data generator. This object knows how to communicate with the underlying language model to get synthetic data."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "1b9ba911",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"synthetic_data_generator = create_openai_data_generator(\n",
|
||||
" output_schema=MedicalBilling,\n",
|
||||
" llm=ChatOpenAI(temperature=1), # You'll need to replace with your actual Language Model instance\n",
|
||||
" prompt=prompt_template,\n",
|
||||
")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "a4198bd6",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## 5. Generate Synthetic Data\n",
|
||||
"Finally, let's get our synthetic data!"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "a424c890",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"synthetic_results = synthetic_data_generator.generate(\n",
|
||||
" subject=\"medical_billing\",\n",
|
||||
" extra=\"the name must be chosen at random. Make it something you wouldn't normally choose.\",\n",
|
||||
" runs=10,\n",
|
||||
")\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "fa4402e9",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"This command asks the generator to produce 10 synthetic medical billing records. The results are stored in `synthetic_results`. The output will be a list of the MedicalBilling pydantic models."
|
||||
]
|
||||
},
|
||||
{
|
||||
"attachments": {},
|
||||
"cell_type": "markdown",
|
||||
"id": "53a4cbf9",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Other implementations\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "9e715d94",
|
||||
"metadata": {
|
||||
"scrolled": true
|
||||
@@ -429,7 +612,7 @@
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.9.16"
|
||||
"version": "3.11.4"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
|
||||
Reference in New Issue
Block a user