mirror of
https://github.com/hwchase17/langchain.git
synced 2025-09-02 11:39:18 +00:00
[performance]: Adding benchmarks for common langchain-core
imports (#30747)
The first in a sequence of PRs focusing on improving performance in core. We're starting with reducing import times for common structures, hence the benchmarks here. The benchmark looks a little bit complicated - we have to use a process so that we don't suffer from Python's import caching system. I tried doing manual modification of `sys.modules` between runs, but that's pretty tricky / hacky to get right, hence the subprocess approach. Motivated by extremely slow baseline for common imports (we're talking 2-5 seconds): <img width="633" alt="Screenshot 2025-04-09 at 12 48 12 PM" src="https://github.com/user-attachments/assets/994616fe-1798-404d-bcbe-48ad0eb8a9a0" /> Also added a `make benchmark` command to make local runs easy :). Currently using walltimes so that we can track total time despite using a manual proces.
This commit is contained in:
37
.github/workflows/codspeed.yml
vendored
Normal file
37
.github/workflows/codspeed.yml
vendored
Normal file
@@ -0,0 +1,37 @@
|
||||
name: CodSpeed
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
# `workflow_dispatch` allows CodSpeed to trigger backtest
|
||||
# performance analysis in order to generate initial data.
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
codspeed:
|
||||
name: Run benchmarks
|
||||
runs-on: codspeed-macro
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
# We have to use 3.12, 3.13 is not yet supported
|
||||
- name: Install uv
|
||||
uses: astral-sh/setup-uv@v5
|
||||
with:
|
||||
python-version: "3.12"
|
||||
|
||||
# Using this action is still necessary for CodSpeed to work
|
||||
- uses: actions/setup-python@v3
|
||||
with:
|
||||
python-version: "3.12"
|
||||
|
||||
- name: install deps
|
||||
run: uv sync --group test
|
||||
working-directory: ./libs/core
|
||||
|
||||
- name: Run benchmarks
|
||||
uses: CodSpeedHQ/action@v3
|
||||
with:
|
||||
token: ${{ secrets.CODSPEED_TOKEN }}
|
||||
run: |
|
||||
cd libs/core
|
||||
uv run --no-sync pytest ./tests/benchmarks --codspeed
|
Reference in New Issue
Block a user