chore: use poetry as dependency manager (#242)

* Adopts [Poetry](https://python-poetry.org/) as a dependency manager
* Introduces dependency version requirements
* Deprecates Python 3.7 support

**TODO**
- [x] Update developer guide
- [x] Add back `playwright`, `manifest-ml`, and `jupyter` to dependency
group

**Not Doing => Fast Follow**
- Investigate single source for version, perhaps relying on GitHub tags
and [tackling this
issue](https://github.com/hwchase17/langchain/issues/26)

.github/workflows/lint.yml

@@ -1,23 +1,35 @@
 name: lint
 
-on: [push, pull_request]
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+env:
+  POETRY_VERSION: "1.2.0"
 
 jobs:
   build:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.7"]
+        python-version: 
+         - "3.8"
+         - "3.9"
+         - "3.10"
     steps:
     - uses: actions/checkout@v3
+    - name: Install poetry
+      run: |
+        pipx install poetry==$POETRY_VERSION
     - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v3
+      uses: actions/setup-python@v4
       with:
         python-version: ${{ matrix.python-version }}
+        cache: poetry
     - name: Install dependencies
       run: |
-        python -m pip install --upgrade pip
+        poetry install
-        pip install -r test_requirements.txt
     - name: Analysing the code with our lint
       run: |
         make lint

.github/workflows/test.yml

@@ -1,23 +1,33 @@
 name: test
 
-on: [push, pull_request]
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+env:
+  POETRY_VERSION: "1.2.0"
 
 jobs:
   build:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ["3.7"]
+        python-version:
+         - "3.8"
+         - "3.9"
+         - "3.10"
     steps:
     - uses: actions/checkout@v3
+    - name: Install poetry
+      run: pipx install poetry==$POETRY_VERSION
     - name: Set up Python ${{ matrix.python-version }}
-      uses: actions/setup-python@v3
+      uses: actions/setup-python@v4
       with:
         python-version: ${{ matrix.python-version }}
+        cache: 'poetry'
     - name: Install dependencies
-      run: |
+      run: poetry install
-        python -m pip install --upgrade pip
-        pip install -r test_requirements.txt
     - name: Run unit tests
       run: |
         make tests

MANIFEST.in

@@ -1,3 +0,0 @@
-include langchain/py.typed
-include langchain/VERSION
-include LICENSE

Makefile

@@ -1,17 +1,17 @@
 .PHONY: format lint tests integration_tests
 
 format:
-	black .
+	poetry run black .
-	isort .
+	poetry run isort .
 
 lint:
-	mypy .
+	poetry run mypy .
-	black . --check
+	poetry run black . --check
-	isort . --check
+	poetry run isort . --check
-	flake8 .
+	poetry run flake8 .
 
 tests:
-	pytest tests/unit_tests
+	poetry run pytest tests/unit_tests
 
 integration_tests:
-	pytest tests/integration_tests
+	poetry run pytest tests/integration_tests

README.md

@@ -99,27 +99,90 @@ both at a short term but also at a long term level. The concept of "Memory" exis
 
 ## 🤖 Developer Guide
 
-To begin developing on this project, first clone to the repo locally.
+To begin developing on this project, first clone the repo locally.
-To install requirements, run `pip install -r requirements.txt`.
+
-This will install all requirements for running the package, examples, linting, formatting, and tests.
+### Quick Start
+
+This project uses [Poetry](https://python-poetry.org/) as a dependency manager. Check out Poetry's own [documentation on how to install it](https://python-poetry.org/docs/#installation) on your system before proceeding.
+
+To install requirements:
+
+```bash
+poetry install -E all
+```
+
+This will install all requirements for running the package, examples, linting, formatting, and tests. Note the `-E all` flag will install all optional dependencies necessary for integration testing.
+
+Now, you should be able to run the common tasks in the following section.
+
+### Common Tasks
+
+#### Code Formatting
 
 Formatting for this project is a combination of [Black](https://black.readthedocs.io/en/stable/) and [isort](https://pycqa.github.io/isort/).
-To run formatting for this project, run `make format`.
+
+To run formatting for this project:
+
+```bash
+make format
+```
+
+#### Linting
 
 Linting for this project is a combination of [Black](https://black.readthedocs.io/en/stable/), [isort](https://pycqa.github.io/isort/), [flake8](https://flake8.pycqa.org/en/latest/), and [mypy](http://mypy-lang.org/).
-To run linting for this project, run `make lint`.
+
+To run linting for this project:
+
+```bash
+make lint
+```
+
 We recognize linting can be annoying - if you do not want to do it, please contact a project maintainer and they can help you with it. We do not want this to be a blocker for good code getting contributed.
 
+#### Testing
+
 Unit tests cover modular logic that does not require calls to outside apis.
-To run unit tests, run `make tests`.
+
+To run unit tests:
+
+```bash
+make tests
+```
+
 If you add new logic, please add a unit test.
 
 Integration tests cover logic that requires making calls to outside APIs (often integration with other services).
-To run integration tests, run `make integration_tests`.
+
+To run integration tests:
+
+```bash
+make integration_tests
+```
+
 If you add support for a new external API, please add a new integration test.
 
-If you are adding a Jupyter notebook example, you can run `pip install -e .` to build the langchain package from your local changes, so your new logic can be imported into the notebook.
+#### Adding a Jupyter Notebook
+
+If you are adding a Jupyter notebook example, you'll want to install the optional `dev` dependencies.
+
+To install dev dependencies:
+
+```bash
+poetry install --with dev
+```
+
+Launch a notebook:
+
+```bash
+poetry run jupyter notebook
+```
+
+When you run `poetry install`, the `langchain` package is installed as editable in the virtualenv, so your new logic can be imported into the notebook.
+
+#### Contribute Documentation
 
 Docs are largely autogenerated by [sphinx](https://www.sphinx-doc.org/en/master/) from the code.
+
 For that reason, we ask that you add good documentation to all classes and methods.
+
 Similar to linting, we recognize documentation can be annoying - if you do not want to do it, please contact a project maintainer and they can help you with it. We do not want this to be a blocker for good code getting contributed.

poetry.toml

@@ -0,0 +1,2 @@
+[virtualenvs]
+in-project = true

pyproject.toml

@@ -1,3 +1,53 @@
+[tool.poetry]
+name = "langchain"
+version = "0.0.26"
+description = "Building applications with LLMs through composability"
+authors = []
+license = "MIT"
+readme = "README.md"
+repository = "https://www.github.com/hwchase17/langchain"
+
+[tool.poetry.dependencies]
+python = ">=3.8.1,<4.0"
+pydantic = "^1"
+SQLAlchemy = "^1"
+requests = "^2"
+PyYAML = "^6"
+numpy = "^1"
+faiss-cpu = {version = "^1", optional = true}
+wikipedia = {version = "^1", optional = true}
+elasticsearch = {version = "^8", optional = true}
+manifest-ml = {version = "^0.0.1", optional = true}
+spacy = {version = "^3", optional = true}
+nltk = {version = "^3", optional = true}
+transformers = {version = "^4", optional = true}
+
+[tool.poetry.group.test.dependencies]
+pytest = "^7.2.0"
+pytest-dotenv = "^0.5.2"
+
+[tool.poetry.group.lint.dependencies]
+flake8-docstrings = "^1.6.0"
+black = "^22.10.0"
+isort = "^5.10.1"
+flake8 = "^6.0.0"
+
+[tool.poetry.group.typing.dependencies]
+mypy = "^0.991"
+types-pyyaml = "^6.0.12.2"
+types-requests = "^2.28.11.5"
+
+[tool.poetry.group.dev]
+optional = true
+
+[tool.poetry.group.dev.dependencies]
+jupyter = "^1.0.0"
+playwright = "^1.28.0"
+
+[tool.poetry.extras]
+llms = ["cohere", "openai", "nlpcloud", "huggingface_hub", "manifest-ml"]
+all = ["cohere", "openai", "nlpcloud", "huggingface_hub", "manifest-ml", "elasticsearch", "google-search-results", "faiss-cpu", "sentence_transformers", "transformers", "spacy", "nltk", "wikipedia"]
+
 [tool.isort]
 profile = "black"
 
@@ -5,3 +55,7 @@ profile = "black"
 ignore_missing_imports = "True"
 disallow_untyped_defs = "True"
 exclude = ["notebooks"]
+
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"

requirements.txt

@@ -1,7 +0,0 @@
--r test_requirements.txt
--e '.[all]'
-# For trickier integrations
-playwright
-manifest-ml
-# For development
-jupyter

setup.py

@@ -1,40 +0,0 @@
-"""Set up the package."""
-from pathlib import Path
-
-from setuptools import find_packages, setup
-
-with open(Path(__file__).absolute().parents[0] / "langchain" / "VERSION") as _f:
-    __version__ = _f.read().strip()
-
-with open("README.md", "r", encoding="utf-8") as f:
-    long_description = f.read()
-
-LLM_DEPENDENCIES = ["cohere", "openai", "nlpcloud", "huggingface_hub"]
-OTHER_DEPENDENCIES = [
-    "elasticsearch",
-    "google-search-results",
-    "wikipedia",
-    "faiss-cpu",
-    "sentence_transformers",
-    "transformers",
-    "spacy",
-    "nltk",
-]
-
-
-setup(
-    name="langchain",
-    version=__version__,
-    packages=find_packages(),
-    description="Building applications with LLMs through composability",
-    install_requires=["pydantic", "sqlalchemy", "numpy", "requests", "pyyaml"],
-    long_description=long_description,
-    license="MIT",
-    url="https://github.com/hwchase17/langchain",
-    include_package_data=True,
-    long_description_content_type="text/markdown",
-    extras_require={
-        "llms": LLM_DEPENDENCIES,
-        "all": LLM_DEPENDENCIES + OTHER_DEPENDENCIES,
-    },
-)

test_requirements.txt

@@ -1,12 +0,0 @@
--e .
-# For testing
-pytest
-pytest-dotenv
-# For linting
-black
-isort
-mypy
-flake8
-flake8-docstrings
-types-requests
-types-PyYAML