mirror of
https://github.com/nomic-ai/gpt4all.git
synced 2026-07-17 10:58:08 +00:00
Compare commits
1 Commits
duplicates
...
gptj_eval
| Author | SHA1 | Date | |
|---|---|---|---|
|
af7a4004c8 |
10
.gitignore
vendored
10
.gitignore
vendored
@@ -1,6 +1,3 @@
|
||||
*.pkl
|
||||
ckpts*
|
||||
.deepspeed_env
|
||||
*.jsonl
|
||||
*tar.gz
|
||||
ckpts**
|
||||
@@ -164,9 +161,4 @@ cython_debug/
|
||||
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
|
||||
# and can be added to the global gitignore or merged into this file. For a more nuclear
|
||||
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
|
||||
#.idea/
|
||||
|
||||
|
||||
# vs code
|
||||
.vscode
|
||||
*.bin
|
||||
#.idea/
|
||||
3
.gitmodules
vendored
3
.gitmodules
vendored
@@ -1,3 +1,6 @@
|
||||
[submodule "transformers"]
|
||||
path = transformers
|
||||
url = https://github.com/huggingface/transformers.git
|
||||
[submodule "peft"]
|
||||
path = peft
|
||||
url = https://github.com/huggingface/peft.git
|
||||
|
||||
17
GPT-J_MAP.md
17
GPT-J_MAP.md
@@ -1,17 +0,0 @@
|
||||
# Inference on Training Data
|
||||
|
||||
|
||||
## Run Inference
|
||||
|
||||
```bash
|
||||
torchrun --master_port=29085 --nproc-per-node 8 inference.py --config=configs/inference/gptj.yaml
|
||||
```
|
||||
|
||||
|
||||
## Visualizations
|
||||
|
||||
```bash
|
||||
python build_map.py
|
||||
```
|
||||
|
||||
will build a map in `Atlas`, one using the internal clustering algorithm provided by Nomic and one using the embeddings generated by the finetuned model.
|
||||
19
LICENSE.txt
19
LICENSE.txt
@@ -1,19 +0,0 @@
|
||||
Copyright (c) 2023 Nomic, Inc.
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
207
README.md
207
README.md
@@ -1,92 +1,13 @@
|
||||
<h1 align="center">GPT4All</h1>
|
||||
<p align="center">Demo, data, and code to train open-source assistant-style large language model based on GPT-J and LLaMa</p>
|
||||
<p align="center">
|
||||
<a href="https://static.nomic.ai/gpt4all/2023_GPT4All-J_Technical_Report_2.pdf">:green_book: Technical Report 2: GPT4All-J </a>
|
||||
</p>
|
||||
<p align="center">Demo, data and code to train an assistant-style large language model with ~800k GPT-3.5-Turbo Generations based on LLaMa</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://s3.amazonaws.com/static.nomic.ai/gpt4all/2023_GPT4All_Technical_Report.pdf">:green_book: Technical Report 1: GPT4All</a>
|
||||
<a href="https://s3.amazonaws.com/static.nomic.ai/gpt4all/2023_GPT4All_Technical_Report.pdf">:green_book: Technical Report</a>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://github.com/nomic-ai/pyllamacpp">:snake: Official Python Bindings</a>
|
||||
<a href="https://discord.gg/kvmy6dQB">Discord</a>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://github.com/nomic-ai/gpt4all-ts">:computer: Official Typescript Bindings</a>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://github.com/nomic-ai/gpt4all-ui">:speech_balloon: Official Web Chat Interface</a>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://github.com/nomic-ai/gpt4all-chat">:speech_balloon: Official Chat Interface</a>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://python.langchain.com/en/latest/modules/models/llms/integrations/gpt4all.html">🦜️🔗 Official Langchain Backend</a>
|
||||
</p>
|
||||
|
||||
|
||||
<p align="center">
|
||||
<a href="https://discord.gg/mGZE39AS3e">Discord</a>
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
|
||||
<p align="center">
|
||||
GPT4All is made possible by our compute partner <a href="https://www.paperspace.com/">Paperspace</a>.
|
||||
</p>
|
||||
|
||||
|
||||
|
||||
## GPT4All-J: An Apache-2 Licensed GPT4All Model
|
||||
|
||||
|
||||
Run on an M1 Mac (not sped up!)
|
||||
|
||||
|
||||
### GPT4All-J Chat UI Installers
|
||||
Installs a native chat-client with auto-update functionality that runs on your desktop with the GPT4All-J model baked into it.
|
||||
|
||||
[Mac/OSX](https://gpt4all.io/installers/gpt4all-0.1.0-Darwin.dmg)
|
||||
|
||||
[Windows](https://gpt4all.io/installers/gpt4all-0.1.0-win64.exe)
|
||||
|
||||
[Ubuntu](https://gpt4all.io/installers/gpt4all-0.1.0-Linux.run)
|
||||
|
||||
These files are not yet cert signed by Windows/Apple so you will see security warnings on initial installation. We did not want to delay release while waiting for their process to complete.
|
||||
|
||||
Find the most up-to-date information on the [GPT4All Website](https://gpt4all.io/)
|
||||
|
||||
### Raw Model
|
||||
[ggml Model Download Link](https://gpt4all.io/models/ggml-gpt4all-j.bin)
|
||||
|
||||
Note this model is only compatible with the C++ bindings found [here](https://github.com/nomic-ai/gpt4all-chat). It will not work with any existing llama.cpp bindings as we had to do a large fork of llama.cpp. GPT4All will support the ecosystem around this new C++ backend going forward.
|
||||
|
||||
Python bindings are imminent and will be integrated into this [repository](https://github.com/nomic-ai/pyllamacpp). Stay tuned on the [GPT4All discord](https://discord.gg/mGZE39AS3e) for updates.
|
||||
|
||||
## Training GPT4All-J
|
||||
|
||||
Please see [GPT4All-J Technical Report](https://static.nomic.ai/gpt4all/2023_GPT4All-J_Technical_Report_2.pdf) for details.
|
||||
|
||||
### GPT4All-J Training Data
|
||||
|
||||
- We are releasing the curated training data for anyone to replicate GPT4All-J here: [GPT4All-J Training Data](https://huggingface.co/datasets/nomic-ai/gpt4all-j-prompt-generations)
|
||||
- [Atlas Map of Prompts](https://atlas.nomic.ai/map/gpt4all-j-prompts-curated)
|
||||
- [Atlas Map of Responses](https://atlas.nomic.ai/map/gpt4all-j-response-curated)
|
||||
|
||||
### GPT4All-J Training Instructions
|
||||
|
||||
```bash
|
||||
accelerate launch --dynamo_backend=inductor --num_processes=8 --num_machines=1 --machine_rank=0 --deepspeed_multinode_launcher standard --mixed_precision=bf16 --use_deepspeed --deepspeed_config_file=configs/deepspeed/ds_config_gptj.json train.py --config configs/train/finetune_gptj.yaml
|
||||
```
|
||||
|
||||
|
||||
# Original GPT4All Model (based on GPL Licensed LLaMa)
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -95,7 +16,7 @@ Run on M1 Mac (not sped up!)
|
||||
|
||||
# Try it yourself
|
||||
|
||||
Here's how to get started with the CPU quantized GPT4All model checkpoint:
|
||||
Here's how to get started with the CPU quantized gpt4all model checkpoint:
|
||||
|
||||
1. Download the `gpt4all-lora-quantized.bin` file from [Direct Link](https://the-eye.eu/public/AI/models/nomic-ai/gpt4all/gpt4all-lora-quantized.bin) or [[Torrent-Magnet]](https://tinyurl.com/gpt4all-lora-quantized).
|
||||
2. Clone this repository, navigate to `chat`, and place the downloaded file there.
|
||||
@@ -105,26 +26,20 @@ Here's how to get started with the CPU quantized GPT4All model checkpoint:
|
||||
- Windows (PowerShell): `cd chat;./gpt4all-lora-quantized-win64.exe`
|
||||
- Intel Mac/OSX: `cd chat;./gpt4all-lora-quantized-OSX-intel`
|
||||
|
||||
For custom hardware compilation, see our [llama.cpp](https://github.com/zanussbaum/gpt4all.cpp) fork.
|
||||
For custom hardware compilation, see our [Alpaca C++](https://github.com/zanussbaum/gpt4all.cpp) repository.
|
||||
|
||||
-----------
|
||||
Find all compatible models in the GPT4All Ecosystem section.
|
||||
|
||||
[Secret Unfiltered Checkpoint](https://the-eye.eu/public/AI/models/nomic-ai/gpt4all/gpt4all-lora-unfiltered-quantized.bin) - [[Torrent]](https://the-eye.eu/public/AI/models/nomic-ai/gpt4all/gpt4all-lora-unfiltered-quantized.bin.torrent)
|
||||
|
||||
This model had all refusal to answer responses removed from training. Try it with:
|
||||
- M1 Mac/OSX: `cd chat;./gpt4all-lora-quantized-OSX-m1 -m gpt4all-lora-unfiltered-quantized.bin`
|
||||
- Linux: `cd chat;./gpt4all-lora-quantized-linux-x86 -m gpt4all-lora-unfiltered-quantized.bin`
|
||||
- Windows (PowerShell): `cd chat;./gpt4all-lora-quantized-win64.exe -m gpt4all-lora-unfiltered-quantized.bin`
|
||||
- Intel Mac/OSX: `cd chat;./gpt4all-lora-quantized-OSX-intel -m gpt4all-lora-unfiltered-quantized.bin`
|
||||
- `cd chat;./gpt4all-lora-quantized-OSX-m1 -m gpt4all-lora-unfiltered-quantized.bin`
|
||||
|
||||
-----------
|
||||
Note: the full model on GPU (16GB of RAM required) performs much better in our qualitative evaluations.
|
||||
|
||||
# Python Client
|
||||
## CPU Interface
|
||||
To run GPT4All in python, see the new [official Python bindings](https://github.com/nomic-ai/pyllamacpp).
|
||||
|
||||
The old bindings are still available but now deprecated. They will not work in a notebook environment.
|
||||
To get running using the python client with the CPU interface, first install the [nomic client](https://github.com/nomic-ai/nomic) using `pip install nomic`
|
||||
Then, you can use the following script to interact with GPT4All:
|
||||
```
|
||||
@@ -157,31 +72,18 @@ We are working on a GPT4All that does not have this limitation right now.
|
||||
|
||||
You can pass any of the [huggingface generation config params](https://huggingface.co/docs/transformers/main_classes/text_generation#transformers.GenerationConfig) in the config.
|
||||
|
||||
# GPT4All Compatibility Ecosystem
|
||||
Edge models in the GPT4All Ecosystem. Please PR as the [community grows](https://huggingface.co/models?sort=modified&search=4bit).
|
||||
Feel free to convert this to a more structured table.
|
||||
|
||||
- [gpt4all](https://the-eye.eu/public/AI/models/nomic-ai/gpt4all/gpt4all-lora-quantized.bin) [[MD5 Signature](https://the-eye.eu/public/AI/models/nomic-ai/gpt4all/gpt4all-lora-quantized.bin.md5)]
|
||||
- [gpt4all-ggml-converted](https://the-eye.eu/public/AI/models/nomic-ai/gpt4all/gpt4all-lora-quantized-ggml.bin) [[MD5 Signature](https://the-eye.eu/public/AI/models/nomic-ai/gpt4all/gpt4all-lora-quantized-ggml.bin.md5)]
|
||||
- [gpt4all-unfiltered](https://the-eye.eu/public/AI/models/nomic-ai/gpt4all/gpt4all-lora-unfiltered-quantized.bin) [[MD5 Signature](https://the-eye.eu/public/AI/models/nomic-ai/gpt4all/gpt4all-lora-unfiltered-quantized.bin.md5)]
|
||||
- [ggml-vicuna-7b-4bit](https://huggingface.co/eachadea/ggml-vicuna-7b-4bit)
|
||||
- [vicuna-13b-GPTQ-4bit-128g](https://huggingface.co/anon8231489123/vicuna-13b-GPTQ-4bit-128g)
|
||||
- [LLaMa-Storytelling-4Bit](https://huggingface.co/GamerUntouch/LLaMa-Storytelling-4Bit)
|
||||
- [Alpaca Native 4bit](https://huggingface.co/Sosaka/Alpaca-native-4bit-ggml/tree/main)
|
||||
|
||||
|
||||
# Roadmap
|
||||
## Short Term
|
||||
- <span style="color:green">(Done)</span> Train a GPT4All model based on GPTJ to alleviate llama distribution issues.
|
||||
- <span style="color:green">(Done)</span> Create improved CPU and GPU interfaces for this model.
|
||||
- <span style="color:green">(Done)</span> [Integrate llama.cpp bindings](https://github.com/nomic-ai/pyllamacpp)
|
||||
- <span style="color:green">(Done)</span> [Create a good conversational chat interface for the model.](https://github.com/nomic-ai/gpt4all-ui)
|
||||
- <span style="color:green">(Done)</span> [Allow users to opt in and submit their chats for subsequent training runs](https://github.com/nomic-ai/gpt4all-ui)
|
||||
- <span style="color:green">(IN PROGRESS)</span> Train a GPT4All model based on GPTJ to alleviate llama distribution issues.
|
||||
- <span style="color:green">(IN PROGRESS)</span> Create improved CPU and GPU interfaces for this model.
|
||||
- <span style="color:red">(NOT STARTED)</span> Integrate llama.cpp bindings
|
||||
- <span style="color:red">(NOT STARTED)</span> Create a good conversational chat interface for the model.
|
||||
- <span style="color:red">(NOT STARTED)</span> Allow users to opt in and submit their chats for subsequent training runs
|
||||
|
||||
## Medium Term
|
||||
- <span style="color:red">(NOT STARTED)</span> Integrate GPT4All with [Atlas](https://atlas.nomic.ai) to allow for document retrieval.
|
||||
- BLOCKED by GPT4All based on GPTJ
|
||||
- <span style="color:red">(Done)</span> Integrate GPT4All with Langchain.
|
||||
- <span style="color:red">(NOT STARTED)</span> Integrate GPT4All with Langchain.
|
||||
- <span style="color:green">(IN PROGRESS)</span> Build easy custom training scripts to allow users to fine tune models.
|
||||
|
||||
## Long Term
|
||||
@@ -190,20 +92,15 @@ Feel free to convert this to a more structured table.
|
||||
|
||||
# Reproducibility
|
||||
|
||||
Trained Model Weights:
|
||||
Trained LoRa Weights:
|
||||
- gpt4all-lora (four full epochs of training): https://huggingface.co/nomic-ai/gpt4all-lora
|
||||
- gpt4all-lora-epoch-2 (three full epochs of training) https://huggingface.co/nomic-ai/gpt4all-lora-epoch-2
|
||||
- gpt4all-j (one full epoch of training) (https://huggingface.co/nomic-ai/gpt4all-j)
|
||||
- gpt4all-j-lora (one full epoch of training) (https://huggingface.co/nomic-ai/gpt4all-j-lora)
|
||||
|
||||
Raw Data:
|
||||
- [Training Data Without P3](https://huggingface.co/datasets/nomic-ai/gpt4all_prompt_generations)
|
||||
- Explorer: https://atlas.nomic.ai/map/gpt4all_data_clean_without_p3
|
||||
- [Full Dataset with P3](https://huggingface.co/datasets/nomic-ai/gpt4all_prompt_generations_with_p3)
|
||||
- Explorer: https://atlas.nomic.ai/map/gpt4all_data_clean
|
||||
- [GPT4All-J Dataset](https://huggingface.co/datasets/nomic-ai/gpt4all-j-prompt-generations)
|
||||
- Explorer Indexed on Prompts: https://atlas.nomic.ai/map/gpt4all-j-prompts-curated
|
||||
- Exporer Indexed on Responses: https://atlas.nomic.ai/map/gpt4all-j-response-curated
|
||||
|
||||
We are not distributing a LLaMa 7B checkpoint.
|
||||
|
||||
@@ -213,16 +110,18 @@ You can reproduce our trained model by doing the following:
|
||||
|
||||
Clone the repo
|
||||
|
||||
```
|
||||
git clone --recurse-submodules https://github.com/nomic-ai/gpt4all.git
|
||||
git submodule update --init
|
||||
```
|
||||
`git clone --recurse-submodules https://github.com/nomic-ai/gpt4all.git`
|
||||
|
||||
`git submodule configure && git submodule update`
|
||||
|
||||
Setup the environment
|
||||
|
||||
```
|
||||
python -m pip install -r requirements.txt
|
||||
|
||||
cd transformers
|
||||
pip install -e .
|
||||
|
||||
cd ../peft
|
||||
pip install -e .
|
||||
```
|
||||
@@ -241,7 +140,7 @@ python generate.py --config configs/generate/generate.yaml --prompt "Write a scr
|
||||
|
||||
## Need Help?
|
||||
|
||||
Join the <a href="https://discord.gg/E7q6fp9bmx"> Discord </a> and ask for help in `#gpt4all-help`
|
||||
Join the <a href="https://discord.gg/kvmy6dQB"> Discord </a> and ask for help in `#gpt4all-help`
|
||||
|
||||
# Sample Generations
|
||||
|
||||
@@ -269,8 +168,7 @@ Join the <a href="https://discord.gg/E7q6fp9bmx"> Discord </a> and ask for help
|
||||
|
||||
### Come up with an interesting idea for a new movie plot. Your plot should be described with a title and a summary.
|
||||
|
||||
> 1. The title of your movie plot should be "The Last Stand".
|
||||
> 2. The summary of your movie plot should be: A group of mercenaries are hired to protect a small town from a dangerous criminal who is on the run. However, the town's sheriff is not willing to let them take over and decides to take matters into his own hands.
|
||||
>1. The title of your movie plot should be "The Last Stand". 2. The summary of your movie plot should be: A group of mercenaries are hired to protect a small town from a dangerous criminal who is on the run. However, the town's sheriff is not willing to let them take over and decides to take matters into his own hands.
|
||||
|
||||
### Reverse a string in python.
|
||||
> In Python, you can reverse a list or tuple by using the reversed() function on it. Here's an example of how to use this method with strings:
|
||||
@@ -280,43 +178,42 @@ Join the <a href="https://discord.gg/E7q6fp9bmx"> Discord </a> and ask for help
|
||||
> print(reversed_str) # Output will be reversed version of the inputted string.
|
||||
|
||||
### List 10 dogs.
|
||||
> 1. Labrador Retriever
|
||||
> 2. Golden Retriever
|
||||
> 3. Beagle
|
||||
> 4. Poodle
|
||||
> 5. German Shepherd
|
||||
> 6. Bulldog
|
||||
> 7. Rottweiler
|
||||
> 8. Boxer
|
||||
> 9. Shih Tzu
|
||||
> 10. Pug
|
||||
>1. Labrador Retriever
|
||||
>2. Golden Retriever
|
||||
>3. Beagle
|
||||
>4. Poodle
|
||||
>5. German Shepherd
|
||||
>6. Bulldog
|
||||
>7. Rottweiler
|
||||
>8. Boxer
|
||||
>9. Shih Tzu
|
||||
>10. Pug
|
||||
|
||||
### Write me a poem about the fall of Julius Caesar into a caesar salad in iambic pentameter.
|
||||
|
||||
> The fall of Julius Caesar into a caesar salad<br>
|
||||
> In iambic pentameter,<br>
|
||||
> The mighty Roman emperor<br>
|
||||
> Fell into a salad of lettuce and croutons,<br>
|
||||
> His empire crumbling around him,<br>
|
||||
> As he was devoured by the hungry diners.<br>
|
||||
> The once mighty emperor<br>
|
||||
> Was now just a salad topping,<br>
|
||||
> His legacy reduced to a mere garnish,<br>
|
||||
> As he was consumed by the masses.<br>
|
||||
> The fall of Julius Caesar into a caesar salad,<br>
|
||||
> In iambic pentameter,<br>
|
||||
> Was a tragic end to a great man,<br>
|
||||
> Who had once ruled the world with an iron fist.<br>
|
||||
> But now, he was just a salad topping,<br>
|
||||
> His legacy reduced to a mere garnish,<br>
|
||||
> As he was consumed by the masses.
|
||||
### Write me a poem about the fall of Julius Ceasar into a ceasar salad in iambic pentameter.
|
||||
>The fall of Julius Ceasar into a ceasar salad
|
||||
>In iambic pentameter,
|
||||
>The mighty Roman emperor
|
||||
>Fell into a salad of lettuce and croutons,
|
||||
>His empire crumbling around him,
|
||||
>As he was devoured by the hungry diners.
|
||||
>The once mighty emperor
|
||||
>Was now just a salad topping,
|
||||
>His legacy reduced to a mere garnish,
|
||||
>As he was consumed by the masses.
|
||||
>The fall of Julius Ceasar into a ceasar salad,
|
||||
>In iambic pentameter,
|
||||
>Was a tragic end to a great man,
|
||||
>Who had once ruled the world with an iron fist.
|
||||
>But now, he was just a salad topping,
|
||||
>His legacy reduced to a mere garnish,
|
||||
>As he was consumed by the masses.
|
||||
|
||||
### What is a three word topic describing the following keywords: baseball, football, soccer:
|
||||
> Sports, athletics, games
|
||||
>Sports, athletics, games
|
||||
|
||||
## Citation
|
||||
|
||||
If you utilize this repository, models or data in a downstream project, please consider citing it with:
|
||||
If you utilize this reposistory, models or data in a downstream project, please consider citing it with:
|
||||
```
|
||||
@misc{gpt4all,
|
||||
author = {Yuvanesh Anand and Zach Nussbaum and Brandon Duderstadt and Benjamin Schmidt and Andriy Mulyar},
|
||||
|
||||
@@ -23,7 +23,7 @@ We used the initial parameters:
|
||||
| Weight decay | 0 |
|
||||
| Warmup Steps | 100 |
|
||||
|
||||
We randomly shuffle and set aside 5% of the data for validation.
|
||||
We randomly shuffle and set aside %5 of the data for validation.
|
||||
|
||||
We had an initial bug in logging the training loss but we noticed a decrease in validation loss.
|
||||
|
||||
@@ -160,7 +160,7 @@ We realized that we had two bugs however:
|
||||
- We accidentally duplicated data and effectively trained for 2 epochs instead of 1
|
||||
- We added an eos token to every sequence, even those that we truncated (e.g. long code that exceeds the 1024).
|
||||
|
||||
## Conditional EOS and 1 Epoch
|
||||
## Conditonal EOS and 1 Epoch
|
||||
|
||||
Using the same parameters, we then trained a model using a "conditional" eos token where we only add an `eos` when the inputs are less than the maximum sequence length for one epoch.
|
||||
|
||||
@@ -235,49 +235,3 @@ Taking inspiration from [the Alpaca Repo](https://github.com/tatsu-lab/stanford_
|
||||
Comparing our model LoRa to the [Alpaca LoRa](https://huggingface.co/tloen/alpaca-lora-7b), our model has lower perplexity. Qualitatively, training on 3 epochs performed the best on perplexity as well as qualitative examples.
|
||||
|
||||
We tried training a full model using the parameters above, but found that during the second epoch the model diverged and samples generated post training were worse than the first epoch.
|
||||
|
||||
|
||||
## GPT-J Training
|
||||
|
||||
### Model Training Divergence
|
||||
|
||||
We trained multiple [GPT-J models](https://huggingface.co/EleutherAI/gpt-j-6b) with varying success. We found that training the full model lead to diverged post epoch 1. 
|
||||
|
||||
|
||||
We release the checkpoint after epoch 1.
|
||||
|
||||
|
||||
Using Atlas, we extracted the embeddings of each point in the dataset and calculated the loss per sequence. We then uploaded [this to Atlas](https://atlas.nomic.ai/map/gpt4all-j-post-epoch-1-embeddings) and noticed that the higher loss items seem to cluster. On further inspection, the highest density clusters seemded to be of prompt/response pairs that asked for creative-like generations such as `Generate a story about ...` 
|
||||
|
||||
|
||||
|
||||
### GPT4All-J Hyperparameters
|
||||
|
||||
We varied learning rate, learning rate schedule, and weight decay following suggestions from the [original GPT-J codebase](https://github.com/kingoflolz/mesh-transformer-jax/blob/master/howto_finetune.md) but found no real performance difference (qualitatively or quantitatively) when varying these parameters.
|
||||
|
||||
|
||||
|
||||
The final model was trained using the following hyperparameters with a linear warmup followed by constant learning rate:
|
||||
|
||||
| Hyperparameter | Value |
|
||||
|----------------|-------|
|
||||
| Per Device BS | 32 |
|
||||
| Global BS | 256 |
|
||||
| Learning rate | 2e-5 |
|
||||
| Epochs | 2 |
|
||||
| Max length | 1024 |
|
||||
| Weight decay | 0 |
|
||||
| Warmup Steps | 500 |
|
||||
|
||||
|
||||
The LoRA model was trained using using the following hyperparameters with a linear warmup followed by constant learning rate:
|
||||
|
||||
| Hyperparameter | Value |
|
||||
|----------------|-------|
|
||||
| Per Device BS | 4 |
|
||||
| Global BS | 32 |
|
||||
| Learning rate | 2e-5 |
|
||||
| Epochs | 2 |
|
||||
| Max length | 1024 |
|
||||
| Weight decay | 0 |
|
||||
| Warmup Steps | 500 |
|
||||
|
||||
54
build_map.py
54
build_map.py
@@ -1,54 +0,0 @@
|
||||
import numpy as np
|
||||
from nomic import atlas
|
||||
import glob
|
||||
from tqdm import tqdm
|
||||
from datasets import load_dataset, concatenate_datasets
|
||||
from sklearn.decomposition import PCA
|
||||
|
||||
files = glob.glob("inference/*.jsonl")
|
||||
print(files)
|
||||
df = concatenate_datasets([load_dataset("json", data_files=file, split="train") for file in tqdm(files)])
|
||||
|
||||
print(len(df))
|
||||
print(df)
|
||||
|
||||
df = df.map(lambda example: {"inputs": [prompt + "\n" + response for prompt, response in zip(example["prompt"], example["response"])]},
|
||||
batched=True,
|
||||
num_proc=64)
|
||||
|
||||
df = df.map(lambda example: {"trained_on": [int(t) for t in example["is_train"]]},
|
||||
batched=True,
|
||||
num_proc=64)
|
||||
|
||||
df = df.remove_columns("is_train")
|
||||
|
||||
text = df.remove_columns(["labels", "input_ids", "embeddings"])
|
||||
|
||||
text_df = [text[i] for i in range(len(text))]
|
||||
|
||||
atlas.map_text(text_df, indexed_field="inputs",
|
||||
name="CHANGE ME!",
|
||||
colorable_fields=["source", "loss", "trained_on"],
|
||||
reset_project_if_exists=True,
|
||||
)
|
||||
|
||||
# index is local to train/test split, regenerate
|
||||
data = df.remove_columns(["labels", "input_ids", "index"])
|
||||
data = data.add_column("index", list(range(len(data))))
|
||||
# max embed dim is 2048 for now
|
||||
# note! this is slow in pyarrow/hf datasets
|
||||
embeddings = np.array(data["embeddings"])
|
||||
print("embeddings shape:", embeddings.shape)
|
||||
embeddings = PCA(n_components=2048).fit_transform(embeddings)
|
||||
|
||||
data = data.remove_columns(["embeddings"])
|
||||
columns = data.to_pandas().to_dict("records")
|
||||
|
||||
atlas.map_embeddings(embeddings,
|
||||
data=columns,
|
||||
id_field="index",
|
||||
name="CHANGE ME!",
|
||||
colorable_fields=["source", "loss", "trained_on"],
|
||||
build_topic_model=True,
|
||||
topic_label_field="inputs",
|
||||
reset_project_if_exists=True,)
|
||||
1
clean.py
1
clean.py
@@ -64,7 +64,6 @@ for file in glob.glob(os.path.join(prompt_generation_dir, "*.jsonl")):
|
||||
df = df.dropna(subset=['prompt', 'response'])
|
||||
df = df[df['prompt'] != '']
|
||||
df = df[df['response'] != '']
|
||||
df = df[df["prompt"].str.len() > 1]
|
||||
curr_len = len(df)
|
||||
|
||||
print(f"Removed {prev_len - curr_len} rows")
|
||||
|
||||
@@ -1,48 +0,0 @@
|
||||
{
|
||||
"train_batch_size": "auto",
|
||||
"gradient_accumulation_steps": "auto",
|
||||
"train_micro_batch_size_per_gpu": "auto",
|
||||
"fp16": {
|
||||
"enabled": "auto",
|
||||
"min_loss_scale": 1,
|
||||
"loss_scale_window": 1000,
|
||||
"hysteresis": 2,
|
||||
"initial_scale_power": 32
|
||||
},
|
||||
"bf16": {
|
||||
"enabled": "auto"
|
||||
},
|
||||
"gradient_clipping": 1.0,
|
||||
"zero_optimization": {
|
||||
"stage": 2,
|
||||
"offload_param": {
|
||||
"device": "none"
|
||||
},
|
||||
"offload_optimizer": {
|
||||
"device": "none"
|
||||
},
|
||||
"allgather_partitions": true,
|
||||
"allgather_bucket_size": 5e8,
|
||||
"contiguous_gradients": true
|
||||
},
|
||||
"optimizer": {
|
||||
"type": "AdamW",
|
||||
"params": {
|
||||
"lr": "auto",
|
||||
"betas": [
|
||||
0.9,
|
||||
0.999
|
||||
],
|
||||
"eps": 1e-08
|
||||
}
|
||||
},
|
||||
"scheduler": {
|
||||
"type": "WarmupLR",
|
||||
"params": {
|
||||
"warmup_min_lr": 0,
|
||||
"warmup_max_lr": "auto",
|
||||
"warmup_num_steps": "auto",
|
||||
"warmup_type": "linear"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,48 +0,0 @@
|
||||
{
|
||||
"train_batch_size": "auto",
|
||||
"gradient_accumulation_steps": "auto",
|
||||
"train_micro_batch_size_per_gpu": "auto",
|
||||
"fp16": {
|
||||
"enabled": "auto",
|
||||
"min_loss_scale": 1,
|
||||
"loss_scale_window": 1000,
|
||||
"hysteresis": 2,
|
||||
"initial_scale_power": 32
|
||||
},
|
||||
"bf16": {
|
||||
"enabled": "auto"
|
||||
},
|
||||
"gradient_clipping": 1,
|
||||
"zero_optimization": {
|
||||
"stage": 2,
|
||||
"offload_param": {
|
||||
"device": "cpu"
|
||||
},
|
||||
"offload_optimizer": {
|
||||
"device": "cpu"
|
||||
},
|
||||
"allgather_partitions": true,
|
||||
"allgather_bucket_size": 5e8,
|
||||
"contiguous_gradients": true
|
||||
},
|
||||
"optimizer": {
|
||||
"type": "AdamW",
|
||||
"params": {
|
||||
"lr": "auto",
|
||||
"betas": [
|
||||
0.9,
|
||||
0.999
|
||||
],
|
||||
"eps": 1e-08
|
||||
}
|
||||
},
|
||||
"scheduler": {
|
||||
"type": "WarmupLR",
|
||||
"params": {
|
||||
"warmup_min_lr": 0,
|
||||
"warmup_max_lr": "auto",
|
||||
"warmup_num_steps": "auto",
|
||||
"warmup_type": "linear"
|
||||
}
|
||||
}
|
||||
}
|
||||
15
configs/eval/generate.yaml
Normal file
15
configs/eval/generate.yaml
Normal file
@@ -0,0 +1,15 @@
|
||||
# model/tokenizer
|
||||
model_name: # update with llama 7b
|
||||
tokenizer_name: # update with llama 7b
|
||||
lora: true
|
||||
lora_path: "nomic-ai/gpt4all-lora"
|
||||
|
||||
max_new_tokens: 512
|
||||
temperature: 0.001
|
||||
prompt: |
|
||||
#this code prints a string reversed
|
||||
my_string = "hello how are you"
|
||||
print(len(my_string))
|
||||
|
||||
|
||||
My code above does not work. Can you help me?
|
||||
@@ -1,5 +1,17 @@
|
||||
# model/tokenizer
|
||||
model_name: "zpn/llama-7b"
|
||||
tokenizer_name: "zpn/llama-7b"
|
||||
model_name: # update with llama model name
|
||||
tokenizer_name: # update with llama model name
|
||||
lora: true
|
||||
lora_path: "tloen/alpaca-lora-7b"
|
||||
lora_path: "tloen/alpaca-lora-7b"
|
||||
|
||||
|
||||
|
||||
max_new_tokens: 512
|
||||
temperature: 0.001
|
||||
prompt: |
|
||||
#this code prints a string reversed
|
||||
my_string = "hello how are you"
|
||||
print(len(my_string))
|
||||
|
||||
|
||||
My code above does not work. Can you help me?
|
||||
|
||||
14
configs/eval/generate_full.yaml
Normal file
14
configs/eval/generate_full.yaml
Normal file
@@ -0,0 +1,14 @@
|
||||
# model/tokenizer
|
||||
model_name: # update
|
||||
tokenizer_name: # update
|
||||
lora_path: "no-lora"
|
||||
|
||||
max_new_tokens: 512
|
||||
temperature: 0.001
|
||||
prompt: |
|
||||
#this code prints a string reversed
|
||||
my_string = "hello how are you"
|
||||
print(len(my_string))
|
||||
|
||||
|
||||
My code above does not work. Can you help me?
|
||||
@@ -1,4 +0,0 @@
|
||||
# model/tokenizer
|
||||
model_name: "nomic-ai/gpt4all-warmup-lr-epoch_0"
|
||||
tokenizer_name: "EleutherAI/gpt-j-6b"
|
||||
lora: false
|
||||
@@ -1,5 +0,0 @@
|
||||
# model/tokenizer
|
||||
model_name: "EleutherAI/gpt-j-6b"
|
||||
tokenizer_name: "EleutherAI/gpt-j-6B"
|
||||
lora: true
|
||||
lora_path: "nomic-ai/gpt4all-gptj-lora-epoch_1"
|
||||
@@ -1,5 +0,0 @@
|
||||
# model/tokenizer
|
||||
model_name: "zpn/llama-7b"
|
||||
tokenizer_name: "zpn/llama-7b"
|
||||
lora: true
|
||||
lora_path: "nomic-ai/gpt4all-lora"
|
||||
@@ -1,8 +1,8 @@
|
||||
# model/tokenizer
|
||||
model_name: "nomic-ai/gpt4all-warmup-lr-epoch_1"
|
||||
tokenizer_name: "EleutherAI/gpt-j-6b"
|
||||
model_name: 'nomic-ai/gpt4all-gptj-epoch_0'
|
||||
tokenizer_name: 'EleutherAI/gpt-j-6B'
|
||||
lora: false
|
||||
|
||||
lora_path: 'nolora'
|
||||
|
||||
max_new_tokens: 512
|
||||
temperature: 0.001
|
||||
15
configs/eval/generate_large_2.yaml
Normal file
15
configs/eval/generate_large_2.yaml
Normal file
@@ -0,0 +1,15 @@
|
||||
# model/tokenizer
|
||||
model_name: # update
|
||||
tokenizer_name: # update
|
||||
lora: true
|
||||
lora_path: # update
|
||||
|
||||
max_new_tokens: 512
|
||||
temperature: 0.001
|
||||
prompt: |
|
||||
#this code prints a string reversed
|
||||
my_string = "hello how are you"
|
||||
print(len(my_string))
|
||||
|
||||
|
||||
My code above does not work. Can you help me?
|
||||
15
configs/eval/generate_large_3.yaml
Normal file
15
configs/eval/generate_large_3.yaml
Normal file
@@ -0,0 +1,15 @@
|
||||
# model/tokenizer
|
||||
model_name: # update
|
||||
tokenizer_name: # update
|
||||
lora: true
|
||||
lora_path: # update
|
||||
|
||||
max_new_tokens: 512
|
||||
temperature: 0.001
|
||||
prompt: |
|
||||
#this code prints a string reversed
|
||||
my_string = "hello how are you"
|
||||
print(len(my_string))
|
||||
|
||||
|
||||
My code above does not work. Can you help me?
|
||||
@@ -1,6 +1,6 @@
|
||||
# model/tokenizer
|
||||
model_name: "zpn/llama-7b"
|
||||
tokenizer_name: "zpn/llama-7b"
|
||||
model_name: # REPLACE HERE with the base llama model
|
||||
tokenizer_name: # REPLACE HERE with the llama tokenizer
|
||||
lora: true
|
||||
lora_path: "nomic-ai/gpt4all-lora"
|
||||
|
||||
|
||||
@@ -1,15 +0,0 @@
|
||||
# model/tokenizer
|
||||
model_name: "EleutherAI/gpt-j-6b"
|
||||
tokenizer_name: "EleutherAI/gpt-j-6b"
|
||||
lora: true
|
||||
lora_path: "nomic-ai/gpt4all-gptj-lora-epoch_0"
|
||||
|
||||
max_new_tokens: 512
|
||||
temperature: 0
|
||||
prompt: |
|
||||
#this code prints a string reversed
|
||||
my_string = "hello how are you"
|
||||
print(len(my_string))
|
||||
|
||||
|
||||
My code above does not work. Can you help me?
|
||||
@@ -1,14 +0,0 @@
|
||||
# model/tokenizer
|
||||
model_name: "nomic-ai/gpt4all-warmup-lr-epoch_1"
|
||||
tokenizer_name: "EleutherAI/gpt-j-6B"
|
||||
|
||||
# dataset
|
||||
streaming: false
|
||||
num_proc: 64
|
||||
dataset_path: "nomic-ai/turbo-500k-multi"
|
||||
max_length: 1024
|
||||
batch_size: 32
|
||||
|
||||
# logging
|
||||
seed: 42
|
||||
|
||||
@@ -2,13 +2,12 @@
|
||||
model_name: # add model here
|
||||
tokenizer_name: # add model here
|
||||
gradient_checkpointing: true
|
||||
save_name: # CHANGE
|
||||
save_name: "nomic-ai/gpt4all-full-multi-turn"
|
||||
|
||||
# dataset
|
||||
streaming: false
|
||||
num_proc: 64
|
||||
dataset_path: # update
|
||||
revision: null
|
||||
max_length: 1024
|
||||
batch_size: 32
|
||||
|
||||
@@ -17,7 +16,7 @@ lr: 5.0e-5
|
||||
eval_every: 800
|
||||
eval_steps: 100
|
||||
save_every: 800
|
||||
output_dir: # CHANGE
|
||||
output_dir: "ckpts/gpt4all-full-multi"
|
||||
checkpoint: null
|
||||
lora: false
|
||||
warmup_steps: 100
|
||||
|
||||
@@ -1,34 +0,0 @@
|
||||
# model/tokenizer
|
||||
model_name: "EleutherAI/gpt-j-6B"
|
||||
tokenizer_name: "EleutherAI/gpt-j-6B"
|
||||
gradient_checkpointing: true
|
||||
save_name: # CHANGE
|
||||
|
||||
# dataset
|
||||
streaming: false
|
||||
num_proc: 64
|
||||
dataset_path: # CHANGE
|
||||
revision: null
|
||||
max_length: 1024
|
||||
batch_size: 32
|
||||
|
||||
# train dynamics
|
||||
lr: 2.0e-5
|
||||
min_lr: 0
|
||||
weight_decay: 0.0
|
||||
eval_every: 500
|
||||
eval_steps: 105
|
||||
save_every: 500
|
||||
log_grads_every: 100
|
||||
output_dir: # CHANGE
|
||||
checkpoint: null
|
||||
lora: false
|
||||
warmup_steps: 500
|
||||
num_epochs: 2
|
||||
|
||||
# logging
|
||||
wandb: true
|
||||
wandb_entity: # CHANGE
|
||||
wandb_project_name: # CHANGE
|
||||
seed: 42
|
||||
|
||||
@@ -1,34 +0,0 @@
|
||||
# model/tokenizer
|
||||
model_name: "EleutherAI/gpt-j-6b"
|
||||
tokenizer_name: "EleutherAI/gpt-j-6b"
|
||||
gradient_checkpointing: false
|
||||
save_name: # CHANGE
|
||||
|
||||
# dataset
|
||||
streaming: false
|
||||
num_proc: 64
|
||||
dataset_path: # CHANGE
|
||||
revision: null
|
||||
max_length: 1024
|
||||
batch_size: 1
|
||||
|
||||
# train dynamics
|
||||
lr: 2.0e-5
|
||||
min_lr: 0
|
||||
weight_decay: 0.0
|
||||
eval_every: 500
|
||||
eval_steps: 105
|
||||
save_every: 500
|
||||
log_grads_every: 500
|
||||
output_dir: # CHANGE
|
||||
checkpoint: null
|
||||
lora: true
|
||||
warmup_steps: 500
|
||||
num_epochs: 2
|
||||
|
||||
# logging
|
||||
wandb: true
|
||||
wandb_entity: # CHANGE
|
||||
wandb_project_name: # CHANGE
|
||||
seed: 42
|
||||
|
||||
@@ -2,24 +2,21 @@
|
||||
model_name: # update
|
||||
tokenizer_name: # update
|
||||
gradient_checkpointing: false
|
||||
save_name: # CHANGE
|
||||
save_name: "nomic-ai/gpt4all-lora-multi-turn"
|
||||
|
||||
# dataset
|
||||
streaming: false
|
||||
num_proc: 64
|
||||
dataset_path: # CHANGE
|
||||
revision: null
|
||||
dataset_path: "data_multiturn"
|
||||
max_length: 1024
|
||||
batch_size: 4
|
||||
|
||||
# train dynamics
|
||||
lr: 5.0e-5
|
||||
min_lr: 0
|
||||
weight_decay: 0.0
|
||||
eval_every: 2000
|
||||
eval_steps: 100
|
||||
save_every: 2000
|
||||
output_dir: # CHANGE
|
||||
output_dir: "ckpts/gpt4all-lora-multi"
|
||||
checkpoint: null
|
||||
lora: true
|
||||
warmup_steps: 100
|
||||
|
||||
@@ -1,8 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
export WORKER_IP=$1
|
||||
N_GPUS=8
|
||||
# create dir if doesn't exist
|
||||
sudo mkdir -p /job
|
||||
printf "localhost slots=$N_GPUS\n$WORKER_IP slots=$N_GPUS" | sudo tee /job/hostfile
|
||||
echo /job/hostfile
|
||||
147
data.py
147
data.py
@@ -9,49 +9,44 @@ from transformers import DefaultDataCollator
|
||||
|
||||
def tokenize_inputs(config, tokenizer, examples):
|
||||
max_length = config["max_length"]
|
||||
input_ids = torch.full((len(examples["prompt"]), max_length), tokenizer.pad_token_id)
|
||||
# ignore bos
|
||||
newline_tokens = tokenizer("\n", return_tensors="pt")["input_ids"][0, 1:]
|
||||
|
||||
# hacky backward compatible
|
||||
different_eos = tokenizer.eos_token != "</s>"
|
||||
out = {"labels": [], "input_ids": []}
|
||||
for prompt, response in zip(examples["prompt"], examples["response"]):
|
||||
if different_eos:
|
||||
if response.count("</s> \n") > 0:
|
||||
response = response.replace("</s> \n", f"{tokenizer.eos_token} \n")
|
||||
out = {"labels": [], "attention_mask": []}
|
||||
for i, (prompt, response) in enumerate(zip(examples["prompt"], examples["response"])):
|
||||
input_tokens = tokenizer(prompt, truncation=True, max_length=max_length // 2, return_tensors="pt")["input_ids"].squeeze()
|
||||
input_len = len(input_tokens)
|
||||
|
||||
prompt_len = len(tokenizer(prompt + "\n", return_tensors="pt")["input_ids"][0])
|
||||
# plus one since we remove bos from response
|
||||
# but we subtract one since we want to add eos token
|
||||
remaining_tokens = max_length - input_len - len(newline_tokens) + 1
|
||||
# remove bos
|
||||
target_tokens = tokenizer(response, truncation=True, max_length=remaining_tokens, return_tensors="pt")["input_ids"].squeeze()[1:]
|
||||
|
||||
# hack if our prompt is super long
|
||||
# we need to include some labels so we arbitrarily trunacate at max_length // 2
|
||||
# if the length is too long
|
||||
if prompt_len >= max_length // 2:
|
||||
# if prompt is too long, truncate
|
||||
# but make sure to truncate to at max 1024 tokens
|
||||
new_len = min(max_length // 2, len(prompt) // 2)
|
||||
prompt = prompt[:new_len]
|
||||
# get new prompt length
|
||||
prompt_len = tokenizer(prompt + "\n", return_tensors="pt", max_length=max_length // 2, truncation=True).input_ids.ne(tokenizer.pad_token_id).sum().item()
|
||||
input_ids[i, :input_len] = input_tokens
|
||||
# add newline between prompt and response
|
||||
newline_plus_inputs = input_len + len(newline_tokens)
|
||||
input_ids[i, input_len: newline_plus_inputs] = newline_tokens
|
||||
|
||||
assert prompt_len <= max_length // 2, f"prompt length {prompt_len} exceeds max length {max_length}"
|
||||
# add target tokens, remove bos
|
||||
input_ids[i, newline_plus_inputs: newline_plus_inputs + len(target_tokens)] = target_tokens
|
||||
# add eos token, enforce stopping if we don't truncate
|
||||
# we don't want long code to stop generating if truncated during training
|
||||
if newline_plus_inputs + len(target_tokens) < max_length:
|
||||
input_ids[i, newline_plus_inputs + len(target_tokens)] = tokenizer.eos_token_id
|
||||
|
||||
input_tokens = tokenizer(prompt + "\n" + response + tokenizer.eos_token,
|
||||
truncation=True, max_length=max_length, return_tensors="pt")["input_ids"].squeeze()
|
||||
labels = input_ids[i].clone()
|
||||
labels[: newline_plus_inputs] = -100
|
||||
labels[labels == tokenizer.pad_token_id] = -100
|
||||
# to debug this, can set all values == -100 to the pad token, then assert that tokenizer.decode(labels, skip_special_tokens=True).strip() == response
|
||||
|
||||
labels = input_tokens.clone()
|
||||
labels[:prompt_len] = -100
|
||||
if len(labels) < max_length:
|
||||
# pad to max_length with -100
|
||||
labels = torch.cat([labels, torch.full((max_length - len(labels),), -100)])
|
||||
attention_mask = input_ids[i].ne(tokenizer.pad_token_id).int()
|
||||
|
||||
assert (labels == -100).sum() < len(labels), f"Labels are all -100, something wrong. prompt length {prompt_len} exceeds max length {max_length}"
|
||||
|
||||
if (labels == -100).sum() == len(labels) - 1:
|
||||
print(prompt)
|
||||
print(response)
|
||||
raise
|
||||
|
||||
input_tokens = tokenizer.pad({"input_ids": input_tokens}, padding="max_length", max_length=max_length)["input_ids"]
|
||||
out["labels"].append(labels)
|
||||
out["input_ids"].append(input_tokens)
|
||||
out["attention_mask"].append(attention_mask)
|
||||
|
||||
out["input_ids"] = input_ids
|
||||
|
||||
out = {k: torch.stack(v) if isinstance(v, list) else v for k, v in out.items()}
|
||||
|
||||
@@ -61,24 +56,8 @@ def tokenize_inputs(config, tokenizer, examples):
|
||||
def load_data(config, tokenizer):
|
||||
dataset_path = config["dataset_path"]
|
||||
|
||||
if isinstance(dataset_path, list):
|
||||
all_datasets = []
|
||||
for path in dataset_path:
|
||||
dataset = load_dataset(path, split="train")
|
||||
|
||||
current_columns = dataset.column_names
|
||||
columns_to_keep = ["prompt", "response"]
|
||||
to_remove = set(current_columns) - set(columns_to_keep)
|
||||
|
||||
dataset = dataset.remove_columns(to_remove)
|
||||
if "source" not in current_columns:
|
||||
dataset = dataset.add_column("source", [path.split("/")[-1]] * len(dataset))
|
||||
all_datasets.append(dataset)
|
||||
|
||||
dataset = concatenate_datasets(all_datasets)
|
||||
|
||||
# load local json dataset
|
||||
elif os.path.exists(dataset_path):
|
||||
if os.path.exists(dataset_path):
|
||||
# check if path is a directory
|
||||
if os.path.isdir(dataset_path):
|
||||
files = glob.glob(os.path.join(dataset_path, "*_clean.jsonl"))
|
||||
else:
|
||||
@@ -87,11 +66,9 @@ def load_data(config, tokenizer):
|
||||
print(f"Reading files {files}")
|
||||
|
||||
dataset = load_dataset("json", data_files=files, split="train")
|
||||
|
||||
# read from huggingface
|
||||
|
||||
else:
|
||||
revision = config["revision"]
|
||||
dataset = load_dataset(dataset_path, split="train", revision=revision)
|
||||
dataset = load_dataset(dataset_path)
|
||||
|
||||
dataset = dataset.train_test_split(test_size=.05, seed=config["seed"])
|
||||
|
||||
@@ -106,13 +83,13 @@ def load_data(config, tokenizer):
|
||||
train_dataset = train_dataset.map(
|
||||
lambda ele: tokenize_inputs(config, tokenizer, ele),
|
||||
batched=True,
|
||||
remove_columns=["source", "prompt", "id", "response"],
|
||||
remove_columns=["source", "prompt"],
|
||||
**kwargs
|
||||
)
|
||||
val_dataset = val_dataset.map(
|
||||
lambda ele: tokenize_inputs(config, tokenizer, ele),
|
||||
lambda ele: tokenize_inputs(config, tokenizer, ele),
|
||||
batched=True,
|
||||
remove_columns=["source", "prompt", "id", "response"],
|
||||
remove_columns=["source", "prompt"],
|
||||
**kwargs
|
||||
)
|
||||
|
||||
@@ -134,53 +111,3 @@ def load_data(config, tokenizer):
|
||||
)
|
||||
|
||||
return train_dataloader, val_dataloader
|
||||
|
||||
|
||||
def load_data_for_inference(config, tokenizer):
|
||||
dataset_path = config["dataset_path"]
|
||||
|
||||
if os.path.exists(dataset_path):
|
||||
# check if path is a directory
|
||||
if os.path.isdir(dataset_path):
|
||||
files = glob.glob(os.path.join(dataset_path, "*_clean.jsonl"))
|
||||
else:
|
||||
files = [dataset_path]
|
||||
|
||||
print(f"Reading files {files}")
|
||||
|
||||
dataset = load_dataset("json", data_files=files, split="train")
|
||||
|
||||
else:
|
||||
dataset = load_dataset(dataset_path, split="train")
|
||||
|
||||
dataset = dataset.train_test_split(test_size=.05, seed=config["seed"])
|
||||
|
||||
train_dataset, val_dataset = dataset["train"], dataset["test"]
|
||||
|
||||
train_dataset = train_dataset.add_column("index", list(range(len(train_dataset))))
|
||||
# select first N batches that are divisible by batch_size
|
||||
# gather is a bit annoying (or the way I'm using it) to get uneven batches as it duplicates data
|
||||
train_dataset = train_dataset.select(range((len(train_dataset) // config["batch_size"]) * config["batch_size"]))
|
||||
val_dataset = val_dataset.add_column("index", list(range(len(val_dataset))))
|
||||
val_dataset = val_dataset.select(range((len(val_dataset) // config["batch_size"]) * config["batch_size"]))
|
||||
|
||||
if config["streaming"] is False:
|
||||
kwargs = {"num_proc": config["num_proc"]}
|
||||
else:
|
||||
kwargs = {}
|
||||
|
||||
# tokenize inputs and return labels and attention mask
|
||||
train_dataset = train_dataset.map(
|
||||
lambda ele: tokenize_inputs(config, tokenizer, ele),
|
||||
batched=True,
|
||||
**kwargs
|
||||
)
|
||||
val_dataset = val_dataset.map(
|
||||
lambda ele: tokenize_inputs(config, tokenizer, ele),
|
||||
batched=True,
|
||||
**kwargs
|
||||
)
|
||||
train_dataset = train_dataset.with_format("torch")
|
||||
val_dataset = val_dataset.with_format("torch")
|
||||
|
||||
return train_dataset, val_dataset
|
||||
|
||||
@@ -6,20 +6,18 @@ from matplotlib import pyplot as plt
|
||||
plt.figure()
|
||||
for fpath in glob.glob('./eval_data/*.pkl'):
|
||||
parts = fpath.split('__')
|
||||
model_name = "-".join(fpath.replace(".pkl", "").split("_")[2:])
|
||||
model_name = parts[1].replace('model-', '').replace('.pkl', '')
|
||||
lora_name = parts[2].replace('lora-', '').replace('.pkl', '')
|
||||
with open(fpath, 'rb') as f:
|
||||
data = pickle.load(f)
|
||||
perplexities = data['perplexities']
|
||||
perplexities = np.nan_to_num(perplexities, 100)
|
||||
perplexities = np.clip(perplexities, 0, 100)
|
||||
if 'alpaca' not in fpath:
|
||||
identifier = model_name = "-".join(fpath.replace(".pkl", "").split("eval__model-")[1:])
|
||||
label = 'GPT4all-'
|
||||
label += identifier
|
||||
|
||||
if 'nomic' in fpath:
|
||||
label = 'GPT4all-lora'
|
||||
else:
|
||||
label = 'alpaca-lora'
|
||||
plt.hist(perplexities, label=label, alpha=.5, bins=50)
|
||||
plt.hist(perplexities, label=label, alpha=.5)
|
||||
|
||||
plt.xlabel('Perplexity')
|
||||
plt.ylabel('Frequency')
|
||||
|
||||
@@ -22,24 +22,25 @@ def read_jsonl_file(file_path):
|
||||
return data
|
||||
|
||||
def setup_model(config):
|
||||
model = AutoModelForCausalLM.from_pretrained(config["model_name"], device_map="auto", torch_dtype=torch.float16, output_hidden_states=True)
|
||||
model = AutoModelForCausalLM.from_pretrained(config["model_name"], device_map="auto", torch_dtype=torch.float16, output_hidden_states=True, use_auth_token=True)
|
||||
tokenizer = AutoTokenizer.from_pretrained(config["tokenizer_name"])
|
||||
added_tokens = tokenizer.add_special_tokens({"bos_token": "<s>", "eos_token": "</s>", "pad_token": "<pad>"})
|
||||
if "gptj" in config["model_name"]:
|
||||
tokenizer.pad_token = tokenizer.eos_token
|
||||
else:
|
||||
added_tokens = tokenizer.add_special_tokens({"bos_token": "<s>", "eos_token": "</s>", "pad_token": "<pad>"})
|
||||
|
||||
if added_tokens > 0:
|
||||
model.resize_token_embeddings(len(tokenizer))
|
||||
if added_tokens > 0:
|
||||
model.resize_token_embeddings(len(tokenizer))
|
||||
|
||||
if 'lora' in config and config['lora']:
|
||||
model = PeftModelForCausalLM.from_pretrained(model, config["lora_path"], device_map="auto", torch_dtype=torch.float16, return_hidden_states=True)
|
||||
model.to(dtype=torch.float16)
|
||||
if config["lora"]:
|
||||
model = PeftModelForCausalLM.from_pretrained(model, config["lora_path"], device_map="auto", torch_dtype=torch.float16)
|
||||
|
||||
model.to(dtype=torch.float16)
|
||||
print(f"Mem needed: {model.get_memory_footprint() / 1024 / 1024 / 1024:.2f} GB")
|
||||
|
||||
return model, tokenizer
|
||||
|
||||
|
||||
|
||||
|
||||
def eval_example(model, tokenizer, example, config):
|
||||
|
||||
prompt = example['instruction'] + ' ' + example['instances'][0]['input']
|
||||
@@ -49,6 +50,28 @@ def eval_example(model, tokenizer, example, config):
|
||||
input = tokenizer(prompt, return_tensors="pt")
|
||||
input = {k: v.to(model.device) for k, v in input.items()}
|
||||
|
||||
continuations = []
|
||||
tokenized_continuations = []
|
||||
trajectories = []
|
||||
for i in range(1):
|
||||
with torch.no_grad():
|
||||
outputs = model.generate(input_ids=input['input_ids'],
|
||||
max_new_tokens=config["max_new_tokens"],
|
||||
min_new_tokens=5,
|
||||
temperature=config["temperature"],
|
||||
repetition_penalty=1.0,
|
||||
do_sample=True)
|
||||
decoded = tokenizer.decode(outputs[0], skip_special_tokens=True).strip()
|
||||
|
||||
y = model(input_ids=outputs)
|
||||
trajectory = y.hidden_states[0].detach().cpu().numpy()[0]
|
||||
trajectory = trajectory / np.linalg.norm(trajectory, axis=1, keepdims=True)
|
||||
trajectory = np.cumsum(trajectory, axis=0) / np.arange(1, trajectory.shape[0]+1).reshape(-1, 1)
|
||||
|
||||
trajectories.append(trajectory)
|
||||
continuations.append(decoded)
|
||||
tokenized_continuations.append(tokenizer.tokenize(decoded))
|
||||
|
||||
#compute the ground truth perplexity
|
||||
gt_input = tokenizer(gt, return_tensors="pt")
|
||||
gt_input = {k: v.to(model.device) for k, v in gt_input.items()}
|
||||
@@ -79,23 +102,30 @@ def eval_example(model, tokenizer, example, config):
|
||||
|
||||
print(prompt)
|
||||
print(80*'-')
|
||||
|
||||
for continuation in continuations:
|
||||
print(continuation)
|
||||
print(80*'-')
|
||||
|
||||
return ppl
|
||||
return ppl, trajectories, continuations, tokenized_continuations
|
||||
|
||||
def do_eval(config):
|
||||
eval_data = read_jsonl_file('eval_data/user_oriented_instructions.jsonl')
|
||||
model, tokenizer = setup_model(config)
|
||||
all_trajectories = []
|
||||
all_perplexities = []
|
||||
all_continuations = []
|
||||
all_tokenized_continuations = []
|
||||
for example in tqdm(eval_data):
|
||||
gt_perplexity = eval_example(model, tokenizer, example, config)
|
||||
gt_perplexity, trajectories, continuations, tokenized_continuations = eval_example(model, tokenizer, example, config)
|
||||
all_trajectories.append(trajectories)
|
||||
all_perplexities.append(gt_perplexity)
|
||||
all_continuations.append(continuations)
|
||||
|
||||
|
||||
name = f"eval_data/eval__model-{config['model_name'].replace('/', '_')}{'__lora-' + config['lora_path'].replace('/', '_') if config['lora'] else ''}.pkl"
|
||||
|
||||
with open(name, 'wb') as f:
|
||||
r = {'perplexities': all_perplexities}
|
||||
with open('eval_data/eval__model-{}__lora-{}.pkl'.format(config['model_name'].replace('/', '_'), config['lora_path'].replace('/', '_')), 'wb') as f:
|
||||
r = {'trajectories': all_trajectories,
|
||||
'perplexities': all_perplexities,
|
||||
'continuations': all_continuations,
|
||||
'tokenized_continuations': all_tokenized_continuations}
|
||||
pickle.dump(r, f)
|
||||
|
||||
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 2.3 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 356 KiB |
204
inference.py
204
inference.py
@@ -1,204 +0,0 @@
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
from argparse import ArgumentParser
|
||||
from read import read_config
|
||||
from accelerate.utils import set_seed
|
||||
from data import load_data_for_inference
|
||||
from tqdm import tqdm
|
||||
from datasets import Dataset
|
||||
import torch.distributed as dist
|
||||
from transformers.trainer_pt_utils import nested_numpify
|
||||
from transformers import DefaultDataCollator
|
||||
from torch.utils.data import DataLoader, DistributedSampler
|
||||
import numpy as np
|
||||
import pyarrow as pa
|
||||
from pyarrow import compute as pc
|
||||
|
||||
|
||||
def calc_cross_entropy_no_reduction(lm_logits, labels):
|
||||
# calculate cross entropy across batch dim
|
||||
shift_logits = lm_logits[..., :-1, :].contiguous()
|
||||
shift_labels = labels[..., 1:].contiguous()
|
||||
# Flatten the tokens
|
||||
loss_fct = nn.CrossEntropyLoss(reduction='none')
|
||||
loss = loss_fct(shift_logits.permute(0, 2, 1), shift_labels).mean(dim=1)
|
||||
|
||||
return loss
|
||||
|
||||
|
||||
def rank0_print(msg):
|
||||
if dist.get_rank() == 0:
|
||||
print(msg)
|
||||
|
||||
|
||||
def inference(config):
|
||||
set_seed(config['seed'])
|
||||
|
||||
rank0_print(f"World size: {dist.get_world_size()}")
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(config['tokenizer_name'], model_max_length=config['max_length'])
|
||||
# llama has no pad token, set it to new token
|
||||
if tokenizer.pad_token is None:
|
||||
tokenizer.pad_token = tokenizer.eos_token
|
||||
|
||||
|
||||
train_dataset, val_dataset = load_data_for_inference(config, tokenizer)
|
||||
|
||||
num_processes = dist.get_world_size()
|
||||
local_rank = dist.get_rank()
|
||||
|
||||
train_sampler = DistributedSampler(train_dataset, shuffle=False, drop_last=True, num_replicas=num_processes, rank=local_rank)
|
||||
train_dataloader = DataLoader(
|
||||
train_dataset,
|
||||
collate_fn=DefaultDataCollator(),
|
||||
batch_size=config["batch_size"],
|
||||
sampler=train_sampler,
|
||||
drop_last=True
|
||||
)
|
||||
|
||||
val_sampler = DistributedSampler(val_dataset, shuffle=False, drop_last=True, num_replicas=num_processes, rank=local_rank)
|
||||
val_dataloader = DataLoader(
|
||||
val_dataset,
|
||||
collate_fn=DefaultDataCollator(),
|
||||
batch_size=config["batch_size"],
|
||||
sampler=val_sampler,
|
||||
drop_last=True
|
||||
)
|
||||
|
||||
|
||||
model = AutoModelForCausalLM.from_pretrained(config["model_name"],
|
||||
trust_remote_code=True,
|
||||
torch_dtype=torch.bfloat16,
|
||||
)
|
||||
model.to(f"cuda:{local_rank}")
|
||||
|
||||
with torch.no_grad():
|
||||
train_outputs = {"loss": [], "embeddings": [], "index": []}
|
||||
for batch in tqdm(train_dataloader, disable=local_rank != 0):
|
||||
batch["input_ids"] = batch["input_ids"].to(f"cuda:{local_rank}")
|
||||
batch["labels"] = batch["labels"].to(f"cuda:{local_rank}")
|
||||
outputs = model(input_ids=batch["input_ids"], labels=batch["labels"], output_hidden_states=True)
|
||||
loss = calc_cross_entropy_no_reduction(outputs.logits, batch["labels"])
|
||||
train_outputs["loss"].extend(loss)
|
||||
|
||||
embeddings = outputs.hidden_states[-1]
|
||||
batch_size = batch["input_ids"].shape[0]
|
||||
sequence_lengths = []
|
||||
# since we use mutiturn with multiple <|endoftext|>, we need to find the place where
|
||||
# <|endoftext|> is repeated
|
||||
for item in batch["input_ids"]:
|
||||
indices = torch.where(item == tokenizer.pad_token_id)[0]
|
||||
found = False
|
||||
for index in indices:
|
||||
# case where sequence is less than max length
|
||||
if torch.all(item[index:] == tokenizer.pad_token_id):
|
||||
sequence_lengths.append(index)
|
||||
found = True
|
||||
break
|
||||
# case where sequence is >= max length
|
||||
if not found:
|
||||
sequence_lengths.append(len(item) - 1)
|
||||
|
||||
sequence_lengths = torch.tensor(sequence_lengths)
|
||||
pooled_logits = embeddings[torch.arange(batch_size, device=embeddings.device), sequence_lengths]
|
||||
|
||||
train_outputs["embeddings"].append(pooled_logits)
|
||||
train_outputs["index"].extend(batch["index"].to(model.device))
|
||||
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
train_outputs = nested_numpify(train_outputs)
|
||||
# stack since they're 0-dim arrays
|
||||
train_outputs["index"] = np.stack(train_outputs["index"])
|
||||
train_outputs["loss"] = np.stack(train_outputs["loss"])
|
||||
train_outputs["embeddings"] = np.concatenate(train_outputs["embeddings"])
|
||||
|
||||
df_train = Dataset.from_dict(train_outputs)
|
||||
curr_idx = df_train["index"]
|
||||
|
||||
# compute mask in pyarrow since it's super fast
|
||||
# ty @bmschmidt for showing me this!
|
||||
table = train_dataset.data
|
||||
mask = pc.is_in(table['index'], value_set=pa.array(curr_idx, pa.int32()))
|
||||
filtered_table = table.filter(mask)
|
||||
# convert from pyarrow to Dataset
|
||||
filtered_train = Dataset.from_dict(filtered_table.to_pydict())
|
||||
|
||||
filtered_train = filtered_train.add_column("embeddings", df_train["embeddings"])
|
||||
filtered_train = filtered_train.add_column("loss", df_train["loss"])
|
||||
filtered_train = filtered_train.add_column("is_train", [True] * len(filtered_train))
|
||||
|
||||
filtered_train.to_json(f"inference/epoch_2_embeddings_train_shard_{local_rank}.jsonl", lines=True, orient="records", num_proc=64)
|
||||
|
||||
val_outputs = {"loss": [], "embeddings": [], "index": []}
|
||||
for batch in tqdm(val_dataloader, disable=local_rank != 0):
|
||||
batch["input_ids"] = batch["input_ids"].to(f"cuda:{local_rank}")
|
||||
batch["labels"] = batch["labels"].to(f"cuda:{local_rank}")
|
||||
outputs = model(input_ids=batch["input_ids"], labels=batch["labels"], output_hidden_states=True)
|
||||
loss = calc_cross_entropy_no_reduction(outputs.logits, batch["labels"])
|
||||
val_outputs["loss"].extend(loss)
|
||||
|
||||
embeddings = outputs.hidden_states[-1]
|
||||
batch_size = batch["input_ids"].shape[0]
|
||||
sequence_lengths = []
|
||||
# since we use mutiturn with multiple <|endoftext|>, we need to find the place where
|
||||
# <|endoftext|> is repeated
|
||||
for item in batch["input_ids"]:
|
||||
indices = torch.where(item == tokenizer.pad_token_id)[0]
|
||||
found = False
|
||||
for index in indices:
|
||||
# case where sequence is less than max length
|
||||
if torch.all(item[index:] == tokenizer.pad_token_id):
|
||||
sequence_lengths.append(index)
|
||||
found = True
|
||||
break
|
||||
# case where sequence is >= max length
|
||||
if not found:
|
||||
sequence_lengths.append(len(item) - 1)
|
||||
|
||||
sequence_lengths = torch.tensor(sequence_lengths)
|
||||
pooled_logits = embeddings[torch.arange(batch_size, device=embeddings.device), sequence_lengths]
|
||||
|
||||
val_outputs["embeddings"].append(pooled_logits)
|
||||
val_outputs["index"].extend(batch["index"].to(model.device))
|
||||
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
val_outputs = nested_numpify(val_outputs)
|
||||
val_outputs["index"] = np.stack(val_outputs["index"])
|
||||
val_outputs["loss"] = np.stack(val_outputs["loss"])
|
||||
val_outputs["embeddings"] = np.concatenate(val_outputs["embeddings"])
|
||||
|
||||
df_val = Dataset.from_dict(val_outputs)
|
||||
curr_idx = df_val["index"]
|
||||
|
||||
# compute mask in pyarrow since it's super fast
|
||||
# ty @bmschmidt for showing me this!
|
||||
table = val_dataset.data
|
||||
mask = pc.is_in(table['index'], value_set=pa.array(curr_idx, pa.int32()))
|
||||
filtered_table = table.filter(mask)
|
||||
# convert from pyarrow to Dataset
|
||||
filtered_val = Dataset.from_dict(filtered_table.to_pydict())
|
||||
filtered_val = filtered_val.add_column("embeddings", df_val["embeddings"])
|
||||
filtered_val = filtered_val.add_column("loss", df_val["loss"])
|
||||
filtered_val = filtered_val.add_column("is_train", [False] * len(filtered_val))
|
||||
|
||||
filtered_val.to_json(f"inference/epoch_2_embeddings_val_shard_{local_rank}.jsonl", lines=True, orient="records", num_proc=64)
|
||||
|
||||
|
||||
def main():
|
||||
dist.init_process_group("nccl")
|
||||
parser = ArgumentParser()
|
||||
parser.add_argument("--config", type=str, default="config.yaml")
|
||||
|
||||
args = parser.parse_args()
|
||||
config = read_config(args.config)
|
||||
|
||||
inference(config)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# parse arguments by reading in a config
|
||||
main()
|
||||
|
||||
88
launcher.sh
88
launcher.sh
@@ -1,88 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Display header
|
||||
echo "=========================================================="
|
||||
echo " ██████ ██████ ████████ ██ ██ █████ ██ ██ "
|
||||
echo "██ ██ ██ ██ ██ ██ ██ ██ ██ ██ "
|
||||
echo "██ ███ ██████ ██ ███████ ███████ ██ ██ "
|
||||
echo "██ ██ ██ ██ ██ ██ ██ ██ ██ "
|
||||
echo " ██████ ██ ██ ██ ██ ██ ███████ ███████ "
|
||||
echo " └─> https://github.com/nomic-ai/gpt4all"
|
||||
|
||||
# Function to detect macOS architecture and set the binary filename
|
||||
detect_mac_arch() {
|
||||
local mac_arch
|
||||
mac_arch=$(uname -m)
|
||||
case "$mac_arch" in
|
||||
arm64)
|
||||
os_type="M1 Mac/OSX"
|
||||
binary_filename="gpt4all-lora-quantized-OSX-m1"
|
||||
;;
|
||||
x86_64)
|
||||
os_type="Intel Mac/OSX"
|
||||
binary_filename="gpt4all-lora-quantized-OSX-intel"
|
||||
;;
|
||||
*)
|
||||
echo "Unknown macOS architecture"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
# Detect operating system and set the binary filename
|
||||
case "$(uname -s)" in
|
||||
Darwin*)
|
||||
detect_mac_arch
|
||||
;;
|
||||
Linux*)
|
||||
if grep -q Microsoft /proc/version; then
|
||||
os_type="Windows (WSL)"
|
||||
binary_filename="gpt4all-lora-quantized-win64.exe"
|
||||
else
|
||||
os_type="Linux"
|
||||
binary_filename="gpt4all-lora-quantized-linux-x86"
|
||||
fi
|
||||
;;
|
||||
CYGWIN*|MINGW32*|MSYS*|MINGW*)
|
||||
os_type="Windows (Cygwin/MSYS/MINGW)"
|
||||
binary_filename="gpt4all-lora-quantized-win64.exe"
|
||||
;;
|
||||
*)
|
||||
echo "Unknown operating system"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
echo "================================"
|
||||
echo "== You are using $os_type."
|
||||
|
||||
|
||||
# Change to the chat directory
|
||||
cd chat
|
||||
|
||||
# List .bin files and prompt user to select one
|
||||
bin_files=(*.bin)
|
||||
echo "== Available .bin files:"
|
||||
for i in "${!bin_files[@]}"; do
|
||||
echo " [$((i+1))] ${bin_files[i]}"
|
||||
done
|
||||
|
||||
# Function to get user input and validate it
|
||||
get_valid_user_input() {
|
||||
local input_valid=false
|
||||
|
||||
while ! $input_valid; do
|
||||
echo "==> Please enter a number:"
|
||||
read -r user_selection
|
||||
if [[ $user_selection =~ ^[0-9]+$ ]] && (( user_selection >= 1 && user_selection <= ${#bin_files[@]} )); then
|
||||
input_valid=true
|
||||
else
|
||||
echo "Invalid input. Please enter a number between 1 and ${#bin_files[@]}."
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
get_valid_user_input
|
||||
selected_bin_file="${bin_files[$((user_selection-1))]}"
|
||||
|
||||
# Run the selected .bin file with the appropriate command
|
||||
./"$binary_filename" -m "$selected_bin_file"
|
||||
@@ -2,14 +2,11 @@ accelerate
|
||||
datasets
|
||||
torchmetrics
|
||||
evaluate
|
||||
transformers>=4.28.0
|
||||
transformers
|
||||
wandb
|
||||
pip
|
||||
peft
|
||||
nodelist-inflator
|
||||
deepspeed
|
||||
sentencepiece
|
||||
jsonlines
|
||||
nomic
|
||||
scikit-learn
|
||||
matplotlib
|
||||
jsonlines
|
||||
108
train.py
108
train.py
@@ -1,7 +1,8 @@
|
||||
import os
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer, get_scheduler, LlamaForCausalLM
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
from transformers.trainer_pt_utils import get_parameter_names
|
||||
import torch
|
||||
from torch.optim import AdamW
|
||||
import torch.nn as nn
|
||||
from argparse import ArgumentParser
|
||||
from read import read_config
|
||||
from accelerate import Accelerator
|
||||
@@ -10,9 +11,7 @@ from peft import get_peft_model, LoraConfig, TaskType
|
||||
from data import load_data
|
||||
from torchmetrics import MeanMetric
|
||||
from tqdm import tqdm
|
||||
import wandb
|
||||
|
||||
torch.backends.cuda.matmul.allow_tf32 = True
|
||||
|
||||
def format_metrics(metrics, split, prefix=""):
|
||||
log = f"[{split}]" + prefix
|
||||
@@ -21,12 +20,17 @@ def format_metrics(metrics, split, prefix=""):
|
||||
return log
|
||||
|
||||
|
||||
def evaluate(model, val_dataloader):
|
||||
def evaluate(config, model, val_dataloader):
|
||||
model.eval()
|
||||
val_loss = MeanMetric(nan_strategy="error").to(model.device)
|
||||
val_loss = MeanMetric().to(model.device)
|
||||
|
||||
with torch.no_grad():
|
||||
for batch in tqdm(val_dataloader):
|
||||
for i, batch in enumerate(
|
||||
tqdm(val_dataloader),
|
||||
):
|
||||
if i == config["eval_steps"]:
|
||||
break
|
||||
|
||||
loss = model(**batch).loss
|
||||
|
||||
loss_values = accelerator.gather_for_metrics({"loss": loss.detach()})
|
||||
@@ -42,20 +46,25 @@ def train(accelerator, config):
|
||||
accelerator.print(config)
|
||||
accelerator.print(f"Using {accelerator.num_processes} GPUs")
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(config['tokenizer_name'], model_max_length=config['max_length'])
|
||||
# if no pad token, set it to eos
|
||||
tokenizer = AutoTokenizer.from_pretrained(config['tokenizer_name'])
|
||||
# llama has no pad token, set it to new token
|
||||
if tokenizer.pad_token is None:
|
||||
tokenizer.pad_token = tokenizer.eos_token
|
||||
# these tokens are already in the vocab, just not mapped correctly
|
||||
added_tokens = tokenizer.add_special_tokens({"bos_token": "<s>", "eos_token": "</s>", "pad_token": "<pad>"})
|
||||
|
||||
|
||||
with accelerator.main_process_first():
|
||||
train_dataloader, val_dataloader = load_data(config, tokenizer)
|
||||
|
||||
|
||||
|
||||
checkpoint = config["gradient_checkpointing"]
|
||||
model = AutoModelForCausalLM.from_pretrained(config["model_name"],
|
||||
use_cache=False if checkpoint else True,
|
||||
trust_remote_code=True)
|
||||
|
||||
if added_tokens > 0:
|
||||
model.resize_token_embeddings(len(tokenizer))
|
||||
|
||||
if checkpoint:
|
||||
model.gradient_checkpointing_enable()
|
||||
|
||||
@@ -68,7 +77,7 @@ def train(accelerator, config):
|
||||
model.print_trainable_parameters()
|
||||
|
||||
optimizer_cls = (
|
||||
AdamW
|
||||
torch.optim.AdamW
|
||||
if accelerator.state.deepspeed_plugin is None
|
||||
or "optimizer" not in accelerator.state.deepspeed_plugin.deepspeed_config
|
||||
else DummyOptim
|
||||
@@ -76,35 +85,11 @@ def train(accelerator, config):
|
||||
|
||||
# karpathy doesn't decay embeddding, maybe we should exclude
|
||||
# https://github.com/karpathy/minGPT/commit/bbbdac74fa9b2e55574d70056163ffbae42310c1#diff-2075fa9c224b395be5bda85544dd36572b59c76c54562819eadadbf268602834R157s
|
||||
optimizer = optimizer_cls(model.parameters(), lr=config["lr"], weight_decay=config["weight_decay"])
|
||||
optimizer = optimizer_cls(model.parameters(), lr=config["lr"])
|
||||
|
||||
if accelerator.state.deepspeed_plugin is not None:
|
||||
gradient_accumulation_steps = accelerator.state.deepspeed_plugin.deepspeed_config[
|
||||
"gradient_accumulation_steps"
|
||||
]
|
||||
|
||||
# decay to min_lr instead of 0
|
||||
lr_ratio = config["min_lr"] / config["lr"]
|
||||
accelerator.print(f"Len of train_dataloader: {len(train_dataloader)}")
|
||||
total_num_steps = (len(train_dataloader) / gradient_accumulation_steps) * config["num_epochs"]
|
||||
# instead of decaying to zero, decay to ratio of min_lr / lr
|
||||
total_num_steps += int(total_num_steps * lr_ratio) + config["warmup_steps"]
|
||||
accelerator.print(f"Total training steps: {total_num_steps}")
|
||||
|
||||
# Creates Dummy Scheduler if `scheduler` was spcified in the config file else creates `args.lr_scheduler_type` Scheduler
|
||||
if (
|
||||
accelerator.state.deepspeed_plugin is None
|
||||
or "scheduler" not in accelerator.state.deepspeed_plugin.deepspeed_config
|
||||
):
|
||||
scheduler = get_scheduler(
|
||||
name="cosine",
|
||||
optimizer=optimizer,
|
||||
num_warmup_steps=config["warmup_steps"] * accelerator.num_processes,
|
||||
num_training_steps=total_num_steps,
|
||||
)
|
||||
else:
|
||||
scheduler = DummyScheduler(
|
||||
optimizer, total_num_steps=config["warmup_steps"], warmup_num_steps=config["warmup_steps"]
|
||||
# scheduler defined in Deepspeed config
|
||||
scheduler = DummyScheduler(
|
||||
optimizer, warmup_num_steps=config["warmup_steps"],
|
||||
)
|
||||
|
||||
model, optimizer, train_dataloader, val_dataloader, scheduler = accelerator.prepare(
|
||||
@@ -123,25 +108,21 @@ def train(accelerator, config):
|
||||
accelerator.skip_first_batches(train_dataloader, resume_step)
|
||||
accelerator.print(f"Resuming from step {resume_step}")
|
||||
|
||||
train_loss = MeanMetric().to(model.device)
|
||||
|
||||
# log gradients
|
||||
if accelerator.is_main_process and config["wandb"]:
|
||||
wandb.watch(model, log_freq=config["log_grads_every"], log="all")
|
||||
if accelerator.state.deepspeed_plugin is not None:
|
||||
gradient_accumulation_steps = accelerator.state.deepspeed_plugin.deepspeed_config[
|
||||
"gradient_accumulation_steps"
|
||||
]
|
||||
|
||||
for epoch in range(config["num_epochs"]):
|
||||
train_loss = MeanMetric(nan_strategy="error").to(model.device)
|
||||
for step, batch in enumerate(tqdm(train_dataloader)):
|
||||
model.train()
|
||||
outputs = model(**batch)
|
||||
loss = outputs.loss
|
||||
|
||||
# gather loss before backprop in case of gradient accumulation
|
||||
loss_values = accelerator.gather_for_metrics({"loss": loss.detach().float()})
|
||||
train_loss.update(loss_values["loss"])
|
||||
|
||||
loss = loss / gradient_accumulation_steps
|
||||
|
||||
accelerator.backward(loss)
|
||||
# get gradient norm of all params
|
||||
|
||||
# log LR in case something weird happens
|
||||
if step > 0 and step % (config["eval_every"] // 10) == 0:
|
||||
@@ -154,13 +135,14 @@ def train(accelerator, config):
|
||||
scheduler.step()
|
||||
optimizer.zero_grad()
|
||||
|
||||
loss_values = accelerator.gather_for_metrics({"loss": loss.detach()})
|
||||
train_loss.update(loss_values["loss"])
|
||||
|
||||
if step > 0 and step % config["save_every"] == 0:
|
||||
curr_step = step + epoch * len(train_dataloader)
|
||||
accelerator.save_state(f"{config['output_dir']}/step_{curr_step}")
|
||||
accelerator.save_state(f"{config['output_dir']}/step_{step}")
|
||||
|
||||
if step > 0 and (step % config["eval_every"] == 0 or step == len(train_dataloader) - 1):
|
||||
val_loss = evaluate(model, val_dataloader)
|
||||
if step > 0 and step % config["eval_every"] == 0:
|
||||
val_loss = evaluate(config, model, val_dataloader)
|
||||
|
||||
log_train = {
|
||||
"train_loss": train_loss.compute()
|
||||
@@ -183,20 +165,9 @@ def train(accelerator, config):
|
||||
accelerator.print(f"Pushing to HF hub")
|
||||
accelerator.wait_for_everyone()
|
||||
unwrapped_model = accelerator.unwrap_model(model)
|
||||
try:
|
||||
if accelerator.is_main_process:
|
||||
unwrapped_model.push_to_hub(config["save_name"] + f"-epoch_{epoch}", private=True)
|
||||
if accelerator.is_main_process:
|
||||
unwrapped_model.push_to_hub(config["save_name"] + "_first_epoch", private=True)
|
||||
|
||||
except Exception as e:
|
||||
accelerator.print(e)
|
||||
accelerator.print(f"Failed to push to hub")
|
||||
|
||||
unwrapped_model.save_pretrained(
|
||||
f"{config['output_dir']}/epoch_{epoch}",
|
||||
is_main_process=accelerator.is_main_process,
|
||||
save_function=accelerator.save,
|
||||
state_dict=accelerator.get_state_dict(model),
|
||||
)
|
||||
|
||||
accelerator.wait_for_everyone()
|
||||
unwrapped_model = accelerator.unwrap_model(model)
|
||||
@@ -207,6 +178,9 @@ def train(accelerator, config):
|
||||
state_dict=accelerator.get_state_dict(model),
|
||||
)
|
||||
|
||||
if accelerator.is_main_process:
|
||||
unwrapped_model.push_to_hub(config["save_name"], private=True)
|
||||
|
||||
accelerator.end_training()
|
||||
|
||||
|
||||
|
||||
1
transformers
Submodule
1
transformers
Submodule
Submodule transformers added at cae78c46d6
Reference in New Issue
Block a user