From d5661f0f257663ea07bff5f5d5065f8318062a75 Mon Sep 17 00:00:00 2001 From: digger yu Date: Mon, 27 Nov 2023 18:25:53 +0800 Subject: [PATCH] [nfc] fix typo change directoty to directory (#5111) --- applications/Colossal-LLaMA-2/README.md | 36 +++++++++++++++-------- docs/source/zh-Hans/basics/booster_api.md | 2 +- examples/language/llama2/README.md | 4 +-- 3 files changed, 26 insertions(+), 16 deletions(-) diff --git a/applications/Colossal-LLaMA-2/README.md b/applications/Colossal-LLaMA-2/README.md index ae2e0c6bb..d2b8e0939 100644 --- a/applications/Colossal-LLaMA-2/README.md +++ b/applications/Colossal-LLaMA-2/README.md @@ -5,20 +5,31 @@ ## Table of Contents +- [Table of Contents](#table-of-contents) - [News](#news) - [Colossal-LLaMA-2-7B](#colossal-llama-2-7b) - - [Performance Evaluation](#performance-evaluation) - - [Examples](#examples) - - [Training Logs](#training-logs) - - [Import from Transformers](#import-from-transformers) + - [Performance Evaluation](#performance-evaluation) + - [Examples](#examples) + - [Training Logs](#training-logs) + - [Import from Transformers (Inference)](#import-from-transformers-inference) - [Usage](#usage) - - [Install](#install) - - [How to run](#how-to-run) -- [Technical Insight](#technical-insights) - - [Data](#data) - - [Tokenizer](#tokenizer) - - [Training Strategy](#training-strategy) - - [Bridging Any Domain-specific Large Models](#bridging-any-domain-specific-large-models) + - [Install](#install) + - [0. Pre-requisite](#0-pre-requisite) + - [1. Install required packages](#1-install-required-packages) + - [2. Install `xentropy`, `layer_norm` and `rotary`](#2-install-xentropy-layer_norm-and-rotary) + - [How to run](#how-to-run) + - [1. Init Tokenizer Preparation](#1-init-tokenizer-preparation) + - [2. Init Model Preparation](#2-init-model-preparation) + - [3. Data Preparation](#3-data-preparation) + - [4. Command Line Arguments for Training](#4-command-line-arguments-for-training) + - [5. Running Command](#5-running-command) +- [Technical Insights](#technical-insights) + - [Data](#data) + - [Tokenizer](#tokenizer) + - [Training Strategy](#training-strategy) + - [Multi-stage Training](#multi-stage-training) + - [Bucket-based Training](#bucket-based-training) + - [Bridging Any Domain-specific Large Models](#bridging-any-domain-specific-large-models) - [Citations](#citations) ## News @@ -260,7 +271,7 @@ Here is details about CLI arguments: * Booster plugin: `--plugin`. `gemini`, `gemini_auto`, `zero2`，`zero2_cpu` and `3d` are supported.For more details, please refer to [Booster plugins](https://colossalai.org/docs/basics/booster_plugins/). * Intermediate checkpoint to load: `--load_checkpoint`. Path to the intermediate checkpoint. Saved checkpoint contains the states for `lr_scheduler`, `optimizer`,`running_states.json` and `modelling`. If `load_checkpoint` points to the `modelling` folder, only the model weights will be loaded without any other states to support multi-stage training. * Save interval: `--save_interval`. The interval (steps) of saving checkpoints. The default value is 1000. -* Checkpoint directory: `--save_dir`. The directoty path to save checkpoint and intermediate states. Intermediate states include `lr_scheduler`, `optimizer`,`running_states.json` and `modelling`. +* Checkpoint directory: `--save_dir`. The directory path to save checkpoint and intermediate states. Intermediate states include `lr_scheduler`, `optimizer`,`running_states.json` and `modelling`. * Tensorboard directory: `--tensorboard_dir`. The path to save tensorboard logs. * Configuration file: `--config_file`. The path to save the configuration file. * Number of epochs: `--num_epochs`. Number of training epochs. The default value is 1. @@ -404,5 +415,4 @@ Applying the above process to perform knowledge transfer in any field allows for author={Dao, Tri}, year={2023} } -} ``` diff --git a/docs/source/zh-Hans/basics/booster_api.md b/docs/source/zh-Hans/basics/booster_api.md index f9310374d..bb100964d 100644 --- a/docs/source/zh-Hans/basics/booster_api.md +++ b/docs/source/zh-Hans/basics/booster_api.md @@ -23,7 +23,7 @@ Booster 插件是管理并行配置的重要组件（eg：gemini 插件封装了 gemini 加速方案）。目前支持的插件如下： -**_HybridParallelPlugin:_** HybirdParallelPlugin 插件封装了混合并行的加速解决方案。它提供的接口可以在张量并行，流水线并行以及两种数据并行方法（DDP, Zero）间进行任意的组合。 +**_HybridParallelPlugin:_** HybridParallelPlugin 插件封装了混合并行的加速解决方案。它提供的接口可以在张量并行，流水线并行以及两种数据并行方法（DDP, Zero）间进行任意的组合。 **_GeminiPlugin:_** GeminiPlugin 插件封装了 gemini 加速解决方案，即基于块内存管理的 ZeRO 优化方案。 diff --git a/examples/language/llama2/README.md b/examples/language/llama2/README.md index 83ef99b57..f29b9dcdd 100644 --- a/examples/language/llama2/README.md +++ b/examples/language/llama2/README.md @@ -103,7 +103,7 @@ Here is details about CLI arguments: - Max length: `-l`, `--max_length`. The default value is 4096. - Mixed precision: `-x`, `--mixed_precision`. The default value is "fp16". "fp16" and "bf16" are supported. - Save interval: `-i`, `--save_interval`. The interval (steps) of saving checkpoints. The default value is 1000. -- Checkpoint directory: `-o`, `--save_dir`. The directoty path to save checkpoints. The default value is `checkpoint`. +- Checkpoint directory: `-o`, `--save_dir`. The directory path to save checkpoints. The default value is `checkpoint`. - Checkpoint to load: `-f`, `--load`. The checkpoint path to load. The default value is `None`. - Gradient clipping: `--gradient_clipping`. The default value is 1.0. - Tensorboard log directory: `-t`, `--tensorboard_dir`. The directory path to save tensorboard logs. The default value is `tb_logs`. @@ -217,7 +217,7 @@ Here is details about CLI arguments: - Max length: `-l`, `--max_length`. The default value is 4096. - Mixed precision: `-x`, `--mixed_precision`. The default value is "fp16". "fp16" and "bf16" are supported. - Save interval: `-i`, `--save_interval`. The interval (steps) of saving checkpoints. The default value is 1000. -- Checkpoint directory: `-o`, `--save_dir`. The directoty path to save checkpoints. The default value is `checkpoint`. +- Checkpoint directory: `-o`, `--save_dir`. The directory path to save checkpoints. The default value is `checkpoint`. - Checkpoint to load: `-f`, `--load`. The checkpoint path to load. The default value is `None`. - Gradient clipping: `--gradient_clipping`. The default value is 1.0. - Tensorboard log directory: `-t`, `--tensorboard_dir`. The directory path to save tensorboard logs. The default value is `tb_logs`.