mirror of
https://github.com/hpcaitech/ColossalAI.git
synced 2025-06-24 06:29:09 +00:00
* Add dpo. Fix sft, ppo, lora. Refactor all * fix and tested ppo * 2 nd round refactor * add ci tests * fix ci * fix ci * fix readme, style * fix readme style * fix style, fix benchmark * reproduce benchmark result, remove useless files * rename to ColossalChat * use new image * fix ci workflow * fix ci * use local model/tokenizer for ci tests * fix ci * fix ci * fix ci * fix ci timeout * fix rm progress bar. fix ci timeout * fix ci * fix ci typo * remove 3d plugin from ci temporary * test environment * cannot save optimizer * support chat template * fix readme * fix path * test ci locally * restore build_or_pr * fix ci data path * fix benchmark * fix ci, move ci tests to 3080, disable fast tokenizer * move ci to 85 * support flash attention 2 * add all-in-one data preparation script. Fix colossal-llama2-chat chat template * add hardware requirements * move ci test data * fix save_model, add unwrap * fix missing bos * fix missing bos; support grad accumulation with gemini * fix ci * fix ci * fix ci * fix llama2 chat template config * debug sft * debug sft * fix colossalai version requirement * fix ci * add sanity check to prevent NaN loss * fix requirements * add dummy data generation script * add dummy data generation script * add dummy data generation script * add dummy data generation script * update readme * update readme * update readme and ignore * fix logger bug * support parallel_output * modify data preparation logic * fix tokenization * update lr * fix inference * run pre-commit --------- Co-authored-by: Tong Li <tong.li352711588@gmail.com>
70 lines
1.9 KiB
Python
Executable File
70 lines
1.9 KiB
Python
Executable File
"""
|
|
A class that can be used to calculate the mean of a variable
|
|
"""
|
|
|
|
|
|
class AccumulativeMeanVariable:
|
|
"""
|
|
A class that calculates the accumulative mean of a variable.
|
|
"""
|
|
|
|
def __init__(self):
|
|
self._sum = 0
|
|
self._count = 0
|
|
|
|
def add(self, value, count_update=1):
|
|
"""
|
|
Adds a value to the sum and updates the count.
|
|
|
|
Args:
|
|
value (float): The value to be added.
|
|
count_update (int, optional): The amount to update the count by. Defaults to 1.
|
|
"""
|
|
self._sum += value
|
|
self._count += count_update
|
|
|
|
def get(self):
|
|
"""
|
|
Calculates and returns the accumulative mean.
|
|
|
|
Returns:
|
|
float: The accumulative mean.
|
|
"""
|
|
return self._sum / self._count if self._count > 0 else 0
|
|
|
|
def reset(self):
|
|
"""
|
|
Resets the sum and count to zero.
|
|
"""
|
|
self._sum = 0
|
|
self._count = 0
|
|
|
|
|
|
class AccumulativeMeanMeter:
|
|
"""
|
|
A class for calculating and storing the accumulative mean of variables.
|
|
|
|
Attributes:
|
|
variable_dict (dict): A dictionary to store the accumulative mean variables.
|
|
|
|
Methods:
|
|
add(name, value, count_update=1): Adds a value to the specified variable.
|
|
get(name): Retrieves the accumulative mean value of the specified variable.
|
|
reset(): Resets all the accumulative mean variables to their initial state.
|
|
"""
|
|
|
|
def __init__(self):
|
|
self.variable_dict = {}
|
|
|
|
def add(self, name, value, count_update=1):
|
|
if name not in self.variable_dict:
|
|
self.variable_dict[name] = AccumulativeMeanVariable()
|
|
self.variable_dict[name].add(value, count_update=count_update)
|
|
|
|
def get(self, name):
|
|
return self.variable_dict[name].get()
|
|
|
|
def reset(self):
|
|
for name in self.variable_dict:
|
|
self.variable_dict[name].reset()
|