Skip to content

Commit 5618083

Browse files
yaozheweiyaozhewei
authored
make file name as a hashing string to aviod long file names (deepspeedai#274)
* make file name as a hashing string to aviod long file names * fix format issue
1 parent e320e75 commit 5618083

1 file changed

Lines changed: 1 addition & 0 deletions

File tree

‎applications/DeepSpeed-Chat/training/utils/data/data_utils.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -252,6 +252,7 @@ def create_prompt_dataset(local_rank,
252252
tokenizer_name = tokenizer.init_kwargs['name_or_path'].replace('/', '_')
253253
fname = f"{fname}_split{data_split}_phase{train_phase}_seed{seed}_tokenizer{tokenizer_name}_seqlen{max_seq_len}"
254254
fname = '_'.join(fname.split('/'))
255+
fname = str(hash(fname)) # hash the file name to avoid too long file name
255256
train_fname = f"{output_path}/traindata_{fname}.pt"
256257
eval_fname = f"{output_path}/evaldata_{fname}.pt"
257258

0 commit comments

Comments
 (0)