Skip to content
This repository has been archived by the owner on Oct 25, 2024. It is now read-only.

Commit

Permalink
fixed off by one error on masking (#1193)
Browse files Browse the repository at this point in the history
Signed-off-by: Dillon Laird <dillonalaird@gmail.com>
  • Loading branch information
dillonalaird authored Jan 26, 2024
1 parent 09ddade commit 525076d
Showing 1 changed file with 2 additions and 2 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -335,10 +335,10 @@ def preprocess_v1(

if has_image:
round_len = len(tokenizer_image_token(rou, tokenizer))
instruction_len = len(tokenizer_image_token(parts[0], tokenizer)) - 1
instruction_len = len(tokenizer_image_token(parts[0], tokenizer)) - 2
else:
round_len = len(tokenizer(rou).input_ids)
instruction_len = len(tokenizer(parts[0]).input_ids) - 1
instruction_len = len(tokenizer(parts[0]).input_ids) - 2

target[cur_len : cur_len + instruction_len] = IGNORE_INDEX

Expand Down

0 comments on commit 525076d

Please sign in to comment.