Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Tools] Accelerate model loading. #224

Merged
merged 1 commit into from
Feb 22, 2024
Merged

Conversation

marvin-Yu
Copy link
Contributor

qwen 7b loading time from 24s to 16s;
qwen 72b loading time from 517s to 400s;

@marvin-Yu marvin-Yu force-pushed the Accelerate_model_loading branch from e5ee14b to b9110fc Compare February 19, 2024 08:07
Copy link
Contributor

@pujiang2018 pujiang2018 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. need to be careful to add omp in float16.h, as the outer scope may already assume it is in simple thread, and contains omp at the outside scope.
  2. can we remove filesystem linkage? (as current binary is already big, we may only add new library when it is a MUST)

@Duyi-Wang
Copy link
Contributor

  1. need to be careful to add omp in float16.h, as the outer scope may already assume it is in simple thread, and contains omp at the outside scope.
  2. can we remove filesystem linkage? (as current binary is already big, we may only add new library when it is a MUST)

We have introduced filesystem in debugger.h

@pujiang2018
Copy link
Contributor

  1. need to be careful to add omp in float16.h, as the outer scope may already assume it is in simple thread, and contains omp at the outside scope.
  2. can we remove filesystem linkage? (as current binary is already big, we may only add new library when it is a MUST)

We have introduced filesystem in debugger.h

there is a difference that debugger will ONLY be effective when DEBUG is enabled.

@marvin-Yu marvin-Yu force-pushed the Accelerate_model_loading branch 2 times, most recently from ed1d662 to 5193171 Compare February 22, 2024 05:44
src/common/float16.h Outdated Show resolved Hide resolved
@marvin-Yu marvin-Yu force-pushed the Accelerate_model_loading branch from 5193171 to 956fc3c Compare February 22, 2024 09:18
@marvin-Yu marvin-Yu merged commit 5ea9bf8 into main Feb 22, 2024
1 check passed
@marvin-Yu marvin-Yu deleted the Accelerate_model_loading branch February 22, 2024 09:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants