Skip to content
This repository has been archived by the owner on Oct 25, 2024. It is now read-only.

Enable text-generation with new API #318

Merged
merged 21 commits into from
Sep 18, 2023
Merged

Enable text-generation with new API #318

merged 21 commits into from
Sep 18, 2023

Conversation

changwangss
Copy link
Contributor

@changwangss changwangss commented Sep 14, 2023

Type of Change

    from intel_extension_for_transformers.transformers import (
        MixedPrecisionConfig,
        WeightOnlyQuantConfig,
        SmoothQuantConfig,
        BitsAndBytesConfig

    ) 
    from intel_extension_for_transformers.transformers import AutoModelForCausalLM
    # smooth-quant
    sq_config = SmoothQuantConfig(
                                tokenizer=tokenizer,  # either two of one, tokenizer or calib_func
                               )
    q_model = AutoModelForCausalLM.from_pretrained(model_name_or_path,
                                                   quantization_config=sq_config
                                               )
    
    # weight-only
    woq_config = WeightOnlyQuantConfig()
    woq_model = AutoModelForCausalLM.from_pretrained(model_name_or_path,
                                                quantization_config=woq_config
                                            )
    
    # mp
    mp_config = MixedPrecisionConfig() 
    amp_model = AutoModelForCausalLM.from_pretrained(model_name_or_path,
                                                quantization_config=mp_config
                                            )
  
    # bitsandbytes
    bab_config = BitsAndBytesConfig()
    bab_model = AutoModelForCausalLM.from_pretrained(model_name_or_path,
                                                quantization_config=bab_config
                                            )

Description

detail description
JIRA ticket: https://jira.devtools.intel.com/browse/NLPTOOLKIU-878

Expected Behavior & Potential Risk

the expected behavior that triggered by this PR

How has this PR been tested?

how to reproduce the test (including hardware information)

Dependency Change?

any library dependency introduced or removed

@changwangss changwangss changed the title Enable text-generation with NeuralChat API Enable text-generation with new API Sep 15, 2023
@changwangss
Copy link
Contributor Author

confilct with #297, need align with @PenghuiCheng

@hshen14
Copy link
Contributor

hshen14 commented Sep 15, 2023

confilct with #297, need align with @PenghuiCheng

what's the conflict?

Signed-off-by: changwangss <chang1.wang@intel.com>
Signed-off-by: changwangss <chang1.wang@intel.com>
Signed-off-by: changwangss <chang1.wang@intel.com>
Signed-off-by: changwangss <chang1.wang@intel.com>
Signed-off-by: changwangss <chang1.wang@intel.com>
changwangss and others added 7 commits September 17, 2023 23:11
Signed-off-by: changwangss <chang1.wang@intel.com>
Signed-off-by: changwangss <chang1.wang@intel.com>
Signed-off-by: changwangss <chang1.wang@intel.com>
Signed-off-by: changwangss <chang1.wang@intel.com>
Signed-off-by: Cheng, Penghui <penghui.cheng@intel.com>
Signed-off-by: Cheng, Penghui <penghui.cheng@intel.com>
@changwangss changwangss removed the draft label Sep 18, 2023
Signed-off-by: changwangss <chang1.wang@intel.com>
Signed-off-by: changwangss <chang1.wang@intel.com>
@hshen14 hshen14 merged commit f4dc780 into main Sep 18, 2023
@hshen14 hshen14 deleted the wangchang/neuralchat branch September 18, 2023 23:05
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants