Convert to ONNX

[lilhoser]

I'm trying to convert the alpaca fine-tuned model to ONNX format using the code below, but the input is not correct. Can someone please give me a hand?

    checkpoint_location=r"C:\Downloads\alpaca-lora\ckpt\consolidated.00.pth"
    config_name="decapoda-research/llama-7b-hf"
    # Instantiate the appropriate model
    config = LlamaConfig.from_pretrained(config_name)
    model = LlamaForCausalLM(config)
    # Set to inference mode.
    model = model.eval()
    # Load weights
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    model = model.load_state_dict(torch.load(checkpoint_location, map_location=device))
    model.seqlen = 2048
    # Input to the model
    batch_size = 512
    x = torch.randn(batch_size, 1, 224, 224, requires_grad=True)
    torch_out = model(x)
    # Export the model
    torch.onnx.export(model,                   # model being run
                    x,                         # model input (or a tuple for multiple inputs)
                    output_location,           # where to save the model (can be a file or file-like object)
                    export_params=True,        # store the trained parameter weights inside the model file
                    opset_version=12,          # the ONNX version to export the model to
                    do_constant_folding=True,  # whether to execute constant folding for optimization
                    input_names = ['input'],   # the model's input names
                    output_names = ['output'], # the model's output names
                    dynamic_axes={'input' : {0 : 'batch_size'},    # variable length axes
                                    'output' : {0 : 'batch_size'}})
    # Load the ONNX model
    model = torch.onnx.load("llama-30b-4bit.onnx")
    # Check that the model is well formed
    torch.onnx.checker.check_model(model)
    # Print a human readable representation of the graph
    print(torch.onnx.helper.printable_graph(model.graph))```

[lilhoser]

For whoever cares: https://huggingface.co/nenkoru/alpaca-lora-7b-fp16-onnx

[dengzheng-cloud]

is anybody know about "task" flag and what "with_past" do in model convert? i am trying to convert output of optimum-cli into tensorrt engine, while it always stop when i set dynamic shape, 1x1input could run tensorrt convert.