[feature] Support Edge Deployment

[gaocegege]

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug
/kind feature

What happened:

We are glad to investigate if we can support deploy the model on edge servers/devices.

The workflow will be like: Users downloads the binary from the model registry and use it to server at edge.

Ref https://aws.amazon.com/cn/blogs/aws/amazon-sagemaker-neo-train-your-machine-learning-models-once-run-them-anywhere/

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

[hicaistar]

A brief flow of Amazon SageMaker Neo:

1. Compile a model to an executable binary that can be run by Neo runtime(a python package)
2. Application run on eade device loads Neo runtime
3. Neo runtime loads model and starts serving

Neo runtime is the core. It's very small. Is that compiling serving program together with a model to a binary that can run on different platform be feasible?

[gaocegege]

Is that compiling serving program together with a model to a binary that can run on the different platforms be feasible?

I do not think we should provide one unified binary, I think we will provide one binary for one arch. WDYT

[hicaistar]

Is that compiling serving program together with a model to a binary that can run on the different platforms be feasible?

I do not think we should provide one unified binary, I think we will provide one binary for one arch. WDYT

LGTM

[gaocegege]

It may be related to kleveross/ormb#47

[gaocegege]

/assign @judgeeeeee

[xieydd]

For different platform edge device, there are two keypoints:

• speed is the key point.
• Supported platform , like ios、Android or NVIDIA edge device
  There is no doubt, TVM is a best choice, but TVM is too complicated for users and developer .We have deploy many application on edge device, the pipeline is :

pytorch/tensorflow -> onnx(or direct) -> mnn/ncnn/tnn (We will compare with two framework using internal tools, but an opensource like this https://github.com/AI-performance/embedded-ai.bench)
# the framework model is easy convert to binary.

Looking forward to klever edge deployment Design.