FATE-Serving

Introduction

FATE-Serving is a high-performance, industrialized serving system for federated learning models, designed for production environments.

FATE-Serving now supports

• High performance online Federated Learning algorithms.
• Federated Learning online inference pipeline.
• Dynamic loading federated learning models.
• Can serve multiple models, or multiple versions of the same model.
• Support A/B testing experimental models.
• Real-time inference using federated learning models.
• Support multi-level cache for remote party federated inference result.
• Support pre-processing, post-processing and data-access adapters for the production deployment.

Federated Learning Online Inference Pipeline

Architecture

Deploy

The code directory for FATE-Serving is FATE/fate-serving, which contains three parts:

• fate-serving-core: base api for FATE-Serving.
• federatedml: high performance online Federated Learning algorithms.
• serving-server: Federated Learning online inference service based on GRPC.

After the compilation is complete, the fate-serving-core and federatedml will be included in the lib, and the serving-server is included in the jar package.

FATE-Serving has two profiles, one is the log4j2.xml for log settings, another is serving-server.properties.

serving-server.properties

Key configuration item description:

Configuration item	Configuration item meaning	Configuration item value
ip	listen address for FATE-Serving	default 0.0.0.0
port	listen port for the grpc server of FATE-Serving	default 8000
workMode	the work mode of FATE-Flow	0 for standalone, 1 for cluster
inferenceWorkerThreadNum	inference worker num for async inference	default 10
standaloneStoragePath	the storage path of standalone EggRoll	generally is PYTHONPATH/data
remoteModelInferenceResultCacheSwitch	switch of remote model inference result cache storage	default true
proxy	the address of proxy	custom configuration
roll	the address of roll	custom configuration
OnlineDataAccessAdapter	data access adapter class for obtaining host feature data	default TestFile, read host feature data from host_data.csv on serving-server root directory
InferencePostProcessingAdapter	inference post-processing adapter class for dealing result after model inference	default pass
InferencePreProcessingAdapter	inference pre-processing adapter class for dealing guest feature data before model inference	default pass

Deploy Serving-Server

For detail, please refer to cluster deploy guide at cluster-deploy. Here are some key steps:

• Compile in the arch directory
• Compile in the fate-serving directory
• Create your serving directory by referring to the cluster-deploy/example-dir-tree/serving-server directory
• Copy fate-serving/serving-server/target/fate-serving-server-*.jar to serving-server directory
• Copy fate-serving/serving-server/target/lib to serving-server directory
• Copy fate-serving/serving-server/src/main/resources/* to serving-server/conf
• Modify the configuration file according to the actual situation
• Using the service.sh script to start/stop/restart

Usage

FATE-Serving provide publish model and online inference API.

Publish Model

Please use FATE-Flow Client which in the fate-flow to operate, refer to Online Inference guide at fate_flow_readme.

Inference

Serving currently supports three inference-related interfaces, using the grpc protocol.

• inference: Initiate an inference request and get the result
• startInferenceJob: Initiate an inference request task without getting results
• getInferenceResult: Get the result of the inference by caseid

python examples/inference_request.py ${sering_host}

please refer to this script for inference.

Adapter

Serving supports pre-processing, post-processing and data-access adapters for the actural production.

• pre-processing: Data pre processing before model calculation
• post-processing: Data post processing after model calculation
• data-access: get feature from party's system

At the current stage, you need to put the java code to recompile, and later support to dynamically load the jar in the form of a release.

For now:

• push your pre-processing and post-processing adapter code into fate-serving/serving-server/src/main/java/com/webank/ai/fate/serving/adapter/processing and modify the InferencePreProcessingAdapter/InferencePostProcessingAdapter configuration parameters.
• push your data-access adapter code into fate-serving/serving-server/src/main/java/com/webank/ai/fate/serving/adapter/dataaccess and modify the OnlineDataAccessAdapter configuration parameters.

please refer to PassPostProcessing, PassPreProcessing, TestFile adapter.

Remote party multi-level cache

For federal learning, one inference needs to be calculated by multiple parties. In the production environment, the parties are deployed in different IDCs, and the network communication between multiple parties is one of the bottleneck.

So, fate-serving supports caches multi-party model inference results on the initiator, but never caches feature data. you can turn the remoteModelInferenceResultCacheSwitch which in the configuration.