* add variable for ca files

* rename variable

* divide openfl cert from ca cert folders

* Configure envoy health check period form director

* Period in seconds comment

* Rename listen_ip -> listen_host. Add params to default director.yaml

* Refactor health-check

Co-authored-by: igor-davidyuk <76463150+igor-davidyuk@users.noreply.github.com>

* Complete Tensorflow MNIST test

* Complete PyTorch UNet Kvasir test

* Add run script for tensorflow_mnist experiment

* Add run script for pytorch_kvasir_unet experiment

* Update requirements for pytorch_kvasir_unet experiment

* Add -r key

* Add scikit-image to shard descriptor requirements

* Convert enforce_image_hw to int

* Update envoy bash script

* Pass director port as number

* Move imports to file header

* Remove unused kwargs parameter

* Add train_bs and valid_bs properties

* Split rank_worldsize variable

* Update file extension filtering

* Split rank_worldsize field

* Latest develop changes

* Remove backslash

* Fix argument order

* Remove unused kwargs

* Remove certificate-related parameters in non-TLS environment

* Add new log level - METRIC. Write metrics on aggregator to tensorboard

* Refactor code. Add tensorboardX to setup.py

* fix flake8

* Return back global logger in native.py

* Delete raising exceptions in addLoggingLevel function

* Modify log level checking

* Apply suggestions from code review

Rename tensorboard option

Co-authored-by: Ilya Trushkin <76161256+itrushkin@users.noreply.github.com>

* Remove log level checking

* Pass write_metric function through plan

* Pass write metric function also in workspace

If we dont define this callback in notebook it will be taken from
workspace, if exists

* Add docs for log metric callback

* Clear notebook output

* Add cfssl_scheme.md

* Add diagram as png

* Add description of CA to clients tls connestion

* Fix cfssl scheme

* Set 'metric' tag to train task result in interactive api

It's necessary for logger to parse this tag

* Log metric task result on aggregator additionally.

Collaborator logs metric task result, but sometimes collaborator
and aggregator have different consoles (in interactive api e.g.)

* fix flake8

* Remove default tensorboard option in defaults plan.yaml

* Remove tensorboardX dependency

* Fix some pep8 issues. Remove debugging grpc code.

* VOC shard descriptor

* Remove debugging grpc code.

* First draft working pki with step-ca

* Initial test experiments for new python API

* Director - envoy initial commit

* Base shard descriptor

* Pass shard descriptor to get_collaborator and addition API

* Reorganize project structure for director and collaborator manager

* Move ca commands to separate file

* first LL adaption approach

* fix flake8

* First working example

* Prepare an aggregator workspace and pass initial tensors

* fix flake8

* Adapted launching aggregator on director side

* requirements fix

* Updated the Kvasir notebook

* Add tensorboard to torch_cnn_mnist requirements

* Fix typo

* Initial test experiments for new python API

* Director - envoy initial commit

* Base shard descriptor

* Pass shard descriptor to get_collaborator and addition API

* Reorganize project structure for director and collaborator manager

* VOC shard descriptor

* first LL adaption approach

* Prepare an aggregator workspace and pass initial tensors

* First working example

* CLI commands

* Add tensorboard

* Flake8 and logging

* Flake8

* Setup logging for experiment

* Setup logging for experiment

* Change requirements for grpcio

* Change requirements for grpcio

* Change requirements for grpcio

* Remove unused kwargs

* Resolve aggregator serving

* Resolve aggregator serving

* flake8

* Working notebook

* Create Updated_Kvasir_with_Director.ipynb

* lost fixes after merge

* fixed notebook experimnet start

* Infrastructure for model retrieving

* fixes to the director example

* added requirements file for envoy

* fixed metric logging

* pkg-resources bug fix

* pkg-resources fix

* pip fix

* first try to run and stop aggregator async

* added workspace cleaning in director and envoy

* removing workspace archive on Frontend API
in case of set_new_experiment fail

* Update pki scheme to step-ca flow

* Draft ca integration

* Add ca to utils

* Move ca funtions to separate component

* Add certificates for api layer director and collaborator-manager

* Pass disable_tls option to all grpc clients and servers

* New CLI and fixes for the step-ca component

* flake8 fixes

* Pass certificates name to collaborator and aggregator

* Fix rank_worldsize value

* Fix flake8

* flake8

* Separate install and run command for pki

* Stream metrics to frontend RPC

* UnpackWorkspace context manager

* local changes

* Add step-ca pki mermaid scheme to documentation.

* flake8

* Resolve merge conflicts

* Backward compatibility

* Fix cert api

* make cert paths absolute

* Fix simultaneous requirements installation

* Change default and max token and certificate durations

* Import order

* LogNormal split implementation

* Create DataSplitter interface

* Avoid storing on-disk dataset for lognormal split

* Add random splitter for PyTorch datasets

* Fix linter

* Add dirichlet splits

* Fix linter

* First implementation of API registry

* Added metric logging on frontend

* Tensorboard logs: removed hardcoded name

* Add description of pki workflow to documentation

* Health check and flake8

* Fix pb2 import

* Add info about token live time and other.

* Implement split of shard descriptor

* Fix flake8

* Use numpy arrays in shard descriptor

* Use indices instead of data slices

* Merge fix

* Fix data_splitters package path

* Fix linter

* Update Kvasir shard descriptor

* Fix linter

* Clear notebook

* Add separate splitter for validation data

* Remove PyTorch splitters

* Remove PyTorch dataset from KvasirShardDescriptor

* Change numpy splitter signature

* Fix linter

* Fix Python native tests

* Move Data Splitter functionality to separate shard_descriptor file

* Restore interface signature

* Extract Data Splitter processing in federated_data.py

* Change hash validation in Kvasir Shard Descriptor

* Add docs

* Fix linter

* Fix native tests

Co-authored-by: igor-davidyuk <76463150+igor-davidyuk@users.noreply.github.com>
Co-authored-by: Radionov, Maksim <maksim.radionov@intel.com>
Co-authored-by: maradionov <77286045+maradionov@users.noreply.github.com>
Co-authored-by: Igor Davidyuk <igor.davidyuk@intel.com>
Co-authored-by: amokrov <aleksandr.mokrov@intel.com>
Co-authored-by: aleksandr-mokrov <76171391+aleksandr-mokrov@users.noreply.github.com>

* Use epochs instead  batch_num. Log epoch count

* Fix flake8

* Resolve conversations

* Fix linter

Co-authored-by: Ilya Trushkin <ilya.trushkin@intel.com>

Co-authored-by: igor-davidyuk <76463150+igor-davidyuk@users.noreply.github.com>

…into develop