Introduced cpu performance gaps disscussion

[ShvetsKS]

Two performance problems were discovered in recently merged PRs:

1. thread safety prediction: Make prediction thread safe. #6648

santander training stage	before #6648	thread safe #6648 and master	#6696 PR
full training	136s	165s	140s
PredictRaw	55s	78s	59s

Would it be possible to have not thread safe PredictDMatrix call to reduce overheads for subsampling cases and have initialization buffer only once on first training iteration? I'd appreciate for any ideas.

2. drop binary format: Drop binary format for memory snapshot. #6513 leads to stable 10% perf regression, observed for Higgs1m dataset after this commit (16s vs 14.3s full training).
   Before that we have an option to save old behavior via setting enable-experimental-json-serialization to False but currently there is no such possibility.
   Could you share your thoughts about best way to mitigate it?

[trivialfis]

I'm confused that why is thread-safe prediction has any impact on training? I thought we have enabled prediction cache for both regression and multi-class on CPU?

[trivialfis]

If the prediction happens during evaluation, we can use thread local static storage I believe.

As for the JSON thing, it's occurred during the end of training, which is used to release memory.

[ShvetsKS]

I'm confused that why is thread-safe prediction has any impact on training? I thought we have enabled prediction cache for both regression and multi-class on CPU?

Yes prediction caching is enabled for regression, binary and multiclass classification but for subsample==1 case only

[ShvetsKS]

As for the JSON thing, it's occurred during the end of training, which is used to release memory.

is it possible to have some alternative enable-experimental-json-serialization to reduce observed gaps?

[trivialfis]

Thanks for the reply. I think it's possible to change the subsampling implementation to allow cache.

As for the serialization time, I will try to find a way to remove it, either by BSON or by a better way to release memory.

[trivialfis]

The thread safety was mostly for dask interface, also some other feature requests. In dask, the prediction is done on each block of data in parallel.

[ShvetsKS]

The thread safety was mostly for dask interface, also some other feature requests. In dask, the prediction is done on each block of data in parallel.

so if i correctly understand we could initialize feat_vecs externally in training part function PredictRaw only once on 0-th iteration and propagate it right to PredictDMatrix

[trivialfis]

Closed by #7545 .