[Core] avoid too many cuda context by caching p2p test (#4021)

[Core] avoid too many cuda context by caching p2p test (#4021) #3