Adds caching layer to tensor allocations

[coreylowman]

Instead of memory being immediately freed upon drop, this instead holds it in a device specific cache object. Then when another allocation is requested of the same size, it can re-use the allocated memory.

Related to #541

Summary

Changes the Cuda::Vec and Cpu::Vec to be wrapper objects around both data and a cache object:

#[derive(Debug)]
pub struct CachableVec<E> {
    pub(crate) data: Vec<E>,
    pub(crate) destination: Arc<RwLock<BTreeMap<usize, Vec<BytesPtr>>>>,
}

Where the destination field is the cache that the object will be placed into on Drop.

There are a number of special implentations/usages that are required for this to work:

1. When a new tensor allocation is requested, it can be pulled from the device specific cache object
2. impl<E> Clone for CachableVec<E> should attempt to pull the new memory from the cache
3. impl<E> Drop for CachableVec<E> should insert the memory back into the cache object

Results

Cpu

cargo +nightly bench --bench conv2d -F cpu-mkl-matmul

branch	fwd	bwd
main	320ms	990ms
this	280ms	900ms

cargo bench --bench batchnorm2d

branch	fwd	bwd
main	270ms	401ms
this	263ms	312ms

Cuda

cargo +nightly bench --bench conv2d -F cuda

branch	infer	fwd	bwd
main	4.8ms	5ms	20ms
this	3.7ms	4.2ms	15ms

cargo bench --bench batchnorm2d -F cuda

branch	infer	fwd	bwd
main	3.8ms	8.8ms	25ms
this	1.8ms	4.4ms	9.3ms