MNT Remove version pin of diffusers (huggingface#2162)

The pin was added way back in huggingface#936 and then we forget to ever remove it.

This is now causing trouble, as the old diffusers version still uses
cached_download, which was removed from huggingface_hub:

> ImportError: cannot import name 'cached_download' from 'huggingface_hub'

setup.py

@@ -34,7 +34,7 @@
     "pytest-xdist",
     "parameterized",
     "datasets",
-    "diffusers<0.21.0",
+    "diffusers",
     "scipy",
 ]