Remove xformers as it breaks injections module on mac

[jamie256]

Fixes #161 and also fills the gap in our CI that should have caught this:

• remove xformers from all target, its not strictly needed and can remain as an optional dependency.
• change the poetry dependency install cache key to include the string 'all'
• replace the ci workflow's use of poetry install to use the make install command which aliases poetry install -E all and runs on mac

If we had run the poetry install -E all to setup the dependencies then we could have caught failures in our CI relating to the optional dependencies.

[richard-rogers]

I think faster-fifo may be broken on Apple silicon Macs.