vllm: init at v0.3.1 with rocm support

[CertainLach]

Description of changes

Depends on:

• python311Packages.werkzeug: 2.3.7 -> 3.0.1; python311Packages.flask: 2.3.3 -> 3.0.1; python311Packages.quart: 0.18.4 -> 0.19.4 #258300 (VLLM uses newer versions of those libraries) - done in staging
• RFE: please provide update for werkzeug 3.x psf/httpbin#36 (As the needed for new werkzeug version is not released. VLLM doesn't depend directly on this, but httpbin is still in the dependency graph, and needs to be built.) - fix backported to staging
• ROCm: Allow setting compilation target vllm-project/vllm#2581 (Needed for proper sandbox support) - added patch
• rocmPackages.clr: backport bf16 compilation fix #286720
• python311Packages.openai-triton: remove ptxas --version call when built without cuda support #286763

I am not very familiar with python packaging in nixos (Though, anything is better than requirements.txt)

Couple of changes were made to fix build due to conflicting dependencies:

• It now depends on torch instead of torch-bin, because plain torch dependency is propagated somewhere, together with duplicate openai-triton package
• xformers and openai-triton were also broken for me due to usage of different torch

I think the only working way here is to use nixpkgs.config.torchSupport/nixpkgs.config.cudaSupport, as overriding torch dependency doesn't work very well in this case.

It also builds with CUDA for me, though I haven't properly tested it in runtime

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[mweinelt]

Flask/Werkzeug can't go into master and needs a lot more work on reverse dependencies.

[happysalada]

vllm as is still needed some work and wasn't working properly with cuda. If you've got it working for rocm, we could merge an early version with that only. (if you ever want to submit a PR for vllm, you don't need to credit me at all, you can just copy paste what you feel is useful and put me as a maintainer).
transformers is already at 4.37.1 on master.
there are some elements that could be merged relatively quickly, but I feel it would best be to break it down into smaller PRs that are easier to test and review.

[CertainLach]

Managed to get it working on Cuda with this PR:
#285495

[CertainLach]

Currently re-testing CUDA build after rebase on master, for ROCm it requires

• rocmPackages.clr: backport bf16 compilation fix #286720
• python311Packages.openai-triton: remove ptxas --version call when built without cuda support #286763

[CertainLach]

Looks like vllm setup.py calculates the intersection by itself now, this needs to be removed.

[CertainLach]

Tested with CUDA, it works.
ROCm testing (on top of patches mentioned earlier, + couple of fixups on top of them) in progress.

Edit: retested with cuda, i was testing on base branch without #285249, I have patched dependencies for torch 2.2.0.

[CertainLach]

Works on ROCm.

[happysalada]

@CertainLach thank you very much for this!