FYI: LLaMA with GPT-Q is not deterministic.

[nullgate]

The cuda kernel for GPTQ is not part of pytorch, so its algorithm is not deterministic (e.g.: accumulate float numbers in consistent order).
I tried everything, but it always diverged after sampling a long sequence.

My code (set CUBLAS_WORKSPACE_CONFIG=:4096:8):

        torch.use_deterministic_algorithms(True)
        torch.manual_seed(seed)
        if torch.cuda.is_available():
            torch.cuda.manual_seed_all(seed)

https://pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html#torch.use_deterministic_algorithms
https://docs.nvidia.com/cuda/cublas/index.html#cublasApi_reproducibility