deepspeed and text-generation-webui

[devzzzero]

Hi, I managed to alltalk_tts installed on a fairly recent oobabooga. The base (without deepspeed) ran fine.
But when after I installed deepspeed, ooba started to barf

, restarting ooba caused this error

         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ai/LLM/UI/text-generation-webui/installer_files/env/lib/python3.11/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/home/ai/LLM/UI/text-generation-webui/installer_files/env/lib/python3.11/subprocess.py", line 1955, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/ai/LLM/UI/text-generation-webui/installer_files/env/bin/nvcc'

(and in my machine nvcc and friends are all in /usr/local/cuda)

I then changed the following lines in ooba/start_linux.sh

# export CUDA_PATH="$INSTALL_ENV_DIR"
# export CUDA_HOME="$CUDA_PATH"
export CUDA_PATH="/usr/local/cuda"
export CUDA_HOME="/usr/local/cuda"

and then that error went away..

When ooba is starting up, I saw this

[AllTalk Startup] AllTalk startup Mode   : Text-Gen-webui mode
[AllTalk Startup] WAV file deletion      : Disabled
[AllTalk Startup] DeepSpeed version      : 0.14.2 
[AllTalk Startup] Model is available     : Checking
[AllTalk Startup] Model is available     : Checked

but the deepspeed button does not seem to show up on ooba's chat tab settings for AllTalk.
I then went do the alltalk docs page, clicked deepspeed to True,

and then I did the demo/Test TTS

then I got this on the ooba console
[AllTalk TTSGen] 0.92 seconds. LowVRAM: False DeepSpeed: False

I attached the diagnostics.log from alltalk_tts
diagnostics.log

Thank you.

[erew123]

Hi @devzzzero

So to confirm you went through this setup when you installed deepspeed? https://github.com/erew123/alltalk_tts#-deepspeed-installation-options

DeepSpeed specifically needs access to the CUDA toolkit when it first compiles, otherwise it has issues. hence typically you will ONLY need to set the export paths as per step 7 in the above instructions, at time of doing the pip install deepspeed

The deepspeed installation on Linux looks for nvcc (Nvidia's cuda compiler driver, which helps other processes figure out if a CUDA toolkit exists and what version it is) and THEN it will compile deepspeed, hence your original error, as TGWUI sets it own paths for its own version of PyTorch CUDA (which it typically needs when you start TGWUI, hence we only set them temporarily when compiling deepspeed).

As AllTalk detected DeepSpeed, it should give you the checkbox, BUT, it may not do if DeepSpeed hasn't compiled correctly.

I would suggest you change the paths back to this:

export CUDA_PATH="$INSTALL_ENV_DIR"
export CUDA_HOME="$CUDA_PATH"

Then close your terminal window and try starting TGWUI again.

If you want to restart the DeepSpeed installation process, you can run /cmd_linux.sh in your TGWUI folder (to start the Python environment) and then you can pip uninstall deepspeed

If you still have a problem after that, please let me know. Also, Im not sure what flavour of linux you are on, so that could be handy to know.

Thanks

[devzzzero]

When I reset the ooba/start_inux.sh to have the original CUDA_PATH and CUDA_HOME

I get this error:

[AllTalk Startup]     _    _ _ _____     _ _       _____ _____ ____  
[AllTalk Startup]    / \  | | |_   _|_ _| | | __  |_   _|_   _/ ___| 
[AllTalk Startup]   / _ \ | | | | |/ _` | | |/ /    | |   | | \___ \ 
[AllTalk Startup]  / ___ \| | | | | (_| | |   <     | |   | |  ___) |
[AllTalk Startup] /_/   \_\_|_| |_|\__,_|_|_|\_\    |_|   |_| |____/ 
[AllTalk Startup]
[AllTalk Startup] Config file check      : No Updates required
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
Traceback (most recent call last):
  File "/home/ai/LLM/UI/text-generation-webui/modules/extensions.py", line 37, in load_extensions
CUT-----------------------------------CUT
FileNotFoundError: [Errno 2] No such file or directory: '<PATH>/text-generation-webui/installer_files/env/bin/nvcc'`

Its trying to use nvcc in the ooba env directory.
ooba never installed nvcc there! Neither had atsetup.sh, but it's trying to use the ooba/installer_files/bin to access nvcc instead of the CUDA_HOME that I set when I ran atsetup.sh

ooba DOES start up, but is glitched (i.e. alltalk_tts config is missing from the chat tab)

During the install process of deepspeed (I used atsetup.sh to do it) I did not get any errors.
(and yes, I made sure the use ooba/installer_files/env as the active conda env when I ran atsetup.sh

I'll try a fresh install of alltaltk_tts next....

[devzzzero]

So after a fresh reinstall of alltalk_tts, deepspeed is active (and seems to work), but ooba/installer_files/env/bin still does not have nvcc
and without that gross hack on ooba's start_linux.sh script, alltalk_tts does not start up properly.

Why is alltalk_tts insisting on using ooba/installer_files/env/bin as the prefix to nvcc? Should it not being using the CUDA_HOME variable that was used when it was installed?

I'm going to see if a standalone install works differently. and digging into the code next.

[erew123]

Text-gen-webui along with PyTorch w/CUDA will create the CUDA_HOME and CUDA_PATH at time of installation, to point to the custom Text-gen-webui Python installation (stored in installer_files).

This is done because all the LLM model loaders etc rely on lots of aspects of CUDA and they also need to access the specific versions of files that match the Text-gen-webui PyTorch CUDA version, selected when a user installs Text-gen-webui. Hence, changing the CUDA_HOME and CUDA_PATH that Text-gen-webui & the LLM loaders have access to may well impact various things.

Nvcc is not something that can be automatically installed into a Python environment, however, Nvidia have made an installer that acts as a placeholder to say its not installed https://pypi.org/project/nvidia-cuda-nvcc/

Coming back to DeepSpeed, when Microsoft have written the compilation routine for DeepSpeed, it has to be compiled/built for the specific Linux variant you are on and also customised to the specific Python environment you are in (its features and various other bits).

For it to compile/build, it DeepSpeed checks if nvcc exists as various parts of the compilation script get version numbers from nvcc and also access various features of the Nvidia CUDA Development Toolkit available within PyTorch's CUDA build.

So the way it finds nvcc and the CUDA Development toolkit is to look at the environment variables of CUDA_HOME and CUDA_PATH.

As such, when you start Text-gen-webui, it sets CUDA_HOME and CUDA_PATH to point to the custom path in its installer_files folder, however, to install DeepSpeed, we have to compile it with the Nvidia CUDA Development Toolkit which means we temporarily want to hijack CUDA_HOME and CUDA_PATH in our custom Text-gen-webui Python environment to point to our installed Nvidia CUDA Development Toolkit (nvcc etc), let it compile/build and install, then have our CUDA_HOME and CUDA_PATH set back to the Text-gen-webui installer_files path, for our LLM model loaders to make use of the correct versions of files that match the PyTorch versions.

Needless to say, its annoying, but there's no great way around it that I know of for Linux.

Hope that details what you need.

Thanks

[devzzzero]

So, I think what happened in my case was that I had a partially broken conda env
I basically deleted everything ooba/installer_files and followed the MANUAL installation instructions for ooba.
i.e. I created a brand new miniconda env, activated it, and installed everything for ooba (installed cuda, not cuda-runtime!) according to ooba's MANUAL install instructions into this new miniconda env.
Then I followed the MANUAL install instructions for alltalk_tts (including TEMPORARILY setting CUDA_HOME, PATH, LD_LIBRARY_PATH)
Now everything works! no ugly hack on ooba/start_linux.sh needed.
All done! Thank you for your help!