Coqui Telegram Bot #173

zaptrem · 2023-05-30T18:31:27Z

Prompt-To-Voice Telegram Bot

A demonstration of how Coqui TTS can be integrated with Vocode and telegram to create engaging voice applications.

The bot (based on albertwujj's work) uses the python-telegram-bot library to handle user messages and commands, the WhisperTranscriber class to transcribe voice messages from users, and the ChatGPTAgent class to generate text responses based on a system prompt and the user input. The system prompt is even customized based on the voice name and description of the current voice. The bot also allows the user to select or create different voices using Coqui TTS's voice creation APIs.

The bot supports the following commands for the user to interact with it:

/start: Initializes the user data and sends a welcome message.
/voice <voice_id>: Changes the current voice to the one with the given id and resets the conversation. The voice id must be an integer corresponding to one of the available voices.
/create <voice_description>: Creates a new Coqui TTS voice from a text prompt and switches to it. The voice description must be a string that describes how the voice should sound like.
/list: Lists all the available voices with their ids, names, and descriptions (if any).
/who: Shows the name and description (if any) of the current voice.
/help: Shows a help message with all the available commands.

TODO:

[🚧] Add voice cloning from audio clip feature using Coqui TTS's clone endpoint.
Make the bot work on Replit including use of ReplitDB for conversation/voice persistence between instances.
Get feedback on code cleanliness and readability.
Test the changes to CoquiSynthesizer more thoroughly and handle possible errors or edge cases.
Evaluate if the InMemoryDB wrapper is the best way to handle non-existent users or if there is a better alternative.
Implement Coqui improvements/fix in streaming synthesizer or create separate issue.
Investigate issue on Coqui's side relating to dropped sentences and/or switch to their new (but slower) model.

…provements

ajar98 · 2023-05-30T19:11:30Z

apps/telegram_bot/README.md

@@ -0,0 +1,36 @@
+# client_backend


note for later - and let's add this to linear, this will need to be in docs/

ajar98 · 2023-05-30T19:14:42Z

vocode/turn_based/synthesizer/coqui_synthesizer.py

+                    # Return an AudioSegment object from the audio data
+                    return AudioSegment.from_wav(io.BytesIO(audio_data))  # type: ignore
+
+    def get_request(self, text: str) -> tuple[str, dict[str, str], dict[str, str]]:


i'd guess this line is what is breaking mypy, you'll need to use typing.Tuple instead of tuple and typing.Dict instead of dict

for compatibility with python 3.8

fixed in other PR

ajar98 · 2023-05-30T19:15:25Z

vocode/turn_based/synthesizer/coqui_synthesizer.py

+                continue
+            # Concatenate the current chunk and the sentence, and add a period to the end
+            proposed_chunk = current_chunk + sentence
+            if len(proposed_chunk) > 250:


nit: magic number

fixed in other PR

ajar98 · 2023-05-30T19:17:33Z

vocode/turn_based/synthesizer/coqui_synthesizer.py

+            proposed_chunk = current_chunk + sentence
+            if len(proposed_chunk) > 250:
+                chunks.append(current_chunk.strip())
+                current_chunk = sentence + "."


as kian said before, this will need to preserve the correct sentence ending that it was split on

fixed in other PR

ajar98 · 2023-05-30T19:21:45Z

vocode/turn_based/synthesizer/coqui_synthesizer.py

+        # Create an aiohttp session and post the request asynchronously using await
+        async with aiohttp.ClientSession() as session:
+            async with session.post(url, headers=headers, json=body) as response:
+                assert response.status == 201, (


ok for now, but this sort of assert that is "expected" (not something that should never happen) - should probably be some other error class

added linear followup

ajar98 · 2023-05-30T19:52:06Z

apps/telegram_bot/main.py

+    ) -> None:
+        assert update.effective_chat, "Chat must be defined!"
+        chat_id = update.effective_chat.id
+        if type(self.synthesizer) is not CoquiSynthesizer:


nit isinstance instead of CoquiSynthesizer - in case in the future we subclass CoquiSynthesizer, for example

ajar98 · 2023-05-30T19:52:36Z

apps/telegram_bot/main.py

+        if type(self.synthesizer) is not CoquiSynthesizer:
+            await context.bot.send_message(
+                chat_id=chat_id,
+                text="Sorry, voice creation is only supported for Coqui TTS.",


Suggested change

text="Sorry, voice creation is only supported for Coqui TTS.",

text="Sorry, voice creation is only supported for Coqui.",

since we have a "Coqui TTS" synthesizer which is their OSS thing

don't see the change

ajar98 · 2023-05-30T19:55:03Z

apps/telegram_bot/main.py

+    def get_agent(self, chat_id: int) -> ChatGPTAgent:
+        # Get current voice name and description from DB
+        _, voice_name, voice_description = self.db[chat_id].get(
+            "current_voice", (None, None, None)


nit: we can turn this tuple into a pydantic class:

class Voice(pydantic.BaseModel): voice_id...

ajar98 · 2023-05-30T19:55:53Z

apps/telegram_bot/main.py

+        chat_id = update.effective_chat.id
+        user_voices = self.db[chat_id]["voices"]  # array (id, name, description)
+        # Make string table of id, name, description
+        voices = "\n".join(


nit

Suggested change

voices = "\n".join(

voices_formatted = "\n".join(

ajar98 · 2023-05-30T19:56:16Z

apps/telegram_bot/main.py

+- Use /help to see this help message again.
+"""
+        assert update.effective_chat, "Chat must be defined!"
+        if type(self.synthesizer) is CoquiSynthesizer:


nit isinstance

…egram-bot

ajar98

looks great! nice work

ajar98 · 2023-06-13T06:17:30Z

apps/telegram_bot/main.py

    ) -> None:
        self.transcriber = transcriber
        self.system_prompt = system_prompt
        self.synthesizer = synthesizer
-        self.db = ChatsDB(db if db else {})
+        self.db: Dict[int, Chat] = defaultdict(Chat)


nice, very clean

ajar98 · 2023-06-13T06:17:37Z

apps/telegram_bot/main.py

-        # Initialize an empty dictionary to store user data
-        self.db = db
+# Define a Voice model with id, name and description fields
+class Voice(BaseModel):


ajar98 · 2023-06-13T06:18:23Z

apps/telegram_bot/main.py

+        if type(self.synthesizer) is not CoquiSynthesizer:
+            await context.bot.send_message(
+                chat_id=chat_id,
+                text="Sorry, voice creation is only supported for Coqui TTS.",


don't see the change

…m-bot

zaptrem · 2023-06-15T14:35:04Z

@ajar98 Fixed the branding issue and merged again. Good to go from my end.

* Add async synthesize, xtts, and prompt to coqui TB * add speechrecognition and aiohttp dependencies * add optional memory arg to turn-based ChatGPTAgent * add coqui telegram bot * fix mypy issue * pr feedback * Rename defaultdict * fix py3.8 typing issue * another py3.8 fix * [broken] pydantic progress * use pydantic and defaultdict * more nit fixes * Fix Coqui branding * fix type error

zaptrem added 5 commits May 30, 2023 10:15

Add async synthesize, xtts, and prompt to coqui TB

259d03a

Merge remote-tracking branch 'origin/main' into zaptrem/turn-based-im…

5dfd487

…provements

add speechrecognition and aiohttp dependencies

cdc3661

add optional memory arg to turn-based ChatGPTAgent

bf67bae

add coqui telegram bot

751f4d7

zaptrem mentioned this pull request May 30, 2023

Telegram Bot and Coqui Improvments #144

Closed

7 tasks

fix mypy issue

7ac0c1e

zaptrem changed the title ~~Zaptrem/coqui telegram bot~~ Coqui Telegram Bot May 30, 2023

Kian1354 assigned ajar98 May 30, 2023

ajar98 requested changes May 30, 2023

View reviewed changes

zaptrem added 10 commits May 30, 2023 15:17

pr feedback

84e2d74

Rename defaultdict

8838128

Merge branch 'main' into zaptrem/turn-based-improvements

d3443bd

fix py3.8 typing issue

60b3ed8

another py3.8 fix

b149615

Merge branch 'zaptrem/turn-based-improvements' into zaptrem/coqui-tel…

46c9800

…egram-bot

[broken] pydantic progress

417b57b

use pydantic and defaultdict

65c865e

more nit fixes

7c0fa4d

Reset poetry lock to merge main into zaptrem/coqui-telegram-bot

061e32a

zaptrem requested a review from ajar98 June 11, 2023 15:38

ajar98 approved these changes Jun 13, 2023

View reviewed changes

ajar98 assigned zaptrem and unassigned ajar98 Jun 13, 2023

zaptrem added 2 commits June 15, 2023 16:31

Fix Coqui branding

6338afb

Merge remote-tracking branch 'origin/main' into zaptrem/coqui-telegra…

3390f14

…m-bot

fix type error

c2eb4ff

zaptrem merged commit 258876b into main Jun 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Coqui Telegram Bot #173

Coqui Telegram Bot #173

zaptrem commented May 30, 2023 •

edited

Loading

ajar98 May 30, 2023

ajar98 May 30, 2023

ajar98 May 30, 2023

zaptrem May 30, 2023

ajar98 May 30, 2023

zaptrem May 30, 2023

ajar98 May 30, 2023

zaptrem May 30, 2023

ajar98 May 30, 2023

ajar98 May 30, 2023

ajar98 May 30, 2023

zaptrem Jun 11, 2023

ajar98 May 30, 2023

zaptrem Jun 11, 2023

ajar98 Jun 13, 2023

ajar98 May 30, 2023

zaptrem Jun 11, 2023

ajar98 May 30, 2023

zaptrem Jun 11, 2023

ajar98 May 30, 2023

zaptrem Jun 11, 2023

ajar98 left a comment

ajar98 Jun 13, 2023

ajar98 Jun 13, 2023

ajar98 Jun 13, 2023

zaptrem commented Jun 15, 2023

	text="Sorry, voice creation is only supported for Coqui TTS.",
	text="Sorry, voice creation is only supported for Coqui.",

Coqui Telegram Bot #173

Coqui Telegram Bot #173

Conversation

zaptrem commented May 30, 2023 • edited Loading

Prompt-To-Voice Telegram Bot

TODO:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ajar98 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zaptrem commented Jun 15, 2023

zaptrem commented May 30, 2023 •

edited

Loading