Using marked for chatbot markdown parsing

[dawoodkhan82]

Description

Move chatbot markdown parsing to frontend using marked, and highlight.js.

Please include:

• relevant motivation
• a summary of the change
• which issue is fixed.
• any additional dependencies that are required for this change.

Closes: #4146
Closes: #3680
Closes: #4143
Closes: #4092

Checklist:

• I have performed a self-review of my own code
• I have added a short summary of my change to the CHANGELOG.md
• My code follows the style guidelines of this project
• I have commented my code in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

A note about the CHANGELOG

Hello 👋 and thank you for contributing to Gradio!

All pull requests must update the change log located in CHANGELOG.md, unless the pull request is labeled with the "no-changelog-update" label.

Please add a brief summary of the change to the Upcoming Release > Full Changelog section of the CHANGELOG.md file and include
a link to the PR (formatted in markdown) and a link to your github profile (if you like). For example, "* Added a cool new feature by [@myusername](link-to-your-github-profile) in [PR 11111](https://github.com/gradio-app/gradio/pull/11111)".

If you would like to elaborate on your change further, feel free to include a longer explanation in the other sections.
If you would like an image/gif/video showcasing your feature, it may be best to edit the CHANGELOG file using the
GitHub web UI since that lets you upload files directly via drag-and-drop.

[pngwn]

looks so cool.

Maybe if we add a min height to the chatbubbles that is roughly equal to one line, it will prevent that initial jumpy feeling?

[pngwn]

Maybe consider prism over highlightjs for syntax highlighting. I think it is a little more complete and slightly better maintained. We're also using it elsewhere in the repo.

[abidlabs]

@dawoodkhan82 this is still a draft or ready to be reviewed?

[gradio-pr-bot]

All the demos for this PR have been deployed at https://huggingface.co/spaces/gradio-pr-deploys/pr-4150-all-demos

[dawoodkhan82]

@abidlabs will be ready for review later today. have to make some fixes first

[aliabid94]

Great work @dawoodkhan82. Can we make the codeboxes more clearly separate from the rest of the content?

There is more of a visible difference on main:

[abidlabs]

Haven't taken a deep look at this PR yet (having some issues building with pnpm) but wondering if this also solves (or can easily be adapted to solve) some of the other Markdown-related issues in the Chatbot, like rendering headings, lists and tables

Also, does this approach play nicely with LaTeX rendering in the backend?

[dawoodkhan82]

See updated screenshot for fixes. Ready for re-review @abidlabs @pngwn

[abidlabs]

Very cool! Taking a look now

[abidlabs]

This is fantastic @dawoodkhan82! Works quite well in my tests. This also closes #3680 I believe so I've added that to the parent comment.

I did notice some design nits:

Code

1. In Dark mode, code with a specified language has a weird gold border and gold background color:

Code to reproduce:

import gradio as gr

code = """
This is some code in Python:

```python
def test(x):
  return x
```
"""

value = [
    ("Hello", code),
    (code, "Nice!")
]

with gr.Blocks() as demo:
    gr.Chatbot(value)
    
demo.launch()

In light mode, it looks great. It would be good to check the other built-in gradio themes to confirm that they work appropriately

2. Code with no language specified has reduced margin

There's slightly less vertical margin around the code if no language is specified. I think it should be a little more:

Code to reproduce:

import gradio as gr

code = """
This is some code in Python:

```
def test(x):
  return x
```
"""

value = [
    ("Hello", code),
    (code, "Nice!")
]

with gr.Blocks() as demo:
    gr.Chatbot(value)
    
demo.launch()

[abidlabs]

Tables and Lists

3. It would be good to add a tiny bit of vertical margin above and below the tables, just like there is with the code:

4. In Dark mode, there is no visible table border when the table is on the right side of the chat:

It would be good to test with the other built-in themes as well

5. The lists look good when they are on the left side of the chat. When they are on the right side, there isn't enough margin on the left side:

Code to reproduce the above:

import gradio as gr

code = """
And here is a table:
| Syntax      | Description |
| ----------- | ----------- |
| Header      | Title       |
| Paragraph   | Text        |

And here is a list:
1. ad
2. ad
3. ad

And here are bullets
* sdf
* adslkfjasldkf
* dasf
"""

value = [
    ("Hello", code),
    (code, "Nice!")
]

with gr.Blocks() as demo:
    gr.Chatbot(value)
    
demo.launch()

[dawoodkhan82]

Fixed the styling issues @abidlabs

[abidlabs]

Looks lovely!

[dawoodkhan82]

@abidlabs Sure, I can look into latex support in a separate pr

[abidlabs]

Based on @aliabid94’s benchmarking, this should close #4092 as well

[Keldos-Li]

little suggestion:

How should you fix list styling:

message-wrap :global(ol),
message-wrap :global(ul) {
    padding-inline-start: 2em;
}

[Keldos-Li]

In our own project built with gradio, we rendered latex using mathjax:

iShot_2023-04-25_.1.34.09.mp4

BUT PLEASE give users or developers a choice of whether or not to render in-line formulas! Because the $ symbol can really cause a lot of problems. The $ in the table or the $ in the code block displaying shell code is likely to cause the entire display to be confused. Do not add formula rendering directly by default please. As the developer who actually uses Gradio for user-oriented projects, I believe we have encountered more issues with $ than you think...

[Keldos-Li]

In fact, I felt really upset when I saw this PR 🤧. Maybe you can let developers decide what to do with chatbot messages?😢 The origin method using python to post_process messages allows us to achieve some cool effects such as:

And for us, at least we believe and implement the principle that all messages sent by users should remain in their original format without being rendered by markdown or something. Though I know there may be different opinions, but at least, the origin method makes it possible for us to override and take control of the styles of the output... I'm now very nervous that we may lose our features with this gradio update.

I would also say this should be written in breaking change rather than other changes😕

or maybe something cool may happen with svelte? I'm not quite sure with that.

[abidlabs]

Hi @Keldos-Li I think it should still be possible to achieve what you're showing in the screenshot. Basically the difference between the two is that in one case, you escape the Markdown, while in the other, you do not, correct? If you escape the Markdown and send it to the frontend, it will (should?) remain in its raw form even if the rendering is moved the frontend.

Are there any other concerns that you have? There are a variety of reasons that we moved the rendering to the frontend (you can take a look at the linked issues to see, among them being: (1) more expected results when using Gradio Chatbots via APIs and (2) less latency

[abidlabs]

little suggestion:

How should you fix list styling:

message-wrap :global(ol),
message-wrap :global(ul) {
    padding-inline-start: 2em;
}

Adding a little bit of padding on the left of the lists is not a bad idea @dawoodkhan82

[Keldos-Li]

If you escape the Markdown and send it to the frontend, it will (should?) remain in its raw form even if the rendering is moved the frontend.

That makes sense！We'll try to handle this. Thanks for the reply!

Now I understand why it's important for you to move rendering to the front-end. You are right.

Yeah... I think I'll just have no more concerns for now if we still get a chance to take control of the message output. Thanks.

Gradio is such a great app, we really appreciate your contribution.

[pseudotensor]

Awesome!

[Keldos-Li]

Hi @Keldos-Li I think it should still be possible to achieve what you're showing in the screenshot. Basically the difference between the two is that in one case, you escape the Markdown, while in the other, you do not, correct? If you escape the Markdown and send it to the frontend, it will (should?) remain in its raw form even if the rendering is moved the frontend.

Are there any other concerns that you have? There are a variety of reasons that we moved the rendering to the frontend (you can take a look at the linked issues to see, among them being: (1) more expected results when using Gradio Chatbots via APIs and (2) less latency

@abidlabs Sorry to bother you again. We have encountered a problem that we cannot solve... So I'm here to ask for help.

These days we have been working hard to adapt to the new Gradio, but we have encountered difficulties in achieving our original feature in the screenshot above. Perhaps we could write a function to convert the special markdown control characters in advance, but our main issue at the moment is that we are unable to switch the display of rendered and not-rendered text.

Previously, we developed new postprocess and postprocess_chat_messages functions to override the original Gradio functions. This allowed us to parse chat_message through our additional functions. The core code for this part of the function is roughly as follows:

def convert_mdtext(md_text):
    ...
    raw = f'<div class="raw-message hideM">{html.escape(md_text)}</div>'
    result = ...
    output = f'<div class="md-message">{result}</div>'
    output += raw
    ...
    return output

Then, we use CSS and JavaScript to control the display of these two divs through the click event of the button we created.

However, after {@html marked.parse(message)}, everything is invalid... I saw that you configured it for marked:

marked.setOptions({
    ...
    sanitize: true,
    ...
});

gradio/js/chatbot/src/ChatBot.svelte

Lines 33 to 41 in 8242c06

	marked.setOptions({
	renderer: new marked.Renderer(),
	gfm: true,
	breaks: true,
	pedantic: false,
	sanitize: true,
	smartLists: true,
	smartypants: false
	});

However, in the marked documentation, it says:

Although it does mention that this may bring XSS attacks, turning off sanitize may be crucial for our project. Moreover, marked also recommends not to use its sanitize directly (but to use other libraries).

Warning: 🚨 Marked does not sanitize the output HTML. If you are processing potentially unsafe strings, it's important to filter for possible XSS attacks. Some filtering options include DOMPurify (recommended), js-xss, sanitize-html and insane on the output HTML! 🚨

I was wondering if it's possible to set the sanitize option in marked.setOptions to false? Or maybe provide an option for us to choose whether to sanitize... (I know it's unlikely, sigh)

Alternatively, do you know of any other methods to prevent HTML tags from being rendered by marked?

Or perhaps you could think of some other ways for us to achieve the desired effect?

Or... maybe Gradio has considered adding a feature that allows switching between markdown rendering and copying raw output? It shouldn't be too difficult to implement with Svelte... (we'd be happy to help) If Gradio had these features natively, we wouldn't be having this problem.

Sorry for replying in this PR again, please forgive me if I'm mistaken.

[abidlabs]

I was wondering if it's possible to set the sanitize option in marked.setOptions to false?

We are working on doing something like this in this PR in progress: #4360 (feel free to chime in if that does not fix it for you)

Apologies for the breaking change, but we are working on restoring functionality soon & hopefully with the move the frontend, processing should be much faster.

[Keldos-Li]

We are working on doing something like this in this PR in progress: #4360 (feel free to chime in if that does not fix it for you)

Such great news! Thank you!!!

[abidlabs]

Such great news! Thank you!!!

Feel free to test it out by installing pip install https://gradio-builds.s3.amazonaws.com/db234a0f26a2720b1764be97e3cc4bfa70795275/gradio-3.32.0-py3-none-any.whl and letting us know if it works/doesn't work for your use case

[jefffohl]

Thanks for this @dawoodkhan82 - this is great. However, I noticed that there seems to be a problem with dark/light mode handling and the Prism code highlighting. When the operating system has selected "light" or "dark" themes, the Chatbot component works fine. However, when using the dark/light mode theming pattern established elsewhere in Gradio, which uses the class name dark applied to the <body> element, we run into problems. When the OS is indicating the preferred theme to be "light", but then we toggle the them to "dark" using the classname method, the Prism code highlighting is not also toggled.

Example:

then, toggling the "Dark Mode" by clicking the button which applies the class dark to the <body> element of the app, which enables the dark mode of the theme (as applied in the base Gradio theme):

We see that the logic for applying the Prism code highlighting is separate from the CSS pattern described above, which results in what is seen in the second example shown above.

Am I simply implementing this incorrectly, or is there more work to do here to sync up the code highlighting dark mode with the Gradio theme for dark mode handling? If the latter, I would be happy to try and help.

[pseudotensor]

Thanks for giving details @jefffohl !

[dawoodkhan82]

@jefffohl seems like its a bug, where the prism highlighting doesn't react to the dark mode change. Can you open a new issue with this context and assign it to me? I'll take a look. Thanks!

[jefffohl]

Will do - thanks @dawoodkhan82 !

[jefffohl]

@dawoodkhan82 issue created ^^ - I don't seem to have permission to assign it to anyone though. Thanks for your help!