Continue to allow $ symbols to delimit inline math in human messages

[dlqqq]

Description

Closes #1089.

Mostly reverts #1068.

This PR reverts the frontend changes in #1068 and just updates the system prompt to encourage the LLM to express quantities of USD in LaTeX instead of plaintext.

• Users can still use $ to delimit inline math, and must double-escape $ symbols to use them literally. This is aligned with JupyterLab's rendering behavior.

This may not fix rendering of literal $ symbols for all LLMs, but makes a best-effort without breaking support for $ as an inline math delimiter.

Demo

Screen.Recording.2024-11-06.at.10.57.40.AM.mov

[dlqqq]

I've outlined a testing strategy which will determine if we can be reasonably confident that this change doesn't lead to a regression in the rendering of AI messages.

1. Iterate through all models in each provider.
2. For each of the test cases (documented below):
   • Run /clear.
   • Send the test case.
   • Ensure the response is correct and rendered properly.

Test cases:

Given $2x + 3y = 4$ and $3x + 5y = 12$, what are $x$ and $y$?

Alice put \\$100 into a savings account with 1% APY 50 years ago.
Bob put \\$1000 into a savings account with 1% APY 5 years ago.

How much money will be in their accounts today? Show your work.

The price of a European call option under the Black-Scholes model is typically expressed as:

$$C = S_0 N(d_1) - X e^{-rT} N(d_2) $$

Can you explain what $C$, $S_0$, $N$, $X$, and $e^{-rT}$ represent in this expression?

These cases should assert that:

• When using $ delimiters, the LLM should not wrap $ delimiters in LaTeX.
• When using double-escaped $ symbols literally, the LLM should still return dollar quantities as LaTeX.

[dlqqq]

The original revision of this PR causes a regression in Claude 2. I'll remove the escaping logic & update the system prompt to see if all cases can still be handled.

[dlqqq]

Based on testing by me and @srdas, we've determined that this PR doesn't introduce any regressions. When using the latest LLMs (Claude 3.5, GPT-4o, Llama3.x), this PR generally fixes the chat rendering when multiple dollar quantities appear on the same line.

However, the system prompt is typically not respected by older models (e.g. gpt-3.5-turbo from OpenAI, claude-2.0 from Anthropic). When these models are used, the rendering issue persists.

The only change from v2.27.0 is that the system prompt has an additional request to express dollar quantities as LaTeX. Given this, I think this PR is safe to merge.

[dlqqq]

Example of improved rendering in Llama 3.1 405B:

[srdas]

While for weaker LLMs, like GPT4All: WizardLLM it does not do well, for Claude3-Haiku, it works great:

The answer also renders well:

And it also renders perfectly when exported to a .md file and then is added to a MarkDown block in a Jupyter notebook.

I have looked over the prompt changes and it all looks good.

[dlqqq]

@meeseeksdev please backport to v3-dev

[dlqqq]

cc @jtpio @brichet re: jupyter-chat