fix: the cosine similarity is evaluated for top comments and bot comments are ignored #225

gentlementlegen · 2024-12-26T07:33:03Z

Resolves #174

What are the changes

bot comments are ignored when building the prompt, which greatly reduces token usage and innacuracy
if the token count for the built prompt is higher than the model limit, strip the comments down to the 10 most relevant ones based on their cosine similarity with the specification

Rfc @sshivaditya2019

…ents are ignored

0x4007

Very skeptical of tfidf approach. We should go simpler and filter out more instead.

0x4007 · 2024-12-28T22:52:08Z

src/parser/content-evaluator-module.ts

@@ -15,6 +15,9 @@ import {
 import { BaseModule } from "../types/module";
 import { ContextPlugin } from "../types/plugin-input";
 import { GithubCommentScore, Result } from "../types/results";
+import { TfIdf } from "../helpers/tf-idf";
+
+const TOKEN_MODEL_LIMIT = 124000;


This depends on the model and possibly should be an environment variable because we might change models.

Hard coding the 12400 doesn't seem like a solution there either

@0x4007 It is not hard coded but configurable within the config file?
There is no API to retrieve a model max token value as far as I know.

Line 179 is hard coded

What should I do if the configuration is undefined, should I throw an error and stop the run then?

Yes if we don't have it saved in our library or collection of known amounts then it should throw

0x4007 · 2024-12-28T22:52:33Z

src/parser/content-evaluator-module.ts

@@ -61,7 +64,7 @@ export class ContentEvaluatorModule extends BaseModule {
    const allCommentsUnClean = data.allComments || [];
    const allComments: { id: number; comment: string; author: string }[] = [];
    for (const commentObj of allCommentsUnClean) {
-      if (commentObj.user) {
+      if (commentObj.user && commentObj.user.type !== "Bot") {


We should also filter out slash commands? And minimized comments?

Yes I agree, I didn't program this part so was not sure what I should keep / remove.

3cbd808

Rfc @sshivaditya2019 I am not sure why all the comments were sent (commands, bot comments etc), now only user evaluated comments (which also are curated) are sent to evaluate the prompt which seems better, let me know if that introduces some issues on comment evaluation precision.

It shouldn't affect it at all. I would proceed with implicit approval even if shiv doesn't reply in time for merge.

@gentlementlegen It shouldn't impact the comment evaluation at all. I think the prompt was updated, though the original one may have included something related to bot comments.

0x4007 · 2024-12-28T22:55:41Z

src/parser/content-evaluator-module.ts

@@ -178,15 +181,33 @@ export class ContentEvaluatorModule extends BaseModule {
      const dummyResponse = JSON.stringify(this._generateDummyResponse(comments), null, 2);
      const maxTokens = this._calculateMaxTokens(dummyResponse);

-      const promptForComments = this._generatePromptForComments(specification, comments, allComments);
+      let promptForComments = this._generatePromptForComments(specification, comments, allComments);
+      if (this._calculateMaxTokens(promptForComments, Infinity) > TOKEN_MODEL_LIMIT) {


I'm skeptical about this whole TFIDF approach

The tokenizer algorithm is different per model and I'm not sure what model we should be using. I'm starting to see a pattern of OpenAI and Anthropic release cycles alternating so to get the best models perhaps we should also alternate. I heard good things about the new OpenAI model.

I don't fully understand why we need this if we can just filter out a lot more overhead like the bot comments, slash commands

The weaknesses of this approach isn't fully clear to me. We did some earlier research on implementing this method in here before but I recall that it wasn't a good fit. I just don't remember why- perhaps accuracy related.

It should be an edge case for very long conversations which should not happen very often. Advantage about TFIDF is that we keep a bunch of relevant comments instead of arbitrarily keep the first / last n comments.

Can you articulate the weaknesses or concerns

The main concern is that we base the selection solely on similarity, which might be less precise that using the LLM to determine which comments are the most relevant. On the other hand I think it is better than actually writing summaries of comments because this way we keep the content intact during evaluation (just shorten it).

Since this should not happen often (and even less often now that we will properly filter useless comments once I resolve your comment), the easiest and most straightforward solution seemed to be the best choice.

sshivaditya2019 · 2024-12-29T06:03:21Z

I don't think TF-IDF would be the best option for selecting the comments, as it only takes in account the word frequency and does not give any importance to semantics. A better solution might be to switch to a larger context model, like Gemini, which provides a 2 million token context window when we reach the token limit, and exclude bot-generated comments from the selection process.

…pt for LLMs

gentlementlegen · 2024-12-29T06:28:17Z

This was added as a failsafe when we go through the limit. Gemini can be an option, but theoretically we could also go beyond its token limit (even if unlikely). Since we can configure models which all have different token max limits (and third parties could be using smaller and cheaper ones) I think it is important that the technique we use to lower the context size is not based on LLM.

0x4007 · 2024-12-29T07:14:20Z

More careful filtering of comments like removal of bot commands and comments, and possibly text summarization, or concatenation of multiple calls are all more accurate approaches. TFIDF is not the right tool for the job.

gentlementlegen · 2024-12-29T07:37:22Z

The commands and bot comments got also fixed in this PR. I added this as a last resort if this was not enough. As I said before, I don't think it should rely on LLMs itself because third party users could chose a tiny model like gpt-3 and have only 3000 tokens available which would not be capable to even summarize. I can change this to summarize each comment in one sentence, which would use one call to the LLM per comment, which I though would potentially be expensive.

0x4007 · 2024-12-29T07:39:46Z

Doing multiple calls to score everything and then concatenate results seems the most straightforward with no data loss.

gentlementlegen · 2024-12-29T07:59:30Z

@0x4007 Doing the evaluation is not the problem, the problem is the given context to the LLM that gets too big. If an issue has 300 comments, then the prompt would contain these 300 comments while evaluating which would be too many for the token limit, so it has to get smaller. I don't see a way to fix that with no data loss, except if you meant comparing the comment against every other single comment one by one?

0x4007 · 2024-12-30T04:32:35Z

Divide into two and do 150 each call. Receive the results array and concatenate them together

gentlementlegen · 2024-12-30T05:57:02Z

@0x4007 This is not reliable, and if a third party decides to use a model like o4-mini it would break anyways. What should be done in that case?

Plus concatenating would not make sense. I should run the comment against 150 first and 150 last (which would make the context inaccurate) and then probably average the result. I believe you didn't see how comment are evaluated: the problem is that for the context of understanding comments, we send all the comments in the context evaluated against the comments of the user. Refer to:

text-conversation-rewards/src/parser/content-evaluator-module.ts

Line 254 in 66c1e77

return `

0x4007 · 2024-12-31T02:53:06Z

Surely it's a bit of a trade off without all of the comments in one shot, but this approach seems to trades off the least.

The task specification is by far the most important reference point.
We can inject a new line in the prompt explaining that following is part 2/2 and that due to context window limits we had to split the conversation into multiple parts for evaluation.

and if a third party decides to use a model like o4-mini it would break anyways. What should be done in that case?

Why would they complain about using a non default model? We set the default to what we know works.

gentlementlegen added 2 commits December 26, 2024 16:32

fix: the cosine similarity is evaluated for top comments and bot comm…

c3d091b

…ents are ignored

chore: fixed cspell words

8d65638

gentlementlegen marked this pull request as ready for review December 26, 2024 07:39

0x4007 reviewed Dec 28, 2024

View reviewed changes

chore: only curated comments are taken into account to build the prom…

3cbd808

…pt for LLMs

gentlementlegen added 2 commits December 29, 2024 15:36

chore: token limit is configuration through the configuration file

a0695e3

test: fixed configuration test

70705b7

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: the cosine similarity is evaluated for top comments and bot comments are ignored #225

fix: the cosine similarity is evaluated for top comments and bot comments are ignored #225

gentlementlegen commented Dec 26, 2024 •

edited

Loading

0x4007 left a comment

0x4007 Dec 28, 2024

gentlementlegen Dec 29, 2024

0x4007 Dec 29, 2024

gentlementlegen Dec 29, 2024

0x4007 Dec 29, 2024

gentlementlegen Dec 29, 2024

0x4007 Dec 30, 2024

0x4007 Dec 28, 2024

gentlementlegen Dec 28, 2024

gentlementlegen Dec 29, 2024

0x4007 Dec 30, 2024

sshivaditya2019 Dec 30, 2024

0x4007 Dec 28, 2024

gentlementlegen Dec 28, 2024

0x4007 Dec 29, 2024

gentlementlegen Dec 29, 2024

sshivaditya2019 commented Dec 29, 2024

gentlementlegen commented Dec 29, 2024

0x4007 commented Dec 29, 2024 •

edited

Loading

gentlementlegen commented Dec 29, 2024

0x4007 commented Dec 29, 2024 •

edited

Loading

gentlementlegen commented Dec 29, 2024

0x4007 commented Dec 30, 2024

gentlementlegen commented Dec 30, 2024 •

edited

Loading

0x4007 commented Dec 31, 2024 •

edited

Loading

fix: the cosine similarity is evaluated for top comments and bot comments are ignored #225

Are you sure you want to change the base?

fix: the cosine similarity is evaluated for top comments and bot comments are ignored #225

Conversation

gentlementlegen commented Dec 26, 2024 • edited Loading

What are the changes

0x4007 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sshivaditya2019 commented Dec 29, 2024

gentlementlegen commented Dec 29, 2024

0x4007 commented Dec 29, 2024 • edited Loading

gentlementlegen commented Dec 29, 2024

0x4007 commented Dec 29, 2024 • edited Loading

gentlementlegen commented Dec 29, 2024

0x4007 commented Dec 30, 2024

gentlementlegen commented Dec 30, 2024 • edited Loading

0x4007 commented Dec 31, 2024 • edited Loading

gentlementlegen commented Dec 26, 2024 •

edited

Loading

0x4007 commented Dec 29, 2024 •

edited

Loading

0x4007 commented Dec 29, 2024 •

edited

Loading

gentlementlegen commented Dec 30, 2024 •

edited

Loading

0x4007 commented Dec 31, 2024 •

edited

Loading