Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added explain analyze query plan to recommendation prompt #17

Merged
merged 4 commits into from
Aug 10, 2024

Conversation

dimafe6
Copy link
Contributor

@dimafe6 dimafe6 commented May 13, 2024

Description

  • Explain analyze query plan has been added to prompt for more relevant recommendations. This provide more detailed information to the LLM about existing indexes utilization, actual query time etc. so LLM can provide more usefull answers.
    This feature can be enabled/disabled by recommendation_use_explain configuration.
  • Two parameters ignore_explain_queries and ignore_insert_queries have been added into config to provide ability to disable logging EXPLAIN and INSERT queries.
  • extractIndexesAndSchemaFromRecord function was refactored to use schema and indexes from all tables used in SQL query instead of SlowLog table schema and indexes because it doesent provide any usefull information for LLM about query. So, it means if query use multiple tables then each table with indexes and schema will be transferred to LLM for provide more usefull answers.
  • Prompt text was changed to instruct LLM to use query plan and tables schema.

After this changes prompt will use more tokens but LLM will provide more usefull recommendations:

  • Full queries for creating indexes;
  • Completely redesigned optimized initial query;
  • Recommendations for database configurations;
  • Architecture recommendations (partitioning, etc.)
    image
    image

dimafe6 and others added 2 commits May 13, 2024 18:52
…relevant recommendations. Added two settings: ignore_explain_queries, ignore_insert_queries. extractIndexesAndSchemaFromRecord was refactored to use schema and indexes from multiple tables from SQL query instead of SlowLog model schema and indexes. getTableNamesFromRawQuery function has been added. Default prompt has been updated.
Added EXPLAIN ANALYSE query plan into RecommendationService for more …
@halilcosdu halilcosdu self-assigned this May 13, 2024
@halilcosdu halilcosdu added the enhancement New feature or request label May 13, 2024
@halilcosdu
Copy link
Owner

Looks good thanks @dimafe6

dimafe6 added 2 commits May 16, 2024 15:58
…s and log slow queries by individual request time instead of the aggregate time of all queries per request.
@dimafe6
Copy link
Contributor Author

dimafe6 commented May 16, 2024

Some additional changes:

  • Send sql instead of raw_sql into LLM. To anonymize data that is transferred to a third-party service.
  • Logging logic for slow queries has been changed. Listen to all queries and log slow queries by individual request time instead of the aggregate time of all queries per HTTP request. Because for now only one latest query will be log and only if total time of all queries during HTTP request will be greather than treshold. For example if controller excute 10 different requests only latest request will be logged and only if total time of this 10 request is greather than treshold. For my opinion this is not correct logic.

@halilcosdu halilcosdu self-requested a review May 25, 2024 18:36
@halilcosdu
Copy link
Owner

Waiting readme update @dimafe6

@halilcosdu halilcosdu merged commit 1529d09 into halilcosdu:main Aug 10, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants