Feature Request: Support for Sending Images or Audio Files to LLM

[bassamsdata]

Hello,

First of all, thank you for all the hard work on this plugin!

It’s become my go-to tool for interacting with any LLM. I was wondering if it would be possible to add a feature that allows sending images or audio to the LLM (perhaps by providing the file path)?

Thanks again!

[olimorris]

Thanks for your kind words. Your suggestions and bug spotting has been hugely appreciated over the past couple of months.

I've thought about adding vision capabilities to the plugin for a while now. I haven't fully implemented it because there's always been, at least in my opinion, lower hanging fruit to pursue. If you could share your use case so I can understand it better than I'm more likely to bump it up the roadmap.

To be open, I plan on adding some features that are similar to Aider in the coming weeks. The ability to create files on the disk, run tests and modify code until the tests pass and make the plugin capable of completing benchmarking challenges so users can better evaluate what models to use. I'd also like to build a /workspace slash command that gives the LLM context of the users repo.

[bassamsdata]

Your suggestions and bug spotting have been hugely appreciated over the past couple of months.

Thank you.

I haven't fully implemented it because there's always been, at least in my opinion, lower-hanging fruit to pursue.

I totally understand, and I really don’t want to rush it at all.

If you could share your use case so I can understand it better, then I’m more likely to bump it up the roadmap.

Here’s my use case for images in two workflows:

1. I do a lot of data visualizations in R and Python, and sometimes I need the LLM to understand what I mean without writing a lot of description about one chart so sending a screenshot would be ideal, especially when iterating on the same idea or chart.
2. Often, I have one graph (image/screenshot) that contains a whole plan, workflow, or structure, and I just don’t want to switch to the web version to output (different from OCR), explain everything, and then return to the chat buffer and paste.

In both workflows, I usually use screenshots.
One note: I saw that Avante is using the plugin img-clip.nvim for this feature, but I haven't tested it.

For audio, I mainly use it for transcribing and translating, which are far from coding-related tasks. I could probably make a small app for that, so there’s probably no need to include that feature.

To be open, I plan on adding some features that are similar to Aider in the coming weeks. The ability to create files on disk, run tests, modify code until the tests pass, and make the plugin capable of completing benchmarking challenges so users can better evaluate what models to use.

Your plan for the plugin is impressive, and I admire that. I wasn’t aware of Aider, but it looks amazing from what I’ve seen so far.

I'd also like to build a /workspace slash command that gives the LLM context of the user's repo.

I think this feature would be a major milestone for this plugin, and I can’t wait to try it out when it’s ready.

Thank you again.