Customize prompts?

[sam1am]

Is there a way to customize the prompts sent to the API? It would be nice to be able to give it some specific direction and info like the date and time or personality direction and a name and all that.

[brianpetro]

@sam1am yes! I'm trying to keep things simple for now, so we can find an ideal set of presets that works for average users.

But enabling customizing prompts for advanced used is 100% possible.

Date/time inclusion is interesting. This may be able to be done automatically if time based references are detected in the user input.

Personality makes sense, too. Maybe this could be added without requiring editing the whole system prompt. Maybe append the default system prompt with some personality traits.

Is there anything else prompt-wise that you had in mind?

Thanks for your input and helping to make this plugin the best it can be!

[ROlwig]

I'm really enjoying using Smart Connections!

It would be great to consider a library of Templates that included

1. "Instructions" as a Preamble
2. a specified folder to create context.

This would then create something similar an OpenAI GPT.

[brianpetro]

@ROlwig that sounds interesting!

Do you mind elaborating a bit?

🌴

[ROlwig]

@brianpetro thanks for the quick reply and interest. I'm referring to functionality similar to OpenAI GPTs Where the configuration includes the GPT
Name - of the "Smart GPT" or similar
Description - I don't use this.
Instructions - "Here you can provide detailed instructions or guidelines on how the GPT should behave, its functionalities, and any particular behaviors to avoid."
Conversations Starters - I don't use these.
Knowledge - the ability to upload files - in the case of Smart Connections, indicate a folder or folders

Thanks, again.

[brianpetro]

@ROlwig that's great, thanks.

I was already planning on doing something quite similar to what you're describing, I call them "alignment notes."

But, something I didn't consider was specifying folders, and I really like that idea!

🌴

[Cody-W-Tucker]

Using Prompts from inside the Obsidian app by linking to a note would be nice. You could almost do that now, but the primary prompt suggests that the answers can be found in the note itself.

obsidian-smart-connections/src/index.js

Line 3129 in 49568af

this.chat.context = `Anticipate the type of answer desired by the user. Imagine the following ${context.length} notes were written by the user and contain all the necessary information to answer the user's question. Begin responses with "${SMART_TRANSLATION[this.plugin.settings.language].prompt}..."`;

It's also worth considering the implications on unrelated replies if the prompts folder was accessible via embeddings.

[jwhco]

I can already do this. I've created files that include {Assistant, System, User} and a Usage subheading. The Assistant is the context around the prompt itself. Bullet points about how to write, what background to include, and the format I'm looking for in this specific prompt.

The System links to a set of system prompt guidelines. The User subhead provides prompt context, like "Write a newsletter issue for the audience provided ..." Usage is an actual example of using the prompt.

I call this prompt by saying, "Use prompt [[NAME]] for X, Y, Z". The closer my prompt request matches the Usage in file mentioned, the better all the other context works. I can ask it to write a newsletter issue from a group of linked notes and it does a good first draft.

You don't need to code in specific alignment notes; it already does. I modeled this prompt sidecar after what's in Ollama. I've done some that say the particular format I want a response to; it consistently gives it to me.

I know this is an old thread. I'm on Smart Connections v2.2.73

[sam1am]

This is really a necessity for me. I am finding the hardcoded system prompt to be a bit limiting. I love the idea of bring able to use a note or notes as templates, which could be dynamically updated and all sorts of things.

[YesJimPossible]

Although I find this plugin already extremely useful as it is, I definitely agree custom system prompts would make it even more useful. I've grown accustomed to prime GPT with specifically "fine-tuned" system prompts which greatly improve the quality of its responses.

For the meantime, I think having a faux-system prompt in the form of a note that you title and write to encourage the model looks into its embeddings every time might work to some extent. Though of course it's a rather janky and unpredictable band-aid.

[brianpetro]

@YesJimPossible thanks for the feedback.

I've been thinking about how to add this in v2, so if you have any thoughts on:

• what might make this sort of thing most useful?
• what might it look like if it were easy for new users? what about advanced users?
• anything else

I'd love to hear them as I'm still working out the final designs for v2.

🌴

[YesJimPossible]

what might make this sort of thing most useful?

Essentially the developing field of "prompt science" if it can be called that. New research with findings about specific ways to prompt GPT that improve output is published continually, and many users have specific preferences in terms of what approach they like most. A few examples of what I would imagine are the most common system prompt uses:

• Basic instructions that improve output quality which people don't want to have to repeat manually for every request (e.g. "Think step-by-step", "You are an expert at JavaScript").
• Ensuring output follows specific requirements. For example, ensuring GPT outputs adhere to academic writing and referencing guidelines.
• Many people like to use various pseudo-programming languages for prompts to improve performance while optimizing token usage (e.g. https://medium.com/javascript-scene/sudolang-a-powerful-pseudocode-programming-language-for-llms-d64d42aa719b)

what might it look like if it were easy for new users? what about advanced users?

I think following an approach similar to OpenAI's own custom instructions (https://openai.com/blog/custom-instructions-for-chatgpt) would be most user-friendly.

I'd imagine the main distinction between novice and advanced users could be made in the freedom given by default. A simple and sleek custom instruction-like interface by default that lets users add quick and easy optional system prompts that are appended to Smart Connection's main prompt in a predictable manner.

Beyond that, an option to enable full customization of all API parameters would honestly be wonderful. Essentially giving advanced users as much freedom as possible to adjust as many available API parameters and endpoints as possible. Having the ability to use as much of the OpenAI API to one's own desire would of course be really amazing but is probably a lot to ask for a single person to implement.

anything else

I've only recently begun using Obsidian so I'm fairly ignorant about uses that Obsidian veterans would want, but, in a world where you have unlimited development time and no other obligations in your life, I could imagine people wanting to use new features like audio and images as well as new beta features like assistants to make really powerful integrations.

I'm not fully aware what exactly is technically possible in terms of letting GPT interact with Obsidian, notes and potentially other plugins but anything that allows that sort of thing would of course be quite lovely.

Lastly I just want to emphasize again that the plugin in its current state is already absolutely amazing, even when compared to various commercial tools that offer similar functionality (i.e. "talk to pdf" stuff). Even without custom system prompts the synergy between Obsidian, Smart Connections and other plugins like Zotero is extremely powerful and works very well. The effort you've put into developing this is hugely appreciated!

[brianpetro]

@YesJimPossible love it. Thanks for your insights. I'll keep them in mind as a move forward 😊

🌴

[fabioscarsi]

@brianpetro, the answer from @YesJimPossible Is very interesting. I have an opinion too.

Also because I'm neither a technician nor a native English speaker, I'll try to express my view shortly and as simply as I can.

1. Smart Connection is an Obsidian plugin. Obsidian is the choice for many as their PKM infrastructure. Smart Connection can then be the natural vectorization and RAG tool for a big part of the context to give to LLMs for many tasks.
2. Given point 1, Smart Connection has extraordinary potential, but we should go on building on the vector database it creates from Obsidian notes. Right now, the plugin is far from reaching its potential. The closed prompting structure is the main limit.
   2.1 One way could be a merger between Smart Connections and Text Generator, one of the other good AI plugins in Obsidian. Another option, another extraordinary plugin, could be QuickAdd. They seem quite complementary. The Obsidian App developers Team (Kepano and the founders) are, incredibly, completely neglecting artificial intelligence. So, the situation in Obsidian for AI developments is chaotic, and some sort of integration or consolidation in this strategic area probably would be very useful;
   2.2 if Smart connection has to remain a standalone plugin without any deliberate integration with other tools from the ground (and it's not your fault @brianpetro as I clearly remember that at the beginning of the development many months ago, you offered to other developers to use the vector database created by Smart Connections with embeddings), I think you should consider using Obsidian notes as prompts in template logic, with some variables mechanism too;
   2.3 Considering the development you have done for your ChatGPT plugin and also considering the last OpenAI releases, I think our own proprietary context indexed through your plugin could be easily integrated with a RAG mechanism a la Perplexity AI (https://www.perplexity.ai/).

Considering that I think your view on this topic is one of the clearest among developers, I'm sure you have been considering these opportunities for months now, as your ChatGPT plugin clearly demonstrates. The only real problems are related to the big development costs of coding these opportunities. Personally, I'm a supporter and invested heavily many months ago, giving you credit, and I'm sure of the enormous potential already available. I think that the evolution of your tool could have a significant market. But it should be fast, otherwise some other developer will fill the gap between what is available and currently possible.

[ProfSynapse]

You can do this super easily manually. Just head into the JSON file with all the code, search "prompt" and you'll find both the prompt and the intro which you can change. When you update it'll revert back but you can just change it the same way again.

[jritsema]

Sorry, which JSON file are you referring to?

[jagai]

Not sure if this is of any help to anyone, but I'm sharing below some micro-adjustments to main.js @ .obsidian/plugins/smart-connections. Please note that while I might refer to other .js or .json extensions, they're all within main.js. I'm including them to make the search easier within the file.

I adjusted the initial chat message first around build/views.json. Within <div class="sc-message-content">, there's a <span> tag you can modify the initial message on the chat.

e.g:

       <div class="sc-message-content">
          <span>
            Hey X! What would you like to discuss? I'm here to help.
          </span>

I also modified around src/chat/ScTranslations.json the en part. It helped eliminate the "based on your notes..." arbitrary text that is prefixed to every response which I found redundant.

  en: {
    pronouns: ["my", "I", "me", "mine", "our", "ours", "us", "we"],
    prompt: "",
    initial_message: "Hey X! What would you like to discuss? I'm here to help."
  },
  ...

Note that these two are purely visual and has no effect on the prompt itself.

Below is more about having the LLM stay in-character.

I refactored a few bits around src/chat/sc_chat_model.js, all you have to do is just replace that entire section with the one below, and modify the prompt as you see fit. Just make sure your prompt is copied on all three sections marked as Write your prompt here to ensure the LLM doesn't lose context.

I ended the final prompt with the TEXT and the given context params just so it keeps context again.

     TEXT:
    ${params.ct ? " " + params.ct : ""}

Main part to refactor:

// src/chat/sc_chat_model.js
var import_smart_chat_model = __toESM(require_smart_chat_model(), 1);
var ScChatModel = class extends import_smart_chat_model.SmartChatModel {
  async done_handler(full_str) {
    await this.env.chat_ui.new_message(full_str, "assistant");
    this.env.chats.current.add_message({ role: "assistant", content: full_str });
    this.env.chat_ui.clear_streaming_ux();
  }
  async chunk_handler(text_chunk) {
    await this.env.chat_ui.new_message(text_chunk, "assistant", true);
  }
  async request_middlewares(opts) {

    // Insert the system prompt at the start of the conversation
    if (!opts.messages.some(msg => msg.role === "system")) {
      opts.messages.unshift({
        role: "system",
        content: `Write your prompt here`
      });
    }


    await Promise.all(opts.messages.map(async (msg, i) => {
      const context_start = "```sc-context";
      if (msg.role === "tool" && msg.tool_call_id === "lookup") {
        msg.role = "system";
        msg.content = context_start + "\n" + JSON.parse(msg.content).map((c) => c.path).join("\n") + "\n```";
      }
      if (msg.role === "system" && msg.content.includes(context_start)) {
        const context_start_i = msg.content.indexOf(context_start) + context_start.length;
        const context_end_i = msg.content.substring(context_start_i).indexOf("```");
        const raw_contents = msg.content.substring(context_start_i, context_start_i + context_end_i);
        const entities = this.env.plugin.get_entities_from_context_codeblock(raw_contents);
        let context = [];
        let tokens = [];
        await Promise.all(entities.map(async (entity, i2) => {
          if (!entity?.get_as_context) return console.log(entity);
          context[i2] = await entity.get_as_context({ i: i2 });
          tokens[i2] = await this.count_tokens(context[i2]);
        }));
        let total_tokens = 0;
        let ct = 0;
        context = context.reduce((acc, c, i2) => {
          if (!c) return acc;
          if (total_tokens + tokens[i2] > this.max_input_tokens) return acc;
          total_tokens += tokens[i2];
          ct++;
          if (acc) acc += "\n";
          return acc + c;
        }, "");
        msg.content = this.get_prompt_context_prefix({ ct }) + "\n" + context;
      }
      const sys_start = "```sc-system";
      if (msg.role === "system" && msg.content.includes(sys_start)) {
        const sys_start_i = msg.content.indexOf(sys_start) + sys_start.length;
        const sys_end_i = msg.content.substring(sys_start_i).indexOf("```");
        const sys_prompts = msg.content.substring(sys_start_i, sys_start_i + sys_end_i).split("\n").filter((ln) => ln.trim());
        console.log(sys_prompts);
        msg.content = `Write your prompt here`;
        for (const sys_prompt of sys_prompts) {
          const tfile = this.env.smart_connections_plugin.system_prompts.find((file) => file.basename === sys_prompt);
          const note_content = await this.env.smart_connections_plugin.read_file(tfile);
          if (msg.content) msg.content += "\n";
          msg.content += note_content;
        }
      }
      return msg;
    }));
    opts.messages = opts.messages.filter((msg) => msg.role !== "assistant" || msg.content || !msg.tool_calls?.find((call) => call.id === "lookup"));
    console.log(opts.messages);
    return opts;
  }
  get_prompt_context_prefix(params = {}) {
    return `Write your prompt here
    
    TEXT:
    ${params.ct ? " " + params.ct : ""}`;
  }
};

[Yunu1994]

That helps me a lot, thanks