Small Models and In Context Learning

[DataBassGit]

The problem with large corporate language models is that they become out of date very quickly. You also have the issue of large models getting out of date. For example, a python library gets an update and old code is no longer valid. It costs a lot of money to update that model, and the maintainers are not going to update it every time a new update comes out for a python module.

Therefore, in context learning has to be used in order to leverage knowledge that was created after the models training date. (2021 for GPT-4) I.e. you inject a KB article into the prompt for the python module you want to write code with. This approach can be expanded to be used for all implementations of language modules.

Therefore, you don't need 180b parameter modules to do good NLP processing. With a functional and efficient prompt enrichment protocol, small models with enriched prompts can accomplish most of the tasks that large models can. (As long as the context length and attention mechanisms can handle the enriched prompt.) This could allow us to use smaller locally hosted modules that could potentially run faster and on a single GPU for many of our prompt executions, then leverage larger corporate modules only for tasks where they are needed and excel at.

[daveshap]

Yeah that's the idea. I've heard rumors that Gemini will have "up to the second" updates, so either it's got massive in context learning or some other way to slipstream information in at reference

[DataBassGit]

Up to the second sounds like a completely new architecture from the way language models work today, as they rely on massive compute to integrate new information. Maybe having a LoRA that later gets turned into additional training data or pure RLAI, but I'm only guessing at this point. I guess we have to wait.