ACE using "local" and "global" LLMs for different layers. Looking for a team! (Or creating one of my own?)

[ippolite]

Hi,
My name is Aleksei. I come from a weird background of nuclear physics and politics.

What do I want to build?
An implementation of an ACE framework that for some list of tasks uses only locally installed LLMs (for sensitive data).

Think of it as if Agent and Executive layers use some local model, and everything else is relying on OpenAI's API.
Or going completely local if the internet connection goes off.

What do I bring to the table?

• I have worked with a monstrosity of code, the ALIROOT software, which never worked as planned. I come from the C++ environment (so I use ChatGPT to translate my thoughts to Python) and from a team with a terrible working process organisation. Which means, we naturally need a person who knows how a good one (and free, maybe?) looks like;
• I have tried some 7B and 13B models locally on my laptop;
• I helped a friend of mine with his tycoon/rogue-lite game, but it was ultimately meant for the desk drawer;
• I know how a consulting-ish case approach looks like;
• I run an ARM processor and UNIX system. Compatibility issues, yay!;
• I have 10-ish hours per week.

What do I do not have?

• An understanding of a streamlined development medium;
• Resources. I mean, I only have around 10 bucks/month to burn for API and nowhere near enough laptop capabilities to run 64B model and to get an answer in a reasonable time;
• A server. I may ask for one, but it's unlikely that they'll agree to;
• I have never trained a model of my own, due to the points above;
• Enough grammar knowledge to not to make frequent mistakes.

Possible Problems that I Presume (the PPP)

• We'll need another "sanitising layer / bus" that ensures that the info exchanged between "the local" and "the global" LLMs is clean from user-specific or any other top secret data;
• How the hell would buses work? How would they be censored?;
• Some of the local models might be too dumb or undertrained for the task! We need some way of benchmarking them occasionally against bigger models. I see it as if the Global strategy layer have a specific task to produce test objects once in a while and compare somehow the results of a bigger external model to a local one (because I believe in hot swap and plug-and-play);
• How do we tell the layer that it is online-first and local-second? Would it be just a "telemetry data" from the bus? And how do we tell a layer that it is now dealing with a sensitive info and must run local-only? A faux sensor? Software button triggered by a file extension?;
• Can we split layers into "local" and "global" parts? Do we really need to isolate the whole thing, or we can invent a specific code language that would be understood and processed by a "global" LLM and then unciphered and executed by a "local" one?

What is my own alignment?
I figured I disclose it beforehand.
I participate in an anti-war project that utilizes the phone calls and local network of people to battle the current Russian government. You can find us as "агитмашина Навального" (Navalny's machine of agitation).
We have a call center, where a ton of analysis is done by hand and Google Sheets, so three of us have ventured to look for an AI solution.
It neither means that this very ACE project would be shaped by my own goals, nor that it would be used for a political action.
If you're worried that I come from the wrong side, I can ask people for credits.

[daveshap]

Thanks for being the first post! I will be happy to help support your team as you will have interest in local LLMs - which is a big idea! Let me point some people in your direction.

[daveshap]

Also it may be helpful to provide some context in your post title. Maybe "Locally installed LLMs for ACE doing list of tasks" so that people will have a better idea of what to expect

[ippolite]

Hey, I have finished the post and it's heading. Please, look through and tell if anything is out of place and needs to be fixed.
I have also created a corresponding Q&A post about it.

[Qriscuit]

My friend and i'd love to talk to you, we're both game developers who got inspired to try and bring AGI to implementation. That said we're absolute noobs when it comes to web and software development. We're learning as we go. The very first thing we wanted was a reliable vector search so thats what we're in the process of making. This is also not an ask to get hired, but to discuss a high level
implementation

Ping me on discord: qriscuit

[nofx99]

I have all the relative hardware, and no idea what to do or where to start. So I will also be requiring a team of some sort.
I have the starlink and 4090 and and all that good stuff
But I want to learn

[samgriek]

I just saw Dave's video and I was feeling quite disappointed that I'm coming in so late to not have a chance to join the team. Until he mentioned that we can self organize to create more teams and work on his backlog. Does anyone want to work on his backlog here or only on your personal project/idea? (I have my one project too but I want to join a team working on the main backlog).

My background is quite broad but I've been coding since 2003 and now specialize in python. I have plenty of experience designing and building enterprise applications (that I can't show you because they are proprietary) and I like to think I can hold my own in the areas of infrastructure, devops, and frontend (Vue3). I'm also quite experienced in building production grade APIs in FastAPI. For the last two years I've been focused on technology leadership role.

If you want to organize another scrum team, let me know by reaching and quoting my message.

Sam

[ippolite]

Hey, Sam!

What tasks exactly do you want to take on from the backlog?

I just saw Dave's video and I was feeling quite disappointed that I'm coming in so late to not have a chance to join the team. Until he mentioned that we can self organize to create more teams and work on his backlog. Does anyone want to work on his backlog here or only on your personal project/idea? (I have my one project too but I want to join a team working on the main backlog).

If you want to organize another scrum team, let me know by reaching and quoting my message.

[samgriek]

@ippolite - I've changed my team's discussion title. We will pursue the NPC demo. Regarding the main backlog, I have experience in building distributed system, docker, message buses, and data engineering so I wanted to be involved in understanding the infra tech-stack, local setup, and working on some of the infrastructure. In fact I've already created all the layers and busses locally using docker-compose, postgreSQL, and NATS. The layers are listening to each other but I haven't manged to figure out a good way for the Global Strategy layer to keep track of the overall agent's context and re-write the agent agenda, strategy, and plan yet.

In my vision of the ACE Framework, private models are necessary because public models will be censored and the agent won't be able to reason in those censored areas effectively. Right now private models are too resource intensive for my taste. If we manage to secure some sponsorship for my research or a contract with an enterprise, it would be worth exploring.

I have fine-tuned Llama 2 locally on some data from Databricks and it took 8 hours and that's with the lowest settings which result in a low quality model... I've interacted with several other models using some different open source tools that take away some of the complexity, but also don't seem to give me the results I want in training private models on my private data. Also, my GPU is only 12GB and it precludes a lot of experimentation. I do see some used 24GB GPUs for around $600 that may open more doors, but I don't believe all that effort and time is required to create a compelling value proposition.

[Indicaza]

I left a comment on your YouTube video, but to sum things up I've been working on this solo for a month or two now (mostly for games). The idea is to create a tunable prism that the AI can shine through in order to imbue that call with a custom personality (your modular approach is much better than mine!). After the tuning (easy!), we gotta figure out that prism memory.

This is what I have so far, https://github.com/Indicaza/shard-api
email: zreitmeier@wccnet.edu

Don't be scared that I like Javascript, I know Python too, I just don't enjoy it as much.

IDIC, live long and prosper.👽

[Grab-your-parachutes]

I have all the relative hardware, and no idea what to do or where to start. So I will also be requiring a team of some sort. I have the starlink and 4090 and and all that good stuff But I want to learn

I'm essentially in your position. I started watching Dave what must I think be around 2 years ago, certainly feels like time is passing faster these days. I have the beefy pc required (13900k 4090) to do the work but have been stumbling through it myself since then trying to both keep abreast of things and get some work done to try to learn. I'm a retired aircraft mechanic, I've managed to get an OpenAI driven chatbot for Discord and Twitch working but that really is the extent of it. I'm learning Python essentially by absorption.
I'm autistic, unfortunately it comes with a side of ADHD, both of which have been the bane of a difficult experience of life.
I therefore have a lot of time and hyperfocus and my autistic fascination with language, the only true power we have outside of violence, gives me the linguistic ability to work well with Chatgpt.
Dave has been a huge inspiration to me, I thank you Dave! You are the reason I reach out now to any who might be at a similarly low level of experience but would like people to work with nonetheless. I can't honestly say I bring much more than a keen mind but so it goes. I'm at best in the hobbyist bracket!

I expect I won't be the only one who finds their way here so if anything I've said resonates do reach out.

[nofx99]

i managed to start a model, i've been playing so been giving it science and mathematics and python and some LLM,.... Not sure that's a great mix.

Still unsure about most of what i'm doing but it's happening even with some labelling issues for the files.

I like the idea of law enforcement apps with genetic genealogy incorporated. This would be financially beneficial for the whole world.
Government - i'd love to make the government redundant. The government will never agree but if they no longer have control they have no power.
medicine - with all the fitbits and such, i don't see why we still have to go visit a general practitioner. Shortage of doctors yet they sit and write scripts all day without knowing any information on the patient being treated. Diagnosis = AI

I have no idea what i'm doing though and need help with the coding aspect. Whilst i can have the code written for me 'copy and pasting' that code isn't relevant to my computer. I don't even know if i have to run the code every time i start up my computer to have to AI agent (i'm sure i can figure that out).
But yeah, i'll find some more language models because my AI asked for LLU or something LMU?? i dunno
I'll run what people want in terms of databases and make it available if possible

[andrew-tillmann]

Did anyone get a local version running? I was thinking of setting up ACE on LangChain. If implemented on LangChain I believe you should be able to a mix of LLM models part local and others via api like describing. I know David talked about local implementation further down the road map but I wanted to see if anyone has this currently setup.