feat: make list_agents paginated

[cpacker]

Closes #1498

Change in API endpoint

GET request is now paginated, in the style of: https://platform.openai.com/docs/api-reference/assistants/listAssistants

Change in data locations

For @goetzrobin: there's some differences in the data coming back from the GET request (apart from it now accepting additional pagination args):

MemGPT/memgpt/server/server.py

Line 864 in 82ae5a2

for agent_state, return_dict in zip(agents_states, agents_states_dicts):

"persona" / "human"

            return_dict["persona"] = agent_state._metadata.get("persona", None)
            return_dict["human"] = agent_state._metadata.get("human", None)

This should still be accessible in the List[AgentStateModel] being returned

"tools"

            return_dict["tools"] = [tool for tool in all_available_tools if tool.json_schema in memgpt_agent.functions]

This is now in AgentStateModel.metadata["tools"]

NOTE: this has been moved to a separate GET request: see new route def get_agent_tools (router.get("/agents/{agent_id}/tools)

"memory stats"

            # Add information about memory (raw core, size of recall, size of archival)
            core_memory = memgpt_agent.memory
            recall_memory = memgpt_agent.persistence_manager.recall_memory
            archival_memory = memgpt_agent.persistence_manager.archival_memory
            memory_obj = {
                "core_memory": {section: module.value for (section, module) in core_memory.memory.items()},
                "recall_memory": len(recall_memory) if recall_memory is not None else None,
                "archival_memory": len(archival_memory) if archival_memory is not None else None,
            }
            return_dict["memory"] = memory_obj

This is now in AgentStateModel.metadata["memory_stats"], see new Agent.update_state function

"last run"

            # Add information about last run
            # NOTE: 'last_run' is just the timestamp on the latest message in the buffer
            # Retrieve the Message object via the recall storage or by directly access _messages
            last_msg_obj = memgpt_agent._messages[-1]
            return_dict["last_run"] = last_msg_obj.created_at

This is now in AgentStateModel.metadata["last_run"]

"attached sources"

            # Add information about attached sources
            sources_ids = self.ms.list_attached_sources(agent_id=agent_state.id)
            sources = [self.ms.get_source(source_id=s_id) for s_id in sources_ids]
            return_dict["sources"] = [vars(s) for s in sources]

This is now in AgentStateModel.metadata["sources"]

NOTE: this has been moved to a separate GET request: def get_agent_sources (/agents/{agent_id}/sources)

^extra note @goetzrobin these metadata fields may be null for old agents, since they are stored inside of metadata they will get refreshed on .step(), so only agents that have been run post-PR will have the metadata fields. The frontend should have sane defaults / not crash on null accordingly.

[sarahwooders]

you might want to do the conversion in the REST API, since if you just return List[AgentState] then the LocalClient can directly use this