diff --git a/src/public/image/rfcs/rfc135/img.png b/src/public/image/rfcs/rfc135/img.png new file mode 100644 index 00000000..40cf8ba2 Binary files /dev/null and b/src/public/image/rfcs/rfc135/img.png differ diff --git a/src/public/image/rfcs/rfc135/img_1.png b/src/public/image/rfcs/rfc135/img_1.png new file mode 100644 index 00000000..db7862e5 Binary files /dev/null and b/src/public/image/rfcs/rfc135/img_1.png differ diff --git a/src/public/image/rfcs/rfc135/img_10.png b/src/public/image/rfcs/rfc135/img_10.png new file mode 100644 index 00000000..2025e33d Binary files /dev/null and b/src/public/image/rfcs/rfc135/img_10.png differ diff --git a/src/public/image/rfcs/rfc135/img_11.png b/src/public/image/rfcs/rfc135/img_11.png new file mode 100644 index 00000000..2d9fb7b3 Binary files /dev/null and b/src/public/image/rfcs/rfc135/img_11.png differ diff --git a/src/public/image/rfcs/rfc135/img_12.png b/src/public/image/rfcs/rfc135/img_12.png new file mode 100644 index 00000000..57950cbd Binary files /dev/null and b/src/public/image/rfcs/rfc135/img_12.png differ diff --git a/src/public/image/rfcs/rfc135/img_13.png b/src/public/image/rfcs/rfc135/img_13.png new file mode 100644 index 00000000..ac8edb05 Binary files /dev/null and b/src/public/image/rfcs/rfc135/img_13.png differ diff --git a/src/public/image/rfcs/rfc135/img_14.png b/src/public/image/rfcs/rfc135/img_14.png new file mode 100644 index 00000000..deb0ce28 Binary files /dev/null and b/src/public/image/rfcs/rfc135/img_14.png differ diff --git a/src/public/image/rfcs/rfc135/img_15.png b/src/public/image/rfcs/rfc135/img_15.png new file mode 100644 index 00000000..3f77ed48 Binary files /dev/null and b/src/public/image/rfcs/rfc135/img_15.png differ diff --git a/src/public/image/rfcs/rfc135/img_16.png b/src/public/image/rfcs/rfc135/img_16.png new file mode 100644 index 00000000..ed7cd987 Binary files /dev/null and b/src/public/image/rfcs/rfc135/img_16.png differ diff --git a/src/public/image/rfcs/rfc135/img_17.png b/src/public/image/rfcs/rfc135/img_17.png new file mode 100644 index 00000000..4f1f61c9 Binary files /dev/null and b/src/public/image/rfcs/rfc135/img_17.png differ diff --git a/src/public/image/rfcs/rfc135/img_18.png b/src/public/image/rfcs/rfc135/img_18.png new file mode 100644 index 00000000..a52ae16d Binary files /dev/null and b/src/public/image/rfcs/rfc135/img_18.png differ diff --git a/src/public/image/rfcs/rfc135/img_19.png b/src/public/image/rfcs/rfc135/img_19.png new file mode 100644 index 00000000..d7ca9678 Binary files /dev/null and b/src/public/image/rfcs/rfc135/img_19.png differ diff --git a/src/public/image/rfcs/rfc135/img_2.png b/src/public/image/rfcs/rfc135/img_2.png new file mode 100644 index 00000000..dbe441c9 Binary files /dev/null and b/src/public/image/rfcs/rfc135/img_2.png differ diff --git a/src/public/image/rfcs/rfc135/img_20.png b/src/public/image/rfcs/rfc135/img_20.png new file mode 100644 index 00000000..e927dccb Binary files /dev/null and b/src/public/image/rfcs/rfc135/img_20.png differ diff --git a/src/public/image/rfcs/rfc135/img_21.png b/src/public/image/rfcs/rfc135/img_21.png new file mode 100644 index 00000000..9adf73e7 Binary files /dev/null and b/src/public/image/rfcs/rfc135/img_21.png differ diff --git a/src/public/image/rfcs/rfc135/img_22.png b/src/public/image/rfcs/rfc135/img_22.png new file mode 100644 index 00000000..e2deec22 Binary files /dev/null and b/src/public/image/rfcs/rfc135/img_22.png differ diff --git a/src/public/image/rfcs/rfc135/img_3.png b/src/public/image/rfcs/rfc135/img_3.png new file mode 100644 index 00000000..796b5167 Binary files /dev/null and b/src/public/image/rfcs/rfc135/img_3.png differ diff --git a/src/public/image/rfcs/rfc135/img_4.png b/src/public/image/rfcs/rfc135/img_4.png new file mode 100644 index 00000000..ef31e390 Binary files /dev/null and b/src/public/image/rfcs/rfc135/img_4.png differ diff --git a/src/public/image/rfcs/rfc135/img_5.png b/src/public/image/rfcs/rfc135/img_5.png new file mode 100644 index 00000000..1b4e9d71 Binary files /dev/null and b/src/public/image/rfcs/rfc135/img_5.png differ diff --git a/src/public/image/rfcs/rfc135/img_6.png b/src/public/image/rfcs/rfc135/img_6.png new file mode 100644 index 00000000..2442a6ad Binary files /dev/null and b/src/public/image/rfcs/rfc135/img_6.png differ diff --git a/src/public/image/rfcs/rfc135/img_7.png b/src/public/image/rfcs/rfc135/img_7.png new file mode 100644 index 00000000..53d3d263 Binary files /dev/null and b/src/public/image/rfcs/rfc135/img_7.png differ diff --git a/src/public/image/rfcs/rfc135/img_8.png b/src/public/image/rfcs/rfc135/img_8.png new file mode 100644 index 00000000..b9c3435f Binary files /dev/null and b/src/public/image/rfcs/rfc135/img_8.png differ diff --git a/src/public/image/rfcs/rfc135/img_9.png b/src/public/image/rfcs/rfc135/img_9.png new file mode 100644 index 00000000..4cdb5b9b Binary files /dev/null and b/src/public/image/rfcs/rfc135/img_9.png differ diff --git "a/src/rfcs/RFC-116-MetaGPT\344\274\230\345\214\226\346\226\271\346\241\210.md" "b/src/rfcs/RFC-116-MetaGPT\344\274\230\345\214\226\346\226\271\346\241\210.md" index 904a4a98..45c728e8 100644 --- "a/src/rfcs/RFC-116-MetaGPT\344\274\230\345\214\226\346\226\271\346\241\210.md" +++ "b/src/rfcs/RFC-116-MetaGPT\344\274\230\345\214\226\346\226\271\346\241\210.md" @@ -1,289 +1,301 @@ -# [T]RFC-116-MetaGPT优化方案 - -**文档负责人** :马莘权 - -1. 文档修改记录 - -| **日期** | **版本** | **作者** | **修改内容** | -| ---------- | -------- | -------- | -------------- | -| 2023-10-25 | v1 | 马莘权 | 创建 | -| 2023-11-3 | v2 | 马莘权 | 删掉无用的设计 | - - - -2. 文档审核状态 - -| **审核人** | **邀请日期** | **审核日期** | **评论** | -| ---------- | ------------ | ------------ | -------- | -| @洪思睿 | 2023-10-26 | 2023-10-30 | LGTM | -| @林义章 | 2023-10-30 | 2023-10-30 | | -| @王金淋 | 2023-10-30 | 2023-10-30 | | -| @吴承霖 | 2023-10-30 | 2023-10-30 | | -| @沈楚城 | 2023-10-26 | 2023-10-30 | LGTM | - - -## 1. 引言 -### 1.1 背景 - -鉴于MetaGPT框架已经在软件公司、狼人杀等场景得到应用,希望通过总结现有落地过程中发现的问题,来优化MetaGPT框架设计,以简化后续算法同学、第三方同学的开发工作,简化MetaGPT向Agent的迁移工作。 - - -### 1.2 目标 - -明确下一步的MetaGPT框架优化方向,以: - -1. 简化后续算法同学的开发工作 -2. 简化第三方的开发工作 -3. 简化MetaGPT向Agent的迁移工作 - -## 2. 系统设计 - -### 2.1 系统架构 - -#### 2.1.1 MetaGPT内部的消息处理 - -##### 2.1.1.1 现状 - -![img](/image/rfcs/rfc116/8d9c341f-2bd4-4ab3-942f-2a8981f8eabd.png) - -现有设计适合简单、轻量的智能体应用。 - -在处理持续交互、跨网交互方面存在如下问题: - -1. 对于多轮场景,超参“n_round”的值难统一、难推荐。比如挖矿场景、软件开发场景。 -2. 共享式的消息存放不支持跨网的消息消费; -3. 共享式的消息存放不支持个性化role对象的记忆压缩和信息隔离,如下图所示: - -![img](/image/rfcs/rfc116/12423f9c-1f3c-48dc-b91f-ee23f4e95efd.png) - -##### 2.1.1.2 新的架构方案 - -###### 2.1.1.2.1 自适配event loop - -新架构中,Env对象的event loop终止的方式为: - -1. Env对外提供`is_idle`状态; -2. 外部可通过调Env的stop函数来终止event loop; -3. 外部通过调start函数来启动event loop。start参数中可指定event loop是否在空闲时自动终止。 - -![img](/image/rfcs/rfc116/4d85b0bd-ec56-4c8a-9820-f678cd989089.png) - -新架构中: - -1. 所有消息均通过`Environment`对象提供的路由功能,将消息存放到各个`Role`对象私有的消息buffer中; -2. 所有状态数据都存放到一个支持序列化和反序列化的`StateContext`对象中; -3. Env对象通过检查内部各个role是否都空闲来判断是否需要结束event loop; -4. 跨role对象的消息转发统一由`Environment`对象负责。 - -#### 2.1.2 状态数据管理 - -现状中,并未规范状态数据的存放原则。 - -新方案中,`Role`对象私用的状态数据统一存放在Role对象自己的`StateContext`对象中;跨`Role`对象的状态数据统一存放在`Environment`对象的`StateContext`对象中。 - -`Environment`提供`save`和`load`函数,用于对`Environment`对象内部的状态数据存档和恢复。 - -#### 2.1.3 消息结构 - -##### 2.1.3.1 现状 - -现存的消息结构如图所示: - -![img](/image/rfcs/rfc116/19970c7b-dcb9-4c0a-be4f-5f7af1b29261.png) - -其中: - -1. `content`用来存放消息内容; -2. `instruct_content`功能与`content`相同,区别是存放的是结构化的数据。 -3. `role`是OPENAI规范中定义的`role`的值,是调LLM的参数的一部分。本质是meta信息的一部分。 -4. `cause_by`被同时用作分类(一种meta信息)标签和路由标签: - ```Python - async def _observe(self) -> int: - await super()._observe() - # accept the very first human instruction (the debate topic) or messages sent (from opponent) to self, - # disregard own messages from the last round - self._rc.news = [msg for msg in self._rc.news if msg.cause_by == UserRequirement or msg.send_to == self.name] - return len(self._rc.news) - ``` -5. `sent_from`被用作展示时显示的发言者信息。本质是meta信息的一部分。 -6. `send_to`被用作路由参数,用来在从共享消息队列中筛选发给自己的消息: - ```Python - async def _observe(self) -> int: - await super()._observe() - self._rc.news = [ - msg for msg in self._rc.news if msg.send_to == self.profile - ] # only relevant msgs count as observed news - return len(self._rc.news) - ``` -7. `restricted_to`被用作群发(一发多)的路由参数,用来从共享消息队列中筛选发给自己的消息: - ```Python - async def _think(self): - news = self._rc.news[0] - assert news.cause_by == InstructSpeak # 消息为来自Moderator的指令时,才去做动作 - if not news.restricted_to: - # 消息接收范围为全体角色的,做公开发言(发表投票观点也算发言) - self._rc.todo = Speak() - elif self.profile in news.restricted_to.split(","): - # FIXME: hard code to split, restricted为"Moderator"或"Moderator,角色profile" - # Moderator加密发给自己的,意味着要执行角色的特殊动作 - self._rc.todo = self.special_actions[0]() - ``` - -##### 2.1.3.2 新的消息结构 -新的消息结构如图所示: - -![img](/image/rfcs/rfc116/4edb8a4c-245b-4d02-a065-8e39937e732e.png) - - -其中: - -1. 为了支持`Message`类的JSON化,将`cause_by`的类型替换成str,并封装了自动类型转换的功能; -2. 将原来`send_to`的str类型改为`Set`,以支持多标签“或关系”的消息订阅功能。如果`Message`对象在创建时未设置`send_to`的值,则默认是全员可接收(与现有公共memory的效果相同)。 -3. 本次修改,MetaGPT的消息处理参照了消息队列的定义,明确了消息的生产和消费关系。 - - a. 新框架中,所有`Role`类及其子类都是消费者,它们默认订阅了发送给全员、自己对象的类名字、自己对象的`name`属性值的消息,见缺省的消息订阅标签章。 - - b. 所有`Message`对象的创建者都是消息生产者,它们可以通过设置`Message`对象的`send_to`的值来调整让谁来消费这条消息。 - - - -下面是一些新Message类的使用示例: -1. 设置`cause_by`属性: -```Python -m = Message("a", cause_by=Action) -assert m.cause_by == get_class_name(Action) -assert m.cause_by == any_to_str(Action) -``` - -2. 设置`sent_from`属性: -```Python -m = Message("a", sent_from=Action) -assert m.sent_from == get_class_name(Action) -assert m.sent_from == any_to_str(Action) -``` -3. 设置`sent_to`属性: -```Python -m = Message("a", sent_to={"b", Action}) -assert m.send_to == {"b", get_class_name(Action)} -assert m.send_to == any_to_str_set({"b", Action}) -``` - -###### 2.1.3.2.1 缺省的消息订阅标签 -为了简化业务开发工作,框架默认提供了3种消息订阅标签。 -所有`Role`及其子类都已经订阅了这3种标签,开发者可在发消息时直接使用: -1. `Role`对象的`name`属性:对应`Role`类及其子类对象的`name`属性的值。 -2. `Role`类的类名:对应任何`Role`类及其子类对象的类名。比如"Architect"类的类名为"metagpt.roles.architect.Architect"。 -3. 全员:值为"<all>"。本地所有消费者,都能收到这个消息。 - -示例如下: - - -
-假设有如下几个`Role类`的子类对象: - -1. `Moderator`类对象 `r1`, `r1`.`name`为`a`; -2. `Werewolf`类对象 `r2`, `r2`.`name`为`b`; -3. `Werewolf`类对象 `r3`, `r3`.`name`为`c`; -4. `Villager`类对象 `r4`, `r4`.`name`为`d`; -5. `Villager`类对象 `r5`, `r5`.`name`为`e`; -6. `Seer`类对象 `r6`, `r6`.`name`为`f`; - -已知`Environment`类对象`env`: - -1. `r1`发消息给所有`Werewolf`类对象: -```Python -env.publish_message(Message(content="...", send_to=Werewolf)) -``` - 这里的`Werewolf`用的就是`Role`类的类名做标签。这个消息的接收者是所有的`Werewolf`类对象。 - -2. `r1`发消息给`c`和所有`Villager`类对象: -```Python -env.publish_message(Message(content="...", send_to={Villager, "c"})) -``` -这里的`Villager`是`Role`类的类名标签,`c`是`Role`对象`r3`的`name`属性。这个消息的接收者是所有`Villager`类对象,以及`name`属性为`c`的`Role`对象`r3`。 - -3. `r1`发消息给所有对象: -```Python -env.publish_message(Message(content="...", send_to="")) -``` - 这个消息的接收者为所有群成员。 - -6. `r1`发消息给`c`, `d`, `e`: -```Python -env.publish_message(Message(content="...", send_to={"c", "d", "e"})) -``` -这个消息的接收者为`r3`、`r4`、`r5`。 -
- -### 2.2 MetaGPT需改造的模块 -#### 2.2.1 Message结构 -参考`新的消息结构`章节。 - -新的消息处理框架下,缺省情况下`Role`类对象会收到所有消息(因为缺省的`Message`.`send_to`是全员)。因此`Role`类的`_observe`里需要把自己关注的新增消息过滤出来,并以此判断是否有消息需要处理。 - -代码如下: -```Python -async def _observe(self) -> int: - """Prepare new messages for processing from the message buffer and other sources.""" - # Read unprocessed messages from the msg buffer. - news = self._rc.msg_buffer.pop_all() - # Store the read messages in your own memory to prevent duplicate processing. - self._rc.memory.add_batch(news) - # Filter out messages of interest. - self._rc.news = [n for n in news if n.cause_by in self._rc.watch] - - # Design Rules: - # If you need to further categorize Message objects, you can do so using the Message.set_meta function. - # msg_buffer is a receiving buffer, avoid adding message data and operations to msg_buffer. - news_text = [f"{i.role}: {i.content[:20]}..." for i in self._rc.news] - if news_text: - logger.debug(f"{self._setting} observed: {news_text}") - return len(self._rc.news) -``` - -#### 2.2.2 将公共消息存储改造成Role私有的消息存储 -1. 取消公共消息存储,改为`Role`私有的消息存储。 -```Python -class RoleContext(BaseModel): - """Role Runtime Context""" - env: 'Environment' = Field(default=None) - memory: Memory = Field(default_factory=Memory) - long_term_memory: LongTermMemory = Field(default_factory=LongTermMemory) - state: int = Field(default=0) - todo: Action = Field(default=None) - watch: set[str] = Field(default_factory=set) - news: list[Type[Message]] = Field(default=[]) -``` - `Role`中的所有`self._rc.env.memory`操作变更为`self._rc.memory`操作。 - -2. 所有消息转发都由`Environment`类的`Env`对象负责。禁止`Role`对象之间通过访问对方的私有消息存储来交换消息。 - -![img](/image/rfcs/rfc116/32d67809-99f6-496e-9800-2a28c8eb73c5.png) - -3. 所有`Role`对象增加一个私有的消息buffer,来接收异步`put_message`写入的消息。`Role`对象的observe操作(observe-think-act)需要同时看消息buffer和memory。 - -#### 2.2.3 取消超参k -目前流程的终止依靠`Environment`的`run`的超参`k`和`CostManager`的成本超支检查。 -```Python -async def run(self, k=1): - """处理一次所有信息的运行 - Process all Role runs at once - """ - # while not self.message_queue.empty(): - # message = self.message_queue.get() - # rsp = await self.manager.handle(message, self) - # self.message_queue.put(rsp) - for _ in range(k): - futures = [] - for role in self.roles.values(): - future = role.run() - futures.append(future) - - await asyncio.gather(*futures) -``` -新的流程终止条件改为所有的`self.roles`对象的`think`都为空(`Role`对象私有的消息buffer为空,且无action),或者成本超支。 - -#### 2.2.4 规范状态数据的存储 -1. 新建一个支持序列化和反序列化`save`/`load`的`StateContext`类 -2. 所有状态数据都需要放到这个类对象中。 -3. 仅限`Environment`类和`Role`类拥有这个类的对象。 +# [R]RFC-116-MetaGPT Role对象间消息机制优化方案 + +**文档负责人** :马莘权 + +1. 文档修改记录 + +| **日期** | **版本** | **作者** | **修改内容** | +| ---------- | -------- | -------- | -------------- | +| 2023-10-25 | v1 | 马莘权 | 创建 | +| 2023-11-3 | v2 | 马莘权 | 删掉无用的设计 | + +## 1. 引言 + +### 1.1 背景 + +鉴于MetaGPT框架已经在软件公司、狼人杀等场景得到应用,希望通过总结现有落地过程中发现的问题,来优化MetaGPT框架设计,以简化后续算法同学、第三方同学的开发工作,简化MetaGPT向Agent的迁移工作。 + +### 1.2 目标 + +明确下一步的MetaGPT框架优化方向,以: + +1. 简化后续算法同学的开发工作 +2. 简化第三方的开发工作 +3. 简化MetaGPT向Agent的迁移工作 + +## 2. 系统设计 + +### 2.1 系统架构 + +#### 2.1.1 MetaGPT内部的消息处理 + +##### 2.1.1.1 现状 + +![img](../public/image/rfcs/rfc116/8d9c341f-2bd4-4ab3-942f-2a8981f8eabd.png) + +现有设计适合简单、轻量的智能体应用。 + +在处理持续交互、跨网交互方面存在如下问题: + +1. 对于多轮场景,超参“n_round”的值难统一、难推荐。比如挖矿场景、软件开发场景。 +2. 共享式的消息存放不支持跨网的消息消费; +3. 共享式的消息存放不支持个性化role对象的记忆压缩和信息隔离,如下图所示: + +![img](../public/image/rfcs/rfc116/12423f9c-1f3c-48dc-b91f-ee23f4e95efd.png) + +##### 2.1.1.2 新的架构方案 + +###### 2.1.1.2.1 自适配event loop + +新架构中,Env对象的event loop终止的方式为: + +1. Env对外提供`is_idle`状态; +2. 外部可通过调Env的stop函数来终止event loop; +3. 外部通过调start函数来启动event loop。start参数中可指定event loop是否在空闲时自动终止。 + +![img](../public/image/rfcs/rfc116/4d85b0bd-ec56-4c8a-9820-f678cd989089.png) + +新架构中: + +1. 所有消息均通过`Environment`对象提供的路由功能,将消息存放到各个`Role`对象私有的消息buffer中; +2. 所有状态数据都存放到一个支持序列化和反序列化的`StateContext`对象中; +3. Env对象通过检查内部各个role是否都空闲来判断是否需要结束event loop; +4. 跨role对象的消息转发统一由`Environment`对象负责。 + +#### 2.1.2 状态数据管理 + +现状中,并未规范状态数据的存放原则。 + +新方案中,`Role`对象私用的状态数据统一存放在Role对象自己的`StateContext`对象中;跨`Role`对象的状态数据统一存放在`Environment`对象的`StateContext`对象中。 + +`Environment`提供`save`和`load`函数,用于对`Environment`对象内部的状态数据存档和恢复。 + +#### 2.1.3 消息结构 + +##### 2.1.3.1 现状 + +现存的消息结构如图所示: + +![img](../public/image/rfcs/rfc116/19970c7b-dcb9-4c0a-be4f-5f7af1b29261.png) + +其中: + +1. `content`用来存放消息内容; +2. `instruct_content`功能与`content`相同,区别是存放的是结构化的数据。 +3. `role`是OPENAI规范中定义的`role`的值,是调LLM的参数的一部分。本质是meta信息的一部分。 +4. `cause_by`被同时用作分类(一种meta信息)标签和路由标签: + ```Python + async def _observe(self) -> int: + await super()._observe() + # accept the very first human instruction (the debate topic) or messages sent (from opponent) to self, + # disregard own messages from the last round + self._rc.news = [msg for msg in self._rc.news if msg.cause_by == UserRequirement or msg.send_to == self.name] + return len(self._rc.news) + ``` +5. `sent_from`被用作展示时显示的发言者信息。本质是meta信息的一部分。 +6. `send_to`被用作路由参数,用来在从共享消息队列中筛选发给自己的消息: + ```Python + async def _observe(self) -> int: + await super()._observe() + self._rc.news = [ + msg for msg in self._rc.news if msg.send_to == self.profile + ] # only relevant msgs count as observed news + return len(self._rc.news) + ``` +7. `restricted_to`被用作群发(一发多)的路由参数,用来从共享消息队列中筛选发给自己的消息: + ```Python + async def _think(self): + news = self._rc.news[0] + assert news.cause_by == InstructSpeak # 消息为来自Moderator的指令时,才去做动作 + if not news.restricted_to: + # 消息接收范围为全体角色的,做公开发言(发表投票观点也算发言) + self._rc.todo = Speak() + elif self.profile in news.restricted_to.split(","): + # FIXME: hard code to split, restricted为"Moderator"或"Moderator,角色profile" + # Moderator加密发给自己的,意味着要执行角色的特殊动作 + self._rc.todo = self.special_actions[0]() + ``` + +##### 2.1.3.2 新的消息结构 + +新的消息结构如图所示: + +![img](../public/image/rfcs/rfc116/4edb8a4c-245b-4d02-a065-8e39937e732e.png) + +其中: + +1. 为了支持`Message`类的JSON化,将`cause_by`的类型替换成str,并封装了自动类型转换的功能; +2. 将原来`send_to`的str类型改为`Set`,以支持多标签“或关系”的消息订阅功能。如果`Message`对象在创建时未设置`send_to`的值,则默认是全员可接收(与现有公共memory的效果相同)。 +3. 本次修改,MetaGPT的消息处理参照了消息队列的定义,明确了消息的生产和消费关系。 + + a. 新框架中,所有`Role`类及其子类都是消费者,它们默认订阅了发送给全员、自己对象的类名字、自己对象的`name`属性值的消息,见缺省的消息订阅标签章。 + + b. 所有`Message`对象的创建者都是消息生产者,它们可以通过设置`Message`对象的`send_to`的值来调整让谁来消费这条消息。 + +下面是一些新Message类的使用示例: + +1. 设置`cause_by`属性: + +```Python +m = Message("a", cause_by=Action) +assert m.cause_by == get_class_name(Action) +assert m.cause_by == any_to_str(Action) +``` + +2. 设置`sent_from`属性: + +```Python +m = Message("a", sent_from=Action) +assert m.sent_from == get_class_name(Action) +assert m.sent_from == any_to_str(Action) +``` + +3. 设置`sent_to`属性: + +```Python +m = Message("a", sent_to={"b", Action}) +assert m.send_to == {"b", get_class_name(Action)} +assert m.send_to == any_to_str_set({"b", Action}) +``` + +###### 2.1.3.2.1 缺省的消息订阅标签 + +为了简化业务开发工作,框架默认提供了3种消息订阅标签。 +所有`Role`及其子类都已经订阅了这3种标签,开发者可在发消息时直接使用: + +1. `Role`对象的`name`属性:对应`Role`类及其子类对象的`name`属性的值。 +2. `Role`类的类名:对应任何`Role`类及其子类对象的类名。比如"Architect"类的类名为"metagpt.roles.architect.Architect"。 +3. 全员:值为"<all>"。本地所有消费者,都能收到这个消息。 + +示例如下: + + + +
+假设有如下几个`Role类`的子类对象: + +1. `Moderator`类对象 `r1`, `r1`.`name`为`a`; +2. `Werewolf`类对象 `r2`, `r2`.`name`为`b`; +3. `Werewolf`类对象 `r3`, `r3`.`name`为`c`; +4. `Villager`类对象 `r4`, `r4`.`name`为`d`; +5. `Villager`类对象 `r5`, `r5`.`name`为`e`; +6. `Seer`类对象 `r6`, `r6`.`name`为`f`; + +已知`Environment`类对象`env`: + +1. `r1`发消息给所有`Werewolf`类对象: + +```Python +env.publish_message(Message(content="...", send_to=Werewolf)) +``` + +这里的`Werewolf`用的就是`Role`类的类名做标签。这个消息的接收者是所有的`Werewolf`类对象。 + +2. `r1`发消息给`c`和所有`Villager`类对象: + +```Python +env.publish_message(Message(content="...", send_to={Villager, "c"})) +``` + +这里的`Villager`是`Role`类的类名标签,`c`是`Role`对象`r3`的`name`属性。这个消息的接收者是所有`Villager`类对象,以及`name`属性为`c`的`Role`对象`r3`。 + +3. `r1`发消息给所有对象: + +```Python +env.publish_message(Message(content="...", send_to="")) +``` + +这个消息的接收者为所有群成员。 + +6. `r1`发消息给`c`, `d`, `e`: + +```Python +env.publish_message(Message(content="...", send_to={"c", "d", "e"})) +``` + +这个消息的接收者为`r3`、`r4`、`r5`。 + +
+ +### 2.2 MetaGPT需改造的模块 + +#### 2.2.1 Message结构 + +参考`新的消息结构`章节。 + +新的消息处理框架下,缺省情况下`Role`类对象会收到所有消息(因为缺省的`Message`.`send_to`是全员)。因此`Role`类的`_observe`里需要把自己关注的新增消息过滤出来,并以此判断是否有消息需要处理。 + +代码如下: + +```Python +async def _observe(self) -> int: + """Prepare new messages for processing from the message buffer and other sources.""" + # Read unprocessed messages from the msg buffer. + news = self._rc.msg_buffer.pop_all() + # Store the read messages in your own memory to prevent duplicate processing. + self._rc.memory.add_batch(news) + # Filter out messages of interest. + self._rc.news = [n for n in news if n.cause_by in self._rc.watch] + + # Design Rules: + # If you need to further categorize Message objects, you can do so using the Message.set_meta function. + # msg_buffer is a receiving buffer, avoid adding message data and operations to msg_buffer. + news_text = [f"{i.role}: {i.content[:20]}..." for i in self._rc.news] + if news_text: + logger.debug(f"{self._setting} observed: {news_text}") + return len(self._rc.news) +``` + +#### 2.2.2 将公共消息存储改造成Role私有的消息存储 + +1. 取消公共消息存储,改为`Role`私有的消息存储。 + +```Python +class RoleContext(BaseModel): + """Role Runtime Context""" + env: 'Environment' = Field(default=None) + memory: Memory = Field(default_factory=Memory) + long_term_memory: LongTermMemory = Field(default_factory=LongTermMemory) + state: int = Field(default=0) + todo: Action = Field(default=None) + watch: set[str] = Field(default_factory=set) + news: list[Type[Message]] = Field(default=[]) +``` + +`Role`中的所有`self._rc.env.memory`操作变更为`self._rc.memory`操作。 + +2. 所有消息转发都由`Environment`类的`Env`对象负责。禁止`Role`对象之间通过访问对方的私有消息存储来交换消息。 + +![img](../public/image/rfcs/rfc116/32d67809-99f6-496e-9800-2a28c8eb73c5.png) + +3. 所有`Role`对象增加一个私有的消息buffer,来接收异步`put_message`写入的消息。`Role`对象的observe操作(observe-think-act)需要同时看消息buffer和memory。 + +#### 2.2.3 取消超参k + +目前流程的终止依靠`Environment`的`run`的超参`k`和`CostManager`的成本超支检查。 + +```Python +async def run(self, k=1): + """处理一次所有信息的运行 + Process all Role runs at once + """ + # while not self.message_queue.empty(): + # message = self.message_queue.get() + # rsp = await self.manager.handle(message, self) + # self.message_queue.put(rsp) + for _ in range(k): + futures = [] + for role in self.roles.values(): + future = role.run() + futures.append(future) + + await asyncio.gather(*futures) +``` + +新的流程终止条件改为所有的`self.roles`对象的`think`都为空(`Role`对象私有的消息buffer为空,且无action),或者成本超支。 + +#### 2.2.4 规范状态数据的存储 + +1. 新建一个支持序列化和反序列化`save`/`load`的`StateContext`类 +2. 所有状态数据都需要放到这个类对象中。 +3. 仅限`Environment`类和`Role`类拥有这个类的对象。 diff --git "a/src/rfcs/RFC-135-MetaGPT Software Company\345\242\236\351\207\217\351\234\200\346\261\202\345\274\200\345\217\221\346\226\271\346\241\210\350\256\276\350\256\241.md" "b/src/rfcs/RFC-135-MetaGPT Software Company\345\242\236\351\207\217\351\234\200\346\261\202\345\274\200\345\217\221\346\226\271\346\241\210\350\256\276\350\256\241.md" new file mode 100644 index 00000000..2df3a8d7 --- /dev/null +++ "b/src/rfcs/RFC-135-MetaGPT Software Company\345\242\236\351\207\217\351\234\200\346\261\202\345\274\200\345\217\221\346\226\271\346\241\210\350\256\276\350\256\241.md" @@ -0,0 +1,1304 @@ +# [R]RFC-135-MetaGPT Software Company增量需求开发方案设计 + +**文档负责人** :马莘权 + +文档修改记录 + +| 日期 | 版本 | 作者 | 修改内容 | +| ---------- | ---- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------ | +| 2023-11-13 | v1.0 | 马莘权 | 创建 | +| 2023-11-16 | v1.1 | 马莘权 | 修改2.2.3 架构变动章节 | +| 2023-11-17 | v1.2 | 马莘权 | 增加2.2.3.13 输出件和消息分离 章节 | +| 2023-11-23 | v1.3 | 马莘权 | 增加2.2.3.5.1 WriteCode和WriteCodeReview改造 | +| 2023-12-1 | v1.4 | 马莘权 | 更新2.2.3.5 流程变更章节的时序图 | +| 2023-12-5 | v1.5 | 马莘权 | 更新2.2.3.5.5章节,Engineer增加SummarizeCode环节。 | +| 2023-12-13 | v1.6 | 马莘权 | 1. 更新2.2.3.4章节的docs/bugfix.txt
2. 更新2.2.3.5.2章节的WritePRD,新增对bug feedback的处理
3. 更新2.2.3.5.5章节,新增fix bug相关流程。 | + +## 1. **引言** + +### 1.1 **背景** + +现有的MetaGPT的Software Company仅支持从头重新开发软件。 +这种软件开发方式: + +1. 与实际的软件开发不符。软件开发过程是个反复迭代的过程,通过迭代重新认识需求、对齐目标和优化设计。 +2. 用户提出希望Software Company支持增量开发的需求。 + +### 1.2 **目标** + +MetaGPT的Software Company增加在历史版本基础上的增量开发的功能,包括: + +1. 多版本管理 +2. changelog/diff输出 +3. 版本切换 + +![img](../public/image/rfcs/rfc135/img.png) + +### 1.3 **非目标** + +LLM输出结果的优化非本设计的目标。 + +## 2. **系统设计** + +### 2.1 **设计介绍** + +#### 2.1.1 **竞品分析** + +无 + +#### 2.1.2 **现状** + +MetaGPT Software Company软件开发现有流程如下: + +![img](../public/image/rfcs/rfc135/img_1.png) + +##### 2.1.2.1 **Actions** + +###### 2.1.2.1.1 **WritePRD** + +输入:原始用户需求,比如说“snake game” + +输出: + +```json +{ + "Original Requirements": "Create a snake game", + "Search Information": "", + "Requirements": "", + "Product Goals": [ + "Provide an enjoyable gaming experience", + "Ensure smooth and responsive controls", + "Include engaging visuals and sound effects" + ], + "User Stories": [ + "As a player, I want to control the snake using arrow keys to navigate through the game board", + "As a player, I want the snake to grow longer and increase in speed as it consumes food", + "As a player, I want to see my high score and be able to compare it with other players", + "As a player, I want to have different levels of difficulty to challenge myself", + "As a player, I want to be able to pause and resume the game at any time" + ], + "Competitive Analysis": [ + "Snake Game X has a simple and intuitive user interface", + "Snake Game Y offers multiple game modes and power-ups", + "Snake Game Z has a leaderboard feature to compare scores with friends", + "Snake Game A has smooth and responsive controls", + "Snake Game B has visually appealing graphics and animations", + "Snake Game C offers customizable snake skins", + "Snake Game D has challenging obstacles and mazes" + ], + "Competitive Quadrant Chart": "quadrantChart\n title Reach and engagement of snake games\n x-axis Low Reach --> High Reach\n y-axis Low Engagement --> High Engagement\n quadrant-1 Snake Game X: [0.3, 0.6]\n quadrant-2 Snake Game Y: [0.45, 0.23]\n quadrant-3 Snake Game Z: [0.57, 0.69]\n quadrant-4 Snake Game A: [0.78, 0.34]\n Snake Game B: [0.40, 0.34]\n Snake Game C: [0.35, 0.78]\n Snake Game D: [0.5, 0.6]", + "Requirement Analysis": "", + "Requirement Pool": [ + [ + "P0", + "The snake should move smoothly and responsively when controlled by the player." + ], + [ + "P1", + "The game should have different levels of difficulty to cater to players of different skill levels." + ] + ], + "UI Design draft": "The game will have a simple and clean user interface. It will consist of a game board where the snake moves and food appears. The score and high score will be displayed at the top of the screen. There will be buttons to pause and resume the game. The snake and food will have visually appealing graphics.", + "Anything UNCLEAR": "" +} +``` + +###### 2.1.2.1.2 **WriteDesign** + +输入:WritePRD输出 +输出: + +```json +{ + "Implementation approach": "We will use the Pygame library to implement the snake game. Pygame is a popular open-source library for game development in Python. It provides a simple and intuitive API for handling graphics, sound, and user input. Pygame also has good documentation and a large community, making it easy to find resources and support. Additionally, we will follow the PEP8 coding style guidelines to ensure clean and readable code.", + "Python package name": "snake_game", + "File list": ["main.py"], + "Data structures and interface definitions": ''' + classDiagram + class Game{ + -int score + -Snake snake + -Food food + +start_game() : void + +end_game() : void + +pause_game() : void + +resume_game() : void + +move_snake(direction: str) : void + +eat_food() : void + +update_score() : void + } + + class Snake{ + -List[Tuple[int, int]] body + -str direction + +move() : void + +change_direction(new_direction: str) : void + +grow() : void + +collides_with_self() : bool + +collides_with_food(food_position: Tuple[int, int]) : bool + } + + class Food{ + -Tuple[int, int] position + +generate() : void + +get_position() : Tuple[int, int] + } + + Game "1" -- "1" Snake: has + Game "1" -- "1" Food: has + ''', + "Program call flow": ''' + sequenceDiagram + participant M as Main + participant G as Game + participant S as Snake + participant F as Food + + M->>G: start_game() + G->>S: move_snake(direction) + S->>S: move() + S-->>G: move_snake() + G->>S: eat_food() + S->>S: grow() + S->>S: [asdfsa]collides_with_self(asdf) + S->>F: collides_with_food() + F->>F: generate() + G->>G: update_score() + G->>M: end_game() + G->>M: pause_game() + G->>M: resume_game() + ''', + "Anything UNCLEAR": "The requirements are clear to me." +} +``` + +其中:
+    a. `classDiagram`使用的是[mermaid](https://github.com/mermaid-js/mermaid/blob/develop/README.zh-CN.md)类视图语法;
+    b. `sequenceDiagram`使用的是[mermaid](https://github.com/mermaid-js/mermaid/blob/develop/README.zh-CN.md)时序图语法;
+    c. `metagpt/actions/design_api.py`的`_save`中,用`recreate_workspace`函数清空了输出文件夹。 + +输出文件: + +1. `workspace/snake_game/docs/prd.md` + +```markdown +## Original Requirements + +Design a snake game + +## Product Goals + +- Provide an engaging and addictive gameplay experience +- Offer intuitive controls for easy navigation +- Include various levels of difficulty to cater to different players + +## User Stories + +- As a player, I want to control the snake's movement using arrow keys +- As a player, I want to see my score increase as I eat the food +- As a player, I want to challenge myself with different levels of difficulty +- As a player, I want to compete with my friends for the highest score +- As a player, I want to be able to pause and resume the game + +## Competitive Analysis + +- Snake Game X has a simple and intuitive user interface +- Snake Game Y offers multiple game modes and power-ups +- Snake Game Z has a leaderboard feature for competitive play +- Snake Game A has smooth and responsive controls +- Snake Game B provides customizable snake skins +- Snake Game C offers different themes and backgrounds +- Snake Game D has a tutorial mode for beginners + +## Competitive Quadrant Chart + +quadrantChart + title Reach and engagement of snake games + x-axis Low Reach --> High Reach + y-axis Low Engagement --> High Engagement + quadrant-1 Expand + quadrant-2 Promote + quadrant-3 Re-evaluate + quadrant-4 Improve + Snake Game X: [0.6, 0.7] + Snake Game Y: [0.8, 0.5] + Snake Game Z: [0.7, 0.8] + Snake Game A: [0.6, 0.6] + Snake Game B: [0.5, 0.6] + Snake Game C: [0.4, 0.5] + Snake Game D: [0.5, 0.4] + Our Target Game: [0.7, 0.7] + +## Requirement Analysis + +## Requirement Pool + +- ['P0', 'The snake should move smoothly and responsively'] +- ['P1', 'The game should have multiple levels of difficulty'] +- ['P2', 'The game should have a leaderboard feature for competitive play'] + +## UI Design draft + +The game will have a simple and clean user interface. It will consist of a game board where the snake and food will be displayed. The score will be shown at the top of the screen. The controls will be arrow keys for movement. The game board will have a grid layout to represent the snake's movement. The colors used will be vibrant and visually appealing. The UI will be designed to be intuitive and easy to navigate. + +## Anything UNCLEAR +``` + +2. `workspace/snake_game/resources/competitive_analysis.mmd` + +```text +quadrantChart + title Reach and engagement of snake games + x-axis Low Reach --> High Reach + y-axis Low Engagement --> High Engagement + quadrant-1 Expand + quadrant-2 Promote + quadrant-3 Re-evaluate + quadrant-4 Improve + Snake Game X: [0.6, 0.7] + Snake Game Y: [0.8, 0.5] + Snake Game Z: [0.7, 0.8] + Snake Game A: [0.6, 0.6] + Snake Game B: [0.5, 0.6] + Snake Game C: [0.4, 0.5] + Snake Game D: [0.5, 0.4] + Our Target Game: [0.7, 0.7] +``` + +3. `workspace/snake_game/resources/data_api_design.mmd` + +```text +classDiagram + class Game{ + -int score + -Snake snake + -Food food + +start_game() : void + +end_game() : void + +pause_game() : void + +resume_game() : void + +move_snake(direction: str) : void + +eat_food() : void + +update_score() : void + } + + class Snake{ + -List[Tuple[int, int]] body + -str direction + +move() : void + +change_direction(new_direction: str) : void + +grow() : void + +collides_with_self() : bool + +collides_with_food(food_position: Tuple[int, int]) : bool + } + + class Food{ + -Tuple[int, int] position + +generate() : void + +get_position() : Tuple[int, int] + } + + Game "1" -- "1" Snake: has + Game "1" -- "1" Food: has +``` + +4. `workspace/snake_game/resources/seq_flow.mmd` + +```text +sequenceDiagram + participant M as Main + participant G as Game + participant S as Snake + participant F as Food + + M->>G: start_game() + G->>S: move_snake(direction) + S->>S: move() + S-->>G: move_snake() + G->>S: eat_food() + S->>S: grow() + S->>S: collides_with_self() + S->>F: collides_with_food() + F->>F: generate() + G->>G: update_score() + G->>M: end_game() + G->>M: pause_game() + G->>M: resume_game() +``` + +5. `workspace/snake_game/docs/system_design.md` + +```markdown +## Implementation approach + +We will use the Pygame library to implement the snake game. Pygame is a popular open-source library for game development in Python. It provides a simple and intuitive API for handling graphics, sound, and user input. Pygame also has good documentation and a large community, making it easy to find resources and support. Additionally, we will follow the PEP8 coding style guidelines to ensure clean and readable code. + +## Python package name + +snake_game + +## File list + +- main.py + +## Data structures and interface definitions + + classDiagram + class Game{ + -int score + -Snake snake + -Food food + +start_game() : void + +end_game() : void + +pause_game() : void + +resume_game() : void + +move_snake(direction: str) : void + +eat_food() : void + +update_score() : void + } + + class Snake{ + -List[Tuple[int, int]] body + -str direction + +move() : void + +change_direction(new_direction: str) : void + +grow() : void + +collides_with_self() : bool + +collides_with_food(food_position: Tuple[int, int]) : bool + } + + class Food{ + -Tuple[int, int] position + +generate() : void + +get_position() : Tuple[int, int] + } + + Game "1" -- "1" Snake: has + Game "1" -- "1" Food: has + +## Program call flow + + sequenceDiagram + participant M as Main + participant G as Game + participant S as Snake + participant F as Food + + M->>G: start_game() + G->>S: move_snake(direction) + S->>S: move() + S-->>G: move_snake() + G->>S: eat_food() + S->>S: grow() + S->>S: collides_with_self() + S->>F: collides_with_food() + F->>F: generate() + G->>G: update_score() + G->>M: end_game() + G->>M: pause_game() + G->>M: resume_game() + +## Anything UNCLEAR + +The requirements are clear to me. +``` + +###### 2.1.2.1.3 **WriteTasks** + +输入:WriteDesign输出 +输出: + +1. `workspace/snake_game/docs/api_spec_and_tasks.md` + +```json + +{ + "Required Python third-party packages": [ + "pygame==2.0.1" + ], + "Required Other language third-party packages": [ + "No third-party ..." + ], + "Full API spec": """ + openapi: 3.0.0 + ... + description: A JSON object ... + """, + "Logic Analysis": [ + ["main.py", "Game"], + ["main.py", "Snake"], + ["main.py", "Food"] + ], + "Task list": [ + "main.py" + ], + "Shared Knowledge": """ + 'main.py' contains the implementation of the Game class, Snake class, and Food class. The Game class is responsible for managing the game flow, including starting, ending, pausing, and resuming the game. The Snake class represents the snake in the game, handling its movement, growth, and collision detection. The Food class represents the food in the game, generating new positions and checking for collisions with the snake. + """, + "Anything UNCLEAR": "No unclear points." +} +``` + +2. `workspace/snake_game/requirements.txt` + +```markdown +pygame==2.0.1 +``` + +###### 2.1.2.1.4 **WriteCode** + +输入:WriteTasks输出 + WriteDesign输出+ 已有Code memory + +```text + +{ + "Required Python third-party packages": [ + "pygame==2.0.1" + ], + "Required Other language third-party packages": [ + "No third-party ..." + ], + "Full API spec": """ + openapi: 3.0.0 + ... + description: A JSON object ... + """, + "Logic Analysis": [ + ["main.py", "Game"], + ["main.py", "Snake"], + ["main.py", "Food"] + ], + "Task list": [ + "main.py" + ], + "Shared Knowledge": """ + 'main.py' contains the implementation of the Game class, Snake class, and Food class. The Game class is responsible for managing the game flow, including starting, ending, pausing, and resuming the game. The Snake class represents the snake in the game, handling its movement, growth, and collision detection. The Food class represents the food in the game, generating new positions and checking for collisions with the snake. + """, + "Anything UNCLEAR": "No unclear points." +} + + +{ + "Implementation approach": "We will use the Pygame library to implement the snake game. Pygame is a popular open-source library for game development in Python. It provides a simple and intuitive API for handling graphics, sound, and user input. Pygame also has good documentation and a large community, making it easy to find resources and support. Additionally, we will follow the PEP8 coding style guidelines to ensure clean and readable code.", + "Python package name": "snake_game", + "File list": ["main.py"], + "Data structures and interface definitions": ''' + classDiagram + class Game{ + -int score + -Snake snake + -Food food + +start_game() : void + +end_game() : void + +pause_game() : void + +resume_game() : void + +move_snake(direction: str) : void + +eat_food() : void + +update_score() : void + } + + class Snake{ + -List[Tuple[int, int]] body + -str direction + +move() : void + +change_direction(new_direction: str) : void + +grow() : void + +collides_with_self() : bool + +collides_with_food(food_position: Tuple[int, int]) : bool + } + + class Food{ + -Tuple[int, int] position + +generate() : void + +get_position() : Tuple[int, int] + } + + Game "1" -- "1" Snake: has + Game "1" -- "1" Food: has + ''', + "Program call flow": ''' + sequenceDiagram + participant M as Main + participant G as Game + participant S as Snake + participant F as Food + + M->>G: start_game() + G->>S: move_snake(direction) + S->>S: move() + S-->>G: move_snake() + G->>S: eat_food() + S->>S: grow() + S->>S: collides_with_self() + S->>F: collides_with_food() + F->>F: generate() + G->>G: update_score() + G->>M: end_game() + G->>M: pause_game() + G->>M: resume_game() + ''', + "Anything UNCLEAR": "The requirements are clear to me." +} + +----------- +# a.py +.... +----------- +# b.py +.... +----------- +``` + +输出:\*.py + +```python +""" +main.py +""" + +import pygame +import random + +class Game: + def __init__(self): + self.score = 0 + self.snake = Snake() + self.food = Food() + + def start_game(self): + pygame.init() + self.screen = pygame.display.set_mode((800, 600)) + pygame.display.set_caption("Snake Game") + self.clock = pygame.time.Clock() + self.running = True + + while self.running: + for event in pygame.event.get(): + if event.type == pygame.QUIT: + self.end_game() + elif event.type == pygame.KEYDOWN: + if event.key == pygame.K_p: + self.pause_game() + elif event.key == pygame.K_r: + self.resume_game() + elif event.key == pygame.K_UP: + self.move_snake("UP") + elif event.key == pygame.K_DOWN: + self.move_snake("DOWN") + elif event.key == pygame.K_LEFT: + self.move_snake("LEFT") + elif event.key == pygame.K_RIGHT: + self.move_snake("RIGHT") + + self.screen.fill((0, 0, 0)) + self.snake.move() + self.check_collision() + self.draw_snake() + self.draw_food() + self.update_score() + pygame.display.update() + self.clock.tick(10) + + def end_game(self): + pygame.quit() + self.running = False + + def pause_game(self): + self.running = False + + def resume_game(self): + self.running = True + + def move_snake(self, direction): + self.snake.change_direction(direction) + + def eat_food(self): + self.snake.grow() + self.food.generate() + self.score += 1 + + def update_score(self): + font = pygame.font.Font(None, 36) + text = font.render(f"Score: {self.score}", True, (255, 255, 255)) + self.screen.blit(text, (10, 10)) + + def check_collision(self): + if self.snake.collides_with_self(): + self.end_game() + elif self.snake.collides_with_food(self.food.get_position()): + self.eat_food() + + def draw_snake(self): + for segment in self.snake.body: + pygame.draw.rect(self.screen, (0, 255, 0), (segment[0], segment[1], 20, 20)) + + def draw_food(self): + pygame.draw.rect(self.screen, (255, 0, 0), (self.food.position[0], self.food.position[1], 20, 20)) + + +class Snake: + def __init__(self): + self.body = [(100, 100), (80, 100), (60, 100)] + self.direction = "RIGHT" + + def move(self): + if self.direction == "UP": + head = (self.body[0][0], self.body[0][1] - 20) + elif self.direction == "DOWN": + head = (self.body[0][0], self.body[0][1] + 20) + elif self.direction == "LEFT": + head = (self.body[0][0] - 20, self.body[0][1]) + elif self.direction == "RIGHT": + head = (self.body[0][0] + 20, self.body[0][1]) + + self.body.insert(0, head) + self.body.pop() + + def change_direction(self, new_direction): + if new_direction == "UP" and self.direction != "DOWN": + self.direction = new_direction + elif new_direction == "DOWN" and self.direction != "UP": + self.direction = new_direction + elif new_direction == "LEFT" and self.direction != "RIGHT": + self.direction = new_direction + elif new_direction == "RIGHT" and self.direction != "LEFT": + self.direction = new_direction + + def grow(self): + tail = self.body[-1] + self.body.append(tail) + + def collides_with_self(self): + head = self.body[0] + return head in self.body[1:] + + def collides_with_food(self, food_position): + head = self.body[0] + return head == food_position + + +class Food: + def __init__(self): + self.position = (random.randint(1, 39) * 20, random.randint(1, 29) * 20) + + def generate(self): + self.position = (random.randint(1, 39) * 20, random.randint(1, 29) * 20) + + def get_position(self): + return self.position + + +if __name__ == "__main__": + game = Game() + game.start_game() + +``` + +##### 2.1.2.2 **PRD** + +PRD是WritePRD action输出的结果。 + +PRD的内容包括如下几部分: + + +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
字段名是否可增量修改字段值示例
Original Requirements"Create a snake game"
Search Information""
Project Name如果想修改,请通过CLI命令参数 --project-name进行
Requirements""
Product Goals +[
+    "Provide an enjoyable gaming experience",
+    "Ensure smooth and responsive controls",
+    "Include engaging visuals and sound effects"
+]
User Stories +[
+    "As a player, I want to control the snake using arrow keys to navigate +through the game board",
+    "As a player, I want the snake to grow longer and increase in speed as +it consumes food",
+    "As a player, I want to see my high score and be able to compare it with +other players",
+    "As a player, I want to have different levels of difficulty to challenge +myself",
+    "As a player, I want to be able to pause and resume the game at any +time"
+]
Competitive Analysis +[
+    "Snake Game X has a simple and intuitive user interface",
+    "Snake Game Y offers multiple game modes and power-ups",
+    "Snake Game Z has a leaderboard feature to compare scores with +friends",
+    "Snake Game A has smooth and responsive controls",
+    "Snake Game B has visually appealing graphics and animations",
+    "Snake Game C offers customizable snake skins",
+    "Snake Game D has challenging obstacles and mazes"
+]
Competitive Quadrant Chart +"quadrantChart\n title Reach and engagement of snake games\n x-axis Low Reach --> High Reach\n y-axis Low Engagement --> High Engagement\n quadrant-1 Snake Game X: [0.3, 0.6]\n quadrant-2 Snake Game Y: [0.45, 0.23]\n quadrant-3 Snake Game Z: [0.57, 0.69]\n quadrant-4 Snake Game A: [0.78, 0.34]\n Snake Game B: [0.40, 0.34]\n Snake Game C: [0.35, 0.78]\n Snake Game D: [0.5, 0.6]" +
Requirement Analysis""
Requirement Pool +[
+    [
+        "P0",
+        "The snake should move smoothly and responsively when controlled by the +player."
+    ],
+    [
+        "P1",
+        "The game should have different levels of difficulty to cater to players +of different skill levels."
+    ]
+]
UI Design draft +"The game will have a simple and clean user interface. It will consist +of a game board where the snake moves and food appears. The score and +high score will be displayed at the top of the screen. There will be +buttons to pause and resume the game. The snake and food will have +visually appealing graphics."
Anything UNCLEAR""
+ +##### 2.1.2.3 **[Mermaid](https://mermaid.js.org/syntax/classDiagram.html)类视图和时序图** + +类视图和时序图都是WriteDesign的输出的一部分。 + +类视图与时序图都使用了基于[Mermaid](https://mermaid.js.org/syntax/classDiagram.html)的纯文本UML语法,它描述了某个use +case下的控制流和数据流。因此,对类视图或时序图的修改,需要同时重写类视图和时序图。 + +类视图示例: + +```text +classDiagram + class Game{ + -int score + -Snake snake + -Food food + +start_game() : void + +end_game() : void + +pause_game() : void + +resume_game() : void + +move_snake(direction: str) : void + +eat_food() : void + +update_score() : void + } + + class Snake{ + -List[Tuple[int, int]] body + -str direction + +move() : void + +change_direction(new_direction: str) : void + +grow() : void + +collides_with_self() : bool + +collides_with_food(food_position: Tuple[int, int]) : bool + } + + class Food{ + -Tuple[int, int] position + +generate() : void + +get_position() : Tuple[int, int] + } + + Game "1" -- "1" Snake: has + Game "1" -- "1" Food: has +``` + +时序图示例: + +```text +sequenceDiagram + participant M as Main + participant G as Game + participant S as Snake + participant F as Food + + M->>G: start_game() + G->>S: move_snake(direction) + S->>S: move() + S-->>G: move_snake() + G->>S: eat_food() + S->>S: grow() + S->>S: collides_with_self() + S->>F: collides_with_food() + F->>F: generate() + G->>G: update_score() + G->>M: end_game() + G->>M: pause_game() + G->>M: resume_game() +``` + +##### 2.1.2.4 **WriteTasks** + +WriteTasks的输出定义了类视图中的类对应的文件名。其中,Logic +Analysis中的类名引用的是WriteDesign类视图中的类名。 + +修改WriteTasks仅能改变要生成那些代码文件的逻辑。 + +```json + "Logic Analysis": [ + ["main.py", "Game"], + ["main.py", "Snake"], + ["main.py", "Food"] + ], + "Task list": [ + "main.py" + ], +``` + +##### 2.1.2.5 **WriteCode和QA** + +WriteCode依赖上游的类视图和时序图。 +如果人工修改WriteCode的输出代码,那么这种修改必须是基于类视图和时序图的。比如修改代码bug。 +QA阶段是对WriteCode的输出进行测试,这里可以增加人机实时交互,以指导AI丰富测试方式和测试方向。 +但对于简单的代码而言,重跑QA要比人机交互方式的QA要简单直接。 + +### 2.2 **系统架构** + +#### 2.2.1 **前提约束** + +1. MetaGPT的控制台交互方式决定了用户仅能**通过文本方式来发起增量变更**; +2. PRD使用的Mermaid语法缺少Use Case Diagram和Activity + Diagram描述能力,决定了所开发的**软件复杂度不高**; + +![img](../public/image/rfcs/rfc135/img_2.png) + +#### 2.2.2 **可进行的变更** + +基于上面的框架前提约束,再结合`现状`章节中各个action的输入输出关系,Software +Company可支持的变更如下: + +1. 在Write PRD阶段,变更:
+     a. `Requirement Pool`(p0)
+     b. `Product Goals`
+     c. `User Stories`
+     d. `UI Design draft` + +2. 在Write Design阶段,变更:
+     a. 类视图中的类型、命名、返回值类型等
+     b. 在时序图中变更流程的跳转条件、交互步骤等 + +3. 在Write Code及其后阶段:
+     a. 重新生成指定文件的代码、QA
+     b. 修改指定文件的代码,并重跑其QA + +#### 2.2.3 **架构变动** + +##### 2.2.3.1 **变动原则** + +1. 将旧框架中Message负责传递数据内容的方式,调整为**用Message来传递参考文件名**,由action从参考文件中加载数据内容的方式; +2. 所有**增量信息**,都先落地成文件,通过**文件名来引用**和使用。 +3. **token消耗最小化**。 + +###### 2.2.3.1.1 **消息编号** + +将文件数据从Message内剥离后,会出现消息内容相同但事件意义不同的情况,如下图所示: + +![img](../public/image/rfcs/rfc135/img_3.png) + +为此,在Message中增加id属性,来区分是否是同一个Message。 + +```python +class Message(BaseModel): + """list[: ]""" + + id: str + content: str + instruct_content: BaseModel = Field(default=None) + role: str = "user" # system / user / assistant + cause_by: str = "" + sent_from: str = "" + send_to: Set = Field(default_factory={MESSAGE_ROUTE_TO_ALL}) +``` + +##### 2.2.3.2 **`.dependencies.json`文件** + +根据本次框架的变动原则,在当前项目的输出根目录下,新增`.dependencies.json`文件。`.dependencies.json`文件可通过`FileReporitory`类访问。关于`FileReporitory`类,参见项目文件夹结构章节。 +`.dependencies.json`文件中存放了文件**显式依赖**关系。 +其JSON格式如下: + +```json +{ + "": ["", "", ...] +} +``` + +比如`docs/system_design.md`是`WriteDesign` action根据`docs/prd.md`生成的,所以有这样一条记录: + +```json +{ + "docs/system_design.md": ["docs/prd.md"] +} +``` + +**禁止使用隐性依赖**来描述文件之间的依赖关系。 +假如“a”文件依赖“b”、“c”文件;“b”文件依赖“c”文件。 +正确的`.dependencies.json`文件内容如下: + +```json +{ + "a": ["b", "c"], + "b": ["c"] +} +``` + +**不支持隐性依赖**,比如下面这种写法就是错误的: + +```json +{ + "a": ["b"], + "b": ["c"] +} +``` + +这种写法将被解读为“a”文件依赖“b”文件;“b”文件依赖“c”文件。 + +##### 2.2.3.3 **项目文件的版本管理** + +当用户想在某个项目版本上,继续迭代,那么这个被选中的项目版本就称为“基线版本”。 +用户对基线版本的变更可通过两种方式进行: +通过main参数idea,传入增量的需求内容。 +直接修改基线版本的项目的文件。 +本文设计采用git来管理项目版本(参阅`集成git功能`章节): + +1. 以解决用户在项目迭代过程中的版本管理需求; +2. 方便MetaGPT对用户在基线版本上的修改进行识别: + +```shell +>>> from git import Repo +>>> repo_path = '/Users/iorishinier/github/MetaGPT' +>>> repo = Repo(repo_path) +>>> diff = repo.index.diff(None) +>>> for item in diff: +... print(f"文件:{item.a_path}, 差异:{item.change_type}") +... print(f"差异内容:\n{item.diff}\n") +... +文件:config/config.yaml, 差异:M +差异内容: + + +文件:requirements.txt, 差异:M +差异内容: + + +>>> + +``` + +##### 2.2.3.4 **项目文件夹结构** + +| 路径 | 用户可编辑 | 说明 | +| ------------------------------ | ---------- | ------------------------------------------------------------------------------------- | +| .dependencies.json | 否 | 存储文件间的显式依赖关系 | +| docs/requirement.txt | 是 | 存放本轮**新增的需求**。项目处理过程中,里面的内容会被拆分、合并到docs/prds/下 | +| docs/bugfix.txt | 否 | 存放本轮bug feedback信息。
迭代时作将bug feedback为需求输入,不要直接改这个文件。 | +| docs/prds | 是 | 项目最终拆分完的需求 | +| docs/system_designs | 是 | 项目最终的系统设计 | +| docs/tasks | 是 | 项目的编码任务 | +| docs/code_summaries | 是 | 对全部代码的复盘结果 | +| resources/competitive_analysis | 否 | 竞品分析 | +| resources/data_api_design | 否 | 类视图文档 | +| resources/seq_flow | 否 | 时序图文档 | +| resources/system_design | 否 | 系统设计文档 | +| resources/prd | 否 | PRD文档 | +| resources/api_spec_and_tasks | 否 | 编码任务的tasks文档 | +| tmp | 否 | 项目处理过程中生成的中间文件。这些文件不会被git归档。 | +| <workspace> | 是 | 项目源代码文件夹 | +| tests | 是 | 单元测试代码 | +| test_outputs | 否 | 单测执行的结果 | + +##### 2.2.3.5 **流程变更** + +![img](../public/image/rfcs/rfc135/img_4.png) + +整体流程调整: + +1. `ProductManager`对象**新增**`PrepareDocuments` Action类; +2. 所有输出文档(不包括代码)均采用**可回溯的文件命名**方式,为防止文件名冲突,**采用(YYYYmmddHHMMSS)时间格式的文件名**。比如: + +```shell +docs/prds/202311161517.md +docs/system_designs/202311161517.md +tmp/class_diagrams/202311161517.mmd +tmp/sequence_diagrams/202311161517.mmd +``` + +3. 项目完整的文件夹结构示例如下所示: + ![img](../public/image/rfcs/rfc135/img_5.png) + +对项目文件夹的访问操作将封装成`FileRepository`类。 + +###### 2.2.3.5.1 **PrepareDocuments** + +这个action负责: + +1. 如果main参数没有传入`project_path`参数(参阅`新增main参数`章节),那么action需要创建和初始化workspace文件夹、初始化git环境; +2. 将main参数`idea`中的新增需求写入`docs/requirement.txt` +3. 发送`Message`消息,通知`WritePRD` action使用`docs/requirement.txt`和`docs/prds/`处理需求。 +4. `WritePRD` action的下游action可通过git head diff(封装在`FileRepository`里)来判断文件是否被修改过。参考下面的示例: + +```shell +>>> file_path = 'config/config.yaml' +>>> head_commit = repo.head.commit +>>> file_content = head_commit.tree[file_path].data_stream.read().decode('utf-8') +>>> print(f"HEAD 版本的 {file_path} 内容:\n{file_content}") +HEAD 版本的 config/config.yaml 内容: +# DO NOT MODIFY THIS FILE, create a new key.yaml, define OPENAI_API_KEY. +# The configuration of key.yaml has a higher priority and will not enter git + +#### if OpenAI +## The official OPENAI_API_BASE is https://api.openai.com/v1 +## If the official OPENAI_API_BASE is not available, we recommend using the [openai-forward](https://github.com/beidongjiedeguang/openai-forward). +## Or, you can configure OPENAI_PROXY to access official OPENAI_API_BASE. +OPENAI_API_BASE: "https://api.openai.com/v1" +#OPENAI_PROXY: "http://127.0.0.1:8118" +#OPENAI_API_KEY: "YOUR_API_KEY" +OPENAI_API_MODEL: "gpt-4" + +``` + +Action结束后,`docs/requirement.txt`可用、workspace可用。 +![img](../public/image/rfcs/rfc135/img_6.png) + +###### 2.2.3.5.2 **WritePRD** + +这个action负责: + +1. 判断当前的输入是bug feedback还是需求。如果是bug feedback,则发消息给`Engineer` role处理。 +2. 判断哪些需求文档需要重写:调LLM判断新增需求与prd是否相关,若相关就rewrite + prd。 + +![img](../public/image/rfcs/rfc135/img_7.png) + +其中:
+    a. 文件名优先使用旧文件名。如果没有对应的旧文件,新建时间格式的文件名。这条规则实际上是通过规则来表达dependency,效果等价于`2.2.3.2 .denpendencies.json`文件。
+    b. 旧方案中,PRD相关的文档保存被放到了`WriteDesign`中,新方案将功能移到`WritePRD`中。由于新方案允许需求分片处理,因此,原`resources`下的`competitive_analysis.mmd`,改为`resources/competitive_analysis/`下的多个文件;供人阅读的markdown格式的PRD文档放在`resources/prd/`下。 + +3. 如果没有任何PRD,就使用`docs/requirement.txt`生成一个prd。 +4. 等`docs/prds/`下所有文件都与新增需求对比完后,再触发publish message让工作流跳转到下一环节。如此设计是为了给后续做全局优化留空间。 + +Action结束后,`docs/prds/`可用。 + +![img](../public/image/rfcs/rfc135/img_8.png) + +###### 2.2.3.5.3 **`WriteDesign`** + +这个action负责: + +1. 通过git diff来识别`docs/prds`下哪些PRD文档发生了变动; +2. 通过git diff来识别`docs/system_designs`下那些设计文档发生了变动; +3. 对于那些发生变动的PRD和设计文档,重新生成设计内容; + +![img](../public/image/rfcs/rfc135/img_9.png) + +    a. 由于新方案允许需求分片处理,因此:
+        i. 原`resources`下的`data_api_design.mmd`改为`resources/data_api_design/`下的多个文件;
+        ii. 原`resources`下的`seq_flow.mmd`改为`resources/seq_flow/`下的多个文件;
+        iii. 原`docs`下的`system\_design.md`改为`resources/system_design`下的多个文件。
+    b. 旧方案与PRD保存相关的操作移到`WritePRD`下。
4. 等`docs/system_designs/`下所有文件都处理完才发publish message,给后续做全局优化留空间。 + +Action结束后,`docs/system_designs/`可用。 + +![img](../public/image/rfcs/rfc135/img_10.png) + +###### 2.2.3.5.4 **`WriteTasks`** + +这个action负责: + +1. 根据`docs/system_designs/`下的git head diff识别哪些task文档需要重写; +2. 根据`docs/tasks/`下的git head diff识别哪些task文件被用户修改了,需要重写; +3. 对于那些需要重写的task文档,重新生成task内容; + +![img](../public/image/rfcs/rfc135/img_11.png) + +其中:
+    a. 用于`WriteCode`的原始数据文件保存在`docs/tasks/`。
+    b. 原流程将task的markdown文档内容保存到`docs/api_spec_and_tasks.md`中,由于新流程支持需求分片,因此将文档分片改放`resource/api_spec_and_tasks/`下;
+    c. 原流程会生成`requirements.txt`文件。新流程需要将每个分片生成的`requirements.txt`追加合并到同一个`requirements.txt`中。4. 等`docs/tasks/`下所有文件都处理完才发publish message,给后续做全局优化留空间。 +Action结束后,`docs/tasks/`可用。 + +![img](../public/image/rfcs/rfc135/img_12.png) + +###### 2.2.3.5.5 **`Engineer`** + +`Engineer`的`_observe`需要改造,关于消息内容的处理逻辑移到`_think`中: + +![img](../public/image/rfcs/rfc135/img_13.png) + +其中,`_new_code_actions`流程如下: + +![img](../public/image/rfcs/rfc135/img_14.png) + +`_new_summarize_actions`流程如下: + +![img](../public/image/rfcs/rfc135/img_15.png) + +![img](../public/image/rfcs/rfc135/img_16.png) + +1. 原流程默认只有一个task文件需要处理,现在默认有多个task文件。因此原流程中到memory中搜`WriteDesign`消息的流程已不适用。 +2. 新的流程将 “task文件”、“要写代码的文件”放在同一个`Document`类里,交给一个`WriteCode` action对象去执行。其中,“与task相关的`WriteDesign`设计文件”这个参数通过依赖关系由action对象自行查找。 + +![img](../public/image/rfcs/rfc135/img_17.png) + +3. 与旧流程一样,`WriteCode`action需基于已有的代码,来写当前代码。 +4. `WriteCode`在写代码时,会参考历史`SummarizeCode`发现的问题和用户反馈的bug feedback。 +5. `WriteCode`在保存代码的时候,需要更新文件依赖关系。 +6. `WriteCode`在写代码的时候,如果git diff发现code有变更,需要让LLM参考code文件内容进行重写。 + +![img](../public/image/rfcs/rfc135/img_18.png) + +2.2.3.5.5.1 **`WriteCode`和`WriteCodeReview`改造** + +现有代码中,`WriteCode`和`WriteCodeReview`的传参方式与基类`Action`差异较大: + +1. `Action`类的`context`是在创建对象时传入的,而`WriteCode`和`WriteCodeReview`是在`run`时才传参; +2. 根据think-act原则,think阶段就应该定义好act的内容和范围,而不是在act执行过程中动态调整。从这个设计角度出发,用run传context的方式是在运行时才决定工作内容和范围,这与框架的think-act设计原则相违背。 + +```python +code = await WriteCode().run(context=context_str, filename=todo) +``` + +```python +rewrite_code = await WriteCodeReview().run(context=context_str, code=code, filename=todo) +``` + +因此,本轮优化会将WriteCode和WriteCodeReview的传参移到创建对象时。 + +2.2.3.5.5.2 **`SummarizeCode`** + +![img](../public/image/rfcs/rfc135/img_19.png) + +`SummarizeCode`会根据结果,判断跳转到哪里: + +1. 如果代码中仍然存在问题,则发消息给自己,重新进行`WriteCode`; +2. 如果代码没有发现问题,则发消息给`QaEngineer`,进入单测环节; + +###### 2.2.3.5.6 **`WriteTest`** + +新的流程有两种条件会触发`WriteTest` action: + +1. 上游依赖的code文件发生变化; +2. 用户指定重写单测。这种情况下,触发`WriteTest`的消息由上游`Engineer`对象负责发送。 + +###### 2.2.3.5.7 **QaEngineer & WriteTest & RunCode & DebugCode** + +QA过程中伴随有大量的Message传文件内容的操作,已经统一改为传文件名方式。 + +![img](../public/image/rfcs/rfc135/img_20.png) + +##### 2.2.3.10 **新增main参数** + +| 参数名 | 参数值类型 | 可选/必需 | 说明 | +| ------------------ | ---------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------- | +| `--project-path` | path | 可选 | 从哪里加载上个版本的项目文件 | +| `--reqa-file` | File name | 可选 | 要重新生成单元测试的文件的名字。基于源代码文件夹的相对路径。例如,对于下图场景中的"main.py",可以使用以下命令:
`--reqa-file "main.py"` | +| `--project-name` | str | 可选 | 指定使用的项目名。项目名需同时遵守文件夹和代码变量命名规则。 | +| `--inc`/`--no-inc` | bool | 可选 | 是否采用增量实现需求。`--inc`: 基于`--project-path`增量实现需求更新;`--no-inc`: 如果`--project-path`不空,将清空`--project-path`。 | + +##### 2.2.3.11 **集成git功能** + +git功能作为skill集成到MetaGPT中,方便后续人机交互。 +参阅:[利用python代码操作git - tomjoy - 博客园](https://www.cnblogs.com/guapitomjoy/p/12382605.html) + +##### 2.2.3.12 **集成changelog功能** + +changelog功能作为skill集成到MetaGPT中,方便后续人机交互。 +基于git的changelog生成参阅:[git自动生成changelog及项目版本管理 - +掘金](https://juejin.cn/post/6844904147892830221) + +##### 2.2.3.13 **输出件和消息分离** + +###### 2.2.3.13.1 **现状** + +使用消息来传递所有数据。 + +###### 2.2.3.13.2 **新方案** + +新方案依托FileRepository类和workspace,对消息内容进一步分类,以简化消息的处理逻辑: + +![img](../public/image/rfcs/rfc135/img_21.png) + +如此设计是为了解决如下问题: + +1. 简化数据同步逻辑
+     a. 旧方案依靠Message来负责各个数据副本之间的数据同步。这让数据的消费者被无端卷入不必要的数据同步的麻烦。
+     b. 新方案将这类数据剥离出来,成为独立的文件。如此数据的同步问题就简化很多。
+     c. 新方案使得Message能够传输多媒体文件。
+     d. 后续优化会持续将Message事件化,弱化将其作为数据传输手段的设计。 + +![img](../public/image/rfcs/rfc135/img_22.png) + +2. 简化数据版本管理和迁移的问题。 + +## 3. **附录** + +### 3.1 **术语及定义** + +| 术语 | 定义 | +| -------- | ---------------------------------------------------------------------------- | +| 基线版本 | 用户想在某个项目版本上,继续迭代,那么这个被选中的项目版本就称为“基线版本”。 | + +### 3.2 **参考** + +- MetaGPT TODOs