Add support for different list merging strategies (REPLACE, EXTEND, E…

…XTEND_UNIQUE) (#1082)
omry · Jul 2, 2023 · 7dae67e · 7dae67e
1 parent a68b149
commit 7dae67e
Show file tree

Hide file tree

Showing 9 changed files with 265 additions and 13 deletions.
diff --git a/docs/notebook/Tutorial.ipynb b/docs/notebook/Tutorial.ipynb
@@ -1,6 +1,7 @@
 {
  "cells": [
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -14,6 +15,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -45,6 +47,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -79,6 +82,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -112,6 +116,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -149,6 +154,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -190,6 +196,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -227,6 +234,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -265,6 +273,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -303,6 +312,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -334,6 +344,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -365,6 +376,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -396,6 +408,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -416,6 +429,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -436,6 +450,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -456,6 +471,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -487,6 +503,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -523,6 +540,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -534,6 +552,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -594,6 +613,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -627,6 +647,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -661,6 +682,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -694,6 +716,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -714,6 +737,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -771,6 +795,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -811,13 +836,15 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "## Decoding strings with interpolations"
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -876,13 +903,15 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "## Custom interpolations"
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -917,6 +946,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -946,6 +976,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -1004,6 +1035,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -1112,6 +1144,85 @@
     "conf.merge_with_cli()\n",
     "print(OmegaConf.to_yaml(conf))"
    ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "By default, `merge()` is replacing the target list with the source list.\n",
+    "Use `list_merge_mode` to control the merge behavior for lists.\n",
+    "Currently there are three different merge modes:\n",
+    "* `REPLACE`: Replaces the target list with the new one (default)\n",
+    "* `EXTEND`: Extends the target list with the new one\n",
+    "* `EXTEND_UNIQUE`: Extends the target list items with items not present in it"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "`example2.yaml` file:\n",
+    "```yaml\n",
+    "server:\n",
+    "  port: 80\n",
+    "users:\n",
+    "  - user1\n",
+    "  - user2\n",
+    "```"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "`example4.yaml` file:\n",
+    "```yaml\n",
+    "users:\n",
+    "  - user3\n",
+    "  - user2\n",
+    "```"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "If you load them and merge them with `list_merge_mode=ListMergeMode.EXTEND_UNIQUE` you will get this:\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from omegaconf import OmegaConf, ListMergeMode\n",
+    "\n",
+    "cfg_example2 = OmegaConf.load('../source/example2.yaml')\n",
+    "cfg_example4 = OmegaConf.load('../source/example4.yaml')\n",
+    "\n",
+    "conf = OmegaConf.merge(cfg_example2, cfg_example4, list_merge_mode=ListMergeMode.EXTEND_UNIQUE)\n",
+    "print(OmegaConf.to_yaml(conf))"
+   ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "```yaml\n",
+    "server:\n",
+    "  port: 80\n",
+    "users:\n",
+    "- user1\n",
+    "- user2\n",
+    "- user3\n",
+    "```"
+   ]
   }
  ],
  "metadata": {
@@ -1130,7 +1241,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.8.5"
+   "version": "3.10.8"
   },
   "pycharm": {
    "stem_cell": {

diff --git a/docs/source/example4.yaml b/docs/source/example4.yaml
@@ -0,0 +1,3 @@
+users:
+  - user3
+  - user2
diff --git a/docs/source/usage.rst b/docs/source/usage.rst
@@ -492,6 +492,43 @@ Note how the port changes to 82, and how the users lists are combined.
       file: log.txt
     <BLANKLINE>
 
+By default, ``merge()`` is replacing the target list with the source list.
+Use ``list_merge_mode`` to control the merge behavior for lists.
+This Enum is defined in ``omegaconf.ListMergeMode`` and defines the following modes:
+* ``REPLACE``: Replaces the target list with the new one (default)
+* ``EXTEND``: Extends the target list with the new one
+* ``EXTEND_UNIQUE``: Extends the target list items with items not present in it
+
+**example2.yaml** file:
+
+.. include:: example2.yaml
+   :code: yaml
+
+**example4.yaml** file:
+
+.. include:: example4.yaml
+   :code: yaml
+
+If you load them and merge them with ``list_merge_mode=ListMergeMode.EXTEND_UNIQUE`` you will get this:
+
+.. doctest::
+
+    >>> from omegaconf import OmegaConf, ListMergeMode
+    >>>
+    >>> cfg_1 = OmegaConf.load('source/example2.yaml')
+    >>> cfg_2 = OmegaConf.load('source/example4.yaml')
+    >>> 
+    >>> mode = ListMergeMode.EXTEND_UNIQUE
+    >>> conf = OmegaConf.merge(cfg_1, cfg_2, list_merge_mode=mode)
+    >>> print(OmegaConf.to_yaml(conf))
+    server:
+      port: 80
+    users:
+    - user1
+    - user2
+    - user3
+    <BLANKLINE>
+
 OmegaConf.unsafe_merge()
 ^^^^^^^^^^^^^^^^^^^^^^^^