Can't use same top-level keys in two different parameters.yaml

[puneeter]

Description

I can't use same high-level keys in two different parameters.yaml files for namespacing even though they have separate sub-namespace.

Context

I am trying to namespace my subpipelines and create parameters for these subpipelines separately in their respective parameters files. Since in above example both the subpipelines belonged to pipeline1, I want the parameters for both to start with pipeline1.subpipeline1_x. And in case I wanted all params for pipeline1 I can easily grab using params:pipeline1.

Steps to Reproduce

# conf/base/pipeline1/subpipeline1_1/parameters.yaml
pipeline1:
  subpipeline1_1:
    key1: value1

# conf/base/pipeline1/subpipeline1_2/parameters.yaml
pipeline1:
  subpipeline1_2:
    key2: value2

Expected Result

This should ideally not throw an error and if I want params:pipeline1 it should be a merge dict based on both of these configs.

Actual Result

Traceback (most recent call last):
  File "/Users/Puneet_Saini/anaconda3/envs/project/lib/python3.9/site-packages/IPython/core/shellapp.py", line 282, in init_extensions
    self.shell.extension_manager.load_extension(ext)
  File "/Users/Puneet_Saini/anaconda3/envs/project/lib/python3.9/site-packages/IPython/core/extensions.py", line 76, in load_extension
    return self._load_extension(module_str)
  File "/Users/Puneet_Saini/anaconda3/envs/project/lib/python3.9/site-packages/IPython/core/extensions.py", line 93, in _load_extension
    if self._call_load_ipython_extension(mod):
  File "/Users/Puneet_Saini/anaconda3/envs/project/lib/python3.9/site-packages/IPython/core/extensions.py", line 145, in _call_load_ipython_extension
    mod.load_ipython_extension(self.shell)
  File "/Users/Puneet_Saini/anaconda3/envs/project/lib/python3.9/site-packages/kedro/ipython/__init__.py", line 62, in load_ipython_extension
    reload_kedro()
  File "/Users/Puneet_Saini/anaconda3/envs/project/lib/python3.9/site-packages/kedro/ipython/__init__.py", line 123, in reload_kedro
    catalog = context.catalog
  File "/Users/Puneet_Saini/anaconda3/envs/project/lib/python3.9/site-packages/kedro/framework/context/context.py", line 187, in catalog
    return self._get_catalog()
  File "/Users/Puneet_Saini/anaconda3/envs/project/lib/python3.9/site-packages/kedro/framework/context/context.py", line 238, in _get_catalog
    feed_dict = self._get_feed_dict()
  File "/Users/Puneet_Saini/anaconda3/envs/project/lib/python3.9/site-packages/kedro/framework/context/context.py", line 253, in _get_feed_dict
    params = self.params
  File "/Users/Puneet_Saini/anaconda3/envs/project/lib/python3.9/site-packages/kedro/framework/context/context.py", line 198, in params
    params = self.config_loader["parameters"]
  File "/Users/Puneet_Saini/anaconda3/envs/project/lib/python3.9/site-packages/kedro/config/omegaconf_config.py", line 199, in __getitem__
    base_config = self.load_and_merge_dir_config(  # type: ignore[no-untyped-call]
  File "/Users/Puneet_Saini/anaconda3/envs/project/lib/python3.9/site-packages/kedro/config/omegaconf_config.py", line 326, in load_and_merge_dir_config
    self._check_duplicates(seen_file_to_keys)
  File "/Users/Puneet_Saini/anaconda3/envs/project/lib/python3.9/site-packages/kedro/config/omegaconf_config.py", line 433, in _check_duplicates
    raise ValueError(f"{dup_str}")
ValueError: Duplicate keys found in /Users/Puneet_Saini/repos/project/conf/base/pipeline1/subpipeline1_1/parameters.yaml and /Users/Puneet_Saini/repos/project/conf/base/pipeline1/subpipeline1_2/parameters.yaml: pipeline1

Your Environment

• Kedro version used (pip show kedro or kedro -V): 0.19.3
• Python version used (python -V): 3.9.19
• Operating system and version: Mac OS

Link to discussion: https://kedro-org.slack.com/archives/C03RKP2LW64/p1723193517960259