Support nested (non-standard) DFP structure?

[imi415]

According to Open-CMSIS specifications clause 2, CMSIS pack structure should have .pdsc file located at the base directory of the archive:

~ unzip -Z1 ~/Downloads/NXP.K32L2A31A_DFP.15.0.0.pack | grep pdsc
...
NXP.K32L2A31A_DFP.pdsc
...

However it seems Keil MDK also supports reading pack contents from a sub-directory, resulting the pack with the following structure becomes valid for MDK:

~ unzip -Z1 Renesas.RA_DFP.4.0.0.pack | grep pdsc
...
Renesas.RA_DFP.4.0.0/Renesas.RA_DFP.pdsc
...

pyOCD, however, is using a mix of both methods by finding the .pdsc file recursively, and read the rest of the files relative to the base directory:

pyOCD/pyocd/target/pack/cmsis_pack.py

Lines 97 to 105 in 3042769

	for name in self._pack_file.namelist():
	if name.endswith('.pdsc'):
	self._pdscName = name
	break
	else:
	raise MalformedCmsisPackError(f"CMSIS-Pack '{file_or_path}' is missing a .pdsc file")
	with self._pack_file.open(self._pdscName) as pdscFile:
	self._pdsc = CmsisPackDescription(self, pdscFile)

pyOCD/pyocd/target/pack/cmsis_pack.py

Lines 122 to 132 in 3042769

	def get_file(self, filename) -> IO[bytes]:
	"""@brief Return file-like object for a file within the pack.
	@param self
	@param filename Relative path within the pack. May use forward or back slashes.
	@return A BytesIO object is returned that contains all of the data from the file
	in the pack. This is done to isolate the returned file from how the pack was
	opened (due to particularities of the ZipFile implementation).
	"""
	filename = filename.replace('\\', '/')
	return io.BytesIO(self._pack_file.read(filename))

In the above case for Renesas RA series, the pdsc file could be located, however the flash algorithms can not be loaded due to the FLM path in XML file is relative to the .pdsc directory.
This will raise the following bogos error message indicating the device is not supported:

Error Log

pyocd gdbserver -t R7FA6M5BH --pack ../../Packs/Renesas.RA_DFP.4.0.0.pack        
0000871 W Overlapping memory regions in file ../../Packs/Renesas.RA_DFP.4.0.0.pack (R7FA6M4AF3CFB_dual); deleting outer region. Further warnings will be suppressed for this file. [cmsis_pack]
0000876 C Target type r7fa6m5bh not recognized. Use 'pyocd list --targets' to see currently available target types. See <https://pyocd.io/docs/target_support.html> for how to install additional target support. [__main__]
Traceback (most recent call last):
  File "/home/imi415/Documents/Sources/scm/Embedded/pyOCD/pyocd/board/board.py", line 111, in __init__
    self.target = TARGET[self._target_type](session)
  File "/home/imi415/Documents/Sources/scm/Embedded/pyOCD/pyocd/target/pack/pack_target.py", line 117, in _pack_target__init__
    super(self.__class__, self).__init__(session, self._pack_device.memory_map)
  File "/home/imi415/Documents/Sources/scm/Embedded/pyOCD/pyocd/target/pack/cmsis_pack.py", line 682, in memory_map
    self._build_flash_regions()
  File "/home/imi415/Documents/Sources/scm/Embedded/pyOCD/pyocd/target/pack/cmsis_pack.py", line 511, in _build_flash_regions
    packAlgo = self._load_flash_algo(algo_element.attrib['name'])
  File "/home/imi415/Documents/Sources/scm/Embedded/pyOCD/pyocd/target/pack/cmsis_pack.py", line 645, in _load_flash_algo
    algo_data = self.pack.get_file(filename)
  File "/home/imi415/Documents/Sources/scm/Embedded/pyOCD/pyocd/target/pack/cmsis_pack.py", line 132, in get_file
    return io.BytesIO(self._pack_file.read(filename))
  File "/usr/lib/python3.10/zipfile.py", line 1475, in read
    with self.open(name, "r", pwd) as fp:
  File "/usr/lib/python3.10/zipfile.py", line 1514, in open
    zinfo = self.getinfo(name)
  File "/usr/lib/python3.10/zipfile.py", line 1441, in getinfo
    raise KeyError(
KeyError: "There is no item named 'Flash/RA6M5_2M.FLM' in the archive"

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/imi415/Documents/Sources/scm/Embedded/pyOCD/pyocd/__main__.py", line 161, in run
    status = cmd.invoke()
  File "/home/imi415/Documents/Sources/scm/Embedded/pyOCD/pyocd/subcommands/gdbserver_cmd.py", line 176, in invoke
    session = Session(probe_proxy,
  File "/home/imi415/Documents/Sources/scm/Embedded/pyOCD/pyocd/core/session.py", line 216, in __init__
    self._board = probe.create_associated_board() or Board(self)
  File "/home/imi415/Documents/Sources/scm/Embedded/pyOCD/pyocd/board/board.py", line 113, in __init__
    raise exceptions.TargetSupportError(
pyocd.core.exceptions.TargetSupportError: Target type r7fa6m5bh not recognized. Use 'pyocd list --targets' to see currently available target types. See <https://pyocd.io/docs/target_support.html> for how to install additional target support.

So, should we support this kind of packs by using relative path to the .pdsc file, which stays compatible with MDK, or change the pdsc search method to base directory only?

[imi415]

Found this issue is a duplicate of #1446

[flit]

Thanks for the details, @imi415. Given that there are already packs like the Renesas RA series that already cannot be used, we don't have much choice but to change pyocd to support the non-standard structure. So thanks for #1461!

I'm also adding this to my list of issues with the Open-CMSIS-Pack spec (for which issues need to created in the spec repo…).