Callframe information parsing issues

[Maxicu5]

For the ELF files (generated by IAR) where the referenced CIE may go after the FDE. CIE is loaded in the scope of FDE and placed into a cache, then when the queue comes to that CIE it takes it from the cache and doesn't move a stream offset, so this function stuck:

    def _parse_entries(self):
        entries = []
        offset = 0
        while offset < self.size:
            entries.append(self._parse_entry_at(offset))
            offset = self.stream.tell()
        return entries

locally (temporary) fixed like this (it works):

  def _parse_entries(self):
    entries = []
    offset = 0
    while offset < self.size:
      e = self._parse_entry_at(offset)
      entries.append(e)
      offset = e.offset + e.header.length +e.structs.initial_length_field_size()
    return entries

Also, CIE of version 4 is parsed by the structs of version 2, but there are two additional fields, so after the header everything is shifted, I had to add the fix in two places, seems it works:

        if self.for_eh_frame:
            is_CIE = CIE_id == 0
        else:
            is_CIE = (
                (dwarf_format == 32 and CIE_id == 0xFFFFFFFF) or
                CIE_id == 0xFFFFFFFFFFFFFFFF)

        # Parse the header, which goes up to and excluding the sequence of
        # instructions.
        if is_CIE:
>           version = struct_parse(entry_structs.Dwarf_uint8(''), self.stream)
>           entry_structs = DWARFStructs(
>               little_endian=self.base_structs.little_endian,
>               dwarf_format=dwarf_format,
>               address_size=self.base_structs.address_size,
>                dwarf_version=version)
            header_struct = (entry_structs.EH_CIE_header
                             if self.for_eh_frame else
                             entry_structs.Dwarf_CIE_header)
            header = struct_parse(
                header_struct, self.stream, offset)
        else:
            header = self._parse_fde_header(entry_structs, offset)


        # If this is DWARF version 4 or later, we can have a more precise
        # address size, read from the CIE header.
        if not self.for_eh_frame and entry_structs.dwarf_version >= 4:
            entry_structs = DWARFStructs(
                little_endian=entry_structs.little_endian,
                dwarf_format=entry_structs.dwarf_format,
                address_size=header.address_size,
>                dwarf_version=entry_structs.dwarf_version)

P.S. Very good library. Looking forward removing my workarounds and use fixed version soon.

[sevaa]

Thanks for letting us know. Lack of CIEv4 header support is clearly an oversight. In our readelf autotest corpus, there are no files with V4 CIEs - I've checked, only 1 and 3 (despite DWARF proper being anywhere from 2 to 5). So it never came up. That I'll fix in a PR. Can you share a binary for our autotest, please? Doesn't have to be a real one, a "Hello world" would suffice - as long as it has V4 CIEs.

The caching issue is slightly more subtle. The only user facing function in CallFrameInfo is get_entries() - and by its parse-and-store nature, it doesn't suffer from the cache/stream offset issue. The underscore prefixed class methods are considered, by Pythonic convention, an implementation detail - sort of private (in the OOP sense), if you will. To the best of my knowledge, the frames section doesn't allow for random or indexed access - only sequential. What is your usage scenario exactly so that calling _parse_entries() while some entries are already cached came up?

[Maxicu5]

Related to the test ELF file, will see what can I do, I'm very busy for now, but maybe it will help you: as I remember it was not the part of my code. My code has generated CIEv2 or 3, but that was IAR built-in linked library for ARM double precision float point processing, probably like this one 'm7M_tls.a'. I can't attach it as it is a part of licensed product, but it should be easy to find it, I think.
Related to the caching, probably I was bad in explanation:
When CIE goes before all the references to it, there will be no issues, but in my case, the ELF file was generated like this:
FDE->CIE->FDE->FDE->FDE
All this FDEs are pointing to the same CIE, and the first FDE is going ahead, so in the scope of loading first FDE, you request the CIE which is not loaded yet, and it is cached after that. Then, after the first FDE, you request the CIE which is already in the cache and the stream offset is not updated.
It is probably compiler specific, I haven't try to find the requirements to the sequence in the specs, but the fact it is possible to get such sequence on practice.

[sevaa]

So the v4 CIEs sit in the debug info of a run-time library that ships with a proprietary compiler. It would probably be a copyright violation to share the RTL file by itself, but it definitely won't be if you build a binary of your own, linked against said RTL, and share that. If you do so, put some calls to make sure the library is not optimized away.

Understood re: the FDE ahead of its CIE issue. The logic in _parse_cie_for_fde() will throw off the parse/cache loop, if it looks ahead instead of back. Will fix.

@Maxicu5: EDIT: I have a patch in the works, but I really need a test binary. At least for the first issue.

[Maxicu5]

I've rechecked the library with CIE v4 and I was wrong, it is not the one from proprietary compiler, it is this one 'stm32wb_zigbee_wb_lib.a', it can be simply found on the github. I've tried to build something with it, but any single function has a lot of dependencies on some interfaces, and the examples have a lot of build errors out of the box, maybe you will be able to find something prebuilt with it or to use the objects itself from inside of this lib archive as the "debug_frame" looks the same for ELF and for the obj

[sevaa]

Can you please track one of the v4 CIEs to a specific function in said library? The debug_frame section in object files within the library looks weird, I suspect the linker is doing a lot of magic while building the executable.

[sevaa]

@eliben: it's crazier than I thought. In an object file, such as those found in this static library, there can be multiple sections with the same name - debug ones included. Specifically, there are two sections called .debug_frames in an object file I'm staring at right now. GNU readelf handles that fine, pyelftools doesn't, and it will necessitate an API change. How do you want to approach that? Superficially, object files are ELF and pyelftools opens them - if you disable relocation.

Linking-type relocation for an object file means something completely different than loading time relocation.

I can sort of see how this might work specifically for debug_frames, since it has no cross-section references, but I don't think it can be made to work with other DWARF sections (except maybe ones that link out but not in - e. g. aranges).

[eliben]

How does readelf handle it? We can borrow the approach

[sevaa]

Moved the .o stuff into #564. I'll take a closer look at this once I'm done poking around readelf's bugs.

[Maxicu5]

I have just realized that I don't need to build working firmware for you, it only asks me to give him some symbols, so I've copied them from error log into the code and here you are)
EwarmProj.zip

[Maxicu5]

Just to clarify, CIEv4 doesn't generate any errors or exceptions in the pyelftool, it returns wrong content, for this core should always be:
code_alignment_factor = 2
data_alignment_factor = 4
but for CIEv4 it reads wrong fields:
code_alignment_factor = address_size = 4
data_alignment_factor = segment_size = 0
And all the data is shifted after that

[sevaa]

Is that with the most recent patch?

[Maxicu5]

It is with the head of main branch. There is declared the structs for CIEv4, but they are not used, because the "entry_structs" are always created without the "version" argument, so it is set to default. See my first post in this issue, I've added "dwarf_version" everywhere and some extra read of the version before the creation.

[Maxicu5]

Ah, no, I have not noticed the one made 19hours ago, it seems should work now, will try it as soon as I can. Thank you

[Maxicu5]

Yep, it works

[sevaa]

@Maxicu5 Would it be possible to rebuild the binary with the IAR 9.40?

[Maxicu5]

@sevaa Which binary? Anyway, our license doesn't cover 9.40. That was just a guy from IAR support, who can try any version

[sevaa]

I thought the test binary in #563 was the Jun 16 one, but the timing doesn't match. I have to recall where did it come from...

[sevaa]

The binary that became pyelftools' test/../dwarf_v4cie.elf is a dummy SO library built by me using Android NDK against stm32wb_zigbee_wb_lib.a. Their last release was on July 2 - before I've pulled the library, built the binary and made it a part of the test corpus. So it's the makers of the library that should rebuild and publish: STMicroelectronics/STM32CubeWB#109