Unable to read newer Panasonic RW2 Files

[akjohnston]

ImageMagick version

8.0.4?

Operating system, version and so on

Windows 10-64, using ImageMagick.Net wrapper

Description

Hi,
I'm not sure if this is the right location for this query, if not please could someone provide an accurate redirect?

I'm trying to read camera raw images using ImageMagic.Net (version 14.4.0). This works well for most camera types, including older Panasonic models such as the G9 (which has a default bit depth of 12). However it doesn't work for newer Pansonic models such as the G9ii or S5ii which have a 14-bit depth. I have tried specifying a settings object and setting different "Depth" values, but that doesn't seem to help.

I would like to provide a sample file, but the typical size is 36MB zipped, which exceeds the site limit. Please advise how to address this.

Please could someone provide me with guidance on how to check that these files are supported, and if so what settings may be required to read them correctly?

Thanks
Andrew

[dlemstra]

It might be possible that Libraw has not yet added support for those image. We will probably need to wait for them to do this.

The trick to upload a large file is to create a "multi-part ZIP archive" and upload the separate files

[akjohnston]

Hi,
According to https://www.libraw.org/supported-cameras-202403 the Panasonic G9ii and GH6 are both supported. How can I tell if that support has been implemented in ImageMagick and ImageMagick.Net?
Generally, is there something I can do either before or after attempting to read the file to find out if the camera is supported or not?
Thanks
Andrew

[snibgo]

Note that:

https://www.libraw.org/supported-cameras "LibRaw 0.21 supported cameras" doesn't include "DC-G9 Mark II".

https://www.libraw.org/supported-cameras-202403 "LibRaw 202403 snapshot supported camera list" says "DC-G9 Mark II" is supported.

I suppose IM doesn't use the snapshot.

[dlemstra]

We are using the latest snapshot in the Windows build.

[akjohnston]

Thanks. I do need a way to find out if an image is properly supported or not. It probably doesn't help that Panasonic have retained the same .RW2 extension for the newer cameras, although there have been some internal changes (e.g. from 12- to 14-bit depth for raw colour data).

What might help is if I could get the verbose output from the IM read operation, but I can't find any examples of how to do that. My current VB.Net code looks like this:

Private WithEvents oMagick As MagickImage
Dim oSettings As New MagickReadSettings
oMagick = New MagickImage(sFilename, oSettings)

This executes in a try/catch block, but no exception is triggered. I have an event handler for oMagic.Warning, but that's not triggered either. My last option is that if I can get the verbose output from the read operation I might be able to see a difference between the output for a valid file (older model) and unsupported file (newer model).

Does anyone know how to do this? Is there a working example in any .Net language?

Thanks

[dlemstra]

Can you clarify what you mean by that it cannot read the image? If the image cannot be read this should throw an exception. Or are you seeing information that is incorrect?

And are you using the most recent version?

[akjohnston]

Newer Panasonic files apparently read OK (no error or warning from IM.Net), but the bitmap created is just garbage - see attached screenshots. Older Panasonic files are fine.

I am definitely using the latest version, 14.4.0

Bitmap created by IM.Net:

Same Panasonic G9ii image using XnView:

[dlemstra]

This could be a bug inside Libraw or inside ImageMagick. Can you share that image? You might need to zip it before you can upload it here.

[akjohnston]

Hi,
Above G9ii RAW image attached, split across 2 zip files as requested, but please note I had to change the .z01 file to .z01.zip to get past your upload filter.
240214_G9ii_1002396.z01.zip
240214_G9ii_1002396.zip

The Libraw snapshot page definitely states that the G9ii and a couple of other problematic new models are supported, but that doesn't mean they may not have a bug.
Thanks
Andrew

[snibgo]

On Windows 11, with pre-built binaries of IM, I can't reproduce the problem at the command line.

IM v7.1.1-20 has a problem with the file:

magick 240214_G9ii_1002396.RW2 x2.tiff
magick: Data corrupted at `5071872' @ warning/dng.c/LibRawDataError/392.

But with IM 71.1.1-38, the same command succeeds, and makes an image that seems to be as desired.

magick 240214_G9ii_1002396.RW2 -resize 400 -quality 40 x.jpg

x.jpg is:

The versions of libraw seem a little weird, with the older version of IM (IM v7.1.1-20) using a newer version of libraw.

C:\web\im>%IMG7%magick -list format |grep -i rw2
      RW2  r--   Panasonic Lumix Raw Image (0.21.1-Release)

C:\web\im>%IMG71138%magick -list format |grep -i rw2
      RW2  r--   Panasonic Lumix Raw Format (0.21.0-Beta1)

[akjohnston]

Thanks. That seems to confirm what's implied by the LibRaw support pages, that the latest version of IM should handle the newer files, but older ones won't.

However, that doesn't explain why I'm getting the error using the latest version of Magick.Net. How can I check what component versions (of LibRaw and IM) are being used?

Also I'd ideally like to be able to see the output and detect for myself when there's a problem reading the file. Is there any way with Magick.Net to get the verbose output?

Thanks

[ryza1]

"That seems to confirm what's implied by the LibRaw support pages, that the latest version of IM should handle the newer files, but older ones won't."

If I am reading this correctly: some version of IM has OLDER version of libraw as it is preferring the beta release: 0.21.0-Beta which works. My IM 7.11-.43 list format is even newer for libraw version and does not work

RW2  DNG       r--   Panasonic Lumix Raw Format (0.21.3-Release)

magick SameFile.RW2 output.jpeg

Data corrupted at `5071872' @ warning/dng.c/LibRawDataError/347.

Not sure if this helps, but it seems most likely that libraw change we are waiting for is in 0.21.0-Beta that is not implemented in Releases since then

[snibgo]

Possibly the libraw developers implemented the new RW2 format, but then discovered bugs and withdrew support. But that would be a question for https://www.libraw.org/forum .

[dlemstra]

They are also on GitHub: https://github.com/LibRaw/LibRaw. I do think you should ask your question there. It might be possible that they made a bug in the most recent version.

[akjohnston]

Hi. I managed to sort out getting the output from Magick in my app, and then detecting whether or not the raw file was read correctly. It would really have helped if someone could have directed me to https://github.com/dlemstra/Magick.NET/blob/main/docs/DetailedDebugInformation.md, which I eventually found after a lot of Googling! (@dlemstra - it might have been useful to keep the second thread going under the Magick.Net discussion...)

For completeness, and to help the next person looking for this, here's how it works.

The VB.Net code looks a lot like the C# example. The first thing is to create an event handler for each output item. Mine writes any item with "error" or "warning" to the debug log. I then used that to establish that a failure of the raw read would result in a line including the keyword "LibRawDataError" :

    Private Sub Magick_Log(sender As Object, e As LogEventArgs)
            If e.Message.ToLower.ContainsAny({"error", "warning"}) Then Debug.Print(e.Message)
            If e.Message.Contains("LibRawDataError") Then
                bImageValid = False
                RaiseEvent ErrorRaised(New ErrorDetail(LSError, "Error Reading " & sFileBeingRead))
            End If
    End Sub

When Magick.Net is first initialised, you need to hook up the event handler, e.g.:

          MagickNET.Initialize()
          MagickNET.SetLogEvents(LogEventTypes.All)           ' This could be changed to make the output more restrictive
          AddHandler MagickNET.Log, AddressOf Magick_Log

Finally you can optionally set Verbose mode before the read operation:

          Dim oSettings As New MagickReadSettings
          oSettings.Debug = True
          oSettings.Verbose = True                ' Optional, omit if you only want critical output items    
          oMagick = New MagickImage(sFilename, oSettings)

[akjohnston]

This leaves the problem of why the Panasonic G9ii images are not being read correctly, despite the fact they should be supported.

The first thing to remember is that I am using Magick.Net, not command-line ImageMagick. Please could someone who knows about Magick.Net please therefore answer the following questions:

"For the nuget release packages of Magick.Net, is the version of LibRaw used downloaded and compiled directly into Magick.Net, or does Magick.Net just use the version already linked into ImageMagick? How can we confirm which version of LibRaw is being used in a given application calling Magick.Net?"

Thanks

[dlemstra]

You can find the information in the NOTICE file of the NuGet package. You can also view that in the browser: https://nuget.info/packages/Magick.NET-Q16-x64/14.4.0. And then you will see that the (currently) most recent version use this version:

[ raw 0.21.3 (2024-09-24) ] copyright:

 ** LibRaw: Raw images processing library **

This is the most recent version of the libraw library so I suspect that between the releases mention here and the latest release a bug was introduced in that library.

[akjohnston]

Thanks Dirk. Is there any way to get the information at runtime, just in case for some reason the physical build has got out of step with the documentation?

[dlemstra]

There is no way to get the information at runtime. That document was generated at compile time by looking at files in the repository that tell us which version is used. The document is correct.

[akjohnston]

Also I confess to being completely confused by LibRaw versioning. According to https://www.libraw.org/ "Minor updates (0.20.1, 0.20.2…) do not change the API/ABI and new cameras are not added; generally they are just bugfixes." Therefore I would expect 21.3 to just be a bugfix on 21.0.

To get support for the newer cameras we need the "202403 Snapshot" https://www.libraw.org/supported-cameras-202403, but that number has no relation to the version numbers quoted above, and neither that page nor the main page states clearly what version number it should display.

Does anyone know definitely what version number we should be seeing for the "202403 Sanpshot" version?

Thanks

[fmw42]

Contact the libraw developers.

[dlemstra]

"202403 Snapshot" is older than release 0.21.3. We keep a copy of their code and you can view the changes that we made here: https://github.com/ImageMagick/raw/commits/main/.

I would really advise you to contact the libraw developers instead, there is nothing we can do for you.

[akjohnston]

I have tried to post a query to the LibRaw forum, but it looks like new posts have to be vetted first, so it's not yet public. I'll let you know when it's published and about any response.

However, I think we may be able to work out what's going on. The key is the text I posted above about the LibRaw "minor release" policy. That clearly states that minor updates (e.g. 21.3) do NOT contain the latest camera support - for that you have to specifically target the latest camera support snapshots. If IM / Magick.Net have been compiled with the "latest" version then they may have picked up a "bug fix" release with old camera support. (Personally I think that's a very odd approach and if it was down to me I'd slipstream the latest camera support into each release, but their policy is as stated.)

Do we know if we have a version of Magick.Net which definitely used the 202403 snapshot? If so I could test that to see if my theory is correct?
Thanks

[akjohnston]

The information I got from the LibRaw forum is as follows:

"Version defines for 202403 snapshot are:
#define LIBRAW_MAJOR_VERSION 0
#define LIBRAW_MINOR_VERSION 22
#define LIBRAW_PATCH_VERSION 0
#define LIBRAW_VERSION_TAIL Devel202403"

This suggests that version is marked as 0.22.0. IM currently seems to be using 0.21.X (various values reported above), so I suspect we're not yet picking up the full camera support. Is there any way to check this?

Thanks

[dlemstra]

Thanks for all your research and I finally understand all the confusion. When we switched to the snapshot on Windows we used the version number that was then provided. This was changed in a follow up commit that we missed. And that is why I thought the snapshot was some kind of pre-release. But it turns out it wasn't.

I have revert to the snapshot version again and the next release of Magick.NET and ImageMagick will be able to read this image again.

[kmilos]

And that is why I thought the snapshot was some kind of pre-release. But it turns out it wasn't.

It still is. But just because the date of the snapshot/pre-release is older than the patch release, it doesn't mean it's using the older code base. LibRaw snapshots/pre-releases always use the newer code base (master, development branch) and provide support for more formats/cameras, then their patch releases (branched of months/years ago and only ship critical fixes, even if they come after the snapshot). Confusing, but that's how they do it. The current snapshot (i.e. a commit on git master) is the only pre-release for the 0.22.x series, which has newer features, regardless of how many 0.21.x patches they do in the meantime.

[akjohnston]

Brilliant, thanks. Please post a quick comment here when it's available.