Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix various Gif Decoder/Encoder behaviors. #2289

Merged
merged 11 commits into from
Nov 15, 2022
40 changes: 25 additions & 15 deletions src/ImageSharp/Formats/Gif/GifDecoderCore.cs
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,10 @@ public Image<TPixel> Decode<TPixel>(BufferedReadStream stream, CancellationToken
}

this.ReadFrame(ref image, ref previousFrame);

// Reset per-frame state.
this.imageDescriptor = default;
this.graphicsControlExtension = default;
}
else if (nextFlag == GifConstants.ExtensionIntroducer)
{
Expand Down Expand Up @@ -464,7 +468,7 @@ private void ReadFrameColors<TPixel>(ref Image<TPixel> image, ref ImageFrame<TPi
image = new Image<TPixel>(this.configuration, imageWidth, imageHeight, this.metadata);
}

this.SetFrameMetadata(image.Frames.RootFrame.Metadata);
this.SetFrameMetadata(image.Frames.RootFrame.Metadata, true);

imageFrame = image.Frames.RootFrame;
}
Expand All @@ -475,9 +479,9 @@ private void ReadFrameColors<TPixel>(ref Image<TPixel> image, ref ImageFrame<TPi
prevFrame = previousFrame;
}

currentFrame = image.Frames.AddFrame(previousFrame); // This clones the frame and adds it the collection
currentFrame = image.Frames.CreateFrame();

this.SetFrameMetadata(currentFrame.Metadata);
this.SetFrameMetadata(currentFrame.Metadata, false);

imageFrame = currentFrame;

Expand Down Expand Up @@ -554,7 +558,7 @@ private void ReadFrameColors<TPixel>(ref Image<TPixel> image, ref ImageFrame<TPi
{
for (int x = descriptorLeft; x < descriptorRight && x < imageWidth; x++)
{
int index = Numerics.Clamp(Unsafe.Add(ref indicesRowRef, x - descriptorLeft), 0, colorTableMaxIdx);
int index = Numerics.Clamp(Unsafe.Add(ref indicesRowRef, x - descriptorLeft), 0, Math.Max(transIndex, colorTableMaxIdx));
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some images use a transparent index that is the same as the length of the palette.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I've seen, most viewers treat OOB transparent index as 0. That's also the behavior written into the PNG spec. Here's an example where 0 works and last index doesn't.

bad_transparent_idx

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's interesting. I found that the second gif in the referenced issue had a frame with a local table that contained 64 colors with a transparent index of 64. Browsers/Windows were rendering it just fine, but we were not.

image

Here's your input gif processed with this PR code.

OptionalExtensionsShouldBeHandledProperly_Rgba32_issue_2288_3

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... yeah, it seems like the behavior I saw was a side-effect of the way I handle OOB values in general. I had just noticed that ImageSharp 2.1.3 was mangling that image, and I knew it had an invalid transparent index. The viewer I've been using (Microsoft GIF Animator) crashes on images like that. What's the viewer you screenshotted there?

I just looked at some web browser code, and it looks like they're just treating any OOB colors as transparent if the GCE has the transparency flag. They take an OOB transparent index as a hint that OOB color values are expected in the frame.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I use this. Stumbled upon it a few years back and it's proven incredibly useful

https://github.com/ata4/gifiddle

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh wow, that's great, thanks!

I see what's happening here now. Before this fix, you're not setting a transparent color because it's out of range, but you're clamping the image values into palette range, so pixels meant to be transparent are taking the last color. This was the result.

oob_color_is

Now you're treating the last palette color as transparent and clamping OOB image values to that color, making the transparency work, but also making any pixels that were supposed to be that last palette color transparent as well. It's not creating artifacts as visible as skipping transparency, but it's also not completely correct.

In my implementation, I made the same mistake, but in the other direction. I use shared code for all animated image formats, so I followed the PNG rules of treating OOB image values as the first color in the palette. That meant I also had to treat invalid transparency index as meaning the first color was supposed to be transparent to get things to look right. Hence my initial comment 😅

So yeah, the browser behavior is better. Don't clamp the values but rather assume the palette is actually bigger than the image says.

Copy link
Member Author

@JimBobSquarePants JimBobSquarePants Nov 9, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed a better fix now. I treat anything greater than the max palette index or equal to the flagged transparent index as transparent.

I only clamp following that check to ensure we don't hit OOB when reading the palette.

Thanks for your help here!

if (transIndex != index)
{
ref TPixel pixel = ref Unsafe.Add(ref rowRef, x);
Expand Down Expand Up @@ -592,7 +596,7 @@ private void RestoreToBackground<TPixel>(ImageFrame<TPixel> frame)
return;
}

var interest = Rectangle.Intersect(frame.Bounds(), this.restoreArea.Value);
Rectangle interest = Rectangle.Intersect(frame.Bounds(), this.restoreArea.Value);
Buffer2DRegion<TPixel> pixelRegion = frame.PixelBuffer.GetRegion(interest);
pixelRegion.Clear();

Expand All @@ -603,28 +607,34 @@ private void RestoreToBackground<TPixel>(ImageFrame<TPixel> frame)
/// Sets the frames metadata.
/// </summary>
/// <param name="meta">The metadata.</param>
/// <param name="isRoot">Whether the metadata represents the root frame.</param>
[MethodImpl(MethodImplOptions.AggressiveInlining)]
private void SetFrameMetadata(ImageFrameMetadata meta)
private void SetFrameMetadata(ImageFrameMetadata meta, bool isRoot)
{
GifFrameMetadata gifMeta = meta.GetGifMetadata();
if (this.graphicsControlExtension.DelayTime > 0)
{
gifMeta.FrameDelay = this.graphicsControlExtension.DelayTime;
}

// Frames can either use the global table or their own local table.
if (this.logicalScreenDescriptor.GlobalColorTableFlag
if (isRoot && this.logicalScreenDescriptor.GlobalColorTableFlag
&& this.logicalScreenDescriptor.GlobalColorTableSize > 0)
{
GifFrameMetadata gifMeta = meta.GetGifMetadata();
gifMeta.ColorTableMode = GifColorTableMode.Global;
gifMeta.ColorTableLength = this.logicalScreenDescriptor.GlobalColorTableSize;
}
else if (this.imageDescriptor.LocalColorTableFlag

if (this.imageDescriptor.LocalColorTableFlag
&& this.imageDescriptor.LocalColorTableSize > 0)
{
GifFrameMetadata gifMeta = meta.GetGifMetadata();
gifMeta.ColorTableMode = GifColorTableMode.Local;
gifMeta.ColorTableLength = this.imageDescriptor.LocalColorTableSize;
}

gifMeta.DisposalMethod = this.graphicsControlExtension.DisposalMethod;
// Graphics control extensions is optional.
if (this.graphicsControlExtension != default)
{
GifFrameMetadata gifMeta = meta.GetGifMetadata();
gifMeta.FrameDelay = this.graphicsControlExtension.DelayTime;
gifMeta.DisposalMethod = this.graphicsControlExtension.DisposalMethod;
}
}

/// <summary>
Expand Down
128 changes: 64 additions & 64 deletions src/ImageSharp/Formats/Gif/GifEncoderCore.cs
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,6 @@ public void Encode<TPixel>(Image<TPixel> image, Stream stream, CancellationToken

// Quantize the image returning a palette.
IndexedImageFrame<TPixel> quantized;

using (IQuantizer<TPixel> frameQuantizer = this.quantizer.CreatePixelSpecificQuantizer<TPixel>(this.configuration))
{
if (useGlobalTable)
Expand Down Expand Up @@ -133,101 +132,102 @@ public void Encode<TPixel>(Image<TPixel> image, Stream stream, CancellationToken
this.WriteApplicationExtensions(stream, image.Frames.Count, gifMetadata.RepeatCount, xmpProfile);
}

if (useGlobalTable)
{
this.EncodeGlobal(image, quantized, index, stream);
}
else
{
this.EncodeLocal(image, quantized, stream);
}

// Clean up.
quantized.Dispose();
this.EncodeFrames(stream, image, quantized, quantized.Palette.ToArray());

stream.WriteByte(GifConstants.EndIntroducer);
}

private void EncodeGlobal<TPixel>(Image<TPixel> image, IndexedImageFrame<TPixel> quantized, int transparencyIndex, Stream stream)
private void EncodeFrames<TPixel>(
Stream stream,
Image<TPixel> image,
IndexedImageFrame<TPixel> quantized,
ReadOnlyMemory<TPixel> palette)
where TPixel : unmanaged, IPixel<TPixel>
{
// The palette quantizer can reuse the same pixel map across multiple frames
// since the palette is unchanging. This allows a reduction of memory usage across
// multi frame gifs using a global palette.
PaletteQuantizer<TPixel> paletteFrameQuantizer = default;
bool quantizerInitialized = false;
PaletteQuantizer<TPixel> paletteQuantizer = default;
bool hasPaletteQuantizer = false;
for (int i = 0; i < image.Frames.Count; i++)
{
// Gather the metadata for this frame.
ImageFrame<TPixel> frame = image.Frames[i];
ImageFrameMetadata metadata = frame.Metadata;
GifFrameMetadata frameMetadata = metadata.GetGifMetadata();
this.WriteGraphicalControlExtension(frameMetadata, transparencyIndex, stream);
this.WriteImageDescriptor(frame, false, stream);
bool hasMetadata = metadata.TryGetGifMetadata(out GifFrameMetadata frameMetadata);
bool useLocal = this.colorTableMode == GifColorTableMode.Local || (hasMetadata && frameMetadata.ColorTableMode == GifColorTableMode.Local);

if (i == 0)
if (!useLocal && !hasPaletteQuantizer && i > 0)
{
this.WriteImageData(quantized, stream);
// The palette quantizer can reuse the same pixel map across multiple frames
// since the palette is unchanging. This allows a reduction of memory usage across
// multi frame gifs using a global palette.
hasPaletteQuantizer = true;
paletteQuantizer = new(this.configuration, this.quantizer.Options, palette);
}
else
{
if (!quantizerInitialized)
{
quantizerInitialized = true;
paletteFrameQuantizer = new PaletteQuantizer<TPixel>(this.configuration, this.quantizer.Options, quantized.Palette);
}

using IndexedImageFrame<TPixel> paletteQuantized = paletteFrameQuantizer.QuantizeFrame(frame, frame.Bounds());
this.WriteImageData(paletteQuantized, stream);
}
this.EncodeFrame(stream, frame, i, useLocal, frameMetadata, ref quantized, ref paletteQuantizer);

// Clean up for the next run.
quantized.Dispose();
quantized = null;
}

paletteFrameQuantizer.Dispose();
paletteQuantizer.Dispose();
}

private void EncodeLocal<TPixel>(Image<TPixel> image, IndexedImageFrame<TPixel> quantized, Stream stream)
private void EncodeFrame<TPixel>(
Stream stream,
ImageFrame<TPixel> frame,
int frameIndex,
bool useLocal,
GifFrameMetadata metadata,
ref IndexedImageFrame<TPixel> quantized,
ref PaletteQuantizer<TPixel> paletteQuantizer)
where TPixel : unmanaged, IPixel<TPixel>
{
ImageFrame<TPixel> previousFrame = null;
GifFrameMetadata previousMeta = null;
for (int i = 0; i < image.Frames.Count; i++)
// The first frame has already been quantized so we do not need to do so again.
if (frameIndex > 0)
{
ImageFrame<TPixel> frame = image.Frames[i];
ImageFrameMetadata metadata = frame.Metadata;
GifFrameMetadata frameMetadata = metadata.GetGifMetadata();
if (quantized is null)
if (useLocal)
{
// Allow each frame to be encoded at whatever color depth the frame designates if set.
if (previousFrame != null && previousMeta.ColorTableLength != frameMetadata.ColorTableLength
&& frameMetadata.ColorTableLength > 0)
// Reassign using the current frame and details.
QuantizerOptions options = null;
int colorTableLength = metadata?.ColorTableLength ?? 0;
if (colorTableLength > 0)
{
QuantizerOptions options = new()
options = new()
{
Dither = this.quantizer.Options.Dither,
DitherScale = this.quantizer.Options.DitherScale,
MaxColors = frameMetadata.ColorTableLength
MaxColors = colorTableLength
};

using IQuantizer<TPixel> frameQuantizer = this.quantizer.CreatePixelSpecificQuantizer<TPixel>(this.configuration, options);
quantized = frameQuantizer.BuildPaletteAndQuantizeFrame(frame, frame.Bounds());
}
else
{
using IQuantizer<TPixel> frameQuantizer = this.quantizer.CreatePixelSpecificQuantizer<TPixel>(this.configuration);
quantized = frameQuantizer.BuildPaletteAndQuantizeFrame(frame, frame.Bounds());
}

using IQuantizer<TPixel> frameQuantizer = this.quantizer.CreatePixelSpecificQuantizer<TPixel>(this.configuration, options ?? this.quantizer.Options);
quantized = frameQuantizer.BuildPaletteAndQuantizeFrame(frame, frame.Bounds());
}
else
{
// Quantize the image using the global palette.
quantized = paletteQuantizer.QuantizeFrame(frame, frame.Bounds());
}

this.bitDepth = ColorNumerics.GetBitsNeededForColorDepth(quantized.Palette.Length);
this.WriteGraphicalControlExtension(frameMetadata, GetTransparentIndex(quantized), stream);
this.WriteImageDescriptor(frame, true, stream);
this.WriteColorTable(quantized, stream);
this.WriteImageData(quantized, stream);
}

quantized.Dispose();
quantized = null; // So next frame can regenerate it
previousFrame = frame;
previousMeta = frameMetadata;
// Do we have extension information to write?
int index = GetTransparentIndex(quantized);
if (metadata != null || index > -1)
{
this.WriteGraphicalControlExtension(metadata ?? new(), index, stream);
}

this.WriteImageDescriptor(frame, useLocal, stream);

if (useLocal)
{
this.WriteColorTable(quantized, stream);
}

this.WriteImageData(quantized, stream);
}

/// <summary>
Expand Down Expand Up @@ -407,7 +407,7 @@ private static void WriteCommentSubBlock(Stream stream, ReadOnlySpan<char> comme
}

/// <summary>
/// Writes the graphics control extension to the stream.
/// Writes the optional graphics control extension to the stream.
/// </summary>
/// <param name="metadata">The metadata of the image or frame.</param>
/// <param name="transparencyIndex">The index of the color in the color palette to make transparent.</param>
Expand Down
8 changes: 7 additions & 1 deletion src/ImageSharp/Formats/Gif/GifFrameMetadata.cs
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,19 @@ public GifFrameMetadata()
/// <param name="other">The metadata to create an instance from.</param>
private GifFrameMetadata(GifFrameMetadata other)
{
this.ColorTableMode = other.ColorTableMode;
this.ColorTableLength = other.ColorTableLength;
this.FrameDelay = other.FrameDelay;
this.DisposalMethod = other.DisposalMethod;
}

/// <summary>
/// Gets or sets the length of the color table for paletted images.
/// Gets or sets the color table mode.
/// </summary>
public GifColorTableMode ColorTableMode { get; set; }

/// <summary>
/// Gets or sets the length of the color table.
/// If not 0, then this field indicates the maximum number of colors to use when quantizing the
/// image frame.
/// </summary>
Expand Down
2 changes: 1 addition & 1 deletion src/ImageSharp/Formats/Gif/GifMetadata.cs
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ private GifMetadata(GifMetadata other)
public int GlobalColorTableLength { get; set; }

/// <summary>
/// Gets or sets the the collection of comments about the graphics, credits, descriptions or any
/// Gets or sets the collection of comments about the graphics, credits, descriptions or any
/// other type of non-control and non-graphic data.
/// </summary>
public IList<string> Comments { get; set; } = new List<string>();
Expand Down
22 changes: 18 additions & 4 deletions src/ImageSharp/Formats/Gif/MetadataExtensions.cs
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,28 @@ public static partial class MetadataExtensions
/// <summary>
/// Gets the gif format specific metadata for the image.
/// </summary>
/// <param name="metadata">The metadata this method extends.</param>
/// <param name="source">The metadata this method extends.</param>
/// <returns>The <see cref="GifMetadata"/>.</returns>
public static GifMetadata GetGifMetadata(this ImageMetadata metadata) => metadata.GetFormatMetadata(GifFormat.Instance);
public static GifMetadata GetGifMetadata(this ImageMetadata source) => source.GetFormatMetadata(GifFormat.Instance);

/// <summary>
/// Gets the gif format specific metadata for the image frame.
/// </summary>
/// <param name="metadata">The metadata this method extends.</param>
/// <param name="source">The metadata this method extends.</param>
/// <returns>The <see cref="GifFrameMetadata"/>.</returns>
public static GifFrameMetadata GetGifMetadata(this ImageFrameMetadata metadata) => metadata.GetFormatMetadata(GifFormat.Instance);
public static GifFrameMetadata GetGifMetadata(this ImageFrameMetadata source) => source.GetFormatMetadata(GifFormat.Instance);

/// <summary>
/// Gets the gif format specific metadata for the image frame.
/// </summary>
/// <param name="source">The metadata this method extends.</param>
/// <param name="metadata">
/// When this method returns, contains the metadata associated with the specified frame,
/// if found; otherwise, the default value for the type of the metadata parameter.
/// This parameter is passed uninitialized.
/// </param>
/// <returns>
/// <see langword="true"/> if the gif frame metadata exists; otherwise, <see langword="false"/>.
/// </returns>
public static bool TryGetGifMetadata(this ImageFrameMetadata source, out GifFrameMetadata metadata) => source.TryGetFormatMetadata(GifFormat.Instance, out metadata);
}
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ namespace SixLabors.ImageSharp.Formats.Gif;
/// processing a graphic rendering block.
/// </summary>
[StructLayout(LayoutKind.Sequential, Pack = 1)]
internal readonly struct GifGraphicControlExtension : IGifExtension
internal readonly struct GifGraphicControlExtension : IGifExtension, IEquatable<GifGraphicControlExtension>
{
public GifGraphicControlExtension(
byte packed,
Expand Down Expand Up @@ -64,6 +64,10 @@ public GifGraphicControlExtension(

int IGifExtension.ContentLength => 5;

public static bool operator ==(GifGraphicControlExtension left, GifGraphicControlExtension right) => left.Equals(right);

public static bool operator !=(GifGraphicControlExtension left, GifGraphicControlExtension right) => !(left == right);

public int WriteTo(Span<byte> buffer)
{
ref GifGraphicControlExtension dest = ref Unsafe.As<byte, GifGraphicControlExtension>(ref MemoryMarshal.GetReference(buffer));
Expand Down Expand Up @@ -101,4 +105,27 @@ Transparent Color Flag | 1 Bit

return value;
}

public override bool Equals(object obj) => obj is GifGraphicControlExtension extension && this.Equals(extension);

public bool Equals(GifGraphicControlExtension other)
=> this.BlockSize == other.BlockSize
&& this.Packed == other.Packed
&& this.DelayTime == other.DelayTime
&& this.TransparencyIndex == other.TransparencyIndex
&& this.DisposalMethod == other.DisposalMethod
&& this.TransparencyFlag == other.TransparencyFlag
&& ((IGifExtension)this).Label == ((IGifExtension)other).Label
&& ((IGifExtension)this).ContentLength == ((IGifExtension)other).ContentLength;

public override int GetHashCode()
=> HashCode.Combine(
this.BlockSize,
this.Packed,
this.DelayTime,
this.TransparencyIndex,
this.DisposalMethod,
this.TransparencyFlag,
((IGifExtension)this).Label,
((IGifExtension)this).ContentLength);
}
Loading