Fix rendering of fill outlines that have a different color than the fill itself

[kkaefer]

Fixes #9690.

We're rendering polygon outlines in the same color as the polygon in order to achieve an antialiasing effect. In this case, the rendering order works like this:

• (opaque pass) draw fill, write to depth buffer
• (translucent pass) draw outline, obeying depth buffer that cuts out half of the outline, so that we only render one "side" of the line

When the outline color didn't match the fill color, we did it this way:

• (opaque pass) draw fill, write to depth buffer
• (translucent pass) draw outline, not obeying depth buffer for this particular layer, but obeying it for other layers

This seems to be the part that broke:

• We no longer ignore the depth buffer when rendering because the sublayer for a colored outline now has value 2 (instead of 0, as it used to be)
• We're writing fragments we draw for outlines to the depth buffer, but should not do this, since the outline shader produces translucent fragments
• We're drawing the outline before drawing the fill when the fill color is translucent

This change restores usage of sublayer 0 for mismatched fill/outline color, and sublayer 2 for identical fill/outline color. It also changes the depth mode to prevent writing to the depth buffer when rendering outlines.

Remaining tasks:

• Add render tests to the test suite
• Verify behavior for DDS fill color/outline color

[jfirebaugh]

Related issues: #8760, #6927, mapbox/mapbox-gl-js#4601.

[jfirebaugh]

It looks like this condition is the opposite of gl-js? Am I reading that right?

[kkaefer]

You're right. However, upon further inspection, we're using different formulas to compute the depth mode:

JS:

const farDepth = 1 - ((1 + this.currentLayer) * this.numSublayers + n) * this.depthEpsilon;
const nearDepth = farDepth - 1 + this.depthRange;

Native:

const float nearDepth = ((1 + currentLayer) * numSublayers + n) * depthEpsilon;
const float farDepth = nearDepth + depthRangeSize;

I'll ticket this as a separate change that we'll have to make.

[kkaefer]

This change also fixes #8760. It requires mapbox/mapbox-gl-js#5099 to be merged simultaneously.