[tx] Subroutinized CFF2 font exceeds allowed stack depth

[readroberts]

Applying 'tx -cff2 +S +b' to fdko/tests/tx_data/input/SHSansJPVFTest.otf produces a CFF2 file whose subroutine depth exceeds the limit. I discovered this when trying to apply 'tx -dcf' to the output font. Fortunately, you can dump the output font with ttx.

It also generates error messages:
'inconsistent region indices detected in item variation store subtable 1'.

[miguelsousa]

Confirmed that the subroutine depth limit is exceeded.
Here's the resulting font: SHSansJPVFTest_SUBR.otf.zip

[miguelsousa]

The output from tx -dcf SHSansJPVFTest_SUBR.otf is

### Header (00000c60-00000c64)
major  =2
minor  =0
hdrSize=5
offSize=0
tx: --- SHSansJPVFTest_SUBR.otf
tx: subr depth: 11

tx: fatal error

but the output from

from fontTools.ttLib import TTFont
from fontbakery.profiles.cff import (
    com_adobe_fonts_check_cff2_call_depth as check)
test_font = TTFont('SHSansJPVFTest_SUBR.otf')
status, message = list(check(test_font))[-1]
print(status, message)

is <Status PASS> Maximum call depth not exceeded.

[cjchapman]

@readroberts I believe I fixed the inconsistent region indices detected in item variation store subtable 1 issue in #835

I'll look into the subroutine depth issue.

[cjchapman]

@readroberts & @miguelsousa Based on my initial analysis, I think the max call depth is that font is 2, not 11+. I'll dig into the tx code to figure out why it thinks differently.

Here are the subroutines in font dict 1:

            <Subrs>
              <!-- The 'index' attribute is only for humans; it is ignored when parsed. -->
              <CharString index="0">
                3 blend
                vlineto
              </CharString>
              <CharString index="1">
                -107 callgsubr
                -30</CharString>
              <CharString index="2">
                3 blend
                hlineto
              </CharString>
              <CharString index="3">
                4 0 5</CharString>
              <CharString index="4">
                -571 -179 54 179 -105 callsubr
              </CharString>
            </Subrs>

...here is the subroutine in font dict 2:

            <Subrs>
              <!-- The 'index' attribute is only for humans; it is ignored when parsed. -->
              <CharString index="0">
                1 blend
                hlineto
              </CharString>
            </Subrs>

...and here is the global subroutine:

    <GlobalSubrs>
      <!-- The 'index' attribute is only for humans; it is ignored when parsed. -->
      <CharString index="0">
        2 blend
        rmoveto
      </CharString>
    </GlobalSubrs>

[cjchapman]

@readroberts & @miguelsousa Sorry, I forgot to reduce the depth count after a subr call when I implemented #812