Glyf table point flags bit 6 gets lost

[jenskutilek]

Compressing a TTF to WOFF2 ignores bit 6 of any glyf table point flags. This bit is marked as reserved in the OT spec, but Apple uses it to determine the presence of overlapping contours in a glyph. See fonttools/fonttools#730 and https://developer.apple.com/fonts/TrueType-Reference-Manual/RM06/Chap6AATIntro.html:

For simple glyphs, you should set bit 6 of the first Outline Flag byte to 1 if the unhinted glyph outline has overlapping contours or if variation controls or hinting controls can ever cause any of the contours to overlap. Otherwise this bit should be set to 0.

Should the woff2_compress code be patched so the flag stays intact? I don’t have the experience in C to do it myself.

[jenskutilek]

Correction: The OT spec 1.8.3 does not list the flag as reserved anymore, so I think that woff2_compress ignores the flag is a legitimate bug.

See https://docs.microsoft.com/de-de/typography/opentype/spec/glyf#simple-glyph-description

[vlevantovsky]

Well, while I agree that this ought to be fixed so that woff2 works on all platforms, strictly speaking it's still not a bug from OT spec point of view. The flag description clearly says "it is valid to have contours overlap without having this flag set."

[khaledhosny]

I think this is a limitation of the WOFF2 transformed glyf table, as it has no place to save this flag. I think the only way around this is to store untransformed glyf table (transformation version 3). The woff2 library does not seem to have an option for this, and it will most certainly increase the file size.

[vlevantovsky]

According to the OpenType spec, "The OVERLAP_SIMPLE flag is used by some rasterizer implementations to ensure that a non-zero-fill algorithm is used rather than an even-odd-fill algorithm. Implementations that always use a non-zero-fill algorithm will ignore this flag. "
In current woff2 implementation the bit 6 is always set to zero as the original setting is lost, which creates an issue for some rasterizers. Hypothetically, I can see two possible ways to deal with this issue. First (using brute force, since the OpenType spec no longer demands this bit to be set to zero), is to always set the regenerated flag bit to "1" - this would be ignored by rasterizers that always use non-zero-fill algorithm, but would force all other rasterizers to apply non-zero-fill algorithm, thus avoiding dropouts. Second (more complicated) option is to extend the decoder implementation and run a overlap check on regenerated glyph contours to see if there is indeed an overlap, and set the bit 6 accordingly.

[jenskutilek]

I see, it's a tough problem ... would be easiest if Apple detected the overlapping glyphs correctly ;)

[anthrotype]

always set the regenerated flag bit to "1" - this would be ignored by rasterizers that always use non-zero-fill algorithm, but would force all other rasterizers to apply non-zero-fill algorithm

originally only MacOS rasterizer looked at those bits while they were reserved, but after the OT table allowed them, FreeType decided to also use them when present to enable oversampling (to mitigate effect of increased pixel coverage when paths overlap inside a glyph), see fonttools/fonttools#2059

[davelab6]

@anthrotype pointed out on chat that fonttools ttLib.woff2 compress script has a --no-glyf-transform option, and automatically disabling glyf transform if any of those flags is detected could be done in fonttools (and this c++ implementation)

[rsheeter]

Potentially we spend a reserved bit (https://www.w3.org/TR/WOFF2/#woff20Header) to flag the existence of a new datastructure that gives you flag bits?

[vlevantovsky]

Potentially we spend a reserved bit (https://www.w3.org/TR/WOFF2/#woff20Header) to flag the existence of a new datastructure that gives you flag bits?

At this point, there is only one flag bit (bit 6) that we are missing in the WOFF2 transformed glyf table structure (the remaining bit 7 is reserved). The added bonus is that this particular bit (unlike all other flag bits) can only be set once, on the first flag for the glyph. So, using one reserved bit in the WOFF 2.0 header to indicate, for example, the presence of a new flag6Bitmap[] (a numGlyphs-long bit array, similar to bboxBitmap[]) in the end of the transformed 'glyf' table would be a backward compatible way of making it happen.

[anthrotype]

What about composite glyphs with overlapping components? They also have their own distinct flag.

[vlevantovsky]

What about composite glyphs with overlapping components?

Composite glyph flags values are already part of the compositeStream[].

[rsheeter]

using one reserved bit in the WOFF 2.0 header to indicate, for example, the presence of a new flag6Bitmap[] (a numGlyphs-long bit array, similar to bboxBitmap[]) in the end of the transformed 'glyf' table would be a backward compatible way of making it happen.

Noob question: what do we have to do to make spec updates to woff2 at this point? - not too clear on the process.

[vlevantovsky]

We'd need to discuss the details with Chris L., but there is definitely a path forward either as a minor version upgrade to 2.1 (which may require going through the same steps of ED/CR/Rec) or, considering the relatively small scope of changes (and if there are other spec corrections needed while we are at it), there may be an opportunity for a lightweight erratum including candidate changes.

The good news is that the maintenance of the WOFF2 spec is part of the existing WG charter so that we can start this project immediately, without any additional approval, and if we go with erratum, once the candidate corrections/additions are reviewed and approved by the community they can immediately be folded into the Recommendation text as normative.

[davelab6]

Wow that's amazing news! Cheers Vlad!

[vlevantovsky]

Proposed as agenda+ topic for WebFonts WG call next week.

[vlevantovsky]

The WebFonts WG has discussed this issue during today's call, please review and submit your comments on the proposed changes. Thank you!