Although this article shares its title with an article from four years ago that was about the excitement associated with attending ATypI Hong Kong 2012, this particular one will focus on efforts to properly support Hong Kong SAR (aka HK or Hong Kong) in the Adobe-branded Source Han Sans and Google-branded Noto Sans CJK typeface families, but also in infrastructure, such as OSes and apps.
In other words, this article is not about traveling to Hong Kong, but rather about properly supporting Hong Kong in OSes, apps, and fonts.
A peculiar series of events that took place on April 1st (no joke) and 2nd of this year led to the discovery of what can only be described as somewhat of a revelation: A small number of CJK Compatibility Ideographs are necessary for China. This is important, because I made the following statement on page 168 of CJKV Information Processing, Second Edition:
One of the most powerful font-development tools available today is tx (Type eXchange), which is included in AFDKO (Adobe Font Development Kit for OpenType) and whose sources are available on GitHub. Despite its two-letter name, this command-line utility is packed with an enormous amount of features and functionality.
Four years ago I wrote a similar article, but it seems like a good time to revisit tx and the useful things that it can do. I still recommend that its “-u” and -h” command-line options be used to explore its vast capabilities.
—Humans make mistakes—
—Anything made by humans has the potential to include mistakes—
The most important things about mistakes are that 1) we recognize them, lest they propagate; 2) we learn from them; 3) we make an effort not to repeat them; and 4) we try to fix them, if possible.
Some mistakes are more easily fixed than others. Mistakes that cannot be fixed must be worked around.
With that said, an interesting event of historical significance occurred in June of 2000:
The first version of the IVD (Ideographic Variation Database) was issued on 2007-12-14, meaning over eight years ago, and there have been three subsequent revisions, the latest being issued on 2014-05-16. There are currently three registered IVD collections: Adobe-Japan1, Hanyo-Denshi, and Moji_Joho. A significant number of IVSes are shared between the latter two IVD collections, 9,685 to be exact. While I cannot speak to the latter two IVD collections, the Adobe-Japan1 one is supported by hundreds of OpenType fonts via the Format 14 (Unicode Variation Sequences) ‘cmap‘ subtable. Furthermore, the number of apps and OSes that support UVSes has reached critical mass.
Much of the thinking that I did with regard to this unregistered—but hopefully soon-to-be-registered—IVD (Ideographic Variation Database) collection was done while visiting my parents in South Dakota, with one of the highlights of that trip being a scenic drive through Badlands National Park.
First and foremost, please forget, or at least ignore, most everything that was written in the 2016-02-13 and 2016-02-20 articles (which makes one wonder why I am linking to them, but I digress). Far too many things have changed, and what I present in this article represents the IVD collection that I hope will be registered later this year.
One of my longer term goals for the open source Source Han Sans project has been to eventually register a Pan-CJK IVD (Ideographic Variation Database) collection that would allow the regional variants to display and be preserved in “plain text” environments, and I think that I may have achieved a breakthrough the other day.
CJK Unified Ideographs is a very deep and fascinating subject, and there are people who sometimes— and anecdotally—claim that it is a bottomless pit (or infinite tunnel, hence the quote in this article’s title). The latter may appear to be accurate, especially when one considers what is happening on that front, such as Extension F with its 7,473 new characters slated for Unicode Version 10.0 in June of 2017, and with work on Extension G commencing.
One of the fringe benefits of moving offices—especially when one has accumulated nearly 25 years of font-related material and it is thus not a pain-free exercise—is discovering historical documents, some of which turn out to be true gems. Our team is preparing to move from the Adobe East Tower to the West one, and part of the process is figuring which material to keep, and which to put into File 13. Anyway, I had been recently looking for a particular presentation that I prepared many years ago, and was fortunate enough to come across it while sifting through my accumulated materials.
It is with great sadness that I write that Unicode Version 9.0, whose beta was authorized yesterday, on the last day of UTC #146, will include no additional CJK Unified Ideographs. The next opportunity for additional CJK Unified Ideographs is therefore Unicode Version 10.0, which is slated for a June 2017 release, and is expected to include 21 Urgently Needed Characters (UNCs) that are appended to the URO (Unified Repertoire & Ordering), along with Extension F (see IRG N2156 for the latest version) that currently includes 7,473 characters.
Interestingly, and as long as Extension F’s block remains stable, there are only 3,088 code points remaining in Plane 2 (SIP), specifically U+2EBF0 through U+2F7FF, along with 1,502 code points at the end of Plane 2, immediately following CJK Compatibility Ideographs Supplement, specifically U+2FA20 through U+2FFFD.
The image above is an excerpt of a PDF that shows what Unicode Version 10.0 is likely to include in terms of ideographs. If you click on the image, you will get the actual PDF. Of course, the yellow stuff is tentative and subject to change.
Updated on 2016-06-26 to reflect the additional UNC appended to the URO at U+9FEA, along with a net decrease of 12 characters in Extension F, reducing it to 7,473 characters.
In late 2015, I collaborated with Daisuke MIURA to submit a proposal (L2/15-328) to the UTC (Unicode Technical Committee) to encode the characters for four tally mark systems. The proposal was discussed during UTC #146, and the result was that the five ideographic tally mark characters were accepted. Good news.
The Script Ad Hoc Committee originally recommended in their report for UTC #146 (see page 9 of L2/15-037) that IDEOGRAPHIC TALLY DIGIT TWO not be encoded, because they felt that it could be unified with U+1D36E (COUNTING ROD TENS DIGIT SIX), but concerns over typographic consistency led to it being accepted as a separate character.
In December of 2015, Unicode launched their Adopt a Character campaign, whose goal is to raise funding for the purpose of encoding a large number of remaining scripts, along with encoding additional characters for scripts that are already encoded. In other words, to help Unicode do its important work. The Unicode Consortium has 501(c)(3) tax status, meaning that donations are tax-deductible in the US, and if your company supports matching grants, you can leverage that to significantly increase the effective donation.
By default, the AFDKO makeotf tool includes Macintosh (platformID=1, encodingID=0, languageID=0) ‘name‘ table strings, and if specified in the “FontMenuNameDB” or “features” files, localized Macintosh ‘name’ table strings will also be included. The next release of AFDKO will include “-omitMacNames” as a new command-line option for makeotf whose purpose is to exclude Macintosh ‘name’ table strings, other than any that are explicitly specified in the “features” file.
IUC39 (The 39th Internationalization & Unicode Conference) took place in Santa Clara earlier this week, and Adobe was once again proud to be a Gold Sponsor. It was another outstanding and successful conference, and as usual, one of the greatest benefits of the conference—besides the many excellent presentations—was the opportunity for face-to-face exchanges with Unicode leaders, experts, and enthusiasts.
(The introductory graphic illustrates how the character 剣 (U+5263) is displayed using the fonts that are introduced in this article. The code point for this character maps to a glyph that displays as “63” in the FDArray Test 257 font, which is the hexadecimal equivalent of the decimal index of the FDArray element to which its glyph is assigned, which is 99. Likewise, the code point for this character maps to a glyph that displays as “52” in the FDArray Test 65535 font, which is the hexadecimal equivalent of the decimal index of the FDArray element to which its glyph is assigned, which is 82.)
I have built several CID-keyed OpenType/CFF fonts that are specifically designed to test various limits, by exercising various implementation limits, such as the number of glyphs (65,535 is the architectural limit), the number of FDArray elements (256 is the architectural limit), and the number of mappings in the ‘cmap‘ table (when the surrogates and non-characters are factored out, Unicode has 1,111,998 possible mappings in its 17 planes). I have sometimes made these fonts available, such as in this May of 2012 article that explains how such fonts can be built.
Anyway, I spent pretty much all day yesterday—except for a somewhat longer than usual lunch break that was actually used to watch The Martian (2015) with my wife—preparing a pair of open source CID-keyed OpenType/CFF fonts that exercise these limits but to different degrees, and I also managed to prepare and release the project on GitHub as FDArray Test.