Mirror of forum.nim-lang.org

132 :: Unicode

• exhu (orginal) [2013-02-27T17:24:46+01:00] view original

  Nimrod / lib / pure / unicode.nim defines irune as int, while it's 64-bit on 64-bit targets, this is wrong, as UCS-4/UTF-32 code point is 32 bits and there only 24-bit code points allowed... The rest of procs also assume utf-8 can contain more than 4 bytes per code point, that's wrong because it's reduced to be compatible with UTF-16...

  p.s. code unit = int8 for utf8, int16 for UTF16 etc., code point is 32-bit, i.e. fully describes a font character. (Although a string is needed to describe a composed character...)

---

• exhu (orginal) [2013-02-27T17:38:20+01:00] view original

  And Nimrod / lib / system / widestrs.nim contains copy-pasta from unicode.nim

  Also widestrings are not portable and this way are discouraged for crossplatform development. It is not supported on iOS for example (vwsprintf and friends just strip higher (>=0x1FF) bits of wchar_t) that brought several hours of pain debugging our game localization =)

  So that I finally agree with http://utf8everywhere.org/ . This site has a lot of recipes for common cases and how unicode should be done in windows the portable way.

---

• Araq (orginal) [2013-02-27T19:12:58+01:00] view original

  I'm well aware of what the Unicode spec says. The unicode module implements a superset of Unicode if that leads to smaller code. ;-)

  Using more bits than necessary is not wrong, but unusual. The code still uses int as Nimrod's type system was heavily biased towards int. If it concerns you that much, change it and make a pull request.

  Believe it or not, the copy-pasta is deliberate as system.nim must not have any dependencies. OK, we could instead introduce yet another include file that contains the common definitions.

  Widestrings are only used for wrapping the Windows API which uses UTF-16 under the hood.

---