Hi,
I want to cast a string generated by flatty (a serialization lib) to a cstring. But when the string contains a char with value over 127 the string gets completly wrong. I guess that has something todo with chars over 127 normaly indicating the beginning of a utf8 char, but not being a correct utf8 char?
Heres the code of a little isolated test of the problem:
import flatty
let str = toFlatty(128)
for c in str: echo ord(c)
echo "-"
for c in cstring(str): echo ord(c)
result: 128 0 - 37 56 48 0
thanks,
chol
A serialization library usually has no special logic for UTF-8, it just see strings as raw bytes.
However int8 can store up to 127 and beyond you need int16.
A serialization library usually has no special logic for UTF-8, it just see strings as raw bytes.
yes i know, I thought that is the problem, that js tries to make the string into valid utf8 or something, which it isnt.
However int8 can store up to 127 and beyond you need int16
yes but a string is a char array, and a char is a uint8 isnt it? and a uint8 goes up to 255. I mean the nim string works just fine.
JS strings (cstrings) are UTF-16, which means the cstring items iterator which returns chars needs to deal with UTF-16 on the JS backend somehow, since they can have "characters" bigger than Nim's char type allows. My guess is this is where the difference happens, but I am not sure if the cstring conversion gives String.fromCodePoint(128, 0) either.
Although 128-255 don't mean anything in UTF-16, so maybe these should be preserved when converting back.
Don't know why you want to convert to cstring here though.
So is there any other way to convert to cstring? Probably just by importing concat method and building the cstring manually from chars or something?
Don't know why you want to convert to cstring here though.
the problem arrived in a server-client webframework Im developing. and for automatic remote-proc-calls I need to send it via xhr and for that I need cstring. And I used flatty because I couldnt find any other lib that supports js
It seems like fixing that will be a bit unnecesary complicated. So Ill probably just try to implement a marshal that works on js. That would be the better solution anyways.