Replacement characters are those black diamonds with a ? in the middle. They are hard to detect. This has info how to do it in a bash script. https://unix.stackexchange.com/questions/474709/how-to-grep-for-unicode-in-a-bash-script/474812#474812
Is there a way to do it in Nim? Thanks.
The replacement character is used to replace any invalid Unicode character found in the text. It is a "visual aid", not a real character.
https://nim-lang.org/docs/unicode.html
import unicode
var s = "hello"
s.add cast[char](247) # check-evading
s.add cast[char](205)
s.add cast[char](257)
s.add "there"
echo s
echo s.validateUtf8 # not -1
for r in s.utf8:
if r.validateUtf8 == -1:
echo r # only good values