newLISP Fan Club

Forum => newLISP in the real world => Topic started by: vetelko on January 27, 2017, 12:37:34 PM

Title: regex doesn't match utf-8 character using \w pattern
Post by: vetelko on January 27, 2017, 12:37:34 PM
Hello newLISP-ers :)



the following code returns nil for utf-8 character

(regex {w} "č")

and so the following

(regex {w} "č" "u")

while this works

(regex {w} "m")



Is this not implemented or am I doing something wrong?

VT
Title: Re: regex doesn't match utf-8 character using \w pattern
Post by: rrq on January 27, 2017, 01:26:26 PM
I believe the magic of regex is explained in http://www.newlisp.org/downloads/pcrepattern.html, where you find the two paragraphs:

------

 A "word" character is an underscore or any character less than 256 that is a letter or digit. The definition of letters and digits is controlled by PCRE's low-valued character tables, and may vary if locale-specific matching is taking place (see "Locale support" in the pcreapi page). For example, in the "fr_FR" (French) locale, some character codes greater than 128 are used for accented letters, and these are matched by w.



In UTF-8 mode, characters with values greater than 128 never match d, s, or w, and always match D, S, and W. This is true even when Unicode character property support is available. The use of locales with Unicode is discouraged.

------
Title: Re: regex doesn't match utf-8 character using \w pattern
Post by: vetelko on January 28, 2017, 03:02:18 AM
So there is no way how to match this character as a word character using pattern?