Add support for (32-bit) Wide-character Unicode

[apatpgenengl]

While 16-bit Wide-character Unicode is supported, 32-bit Unicode isn't.
In my C environment, we are using wchar_t by default for all characters;
encoding UTF-8 --> stemming --> decoding UTF-8 would be quite inefficient for a big number of terms.

At the time, in libstemmer_c/runtime/api.h i'm defining:
typedef wchar_t symbol;
and can so use the ISO-8859 stem variants without conversions.

For some of the provided languages only the UTF-8 variants exist, what leads us to this issue.

[ojwb]

Have you tried running the compiler with -widechars and using typedef wchar_t symbol;?

[apatpgenengl]

I did not run the compiler; i used the libstemmer_c from https://snowballstem.org/dist/libstemmer_c-3.0.1.tar.gz

[apatpgenengl]

Have you tried running the compiler with -widechars and using typedef wchar_t symbol;?

I now used the compiler as suggested, but i still don't see any possibility to avoid the encode UTF-8 --> stem --> decode UTF-8 steps for stemming wchar_t native terms.

Do i miss something?

[ojwb]

While the compiler has supported generating C code for wide characters for over 20 years, it's never been hooked up in libstemmer so currently you'll need to use the slightly lower-level API in runtime/api.h. (At least I'm guessing you're trying to use libstemmer - if not you'll have to show what you're doing that seems to require UTF-8.)

I'm not aware of anyone actively using -widechars though so it is possible the code generation or support code isn't quite right currently. Much of the wide character support is well tested by e.g. Java and Python but the C-specific parts of it probably haven't been exercised since 2003 or before.

I'm fairly sure you're actually the first person to even ask about it which suggests there's not much interest in it, so I'm dubious about extending libstemmer to support this as it would complicate things as we'd need to duplicate all the runtime code (for two different symbol typedefs) and API functions such as sb_stemmer_stem(). It'd also significantly increase the size of the library (currently there are 54 generated stemmers - wide character support would add 34 more).

Perhaps we could have a mode which builds a wide variant of libstemmer, but that would probably need to wait until we redo the build system. So best to try via runtime/api.h API for now.

[apatpgenengl]

Thx @ojwb for the quick reply!

You're right, i'm trying to use libstemmer in C using wchar_t as symbol (which works well for the ISO-8859 generated stemmers).
Since the code will be used in production, i decided not to use widechars in C (too risky, if it hasn't been tested since 2003).

I also fully understand, that you won't implement my request, if it's just for me.

If you ever redo the build system in favor of libstemmer-wide, please let me know!