support for non latin characters

[albalansi]

Hi everyone. i ran one of the examples in the vignette on an Arabic corpus made up of news articles.
there is this one step:

x$text <- txt_clean_word2vec(x$text)

which converts text from UTF-8 to ASCII amongst other things. At first i skipped it and the model ran well i guess and I got some results. I tried again but this time run the command and obviously the text column is now blank because Arabic characters are not part of ASCII characters.
1/ so my question is, is it okay to skip this step? won't it affect latter ones?
2/ when I skip it and I print(info, top_n = c(5,2)) I get some text back but i also get NAs. why is that ?
the results show topics with words but shows NA for articles like so:

print(info, top_n = c(5,2))
Top2vec model with 3 topics
Topic distribution: 0.67 0.13 0.19

Topic 0: زيادة المراحل لاحد نركز وقيمته

(1) NA
(2) حضرت الرياضة فى جوائز الاوسكار هذا العام ففاز لاعب كرة السلة الامريكى السابق كوبى براينت بجائزة الاوسكار لافضل فيلم رسوم متحركة قصير والفيلم بعنوان عزيزتى كرة السلة ومستوحى من قصة حياته الحقيقية وفاز الفيلم الوثائقى ايكاروس للمخرج الامريكى بريان فوجيل...

Topic 1: الحبس الاجنبية خوسيه العظمى لاعبيهم

(1) NA
(2) NA

Topic 2: الحقيقة ويحرص كانو البدني نقلا

(1) NA
(2) NA

I truncated the output but as you can see, I only got one article back and the rest of the results show only words but no articles they are all NA.

thank you for the package at least it runs, i tried to install the python one but I couldn't get it to work due to various package conflicts.

[albalansi]

just to clarify,
I was using urls as doc_ids which contained arabic characters. This I guess might have messed up the results because after I changed doc_id to something like "doc_nbr" it worked just fine.
my first question still holds though, are there any undesirable effects if i use UTF-8 encoding and not perform txt_clean_doc2vec() ?