Repository navigation
Encoding problem #69
Description
Hello, I've a pretty large dataset (> 2 TB) split in six files.
I assumed that UTF-8 were the text encoding of jsonl files. However there are some charachters that apparently are non-UTF.8 and this causes R to fail when I specify the encoding.
Not specifying the encoding results in a messy full_text output
Activity
edsu commentedon Dec 10, 2020
Do you have an example?
LeonardoSanna commentedon Dec 10, 2020
Update, I found I walkaround:
The problem was with the function
stream_inproducing the errorError in FUN(X[[i]], ...) : invalid multibyte string, element 1while streaming the json file in a dataframe.Not specifying while importing solves the issue, though
fileEncoding = "UTF-8"must be specified while writing on the outfileHowever there are still some weird charachters under "full text" I think because of emojis
These are unicode emojis and I'm ok with that
RT @ScottAnthonyUSA: <U+26A0><U+FE0F> IT SHOULD BE NOTED that the CDC initially had an embargo placed on CDC testimony. The TRUMP ADMINISTRATION LIFTED…But what about this? iOS emoji?
RT @StocksUnhinged: $SPY $AAL $APT $EXPE $GOOG $DAL $UAL $BA $LAKE $YUM $CMG $HUM $CI #CDC expected to announce first US case of #Wuhan…edsu commentedon Dec 10, 2020
If you can give me a tweet id that will help me test.
LeonardoSanna commentedon Dec 10, 2020
1219771346768596992this the one with the dollar signs