librelist archives

« back to archive

Thoughts on passing non utf-8 strings via Resque

Thoughts on passing non utf-8 strings via Resque

From:
Thibaut Barrère
Date:
2010-07-18 @ 13:47
Hello,

just bouncing ideas and sharing in case it's useful to someone else,
too. I'm using Resque ~ 1.5.0 and Redis 1.2 here.

I have some content that arrives from a third-party system and get pushed
as ISO-8859-1 XML into a Rails Metal back-end (ubuntu based).
The data is then enqueued as a string in Resque (using json-pure-1.2.1).
Afterwards, a Windows JRuby worker (also using json-pure-1.2.1) grabs the
data, iconv it to UTF-8 and works from there.

In the very large majority of my cases, just pushing the string without much
thinking just works. In some edge cases though, the job reserve fails on the
worker side (this ends up with no error notification at all, and the process
exit).

My understanding is that Redis (1.2) knows nothing about encoding (I
remember to read this somewhere), but the json decoder has expectations.

My solutions are:
1/ either iconv to utf-8 before pushing (which makes it less scalable if it
takes some time)
2/ uuencode/uudecode the payload (which I believe could be slightly faster,
but makes the message unreadable from resque-web, which is convenient).

I'll probably pick the first solution in my case.

Did you have to handle non-utf-8 encodings with Resque ? What was your
solution ?

thanks,

-- Thibaut