librelist archives

« back to archive

(fwd) Re: Python/JSON hackers meetup Saturday at 21st Amendment

(fwd) Re: Python/JSON hackers meetup Saturday at 21st Amendment

2009-12-12 @ 01:16
(forwarding to the list)

----- Forwarded message from -----

Date: Fri, 11 Dec 2009 17:15:08 -0800
Subject: Re: Python/JSON hackers meetup Saturday at 21st Amendment

On Tue, 08 Dec 2009, Bob Ippolito wrote:

> This is a great idea but unfortunately I have other plans this weekend
> and won't be able to attend.
> Some quick ideas:
> - I'd love to have a stream parser in simplejson, but that's not what
> I implemented because it can cause a lot of overhead if/when it does
> round-tripping to Python. I've always wanted to add that, I just don't
> have a use case so I didn't want to write and maintain all of that
> additional code. It's only practical to have only a stream parser if
> you are going to require C code or don't particularly care about
> performance. simplejson was written iteratively starting with pure
> python code.

I've not really looked at the simplejson code (with the exception of _speedups)
too much, but I would presume that adding iterative stream parsing wouldn't be
a huge patch to contribute. The biggest question mark for me is what the API
looks like, the use case (for me at least) comes into play when moving large
JSON documents around a network and wanting the receiver to be able to pass off
processing of parts of that document as they are received. I can think of some
simplistic means for doing this, but nothing I feel particularly fond of:

    >>> for element in yajl.iterloads(stream):
    ...    process(element)


> - Python 3 isn't very practical for anything I'm doing, but I'd
> definitely accept patches to make simplejson more compliant. Almost
> all of the Python parts should be, but I haven't taken any special
> care to make the C code compliant.

Thanks to some patches from a colleague of mine I actually was able to get
py-yajl running properly on Python 3 without too much pain. The biggest hurdles
were: testing, #ifdef'ing all the code using PyString*. There was also a
performance hit (py-yajl is ~20% slower on Python 3 than 2) which I haven't had
the time to determine whether it's "me" or Python 3's C API.

> If someone wants to help and it's license and API compatible I'd be
> more than willing to swap out large parts of the simplejson innards
> with something more elegant. It looks like yajl may fit the bill, but
> I haven't had a chance to look closely yet. I don't really think we
> need this many different implementations of the same fundamental API.

I agree, I originally started with my project as more of a exercise in writing
C modules (which I was doing at $WORK at the time) than a "I'm going to conquer
the Python JSON space!". 

Too bad you won't be able to come out tomorrow, I'm looking forward to it :)

-R. Tyler Ballance

----- End forwarded message -----
-R. Tyler Ballance