Dec 172009

I’ve been experiencing grief trying to work with Ruby based tools to build a web-based server (in Part One) and a client (read on).

So. I have to write some Ruby code to act as a client to a remote server using HTTP GET/POST requests, where parameters are sent using the usual HTTP techniques. Just like a form in a browser would do. No XML. No JSON. Just good old HTTP.

I touched on one particular aspect of the server side of things in Part One.

For maybe nine months we’d been using the HTTP client built into Ruby but a few weeks ago started experiencing serious perfomance issues. If it’s bad enough to interfere with development then it certainly won’t work in production. So we had a look around and settled on Patron. Patron is a nice little gem wrapping libcurl. It is very fast, certainly fast enough for our purposes, has proved reliable, and easy to use. We’ve been using it in production, well, we were until this afternoon but the timing is co-incidental.

Patron has a flaw. (Yeah, I’m going to pick on Patron, even though I seriously doubt it is alone here.)

You see, Patron takes its post data in a hash map. Of course hash maps have unique keys, so if multiple values are to be sent you have to associate an array with the key. Or so we supposed. Yeah, we supposed wrong. Patron accepts that form of data but does something seriously wrong with it. It basically converts the array to a string, a string that looks exactly like an array looks when printed in Ruby. This means that multiple values are silently converted into a single value. Secondly that the single value looks just like the array would look when printed in a debugging log.

Patron then sends the data to the server, which, correctly, sees a single value. This is not how even the monkey patched Rack recognises multiple values (Rack parses the key looking for indicators of multiple values, not the value itself.)

So the server gets the wrong thing, and since one is normally a perfectly valid count for a multiple valued parameter, there are no errors reported and, once again, the debugging log looks perfectly okay. But nothing works.


Anyway, like I said, Patron is a wrapper around C and after a quick perusal of the code it looks as though it flattens the value in C not Ruby, and anyway I’ve never been successful monkey patching a method that maps to the Ruby FFI.

Off I go looking for an alternative.

We ended up using typhoeus. Very nice indeed, all the good things about Patron including wrapping libcurl, but it actually generates a proper post. And our client works now.

Paul Dix wrote typhoeus. He also wrote Feedzirra a really nice feed reading library. Not bad Paul. Two excellent gems.

I’ve put some code in this gist that will demonstrate the issue. Have a look if you’re interested.

 Posted by at 8:07 pm

Ruby/Rack and Multiple Value Request Param Pain — Part One

 Ruby, Tools I use  Comments Off on Ruby/Rack and Multiple Value Request Param Pain — Part One
Dec 162009

Today, in quick succession, I encountered a frustrating situation with each of two tools. It was the same problem in both cases—dealing with multiple value request parameters. There’s something you think about every day. Anyway I was tracking down a problem that has been causing us grief for weeks now, but we never spent the time to actually deal with it. What makes this even more annoying is that both these tools (Rack and Patron) are otherwise really pretty good.

Here’s the situation. The HTML select element supports multiple selections. When the browser posts to the server, the multiple selections are passed back as key-value pairs with the key repeated once for each selection.This is standard HTTP POST behaviour.

I’m going to pick on Rack in Part One. This was especially annoying since a) Rack is so solid; b) Rack is infrastructure that you expect to just work. In other words, this was a surprise.

Rack wants to store parameters in a hash map, which is a perfectly reasonable Ruby thing to do. The thing is, hash maps have unique keys. So if you simply assign the keys and values sent from the browser you’ll write over the previous value, and in the end have only one value (the last one selected). This is what Rack does unless you use naming conventions to signal your desire for an array result. So if you named your <select> as ‘several’ then you’ll never get more than one key-value in the Rack params object. Doesn’t matter how many there were, you get zero or one. If, however, you named your select as ‘several[]’ you’d get a key mapped to an array with all of the values (or so it seems, I didn’t bother trying). There is a more complex naming convention supported as well, but I’ve got little interest in that.

This is not what I expected. And it sure isn’t what I need.

This approach presumes you actually have control over the names of those select elements. As it happens in the application we’re developing we don’t have any such control. Moreover, if you are trying to implement a REST API that uses HTTP post for communication (rather than something like XML or JSON) then you’ve got the potential for similar difficulties, and there’s no way you’ll have control over the names on a published API.

So how to fix this? Well, this is where Ruby comes to the rescue, well supported by the rather clean implementation of Rack.

You monkey patch Rack. A lot of people consider monkey patching a Very Bad Thing. They are wrong. So what’s monkey patching? When a client of a library (that’d be our application, or the examples in this gist) redefines a method, or extends a Class, or something similar, the client is monkey patching. In Ruby we’ve had a lot of mileage out of Ruby’s support for this. It is used to extend the language, and to fix things. Like this problem.

The relevant lines are 15 through 22 in the gist. In Rack’s code base these lines correspond to a single line that overwrites existing key-value mappings. In the monkey patch, I’m allowing for a mapping from a key to an array of values. The rest of the code is untouched and so the other naming conventions will be as before.

The other two files are just two test cases, one without, the other with the monkey patch.

Anyway, that fixes the first issue. The second, I’ll talk about in Part Two.

 Posted by at 9:50 pm