Wednesday, December 23, 2009

Google App Engine Datastore "IN" operator

This is the first Desert Island-related technical post. In this article, I am outlining a couple of gotchas regarding the use of the "IN" operator in GQL.

In, I created a class DesertIslandList, which represents a user's list of his/her 8 desert island albums. It, amongst other properties, contains user_id and id_source. For a user who's logged in via Facebook Connect, the user_id of his/her created list would simply be his Facebook profile id and id_source would be "f" (for Facebook).

In order to display lists created by his/her friends, I, first of all, get his/her friend's id list using friends.get method in the Facebook RESTful API with a version of minifb hacked up for Google App Engine. I then created a GQL query for all DesertIslandList WHERE user_id IN :1 AND id_source = :2 with :1 being the Python list containing all the friend ids as strings, and :2 being "f". Straightforward enough? Well it all seemed so until I stumbled across of a couple of gotchas.

Only 30 items allowed

First of all, I soon got a "too many subqueries (max: 30..." exception. As it turns out, only 30 items are allowed in the list:

(from Note: The IN and != operators use multiple queries behind the scenes. For example, the IN operator executes a separate underlying datastore query for every item in the list. The entities returned are a result of the cross-product of all the underlying datastore queries and are de-duplicated. A maximum of 30 datastore queries are allowed for any single GQL query.

So, my bad, not reading the documentation properly. To get around this, you could either somehow limit the list to 30 items or chop your list up into 30-item chunks and union the results (sorting offline if necessary).

What if the "IN" list is empty?

What if he or she has no friends?

Convention SQL wisdom would suggest that the query should return no DesertIslandList objects. However, in the world of Google App Engine datastore, "IN" means cross-product of all the separate individual queries, so in my GQL query above, an empty "IN" list effectively means get all DesertIslandList WHERE id_source = "f" i.e. not filtering on the user_id field at all!

Not only is this slight counter-intuitive, this can also throw a surprise "no matching index found" exception. The fact that an empty "IN" list does not filter on the user_id field means my GQL instance with an empty "friend ids" list has to access an additional DesertIslandList index with only the id_source field, and not the usual index with both the user_id and the id_source fields. Imagine this, if you have not run a GQL instance with an empty "IN" list in your local dev environment for this particular query, no index (with only the id_source field) will be added to your index.yaml. Up in the cloud, if a person happens to have no Facebook friends, he or she is going to see an ugly stacktrace complaining the the index does not exist!

Of course, the simple solution to this problem is to always check the list to see if it is empty before running the query.

Just another thing to bear in mind!

First time film processing...

Finally got round to getting my hands dirty (not literally) and processing my first film rolls (Ilford HP5 Plus ISO 400) with the assistance of a kind colleague of mine.

Using a changing bag, I tranferred the two rolls of film from the canisters onto the plastic reel, which was then placed into the developing tank. After 30 mins or so of pouring various fluids (developer, fixer, warm water) in and out of the tank, the images finally appeared on the negative films. With the films being hung up by the door to dry overnight, I was extremely eager to scan them into Photoshop to get a proper look at them.

The following day, I gingerly placed the negs onto the glass of my Canon flatbed scanner (using the film adapters that came with the scanner), started up Photoshop and got my first thumbnail previews of the shots. They looked OK, so I went ahead and scanned a few of them at maximum (3200 dpi) resolution. Much to my surprise, they came out extremely grainy, bordering on being rubbish. I love film grains, but what I was seeing was definitely not the level of grains that you would expect from an ISO 400 film. Worse still, one of the rolls had a vertical streak running down every single frame, and I am still not sure whether it was due to a mistake during processing or from the camera itself (I used different cameras for the two different rolls). Since the other roll does not exhibit this defect, I suspect it is more likely to be the latter.

Anyway three of the "better shots" (which I actually quite like) that came out - the first two I took using the my Canonet 19QL (£15 from eBay!) and the last one using my Cosina 35E (£2 from eBay...) - notice the vertical streak in this last one:

Aunt Sally

Welcome to Kat's party!

Notice the vertical streak in the following photo:

Music to walk by

Friday, December 11, 2009

Druid Street Cycles - best LBS in London?

Update: Thor and co. have recently set up an official Facebook group. And he is promising 10% discount for all group members :-)

Their address: 18 Druid Street, London SE1 2EY gmap
Phone: 07551016380

Vintage Sun Solo RacerHaving recently bought an old Sun Solo racer from Gumtree (80s Raleigh "gaspipe" steel frame, 27" x 1 1/4 rims, Weinmann single-pivot brakes, Sturmey Archer stem, Sturmey Archer hub, 5-speed), I took it to the little known Druid Street Cycles near London Bridge for a quick service.

Not a lot has been mentioned on the web about this social business project, apart from a couple of youtube videos, a user comment in a Timeout article and an LCC article (they themselves don't have a website):

Thor, who used to repair tanks in his native Germany, started this project to provide affordable (and yes, really affordable) bike repair service for the local community in Southwark. To say that he, along with the rest of Druid Street Cycles, is passionate about building/fixing bikes is an understatement.

I have been there twice now, and on both occasions, the jobs they did on the bike far exceeded my expectation. The first time round, they did a full checkup of the bike, fixed the rear brake, and replaced the dodgy tyres. He ended up only charging me 30 pounds (the tyres along cost 20 quid!) - he even crimped the cable ends for me, such is his attention to details! Second time round, I went in to see if they have any mudguard that would fit over my 27" wheel, which I knew was very hard to come by. In typical Thor's fashion, he said he would find something, and surely he did, somehow he managed to fit a full-length mudguard beautifully on the 27" wheel frame, which has minimal clearance. Initially he refused to take any money from me, but in the end I managed to convince him to take the money, which no doubt would be put to good use to benefit the local biking community.

I really cannot recommend them enough, and I hope more people will get to know about this secret little place hidden underneath one of London Bridge's dark arches.

Ah how can I forget the mention the fact that they provide free courtesy bikes to customers, so you can continue your commute to work or your cycle home while Thor and co. are busying fixing yours! Beat that!

Looking for a commute route, or wanting to share your route with other cyclists? Visit my cycle route database:!

Tuesday, December 08, 2009

Desert Island Discs for the non-celebs

I have just finished my second Google App Engine project: "Desert Island Discs for the non-celebs"

This is just a little fun project to test out the powerful Facebook Connect API, and to further explore Google App Engine (to find yet another Datastore oddity/feature, which I will blog about soon).

Try it out and add you list!