cam macrae - blog

I got an email from Jay yesterday commenting on Text processing fun. His mail server bounced my reply and also my request for permission to publish his message.

Jay created an elegant generator which over the same dataset of 1m rows, 180 columns processed the data a full 5 seconds faster than the list comprehensions in my code.

Here is my reply:

G'day Jay,

Great to hear from you.

Your code is very fast by comparison to the example, and very pythonic.

The whole thing can pretty much be achieved in one line:

rows = csv.DictReader(open('BigFile.csv', 'rb'))

but that doesn't really make an interesting post about using cProfile ;)

Thanks for your note - I hope you stick with python.

cheers, Cam.

I guess the post succeeded in one respect: Jay used cProfile/profile to prove his more elegant code was faster! But I missed the mark in trying to show that by profiling a quick but poor performing hack, you can satisfice and move on.

Anyhow, thanks Jay. I'd love to publish your note if you get in contact.