Posts tagged with “python”
File reading performance in Python
There are a few ways to read a file in Python, some of which are outlined in this page about their relative performance. I am working on a project right now that involves reading large amounts of data from text files, so I repeated the analysis on Python 2.6.6, the version currently shipping with Ubuntu 10.10. I ran three implementations (below) against a file with 1 million lines.
My test script is available here, and the functions I tested are below. Here were my results:
| Script | Time (sec) | Lines read per sec |
|---|---|---|
| fileread1: | 0.1695 | 5,899,280 lines/sec |
| fileread2: | 1.6387 | 610,236 lines/sec |
| fileread3: | 0.1278 | 7,823,156 lines/sec |
def fileread1():
file = open("test.txt")
while 1:
line = file.readlines()
if not line:
break
pass
file.close()
def fileread2():
for l in fileinput.input("test.txt"):
pass
def fileread3():
file = open("test.txt")
for l in file:
pass
pomodoro.py: A time-management tool for Ubuntu
pomodoro.py is a simple script to help increase your focus and productivity, in the Pomodoro style. It integrates nicely with Ubuntu 9.04+ via libnotify, and disables network access during “focus” periods. OSX support is in the works (but don’t hold your breath).
After you run pomodoro.py, you’ll have ten minutes of uninterrupted work time, which is enforced by disabling your network adapters to prevent you from browsing the internet. When ten minutes is up, you’ve earned a two minute break, and your network adapters will be brought back up.
Download at Github
Let me know if you find this useful (or if you find problems that prevent it from being useful to you)!
python-mysqldb: execute() first
While working on implementing a schema-free, MySQL-backed data store (thanks, Bret Taylor!), I ran into a problem with using MySQLdb to access the database. I’ll eventually post the code I wrote up on this site so others can see my example, but for now the following will suffice. When performing a SELECT, I would get the following error upon attempting to fetch my results.
Incorrect Python:
q = "SELECT body FROM entities WHERE id='%s'" % (entity_id)
self.conn.cursor().execute(q)
entity = self.conn.cursor().fetchone()
Error:
File "datastore.py", line 93, in get
entity = self.conn.cursor().fetchone()
File "/usr/lib/pymodules/python2.6/MySQLdb/cursors.py", line 340, in fetchone
self._check_executed()
File "/usr/lib/pymodules/python2.6/MySQLdb/cursors.py", line 70, in _check_executed
self.errorhandler(self, ProgrammingError, "execute() first")
File "/usr/lib/pymodules/python2.6/MySQLdb/connections.py", line 35, in defaulterrorhandler
raise errorclass, errorvalue
_mysql_exceptions.ProgrammingError: execute() first
The issue was using two separate cursor objects. Here’s the corrected code:
c = self.conn.cursor()
q = "SELECT body FROM entities WHERE id='%s'" % (entity_id)
c.execute(q)
entity = c.fetchone()