Posts tagged with “python” and “programming”
File reading performance in Python
There are a few ways to read a file in Python, some of which are outlined in this page about their relative performance. I am working on a project right now that involves reading large amounts of data from text files, so I repeated the analysis on Python 2.6.6, the version currently shipping with Ubuntu 10.10. I ran three implementations (below) against a file with 1 million lines.
My test script is available here, and the functions I tested are below. Here were my results:
| Script | Time (sec) | Lines read per sec |
|---|---|---|
| fileread1: | 0.1695 | 5,899,280 lines/sec |
| fileread2: | 1.6387 | 610,236 lines/sec |
| fileread3: | 0.1278 | 7,823,156 lines/sec |
def fileread1():
file = open("test.txt")
while 1:
line = file.readlines()
if not line:
break
pass
file.close()
def fileread2():
for l in fileinput.input("test.txt"):
pass
def fileread3():
file = open("test.txt")
for l in file:
pass
python-mysqldb: execute() first
While working on implementing a schema-free, MySQL-backed data store (thanks, Bret Taylor!), I ran into a problem with using MySQLdb to access the database. I'll eventually post the code I wrote up on this site so others can see my example, but for now the following will suffice. When performing a SELECT, I would get the following error upon attempting to fetch my results.
Incorrect Python:
q = "SELECT body FROM entities WHERE id='%s'" % (entity_id)
self.conn.cursor().execute(q)
entity = self.conn.cursor().fetchone()
Error:
File "datastore.py", line 93, in get
entity = self.conn.cursor().fetchone()
File "/usr/lib/pymodules/python2.6/MySQLdb/cursors.py", line 340, in fetchone
self._check_executed()
File "/usr/lib/pymodules/python2.6/MySQLdb/cursors.py", line 70, in _check_executed
self.errorhandler(self, ProgrammingError, "execute() first")
File "/usr/lib/pymodules/python2.6/MySQLdb/connections.py", line 35, in defaulterrorhandler
raise errorclass, errorvalue
_mysql_exceptions.ProgrammingError: execute() first
The issue was using two separate cursor objects. Here's the corrected code:
c = self.conn.cursor()
q = "SELECT body FROM entities WHERE id='%s'" % (entity_id)
c.execute(q)
entity = c.fetchone()