Posts tagged with “python”


Tue 24 Jul

Creating and importing custom Python packages

I'm building a custom Python package for a project I'm working on, and it took me more time than should have been needed to figure out how to achieve the import behavior I wanted for that package. The directory structure looks like this:

project_dir/
    foo.py
    package/
        __init__.py
        Bar.py
        Baz.py
        Qux.py

Each of the files in the package defines some classes; for now, we can assume they each have just one eponymous class. foo.py is the driver script that the user actually runs, which mainly just accepts command line arguments and imports the package.

What I'd like to be able to do in foo.py is say:

import package
b = package.Bar("asdf")
b.something()

If all my classes were in a single Python source file called package.py, this would be the default behavior. Alternatively, I could say from package import * in foo.py, but that would import my modules directly and frankly I think it looks ugly.

The way to achieve the desired behavior is to perform the imports of each module in the package in the __init__.py file, like so:

from Bar import *
from Baz import *
from Qux import *

Now, when we import the package, all the associated modules come with it.

I found this post very helpful in understanding how modules work, as well as the official documentation.

· Tags: ,

Sat 23 Apr

File reading performance in Python

There are a few ways to read a file in Python, some of which are outlined in this page about their relative performance. I am working on a project right now that involves reading large amounts of data from text files, so I repeated the analysis on Python 2.6.6, the version currently shipping with Ubuntu 10.10. I ran three implementations (below) against a file with 1 million lines.

My test script is available here, and the functions I tested are below. Here were my results:

ScriptTime (sec)Lines read per sec
fileread1:0.16955,899,280 lines/sec
fileread2:1.6387610,236 lines/sec
fileread3:0.12787,823,156 lines/sec
def fileread1():
    file = open("test.txt")
    while 1:
        line = file.readlines()
        if not line:
            break
        pass
    file.close()

def fileread2():
    for l in fileinput.input("test.txt"):
        pass

def fileread3():
    file = open("test.txt")
    for l in file:
        pass
· Tags: ,

Sat 4 Sep

pomodoro.py: A time-management tool for Ubuntu

pomodoro.py is a simple script to help increase your focus and productivity, in the Pomodoro style. It integrates nicely with Ubuntu 9.04+ via libnotify, and disables network access during "focus" periods. OSX support is in the works (but don't hold your breath).

After you run pomodoro.py, you'll have ten minutes of uninterrupted work time, which is enforced by disabling your network adapters to prevent you from browsing the internet. When ten minutes is up, you've earned a two minute break, and your network adapters will be brought back up.

Download at Github

Let me know if you find this useful (or if you find problems that prevent it from being useful to you)!


Tue 29 Jun

python-mysqldb: execute() first

While working on implementing a schema-free, MySQL-backed data store (thanks, Bret Taylor!), I ran into a problem with using MySQLdb to access the database. I'll eventually post the code I wrote up on this site so others can see my example, but for now the following will suffice. When performing a SELECT, I would get the following error upon attempting to fetch my results.

Incorrect Python:
q = "SELECT body FROM entities WHERE id='%s'" % (entity_id)
self.conn.cursor().execute(q)
entity = self.conn.cursor().fetchone()

Error:
File "datastore.py", line 93, in get
entity = self.conn.cursor().fetchone()
File "/usr/lib/pymodules/python2.6/MySQLdb/cursors.py", line 340, in fetchone
self._check_executed()
File "/usr/lib/pymodules/python2.6/MySQLdb/cursors.py", line 70, in _check_executed
self.errorhandler(self, ProgrammingError, "execute() first")
File "/usr/lib/pymodules/python2.6/MySQLdb/connections.py", line 35, in defaulterrorhandler
raise errorclass, errorvalue
_mysql_exceptions.ProgrammingError: execute() first

The issue was using two separate cursor objects. Here's the corrected code:
c = self.conn.cursor()
q = "SELECT body FROM entities WHERE id='%s'" % (entity_id)
c.execute(q)
entity = c.fetchone()