Sun 11 Nov

Deploying TIER's Village Base Station in Papua

I'm in Papua, Indonesia right now working on a deployment of a system one of my TIER colleagues, Kurtis Heimerl, developed. This is a little blurb I wrote on the TIER blog about our first week here.


Tue 24 Jul

Creating and importing custom Python packages

I'm building a custom Python package for a project I'm working on, and it took me more time than should have been needed to figure out how to achieve the import behavior I wanted for that package. The directory structure looks like this:

project_dir/
    foo.py
    package/
        __init__.py
        Bar.py
        Baz.py
        Qux.py

Each of the files in the package defines some classes; for now, we can assume they each have just one eponymous class. foo.py is the driver script that the user actually runs, which mainly just accepts command line arguments and imports the package.

What I'd like to be able to do in foo.py is say:

import package
b = package.Bar("asdf")
b.something()

If all my classes were in a single Python source file called package.py, this would be the default behavior. Alternatively, I could say from package import * in foo.py, but that would import my modules directly and frankly I think it looks ugly.

The way to achieve the desired behavior is to perform the imports of each module in the package in the __init__.py file, like so:

from Bar import *
from Baz import *
from Qux import *

Now, when we import the package, all the associated modules come with it.

I found this post very helpful in understanding how modules work, as well as the official documentation.

· Tags: ,

Mon 30 Apr

Goodbye, Yee-Haw

Yee-Haw Industries, a fantastic print shop in my hometown of Knoxville, closed its doors today after a decade and a half of making awesome letterpress and screen print art. They had a unique style that to me epitomized Knoxville, maybe because their shop was responsible for designing so many posters and signs around the area.

I remember Yee-Haw as being the first interesting place I ever noticed on Gay Street, back when it probably was the only interesting place on Gay Street. Every time I go back, I'm impressed by how much more exciting and lively downtown Knoxville has become, a transformation which is in no small part due to creative people like the ones behind Yee-Haw.

As others have noted, they will be missed. Thanks to Yee-Haw for all they've done for Knoxville.


Fri 2 Mar

Gitorious on Ubuntu Server 11.10

I set up a Gitorious installation for my research group today, following the quite good instructions from Lucas here. I had to make a couple minor adjustments to his setup steps, which I'll document below.

I started with a fresh installation of Ubuntu Server 11.10. To get started, run the following:

sudo apt-get install build-essential tcl-dev libgeoip-dev postfix apache2 mysql-server mysql-client apg libsqlite3-dev imagemagick libpcre3-dev zlib1g-dev libyaml-dev libmysqlclient-dev apache2-dev libonig-dev ruby-dev rubygems libmysql-ruby libdbd-mysql-ruby libmagick++-dev zip unzip memcached git-core git-svn git-doc git-cvs irb git sphinxsearch libcurl4-openssl-dev libxslt1-dev libxslt-ruby

You'll be asked to set up a mysql root password; remember what you enter.

sudo gem install rake daemons rmagick stompserver passenger bundler

cd /var/www
git clone git://gitorious.org/gitorious/mainline.git gitorious
git submodule init
git submodule update

cd /var/www/gitorious/doc/templates/ubuntu/ && sudo cp git-daemon git-poller git-ultrasphinx stomp /etc/init.d/ && cd /etc/init.d/ && sudo chmod 755 git-daemon git-poller git-ultrasphinx stomp
sudo update-rc.d git-daemon defaults && sudo     update-rc.d git-poller defaults &&   sudo   update-rc.d git-ultrasphinx defaults && sudo    update-rc.d stomp defaults

sudo ln -s /usr /opt/ruby-enterprise

sudo $(gem contents passenger | grep passenger-install-apache2-module)

At this point, the installer the last script runs will ask you to copy some configuration code into a file. It will look something like what's in the echo below:

echo "LoadModule passenger_module /var/lib/gems/1.8/gems/passenger-3.0.11/ext/apache2/mod_passenger.so
    PassengerRoot /var/lib/gems/1.8/gems/passenger-3.0.11
    PassengerRuby /usr/bin/ruby1.8" | sudo tee /etc/apache2/mods-available/passenger.load

Let's move on.

sudo a2enmod passenger &&  sudo a2enmod rewrite && sudo a2enmod ssl

# NOTE: This step requires thinking. Replace the server name appropriately.
echo "<VirtualHost *:80>
    ServerName your.website.com
        DocumentRoot /var/www/gitorious/public
        </VirtualHost>" | sudo tee /etc/apache2/sites-available/gitorious

# Clearly, use a real SSL cert/key... 
echo "<IfModule mod_ssl.c>
    <VirtualHost _default_:443>
        DocumentRoot /var/www/gitorious/public
        SSLEngine on
        SSLCertificateFile    /etc/ssl/certs/ssl-cert-snakeoil.pem
        SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key
        BrowserMatch ".*MSIE.*" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0
    </VirtualHost>
</IfModule>" | sudo tee /etc/apache2/sites-available/gitorious-ssl

sudo a2dissite default && sudo a2dissite default-ssl && sudo a2ensite gitorious && sudo a2ensite gitorious-ssl

mysql -u root -p

# In mysql, enter:
# mysql> GRANT ALL PRIVILEGES ON *.* TO 'gitorious'@'localhost' IDENTIFIED BY '<your password>' WITH GRANT OPTION;
# mysql> FLUSH PRIVILEGES;

cd /var/www/gitorious/ && sudo bundle install && sudo bundle pack

adduser --system --home /var/www/gitorious/ --no-create-home --group --shell /bin/bash git
chown -R git:git /var/www/gitorious

Alright. At this point we're almost done. The next couple steps have some room for creativity (aka, pick options and paths that make sense for how you want to deploy Gitorious).

sudo su git
cd /var/www/gitorious
mkdir .ssh && touch .ssh/authorized_keys && chmod 700 .ssh && chmod 600 .ssh/authorized_keys && mkdir tmp/pids && mkdir repositories && mkdir tarballs

cp config/database.sample.yml config/database.yml && cp config/gitorious.sample.yml config/gitorious.yml && cp config/broker.yml.example config/broker.yml

Follow the configuration instructions provided by Lucas. The next step, quoting him: "Because of an incompatibility of RubyGems with Rails < 2.3.11 you need to add the following line at the top of config/boot.rb:"

require 'thread'

(I can verify this is still necessary in 11.10). Back to Lucas, let's wrap up:

export RAILS_ENV=production && bundle exec rake db:create && bundle exec rake db:migrate && bundle exec rake ultrasphinx:bootstrap

# NOTE: The path to bundle has changed in 11.10! This is an update.
crontab -e * * * * * cd /var/www/gitorious && /usr/local/bin/bundle exec rake ultrasphinx:index RAILS_ENV=production

env RAILS_ENV=production ruby1.8 script/create_admin

Now, a couple small changes before you're ready to run. Some paths have changed in 11.10, as noted by a very helpful commenter on Lucas' article. First, we need to update /etc/init.d/stomp. Change GEMS_HOME="/usr/local" to GEMS_HOME=”/usr/local”. Next, we need to edit /etc/init.d/git-daemon and /etc/init.d/git-poller. As provided, these are each run with /usr/bin/ruby; I had to modify each to run with bundle exec. So make the following changes to the two respective files:

# /etc/init.d/git-daemon
GIT_DAEMON="$RUBY_HOME/local/bin/bundle exec $GITORIOUS_HOME/script/git-daemon -d"

# /etc/init.d/git-poller
GIT_POLLER="$RUBY_HOME/local/bin/bundle exec $GITORIOUS_HOME/script/poller"

At this point, I believe you should be able to reboot and have a working installation of Gitorious. Please let me know if I missed anything so I can update this (I'm actually re-enabling comments on this blog just for this). A couple gotchya's I ran into while I was setting this up:

  • Gitorious relies on several background scripts, most of which are located in the script directory. It also relies on stompserver for message passing. If the site is very slow, especially if pushes don't work, repository creation takes a long time, etc., check that both stompserver and poller are running. Note also that you can see the log output of the gitorious scripts under $GITORIOUS_HOME/tmp/pids.
  • If you can't push to a repo, stompserver is probably not running.

I'm pretty excited about having a simple way to host our projects internally. While I'm a big fan of Github, and our group pushes most of our public code there, having a private hosting location will also be helpful for things like in-submission papers, configuration files, and sensitive data. Our particular installation takes advantage of the very reliable file storage system that our department offers, so by using it all of our work can take advantage of their replication and automated backup/recovery systems. This is really important to me after hearing a harrowing tale from our sysadmin of a grad student who lost 3 years of work due to the theft of the only server that contained his data and code. Eek!


Wed 14 Dec

The world gets cloudier: AWS goes to Sao Pablo

Amazon launched the Sao Paulo region today for their AWS cloud services. I'm glad to see cloud providers branching out to new markets beyond the US and Europe. Apparently, both South American companies and international companies looking to serve the South American market have been clamoring for such a move from Amazon.

While this will be great for the tech scene there, I'm interested to know how this new infrastructure will benefit the broader community of South American Internet users, and in particular how this will impact the cost of international bandwidth in South America. Historically, international bandwidth costs have been a key limitation for the proliferation of affordable Internet access throughout the developing world. This makes sense: running international fiber is expensive, and it often requires a consortium of telco's working together to make it happen (e.g., EASSy). The major cloud providers, on the other hand, have plenty of capital to finance submarine cable construction. Heck, they could build their own cable infrastructure if they wanted to: I'm sure Google and Amazon have the expertise to manage it well, too. In any case, I'm hopeful that we'll see similar cloud infrastructure built in other traditionally underserved areas, such as Africa and South Asia, in the near future.

· Tags: , , ,
← Previous Next → Page 2 of 11