Saturday, February 27, 2010

Development strategy for google-app-engine

        In developing my site for the cloud I found it a bit of a challenge. There is very little documentation in terms of infrastructure strategy and a lot of sites assume you have a clear methodology. I always strive for a clean infrastructure — having developed code for the quite awhile I have come to know that if you have a solid scalable strategy up front it makes things just work easier later on. That is why I was really turned onto google-app-engine. It gives you a clear way of building and testing your code and then pushing it to the web. It is a great place to start but it isn’t the “everything” I’m looking for.
        Because some of the limitations/restrictions with using google-app-engine (GAE) you need to understand that if you want to use it (i.e. python package) you need to include it in your application. That is where the benefits of a virtualenv come into play. virtualenv (when combined with pip) let you know exactly what packages you have because throughout the application development process if you need it you ‘pip-it’. Then pip gives you a clean manifest on what you have so you always know. But.. (you knew this was coming) GAE doesn’t like a virtual env. In fact if you try it you almost surely run into problems. For me it was a error along this line...

ImportError: No module named unittest

        So what can I do? For me the reasons to use a virtualenv/pip combination is too compelling to simply not use it. I really like knowing what packages I willbe deploying up to the cloud and since I need to package them with my application why not? Furthermore if you are used to running in a virtualenv/pip environment or you have a somewhat stock python environment this is just a good way to keep things straight. Understanding the obvious limitation that you can’t run in a virtualenv we are going to take advantage of everything else this environment has to offer.

General Development Strategy

  • Use Django. While I understand that GAE now support 1.1 I want that control left to me.
  • Keep the development tree streamlined for the entire site. Guido recommends (as I understand him in this video ~11:45) to use a single app-engine application per independent application. This doesn’t make much sense to me as the ability to share information between different models is really extending the multi-dimensionality of the site. So my preference is to keep it all under one hood.
  • Use revision control (for me it’s perforce - I don’t have anything against the others (hg/git/svn) it’s just what I like)
  • Use eclipse/pydev as my development environment
General Directory Structure

I am developing for $SITE ( i.e export SITE=“”)

                        django symlink to ../src/django/django

Getting started

Setup your virtualenv

cd dev

Build up a baseline virtualenv

virtualenv --python=python2.5 --no-site-packages ${SITE}

Install some needed apps.. (make sure you are in your dev directory..)

# pip install -E ${SITE} -e svn+
# pip install -E ${SITE} -e svn+
pip install -E ${SITE} -e svn+
pip install -E ${SITE} yolk

*Notes: 1.1.1 failed 2 tests and trunk failed more.. 1.1 passes all tests

Activate your virtualenv

source ${SITE}/bin/activate

Create your www tree (this is where we house our app). Use the google-app-engine-django helper stuff. It’s well worth it!!

cd ${SITE}
svn export www

Link in your django tree

cd www
ln -s ../src/django/django

At this point you should be able to deactivate yourself and simply verify everything works.

python2.5 test
#Ran 61 tests in 8.058s
python2.5 runserver

Now open up a web browser to and you should get the famous “It works!!”. At this point you should also configure the Google App Engine Launcher to point to your site directory and “www” will be your application.

Build up a quick index and push it to the web.

This is not for the faint. I am going to quickly build up a basic django index page. I want to show it works and we con complete the process.

Shuffle around my tree (personal preference)

Like I indicated above I like to have my apps all under a single apps tree (keep it clean). I also like to put under a settings tree - this allows me to mess with the

So with that hang on..

cd ${SITE}/www
mkdir settings
mv settings/
rm settings.pyc

mkdir apps
python2.5 startapp core
mv core apps/

Build your simple app

Edit your ${SITE}/www/

urlpatterns = patterns('',
url(r'^', include('apps.core.urls')),

Add in your app to the ${SITE}/settings/


Create your apps/core/

from django.http import HttpResponse
def index(request):
return HttpResponse('Hello World -- Django rocks!!')

Get your apps/core/ ready to accept the index view

from django.conf.urls.defaults import *
import views
urlpatterns = patterns('',
url(r'^$', views.index),

Verify it works.. Voila - pretty simple eh??

Next up publish it..


Guido's screencast - it was awesome..
        Rapid Development with Python, Django, and Google App Engine (2008 Google I/O Session Videos and Slides)

Monday, January 4, 2010

Building Python on 2.6.4 Snow Leopard

        •        Fully functioning Python with MySQL, Ldap, P4Python, installed
        •        Use pip

        •        None so far — P4Python was not x86_64/i386 compatible but that now is fixed..

        •        Stock python on Snow Leopard ships as a 32-bit only application (file /System/Library/Frameworks/Python.framework/Versions/2.5/bin/python2.5)

Building readline-6.0

Normally you won’t have to set these variables up but since we want a 32 bit with a 64 bit readline we need to ensure these are set up..

        export MACOSX_DEPLOYMENT_TARGET=10.6
        export CFLAGS=“-arch i386 -arch x86_64”
        export LDFLAGS=“-Wall -arch i386 -arch x86_64”
        sudo make install

Building Python 2.6.4

If you use a custom ~/.pydistuti“s move it out of the way! This will build an intel only version of Python.

        ./configure --enable-framework MACOSX_DEPLOYMENT_TARGET=10.6 --with-universal-archs=intel --enable-universalsdk=/
        sudo make install

After the make you should see errors like this..

        Failed to find the necessary bits to build these modules:
        _bsddb dl gdbm
        imageop linuxaudiodev ossaudiodev
        spwd sunaudiodev
        To find the necessary bits, look in in detect_modules() for the module’s name.

MySQL (Not Python)

Get the latest MySQL 64-bit (of course). Pull the file from
Get the Snow Leopard Preference Panel for 64-bit from here

Getting rolling with pip.

If y’u use a custom ~/.pydistutils put it back..

Get setuptools (0.6c11) from pypi.

        python2.6 build
        python2.6 test
        python2.6 install

Get pip from pypi

        python2.6 build
        python2.6 install

Start getting your apps..

        pip install --install-option=“--prefix=/Users/sklass/perforce/dev/tools/python/Mac” nose

Now build up MySQLdb

        python2.6 build
        python2.6 test
        python2.6 install

Saturday, January 2, 2010

Setting up MacJournal for Code Blocks

So I wanted to get some good syntax highlighting..

        1.        Install SyntaxHighlighter on Blog
        2.        Configure SyntaxHighlighter in MacJournal Blog by turning off Escape &lt; and &gt under the edit server list!
        3.        Create a snippet (pyp) to wrap plain text in &lt;pre class=“brush: python” &gt;)
        4.        Test it out

This is a basic python code block

urldata = urllib.urlopen(settings.PEOPLE_NOTES_BASE_URL).read()
nsoup = BeautifulSoup(urldata)
total = 0
links = nsoup.fetch(‘a’, {‘href’:re.compile(‘\/hr\/.*OpenDocument’)})
for element in links:
if maxNum and total >= maxNum: break
queue.put({u”name”: u”%s” % element.find(text = True),
u’url’: “” % element[‘href’]})
total += 1

for index in range(50):
if maxNum and index >= maxNum: break
thrd = self.ParseUserPage(out_queue, final)
#wait on the queue until everything has been processed

And the important thing is grace..

Continuing this should be fine..

Saturday, January 02, 2010 6:13 AM

Friday, January 1, 2010

Thoughts on pip - whats the big deal?

I have been a python user since python 2.0. Since then I’ve stayed fairly current with the python and the newer methodologies (currently running 2.6.4). I am always striving to make myself a more proficient developer seeking out more efficient ways of doing things and not afraid to give them a try.
I currently am responsible for a cross platform python environment for our engineering teams (>150 ppl) which has spanned various unix’s, macs and the occasional windows machine. I’ve been doing this for the past several years and have managed python transitions from 2.2 - 2.6 with relative ease.
The methodology we employ to maintain this has been around since I started using python and I would expect will remain so in the future. We extend the default python path using a pth file methodology. The pth file is coupled to an environment variable which gives developers the ability to shift an environment variable and quickly be testing their code in their respective sandbox. This coupled with Perforce gives us a nice branched virtualenv without using virtualenv. We use this methodology for both external modules ( django/beautifulSoup/paramiko etc) and internally developed modules. Currently our single pth file looks like this.

import os, site; tech = os.environ.get("TECHROOT", "/"); tech = tech if os.path.isdir(os.path.join(tech, "tools/python/Mac/lib/python2.6/site-packages")) else "/"; site.addsitedir(os.path.join(tech, "tools/python/Mac/lib/python2.6/site-packages"))
import os, site; tech = os.environ.get("TECHROOT", ""); tech = tech if os.path.isdir(os.path.join(tech, "tools/python/modules")) else "/";site.addsitedir(os.path.join(tech, "tools/python/modules"))

Admittedly this may be a bit overkill. However limitations on pth files (paths must be on a single line) force the need. FWIW all this file does is get the environment variable TECHROOT if defined or establish a default path. The first line is used for external modules and the second line is for internally developed modules. Module installation is a then breeze when using a ~/.pydistutils.cfg file which looks like this.

install_lib = $TECHROOT/tools/python/Mac/lib/python$py_version_short/site-packages
install_scripts = $TECHROOT/tools/python/Mac/bin

Again this is all tied to our revision control system (perforce) which allows any developer to easily add external modules or build modules for other to use. By shifting around the environment variable we can thoroughly test our code prior to release. Simple enough.

Over the past couple years I have really come to enjoy easy_install. It certainly worked for everything I needed. It has some limitations which I didn’t like but for the most part it worked. Some of the notable limitations I usually do are as follows.
  • Dependency Tracking must be thought out and tested ahead of time.
  • easy_install should be named easy_install<ver> (easy_install2.6)
  • Any bin files should have its header nailed down (/usr/bin/env python2.6) or be named file<ver> (
  • Un-Installing is a manual process - but simple cause it’s a single file and tweak of on file.
Barring the above limitations, I like easy_install. I like the packaging of egg archives; it’s a simple 1 file. True this forced me to be very conscience of the platforms and make sure that everyone was on the same page with respect to upgrades but that’s simple planning. Not every package builds with easy_install but then again neither does pip.

So I decided that I would use my shift to Snow Leopard and 2.6.4 as the basepoint for using pip. After reading comments like this I figured surely I must be missing something. After playing with pip now for a week I can honestly say - Nope I don’t think so.

To be fair pip is still at release 0.61. It appears pip is really geared towards those which use a virtualenv environment. As detailed above we don’t need virtualenv we effectively have an automated one with perforce, and an environment variable. That being said I found several things I didn’t like with pip and other things which are broken altogether.
  • Dependency Tracking still must be thought out and tested ahead of time.
  • Un-Installing packages doesn’t always work
  • Pip Doesn’t understand the test constructs if provided.
  • Pip Doesn’t always respect ~/pydistutils.cfg install_scripts directive. Most of the time I just needed to pip install --install-option="--prefix=$TECHROOT/tools/python/Mac”
So what’s the big deal? Is it me or am I missing something larger. I thought pip would track dependancies allowing you to have say two versions of a module installed simultaneously. But no. It’s a nice and clean installer (I’ll give you that). The freeze is pretty slick if you are distributing your own system without a real SCM system in place. Overall I’m not overly impressed and I feel pip is overrated or easy_install misleadingly accused. Maybe it’s me?

Sunday, December 13, 2009


So I am playing with pydev and so far I think it’s pretty cool. Here is a sample with a picture..


This is the second post. So far it’s pretty amazing. I love the new stuff in 1.5!!

And then I edited it some more.. Pictures..