Saturday, February 27, 2010

Development strategy for google-app-engine

        In developing my site for the cloud I found it a bit of a challenge. There is very little documentation in terms of infrastructure strategy and a lot of sites assume you have a clear methodology. I always strive for a clean infrastructure — having developed code for the quite awhile I have come to know that if you have a solid scalable strategy up front it makes things just work easier later on. That is why I was really turned onto google-app-engine. It gives you a clear way of building and testing your code and then pushing it to the web. It is a great place to start but it isn’t the “everything” I’m looking for.
        Because some of the limitations/restrictions with using google-app-engine (GAE) you need to understand that if you want to use it (i.e. python package) you need to include it in your application. That is where the benefits of a virtualenv come into play. virtualenv (when combined with pip) let you know exactly what packages you have because throughout the application development process if you need it you ‘pip-it’. Then pip gives you a clean manifest on what you have so you always know. But.. (you knew this was coming) GAE doesn’t like a virtual env. In fact if you try it you almost surely run into problems. For me it was a error along this line...


ImportError: No module named unittest


        So what can I do? For me the reasons to use a virtualenv/pip combination is too compelling to simply not use it. I really like knowing what packages I willbe deploying up to the cloud and since I need to package them with my application why not? Furthermore if you are used to running in a virtualenv/pip environment or you have a somewhat stock python environment this is just a good way to keep things straight. Understanding the obvious limitation that you can’t run in a virtualenv we are going to take advantage of everything else this environment has to offer.

General Development Strategy

  • Use Django. While I understand that GAE now support 1.1 I want that control left to me.
  • Keep the development tree streamlined for the entire site. Guido recommends (as I understand him in this video ~11:45) to use a single app-engine application per independent application. This doesn’t make much sense to me as the ability to share information between different models is really extending the multi-dimensionality of the site. So my preference is to keep it all under one hood.
  • Use revision control (for me it’s perforce - I don’t have anything against the others (hg/git/svn) it’s just what I like)
  • Use eclipse/pydev as my development environment
General Directory Structure

I am developing for $SITE ( i.e export SITE=“foobar.com”)

/dev
        /<$SITE>
                /bin
                /lib
                /include
                /src
                        /django
                /www
                        app.yaml
                        /appengine_django
                        manage.py
                        main.py
                        /apps
                                /app_1
                                /app_2
                                /app_3
                                /app_n
                        /settings
                                __init__.py
                        urls.py
                        django symlink to ../src/django/django

Getting started

Setup your virtualenv

cd dev

Build up a baseline virtualenv

virtualenv --python=python2.5 --no-site-packages ${SITE}

Install some needed apps.. (make sure you are in your dev directory..)

# pip install -E ${SITE} -e svn+http://code.djangoproject.com/svn/django/trunk#egg=Django
# pip install -E ${SITE} -e svn+http://code.djangoproject.com/svn/django/tags/releases/1.1.1/#egg=Django
pip install -E ${SITE} -e svn+http://code.djangoproject.com/svn/django/tags/releases/1.1/#egg=Django
pip install -E ${SITE} yolk


*Notes: 1.1.1 failed 2 tests and trunk failed more.. 1.1 passes all tests

Activate your virtualenv

source ${SITE}/bin/activate

Create your www tree (this is where we house our app). Use the google-app-engine-django helper stuff. It’s well worth it!!

cd ${SITE}
svn export http://google-app-engine-django.googlecode.com/svn/trunk/ www

Link in your django tree

cd www
ln -s ../src/django/django

At this point you should be able to deactivate yourself and simply verify everything works.

deactivate
python2.5 manage.py test
#Ran 61 tests in 8.058s
python2.5 manage.py runserver

Now open up a web browser to http://127.0.0.1:8000/ and you should get the famous “It works!!”. At this point you should also configure the Google App Engine Launcher to point to your site directory and “www” will be your application.

Build up a quick index and push it to the web.

This is not for the faint. I am going to quickly build up a basic django index page. I want to show it works and we con complete the process.

Shuffle around my tree (personal preference)

Like I indicated above I like to have my apps all under a single apps tree (keep it clean). I also like to put settings.py under a settings tree - this allows me to mess with the __init__.py

So with that hang on..

cd ${SITE}/www
mkdir settings
mv settings.py settings/__init__.py
rm settings.pyc

mkdir apps
python2.5 manage.py startapp core
mv core apps/

Build your simple app

Edit your ${SITE}/www/urls.py

urlpatterns = patterns('',
url(r'^', include('apps.core.urls')),
)

Add in your app to the ${SITE}/settings/__init__.py

INSTALLED_APPS = (
'appengine_django',
'apps.core',
)

Create your apps/core/views.py

from django.http import HttpResponse
def index(request):
return HttpResponse('Hello World -- Django rocks!!')

Get your apps/core/urls.py ready to accept the index view

from django.conf.urls.defaults import *
import views
urlpatterns = patterns('',
url(r'^$', views.index),
)

Verify it works.. Voila - pretty simple eh??

Next up publish it..

References:

Guido's screencast - it was awesome..
        Rapid Development with Python, Django, and Google App Engine (2008 Google I/O Session Videos and Slides)

8 comments:

Gary Mayer said...

Nice article.

The other Python framework that works nicely on GAE is Web2py. The interesting thing is, it only deploys the application code with some supporting libraries by simply running appcfg.py. It is an MVC type framework and works similar to Django.

Steven Klass said...

Hey Gary,

Thanks - I've heard of Web2py but I'll haven't played with it. I'll have to check it out thanks!!

patrick said...

Your blog is really cool! I will wait when you finish. Thank you for interesting articles.
more templates easy to download

Brian Thorne said...

I had to also create a __init__.py file in the apps folder, otherwise everything else worked exactly as you said! Cheers!

erwin said...

Appreciated your article. I have one question though:
By deactivating your virtualenv before running manage.py test, aren't you using the "main"-python instead of the one in your virtualenv, defeating the purpose of using virtualenv?

x777 said...

Yeah I am not sure what the point is in using virtualenv in this case. Lets say you do pip install foobar. How do you get foobar deployed on GAE?

geoul oh said...

Read all the related Posts:


Basic of VBScript Language for QTP

Introduction to QTP (QuickTest Professional) Part2

Introduction to QTP (QuickTest Professional) Part3

Introduction to QTP (QuickTest Professional) Part4

Basic of VBScript Language for QTP

ISTQB

ISTQB Practice Test Part 1
ISTQB Practice Test Part 2
ISTQB Practice Test Part 3
ISTQB Practice Test Part 4
ISTQB Practice Test Part 5
ISTQB Practice Test Part 6
ISTQB Practice Test Part 7
ISTQB Practice Test Part 8
ISTQB Practice Test Part 9

Emelia Grase said...

NET is a Programme Several vital some piece of what Windows is running in Run On A Present For
General k like program. Cloud Development This Cool People What do I need It. NET provision that runs the PC That self-important. For the individuals who created, .NET advancement, ASP improvement, SharePoint advancement, Microsoft advancement, programming improvement, Singapore.
Window Phone Apps // iPhone Application Development // Window Phone APPS