eriksmartt.com/blog

Austin Python Users Group meeting tomorrow (May 14th) with guest speaker Greg Wilson

This month’s APUG meeting will feature guest speaker Greg Wilson, author of Beautiful Code, Data Crunching, Parallel Programming Using C++, Practical Parallel Programming, etc.

For more details, see: http://wiki.python.org/moin/AustinPythonUserGroup and http://python.meetup.com/188/.

Hope to see you there!

Reading “The Definitive Guide to Django”; Verdict: A solid learning reference for a beginning/intermediate Django user

Last week I received a review-copy of the new “The Definitive Guide to Django” book from Apress. I hadn’t planned on buying the book since it seemed a little too beginner-focused; but I agreed to give it an honest reading, so I happily dove in with an “it’s Python, of course I’m going to like it” attitude.

Background

The book was written by Adrian Holovaty and Jacob Kaplan-Moss, the creators and “Benevolent Dictators” of the Django Web Framework. It was Holovaty and Kaplan-Moss’ first book, and, I believe, meant to be the first Django book to market. The book was drafted online; open to peer-review and community feedback; and ultimately published under the GNU Free Documentation License.

From the get-go, the print edition had a few inherent market challenges to face: First, the entire book is available online, for free, at: <http://www.djangobook.com/>. Second, in many ways the book is a re-hash of the docs available at <http://www.djangoproject.com/documentation/>, which are also free. Third, the book covers Django 0.96, not SVN. (0.96 is technically the latest-snapshot release, but a lot has changed since 0.96.) And finally, the $45 MSRP could be seen as a little steep for what is effectively a printed copy of a free, online book.

The print experience

Diving in, the book takes the reader through the basic installation process, provides a brief background on how the framework came to be (and why you want one), then steps through the major features (ie., the template system, ORM, URLconfs, generic views, etc.) It’s what you’d expect from a technical reference — no fluff, and straight to the details. There are plenty of code snippets to learn from, and the sidebar notes tend to be insightful.

Since it wasn’t new material for me, the book was a fairly quick read; but the experience of reading Django documentation in book-form was actually quite fascinating. There’s something about settling into a comfortable chair with a book, pen, and highlighter that you just can’t get with online documentation. Perhaps it was just a little more noticeable given the material. When I read the Django docs online, I tend to skim over them while trying to solve a problem. I use them as a reference more then a learning tool, and it’s usually while actively coding, thus my brain is partially distracted with whatever it is I’m building.

With a physical book, you can unplug, step away from the computer, and give the material your undivided attention. This isolation from distraction results in a much deeper understanding of the text. This is the real the value of the printed book — it’s an opportunity to digest online documentation in an environment more conducive to learning and retention.

My general take-aways and observations

• The book definitely has a beginner/intermediate feel to it, but only in the sense of a beginner Django user — not a beginner Web developer or Python programmer. I’m curious how well the book is received by folks who are beginners at Django and dynamic Web development since the text brings up a lot of complex topics in Web development that aren’t really explained. (Ex., database administration, server clustering, manipulating HTTP headers, etc.)
• The breadth of the book is impressive, but in some ways, the book really feeds you through a firehose, so to speak. It throws a lot of new concepts at the reader and doesn’t always explain why you’d need to know them, or how you might use them in the real world. For someone deploying a site with Django, it will be good to know that all these features are available, but it might take awhile before they need to use them (if ever.)
• The book does touch on some of the more advanced Django features (like extending the template system and writing custom middleware), which was nice, but some topics are reserved for the appendix and get limited coverage (ex., model managers and ‘Q’ queries.) Others, like the Sites Framework, are given good exposure, but not so much that the reader is left with a clear picture on when to use them and what their limitations are.
• The forms processing chapter was a bit lighter then what I was hoping for — especially given that the current newforms documentation still trends toward “read the source code.” It provides enough to start using newforms if your form needs are pretty basic, but doesn’t address creating your own widgets, or any of the fun stuff you can do once you start dynamically generating and manipulating newforms objects.
• It might have been nicer if the examples in the book were a little more tied together, perhaps all focused on building a single example project and showing how the various features are used in real-world applications. (The example of the book-publisher’s app was a reoccurring theme, but not so strongly that each chapter applied it’s new learnings to it.)
• The Deploying Django: “Going Big” sub-section provides a nice infrastructure graphics for how high-traffic systems might be setup, but once you get to the point of being “big”, you need to architect for it, and that’s really outside of the scope of this book. For this section, it might have been nice to reference other resources on scaling infrastructure, and perhaps pointing out some of the ways that Django can be optimized for performance and horizontal scaling. (For example, one of the Django projects we put into production at work will happily support 1,200 requests/second, but the database layer and session middleware have been reworked a bit, and the content caching approach is a little different then the standard Django offering.)
• On the more positive side, even as someone who’s been using Django for some time, I still learned a few new tricks, and I was reminded of a few features that I could be taking better advantage of. (And when you do this stuff professionally, every shortcut and productivity gain has monetary value — avoiding even a half-hour of debugging pays for the cost of this book.)
• This book would make a fantastic read for a back-end developer joining a project that is already using Django. I normally tell new developers to go through the Python Tutorial at <http://python.org/doc/tut/> if they’re new to Python, then to complete the Django Tutorials at <http://www.djangoproject.com/documentation/> before trying to grok any in-progress Django project. Now I have a third reference (though I might still suggest that they walk through the tutorials first, so that they have some context when reading the book. Otherwise, there are just too many new concepts to do a straight read-through and still grasp it all.)

Summary

The market needed a good Django book, and this one delivered a solid reference for the framework. Arguably, it’s not really a “Beginner’s Guide to Django”, but hopefully it covers enough of the basics that future books can focus on best practices and more advanced techniques. (On a related note, there’s apparently an upcoming “Practical Django Projects” book, also from Apress, that will focus more on building “reusable Django applications from start to finish”. This might actually make for a better beginner’s book, depending on how it turns out. [Via The B-List: Speaking and writing].)

The million-dollar question then, is “Should you buy this book?” My answer ended up being a bit more positive then I expected, but there are two parts: First, if you’re a front-end developer only, you don’t need this book. You can just read Chapter 4: The Django Template System online, and then use the “Django Templates: Guide for HTML authors” section of the online docs as a reference. For back-end developers, the story is different. If you’re going to just “read it while you hack”, then you might as well just read it online; but if you’re serious about building applications with Django (especially if you’re new to it), then you should consider the book and investing the time to step away from the computer and really let yourself get into it. Unless you are an active contributor to Django (which I’m not, just to be clear), the odds are pretty good that you’ll learn something new, even if you’re already using Django today.

Django “lorem ipsum” generator (and a new contrib.webdesign module)

Django “lorem ipsum” generator (and a new contrib.webdesign module)

The Django Web Framework project just added a new contrib.webdesign module with an amazingly simple, but incredibly handy first feature: a lorem ipsum generator. The idea is that a project’s base templates can include generated lorem ipsum for testing layout and page flow, but inheriting templates can override the generated text once real content is available.

The lorem tag is used like this (via the contrib.webdesign docs):

• {% lorem %} will output the common “lorem ipsum” paragraph.
• {% lorem 3 p %} will output the common “lorem ipsum” paragraph and two random paragraphs each wrapped in HTML <p> tags.
• {% lorem 2 w random %} will output two random Latin words.

In practice, you might do this:

templates/template.html:

<html>
  <head>
    <title>{% block article_title %}{% lorem 5 w %}{% endblock %}</title>
  </head>
  <body>
    <div class="article">
      <div class="article_title">{% block article_title %}{% lorem 5 w %}{% endblock %}</div>
      <div class="article_body">{% block article_body %}{% lorem 4 p %}{% endblock %}</div>
    </div>
  </body>
</html>

And then inherit when you’re ready:

templates/article.html:

{% extends "template.html" %}

{% if article %}
  {% block article_title %}{{ article.title }}{% endblock %}
  {% block article_body %}{{ article.body }}{% endblock %}
{% endif %}

Previously, I used to just paste lorem ipsum text directly into the main template (wrapped in block tags for overridding), but this new tag will let you skip the copy/paste routine. Very nice!

PyCon 2007 wrap-up

I’m back from PyCon 2007. It was a busy weekend, with 593 Pythonistas attending the conference. I took a fair amount of notes, but I’ve pulled out some highlights below:

From Ivan Krstic’s keynote on the One Laptop Per Child project:

• Python is the language of the One Laptop Per Child (OLPC). Everything that can, will be done in Python… and there’s a “view source” button on the keyboard (view layout) so you can view (and edit) the source of your current running application.
• The filesystem (which supports versioning) is called Yellow, and will be released withing a week or so. The GUI is called Sugar, and is available on http://dev.laptop.org/ to play with. You can download the full image (or build the environment on Linux.)
• The OLPC supports 802.11s mesh networking.
• The hand crank was removed for case durability. The OLPC’s are designed to last five years, but the torque from the hand-crank would have stressed the plastic case too much for it to last that long.
• The first OLPC’s will start shipping in August of this year!
• The OLPC hardware was getting ~1100 pystones before optimization. They are now up to ~2300 pystones (on a 366 Mhz AMD Geode processor.) (Note: This means they have better Python performance then Python for S60 is seeing on current S60 phones.)

From the Web Frameworks panel:

• James Tauber, “Reinventing the wheel is great if your goal is to learn more about the wheel.”
• Jonathan Ellis, “When you control the whole stack you can innovate faster.”

From Adele Goldberg’s keynote:

Public school education is so bad that real eLearning solutions can’t go to the schools — they need to be outside of schools so that you don’t have the traditional censorship that comes with public schools — and you don’t have the associates with the bad experiences kids have while at “school”.

From Jacob Kaplan-Moss’ talk, “Becoming an open-source developer: Lessons from the Django project”:

1. Use good tools. “Open source is better because it’s better.”
2. Avoid dogma. Don’t get stuck on what language something is implemented in.
3. Work with (and hire) smart people. The model in open source is that if you’re smart, people listen to you. That’s rough if you’re not smart… But also means that it’s worthwhile to mention when you’re an expert on a topic.
4. “Methodologies” suck. Ex., MVC is cool, but Django abuses it because it doesn’t fit so well with the web.
5. DRY — Don’t Repeat Yourself. The one methodology to use.
6. The business case for open source. You have to make one (to your company.)
   • Money. You’ll get recognized and sell services because of it. (Ex., Ellignton wouldn’t be as successful without Django.)
   • Free labor. (Sad to think of this way, but true when you have an interesting project.)
   • Self-improvement. Knowing that peers will review your code makes you much more careful about the code you submit. This makes the code a lot better.
   • Geek cred — gaining credibility within the geek community makes it easier to hire great people.
   • Moral Argument — If you built a business on open source — it’s time to give something back.
   • Figure out where to draw the line — Django gave away the tools, but not all the apps.
7. Selling open source to other companies. Microsoft’s FUD had been quite successful in some areas. Counter the “communist” argument with a “freedom” argument. Focus on the freedom of data — your data belongs to you; there is no vendor lock-in. Open vs. Lock-in is a better argument then Open vs. Closed.
8. Create a community. This doesn’t just happen because you setup a mailing list. (Gave example of thanking people who post anti-Django blog posts and asking what they didn’t like.) Don’t say anything that would get you kicked out of a bar.
   • Avoid monsters (trolls, vampires, etc.) Detect them early, and ignore them.
   • Spam can’t be an afterthought. Collaborative tools require spam filtering from Day 1. You’ll get spam. Lots of it. Google Groups is pretty good about cutting out spam.
9. listen to the community. But smartly. Sometimes the vocal majority doesn’t represent the wishes of the whole community. Django’s magic-removal was a big risk, driven by the community. You also have to be willing to ask for help. Sometimes you don’t feel comfortable delegating tasks that you think suck — but not everyone has the same definition of “what sucks” — sometimes there’s someone who actually WANTS to do this task!
10. Handling community contributions. You need a defined method for how you take contributions. It helped the Django project when they adopted a system for differentiating between patches that are controversial, and those that aren’t. (ie., simple bug fixes vs. design decisions.) A ticket reviewer makes this decision.
11. Learn to be comfortable saying ‘no’ — there are plenty of Python web frameworks, and maybe someone’s needs are better handled by another framework. “If everyone can check in features, you have PHP.”

From “The absolute minimum an open source developer must know about intellectual property”:

• It’s a lawyer’s job to figure out what will go wrong with your plan. They are professional pessimists.
• Only the “claims” in a patent are covered, not the stuff in the “specification.”
• A header file is a “purely functional” expression, thus NOT-copyrightable.
• If you don’t protect your Trademark, you lose it. This is why companies have to send cease and desist. The “get a first life” situation was important because Lindon explicitly granted them a license to use the Second Life trademark in the parody, thus they were able to demonstrate that they were protecting their mark.
• If you tell someone how to do the work (ie., “work for hire”), then you own it.
• An independent contractor owns their work unless the contract specifically assigns the rights to the company.
• The person who made a patch owns the patch. By giving it to you, you get an applied license to use it, but because it’s implied, it’s fuzzy as to what you can do with it.

From Robert M. Lefkowitz’s keynote:

• Only 2% of the population can read source code. (And free software doesn’t matter if no one can read it!)
• Proprietary software values function. Free software people value the building of the “community of learning” around the software, even if it has fewer features.
• The traditional view is that computer literacy is about one’s ability to use applications, rather then to program. If this is right, then what’s the point? Computers might as well be printing presses.
• In literature, you read the greats (ex., Shakespeare), then try to write like them. So in computer literacy, who are the greats? If we were going to make every high school students memorize a program, what would it be?
• Great programmers break the rules elegantly. Bad programmers break the rules without realizing it.

Heading to PyCon 2007

I’m off to PyCon 2007 (Dallas, TX) in the morning. I managaged to get into the Advanced Django tutorial (which I’m really looking forward to), so I’m heading up a day early. If you happen to be there, hopefully we’ll cross paths!

Moving my blog from WordPress to Django; Part 2: Migrating the data

In Part 1 of this series, I described some of the motivation, and the components being used to build a new blog for myself. In this (lengthy) post, I’ll address the solution I used to move my content archives from WordPress to the new app.

Installing new blog software is generally easy, but if you have legacy content that you need to preserve, the ability to move content between systems becomes of utmost importance. Fortunately, it’s quite common for popular software to provide import/export features; Having good tools to migrate content reduces switching costs, making it easy to try new software without fear of content lock-in. Unfortunately, with a home-grow blog platform, these tools need to be written from scratch.

For my soon-to-be-launched Django-based blog, importing content from my WordPress installation was an early priority — there’s only so much testing you can do with lorem ipsum posts. In tackling this content migration, I considered the following four options:

1. Support the legacy database schema.
2. Export and Import at the database level (ie., SQL dump, some text file munching, and SQL imports.)
3. Write an adapter layer to pull from the existing database and insert into the new database.
4. Export the content into a neutral format, and import from that format.

Regardless of the approach taken, I also added one important requirement: The import solution had to be so easy (and easily repeatable) that I would never hesitate to make a change to the database models when needed. Naturally, it’s nice to freeze the model once you have a stable release, but during development, even the database model should be open to agile iteration. I’ve worked on systems where every model change meant writing accompanying SQL scripts to alter the tables, and while effective, it wastes time, and I wanted the option to simply export, wipe the database clean, and re-import whenever needed. (And preferably by simply running a single script.)

I finally settled on option #4, to export into a neutral format (XML), and write an importer for that format; However, I did briefly consider each of the above options:

1) Supporting the legacy (WordPress) database schema sounds nice on the surface. This would allow the two systems to share the same database (thus eliminating the need to migrate content at all), while making it extremely easy to run the systems side-by-side (perhaps even balancing traffic between the two to test the deployment.) The downside though, is that the custom application would need to maintain the data relationships that WordPress was relying on. It’s certainly doable, but on further investigation, I found that I didn’t actually like everything about the WordPress schema; There was a bit too much de-normalized data that I didn’t want to keep around.

2) Exporting and Importing at the database level would essentially involve a mysqldump, some sed/grep/perl magic, and a SQL import into a new database. This would get the job done, but could very well lead to endless hours of tweaking regex patterns; and the end result would basically be throw-away code.

3) Writing an adapter layer was actually the most tempting at first. I knew that Django contained a tool for generating model definitions based on an existing database schema. If this worked for the WordPress database, then all I would need to do is write a thin layer to fetch content from one model and stick it into another. Sure enough, the `inspectdb` tool did do a good job, and I got so far as having routines for pulling posts and comments before realizing that this also wasn’t as reusable a solution as I wanted. Complicating matters was the need to do all this magic in a single database, since the Multiple Database Support branch of Django is still in development/testing.

With the above options scratched off the list, I went in search of a means to export directly from WordPress into a neutral format. With a little googling, I found some posts about an export/import feature that might be “in development” in the WordPress tree, but I found no documentation on the feature. Fortunately, a few more searches turned up the “WordPress XML Export” plugin, which sounded like an effort to backport the exporting feature to early versions of WordPress. After first installing the XML Export plugin, I found that it didn’t actually work with the version of WordPress on my server, but a quick look through the source code revealed a hardcoded version check that was easy enough to modify. With that change made, the plugin has run like a champ ever since.

The XML Export plugin outputs the full contents of a WordPress blog into a WXR file (WordPress eXtended RSS), which is an RSS 2.0 file, extended with a wordpress export namespace so that it can include extra metadata and comments.

With the content archives now in a massive RSS file, the next task was to write an importer. To parse the XML, I decided to use ElementTree for it’s simplicity in getting the job done. Pulling the file into ElementTree is a one-liner (when wordpress_xml_file is a File object):

tree = ET.parse(wordpress_xml_file)

The entries can be easily iterated:

for item in tree.findall("channel/item"):

Extracting the basic elements was also straight-forward (which I stuck into a Dictionary):


results['link'] = item.find(”link”).text
results['pubDate'] = item.find(”pubDate”).text
results['summary'] = item.find(”description”).text
results['body'] = item.find(”{http://purl.org/rss/1.0/modules/content/}encoded”).text
results['post_date'] = item.find(”{http://wordpress.org/export/1.0/}post_date”).text
results['post_date_gmt'] = item.find(”{http://wordpress.org/export/1.0/}post_date_gmt”).text


Extracting the Categories/Tags was only slightly more work:

results['categories'] = []

categories = item.findall(”category”)

for c in categories:
    results['categories'].append(c.text)

Pulling the comments was the only messy part of the process. The list of comments is easy enough to fetch…

comments = item.findall("{http://wordpress.org/export/1.0/}comment")

…but extracting the actual comment text is a little more work because some comments may contain child nodes. For example, a comment containing a hyperlink, bold tag, or any other HTML will be truncated if you simply use the `.text` attribute. To crawl the comment text and child tags, I used the `getiterator()` method, while concatinating `.text` attributes to assemble the full comment text. While doing this, I also decided to filter out any HTML tags from the comments, which made the process fairly simple:

tmp_comment_list = []

comment_tag = comment.find(”{http://wordpress.org/export/1.0/}comment_content”)

for comment_tag_child in comment_tag.getiterator():
    tmp_comment_text = comment_tag_child.text
    if tmp_comment_text: tmp_comment_list.append(tmp_comment_text)

the_comment['body'] = ‘ ‘.join(tmp_comment_list)

results['comments'].append(the_comment)

By writing an importer for the WXR/RSS 2.0 format, this not only solves the problem at hand, but also sets the groundwork for a reusable RSS importer. IMO, this potential reuse adds additional value to the solution (as opposed to one-off SQL munching or custom adaption layers), which makes it worth any additional work that might have gone into it. With a little re-factoring, the same system could also be extended to support the Movable Type Import Format, making the software very easy to setup and evaluate.

In Part 3, I’ll skip some of the development details and jump into the server issues, with a focus on why the new blog hasn’t launched yet. The answer lies heavily in the challenge of running a Python-based application server in shared hosting environments. The common lack of mod_python, the RAM hit, etc., all add to the complexity in adopting Django.

Moving my blog from WordPress to Django; Part 1: Assemble the wheel, don’t reinvent it

I was hoping to write this post as an announcement for my new blogging solution, but instead (since I haven’t flipped the switch yet), I thought I’d start off with why I’m doing it, and what software I’ve pulled together to keep from reinventing the wheel. (In future posts I’ll address the development itself, the unique features, and the major obstacles in moving from a WordPress installation on a shared server, to a custom web app written using Django. This last bit, the actual hosting of a Django app, is a significant one, as it is the primary issue causing a delay in switching over).

I moved my blog to WordPress software (from PyBlosxom, and a number of home-grown solutions before that) back in April 2005. I’ve been quite happy with WordPress, and would definitely recommend it for anyone who doesn’t enjoy coding (and maintaining) their own web apps. After writing a few custom plugins and a plain, but functional theme, my WordPress-based blog has been churning reliably for well over a year. However, after also using Django for over a year to build other web apps, it became too tempting not to use Django for my own site. (It really is a great framework to work with, particularly if you’re a fan of Python.)

Building a custom app isn’t all roses and cherries. (I’m not sure what that means, but it sounds good.) With an established open source solution like WordPress, you have access to thousands of testers and hackers, all working to ensure that the software is reliable. You have access to good documentation, and plenty of bloggers who post solutions for custom integration problems. Furthermore, with PHP support being almost ubiquitous in shared hosting environments, you can have a WordPress installation up and running in a matter of minutes.

With a home-grown system, you do ALL the heavy lifting in development, testing, and maintenance, and in that regard, you’re re-inventing the wheel in some areas, and leaving a community of support behind. Viewed in this light, it seems a little silly to build a custom solution when a proven, free system already exists. But custom apps can have their advantages if you can still leverage some open source communities while assembling a solution that is architected to address the specific needs you have. In my case, I tried to do as little custom, one-off engineering as possible (expect in the fun areas), while enabling a unique flexibility to re-think content interaction on my blog. I wanted the ability to prototype new feature ideas at the speed of Python + Django (which is to say, very fast), but not get bogged-down debugging ORM’s and template engines. (I’ve spent plenty of time doing that in the past.)

Not wanting to write everything from scratch, my new solution is LAMP based (Linux [Ubuntu to be specific], Apache 2, MySQL, and Python), using the Django framework for it’s generated Admin CMS, object-relation mapping library, templating engine, URL mapping, etc. In other words, the only thing not leveraged from the open source community is the actual business logic of my app (which in a blog, can be quite simple.) I’m even leveraging external services like Akismet (for filtering comment spam), and del.icio.us, flickr, and Technorati for pulling in external content and metrics. I’m also using ElementTree (for the XML parsing in my content import system), Pygments (for syntax highlighting the code embedded in blog posts), simplejson (for generating JSON from Python objects), PyTechnorati (for accessing Technorati’s API’s), the Universal Feed Parser (for pulling in external RSS/ATOM feeds), and the Yahoo! Interface Library (for the CSS Fonts and Grids libraries.) During development, I’ve also relied heavily on Subversion and Bazaar for my revision control needs.

With this arsenal of open source software, I was able to feature-match the bits I wanted from WordPress rather quickly, and then iterate on the presentation and interaction without the burden of implementing everything from scratch. Needless to say, I’m excited about the new site (it’s been running parallel to my WordPress blog for several months), and I’m eager to see what happens when I finally flip the switch and start routing traffic to it!

APUG: Austin (TX) Python Users Group meeting tonight, 7pm at Enthought

Just a reminder, the Austin (TX) Python Users Group meeting is tonight, 7pm, at Enthought, in downtown Austin. Eugene Oden will be giving a presentation on using Pyro (Python Remote Objects.)