eriksmartt.com/blog

Latest Updates: django RSS

• erik 8:08 pm on July 21, 2008 | 0 Permalink | Reply
  Tags: books (32), django, python (82), technical (34), web (72)

  Book Review: “Practical Django Projects”

  Summary:

  • Targeted at developers wanting to learn Django by building example applications rather then (or in addition to) reading the docs and man pages
  • The reader builds three working applications by following along
  • The examples are based on up-to-date Django features (ie., a 2008 build)
  • Lesson’s focused on using Django (not on Django’s inner workings)
  • Doesn’t waste time explaining Python and HTML (nor does it dive deep explaining the how/why of what you’re doing in the examples)
  • Introduces the reader to powerful Django features — covering a wide range of capability
  • Examples focus on designing for code reuse (and leading by example, by integrating with existing reusable apps and Python libraries)
  • Offers an alternative approach to learning, focused on relevant, practical examples

  Background:

  Practical Django Projects (Apress book description) was written by James Bennett, release manager and contributor to the Django Web Framework. It was published by Apress in 2008. This was Bennett’s first book.

  Full disclosure: I was provided with a free, review-copy of the book by Apress.

  The Book:

  Practical Django Projects introduces the reader to the Django Web Framework by example. It takes the reader step-by-step through three example projects: a basic CMS, a blog application (called Coltrane, which powers the author’s personal blog), and a code-sharing/snippets site (called Cab, which powers http://www.djangosnippets.org/.) The examples cover real-world problems (and integration tasks) that developers are likely to be interested in, and leaves the reader with three working Django applications.

  The lessons are spread across eleven chapters:

  1. Welcome to Django — a wonderfully short introduction that wastes no space explaining prerequisites (it assumes the reader knows Python)
  2. Your First Django Site: A Simple CMS — an introduction to the Django Admin and Flatpages
  3. Customizing the Simple CMS — customizing the Admin interface (adding TinyMCE) and developing a simple, reusable search feature
  4. A Django-Powered Weblog — defining the basic models, and using django-tagging and Generic Views
  5. Expanding the Weblog — adding del.icio.us-synced links, and custom categories
  6. Templates for the Weblog — more extensive use of Generic Views, template inheritance, and custom template tags
  7. Finishing the Weblog — using django.contrib.comments and model signals to develop a moderation system with email notification and Akismet integration; Using django.contrib.syndication to add RSS/Atom feeds
  8. A Social Code-Sharing Site — building the initial models, integrating with the pygments syntax highlighter, and writing custom model managers
  9. Form Processing in the Code-Sharing Application — great examples of using newforms (much better then the The Definitive Guide to Django’s chapter on form processing)
  10. Finishing the Code-Sharing Application — more custom template tags, this time used with bookmarking and rating features
  11. Writing Reusable Django Applications — a summary of Bennett’s philosophy on decoupling application features into reusable components (with references to the UNIX saying, “do one thing, and do it well”)

  The examples focus on building applications the “Django way” — meaning that they heavily leverage Django features such as Generic Views, custom template tags, and the django.contrib package. Each section starts by outlining the features to be developed, then walking the reader through model definitions, URLs, template design, and the request-handler (view) code.

  While working through the three example applications, Bennett teaches the reader how to decouple applications from projects, how to think about (and look for) opportunities for code reuse, and how to integrate with other reusable Django applications. The lessons aren’t so much “how does Django work”, but rather “how do you, as a developer, structure your projects to get the most out of the framework.” Depending on your level of comfort using Django and Python, the lessons will either be a breeze, or ridiculously confusing. (ie., there’s a lot of magic going on in the examples, and the book assumes that either you get it, you’re comfortable not knowing, or that you’ll figure out the finer bits when you need them.)

  The Core Message

  Ultimately, the book isn’t so much about learning Django, as it is about learning how to use Django properly (where properly is defined as the way in which the Django developers use Django.) From this perspective, it’s quite successful. The reader is shown a number of patterns and concepts that can be applied to any Django project.

  Bennett wraps up the book with a chapter on design philosophy, but I think the overall lesson of the book is best summarized on page 124, with the following quote:

  …this is the hallmark of a well-built Django application. Installing it shouldn’t involve any more work than the following:

  1. Add it to INSTALLED_APPS and run syncdb.
  2. Add a new URL pattern to route to its default URLConf.
  3. Set up any needed templates.

  This is the zen of pluggable Django applications. It’s the path Bennett wants to help you start down. The value of going down this path will depend on how often you’ll use Django in the future.

  Conclusion:

  Overall, I think the book will be more valuable for someone just getting started with Django, then someone who’s been hacking lower-level with the framework for awhile. It’s a developer-focused, quick-start, “get you on the right foot” kind of book that I certainly would have appreciated more a few years ago. The big question then, is whether this book is for you. The answer depends on a couple things, with the most important being how you like to learn. Do you prefer learning by example, or learning by reading the docs and building things on your own? If you prefer to have an expert guide you step-by-step, then this book is for you. You’ll still need to poke around in the Django documentation to really grok how it all works, but this book will get you up to speed quickly.

  If you’ve read the docs, done the online tutorials, and are still interested in picking up some best-practices on decoupling your code from your specific application (ie., learning how Django supports code reuse), then this may still be a book for you. If you know you’ll be building a large application, the lessons in the book might help prevent you from writing a single, monolithic application, or at least give you some insight into how to organize and package your code. Down the road you’ll thank yourself.

  For me personally, I was actually looking forward to this book before it came out. I think the Django docs online (as great as they are) can sometimes lack in providing best practices. However, I’ve also been using the framework professionally for a number of years (to deploy personal, start-up, and enterprise-class web applications), and I’ve previously built and deployed a pluggable, multi-site, Django-based blog engine (with del.icio.us and Akismet integration, flexible moderation rules, etc.), so the idea of using a blog engine as the core example in the book was a bit disappointing. That said, I did enjoy seeing another developer’s approach on solving the same problem, and I picked up a few nice tips around some of the more recent Django features.

  If you’re looking to build a reusable code library (and you should be, if you’re going to build more then one Django project) and ensure that you’re using Django efficiently, this book will help point you down the right path and have you thinking about decoupling your architecture from the start.

   

  Reply Click here to cancel reply.

  Name (required)

  Email (required)

  Web Site

   

• erik 7:07 pm on December 31, 2007 | 0 Permalink | Reply
  Tags: books (32), django, python (82), web (72)

  Reading “The Definitive Guide to Django”; Verdict: A solid learning reference for a beginning/intermediate Django user

  Last week I received a review-copy of the new “The Definitive Guide to Django” book from Apress. I hadn’t planned on buying the book since it seemed a little too beginner-focused; but I agreed to give it an honest reading, so I happily dove in with an “it’s Python, of course I’m going to like it” attitude.

  Background

  The book was written by Adrian Holovaty and Jacob Kaplan-Moss, the creators and “Benevolent Dictators” of the Django Web Framework. It was Holovaty and Kaplan-Moss’ first book, and, I believe, meant to be the first Django book to market. The book was drafted online; open to peer-review and community feedback; and ultimately published under the GNU Free Documentation License.

  From the get-go, the print edition had a few inherent market challenges to face: First, the entire book is available online, for free, at: <http://www.djangobook.com/>. Second, in many ways the book is a re-hash of the docs available at <http://www.djangoproject.com/documentation/>, which are also free. Third, the book covers Django 0.96, not SVN. (0.96 is technically the latest-snapshot release, but a lot has changed since 0.96.) And finally, the $45 MSRP could be seen as a little steep for what is effectively a printed copy of a free, online book.

  The print experience

  Diving in, the book takes the reader through the basic installation process, provides a brief background on how the framework came to be (and why you want one), then steps through the major features (ie., the template system, ORM, URLconfs, generic views, etc.) It’s what you’d expect from a technical reference — no fluff, and straight to the details. There are plenty of code snippets to learn from, and the sidebar notes tend to be insightful.

  Since it wasn’t new material for me, the book was a fairly quick read; but the experience of reading Django documentation in book-form was actually quite fascinating. There’s something about settling into a comfortable chair with a book, pen, and highlighter that you just can’t get with online documentation. Perhaps it was just a little more noticeable given the material. When I read the Django docs online, I tend to skim over them while trying to solve a problem. I use them as a reference more then a learning tool, and it’s usually while actively coding, thus my brain is partially distracted with whatever it is I’m building.

  With a physical book, you can unplug, step away from the computer, and give the material your undivided attention. This isolation from distraction results in a much deeper understanding of the text. This is the real the value of the printed book — it’s an opportunity to digest online documentation in an environment more conducive to learning and retention.

  My general take-aways and observations

  • The book definitely has a beginner/intermediate feel to it, but only in the sense of a beginner Django user — not a beginner Web developer or Python programmer. I’m curious how well the book is received by folks who are beginners at Django and dynamic Web development since the text brings up a lot of complex topics in Web development that aren’t really explained. (Ex., database administration, server clustering, manipulating HTTP headers, etc.)
  • The breadth of the book is impressive, but in some ways, the book really feeds you through a firehose, so to speak. It throws a lot of new concepts at the reader and doesn’t always explain why you’d need to know them, or how you might use them in the real world. For someone deploying a site with Django, it will be good to know that all these features are available, but it might take awhile before they need to use them (if ever.)
  • The book does touch on some of the more advanced Django features (like extending the template system and writing custom middleware), which was nice, but some topics are reserved for the appendix and get limited coverage (ex., model managers and ‘Q’ queries.) Others, like the Sites Framework, are given good exposure, but not so much that the reader is left with a clear picture on when to use them and what their limitations are.
  • The forms processing chapter was a bit lighter then what I was hoping for — especially given that the current newforms documentation still trends toward “read the source code.” It provides enough to start using newforms if your form needs are pretty basic, but doesn’t address creating your own widgets, or any of the fun stuff you can do once you start dynamically generating and manipulating newforms objects.
  • It might have been nicer if the examples in the book were a little more tied together, perhaps all focused on building a single example project and showing how the various features are used in real-world applications. (The example of the book-publisher’s app was a reoccurring theme, but not so strongly that each chapter applied it’s new learnings to it.)
  • The Deploying Django: “Going Big” sub-section provides a nice infrastructure graphics for how high-traffic systems might be setup, but once you get to the point of being “big”, you need to architect for it, and that’s really outside of the scope of this book. For this section, it might have been nice to reference other resources on scaling infrastructure, and perhaps pointing out some of the ways that Django can be optimized for performance and horizontal scaling. (For example, one of the Django projects we put into production at work will happily support 1,200 requests/second, but the database layer and session middleware have been reworked a bit, and the content caching approach is a little different then the standard Django offering.)
  • On the more positive side, even as someone who’s been using Django for some time, I still learned a few new tricks, and I was reminded of a few features that I could be taking better advantage of. (And when you do this stuff professionally, every shortcut and productivity gain has monetary value — avoiding even a half-hour of debugging pays for the cost of this book.)
  • This book would make a fantastic read for a back-end developer joining a project that is already using Django. I normally tell new developers to go through the Python Tutorial at <http://python.org/doc/tut/> if they’re new to Python, then to complete the Django Tutorials at <http://www.djangoproject.com/documentation/> before trying to grok any in-progress Django project. Now I have a third reference (though I might still suggest that they walk through the tutorials first, so that they have some context when reading the book. Otherwise, there are just too many new concepts to do a straight read-through and still grasp it all.)

  Summary

  The market needed a good Django book, and this one delivered a solid reference for the framework. Arguably, it’s not really a “Beginner’s Guide to Django”, but hopefully it covers enough of the basics that future books can focus on best practices and more advanced techniques. (On a related note, there’s apparently an upcoming “Practical Django Projects” book, also from Apress, that will focus more on building “reusable Django applications from start to finish”. This might actually make for a better beginner’s book, depending on how it turns out. [Via The B-List: Speaking and writing].)

  The million-dollar question then, is “Should you buy this book?” My answer ended up being a bit more positive then I expected, but there are two parts: First, if you’re a front-end developer only, you don’t need this book. You can just read Chapter 4: The Django Template System online, and then use the “Django Templates: Guide for HTML authors” section of the online docs as a reference. For back-end developers, the story is different. If you’re going to just “read it while you hack”, then you might as well just read it online; but if you’re serious about building applications with Django (especially if you’re new to it), then you should consider the book and investing the time to step away from the computer and really let yourself get into it. Unless you are an active contributor to Django (which I’m not, just to be clear), the odds are pretty good that you’ll learn something new, even if you’re already using Django today.

   

• erik 9:52 am on March 30, 2007 | 0 Permalink | Reply
  Tags: code (25), django, python (82), technical (34), web (72)

  Django “lorem ipsum” generator (and a new contrib.webdesign module)

  Django “lorem ipsum” generator (and a new contrib.webdesign module)

  The Django Web Framework project just added a new contrib.webdesign module with an amazingly simple, but incredibly handy first feature: a lorem ipsum generator. The idea is that a project’s base templates can include generated lorem ipsum for testing layout and page flow, but inheriting templates can override the generated text once real content is available.

  The lorem tag is used like this (via the contrib.webdesign docs):

  • {% lorem %} will output the common “lorem ipsum” paragraph.
  • {% lorem 3 p %} will output the common “lorem ipsum” paragraph and two random paragraphs each wrapped in HTML <p> tags.
  • {% lorem 2 w random %} will output two random Latin words.

  In practice, you might do this:

  templates/template.html:

  
  <html>
    <head>
      <title>{% block article_title %}{% lorem 5 w %}{% endblock %}</title>
    </head>
    <body>
      <div class="article">
        <div class="article_title">{% block article_title %}{% lorem 5 w %}{% endblock %}</div>
        <div class="article_body">{% block article_body %}{% lorem 4 p %}{% endblock %}</div>
      </div>
    </body>
  </html>
  

  And then inherit when you’re ready:

  templates/article.html:

  
  {% extends "template.html" %}
  
  {% if article %}
    {% block article_title %}{{ article.title }}{% endblock %}
    {% block article_body %}{{ article.body }}{% endblock %}
  {% endif %}
  

  Previously, I used to just paste lorem ipsum text directly into the main template (wrapped in block tags for overridding), but this new tag will let you skip the copy/paste routine. Very nice!

   

• erik 11:16 am on February 27, 2007 | 1 Permalink | Reply
  Tags: django, python (82), travel (25)

  PyCon 2007 wrap-up

  I’m back from PyCon 2007. It was a busy weekend, with 593 Pythonistas attending the conference. I took a fair amount of notes, but I’ve pulled out some highlights below:

  From Ivan Krstic’s keynote on the One Laptop Per Child project:

  • Python is the language of the One Laptop Per Child (OLPC). Everything that can, will be done in Python… and there’s a “view source” button on the keyboard (view layout) so you can view (and edit) the source of your current running application.
  • The filesystem (which supports versioning) is called Yellow, and will be released withing a week or so. The GUI is called Sugar, and is available on http://dev.laptop.org/ to play with. You can download the full image (or build the environment on Linux.)
  • The OLPC supports 802.11s mesh networking.
  • The hand crank was removed for case durability. The OLPC’s are designed to last five years, but the torque from the hand-crank would have stressed the plastic case too much for it to last that long.
  • The first OLPC’s will start shipping in August of this year!
  • The OLPC hardware was getting ~1100 pystones before optimization. They are now up to ~2300 pystones (on a 366 Mhz AMD Geode processor.) (Note: This means they have better Python performance then Python for S60 is seeing on current S60 phones.)

  From the Web Frameworks panel:

  • James Tauber, “Reinventing the wheel is great if your goal is to learn more about the wheel.”
  • Jonathan Ellis, “When you control the whole stack you can innovate faster.”

  From Adele Goldberg’s keynote:

  Public school education is so bad that real eLearning solutions can’t go to the schools — they need to be outside of schools so that you don’t have the traditional censorship that comes with public schools — and you don’t have the associates with the bad experiences kids have while at “school”.

  From Jacob Kaplan-Moss’ talk, “Becoming an open-source developer: Lessons from the Django project”:

  1. Use good tools. “Open source is better because it’s better.”
  2. Avoid dogma. Don’t get stuck on what language something is implemented in.
  3. Work with (and hire) smart people. The model in open source is that if you’re smart, people listen to you. That’s rough if you’re not smart… But also means that it’s worthwhile to mention when you’re an expert on a topic.
  4. “Methodologies” suck. Ex., MVC is cool, but Django abuses it because it doesn’t fit so well with the web.
  5. DRY — Don’t Repeat Yourself. The one methodology to use.
  6. The business case for open source. You have to make one (to your company.)
     • Money. You’ll get recognized and sell services because of it. (Ex., Ellignton wouldn’t be as successful without Django.)
     • Free labor. (Sad to think of this way, but true when you have an interesting project.)
     • Self-improvement. Knowing that peers will review your code makes you much more careful about the code you submit. This makes the code a lot better.
     • Geek cred — gaining credibility within the geek community makes it easier to hire great people.
     • Moral Argument — If you built a business on open source — it’s time to give something back.
     • Figure out where to draw the line — Django gave away the tools, but not all the apps.
  7. Selling open source to other companies. Microsoft’s FUD had been quite successful in some areas. Counter the “communist” argument with a “freedom” argument. Focus on the freedom of data — your data belongs to you; there is no vendor lock-in. Open vs. Lock-in is a better argument then Open vs. Closed.
  8. Create a community. This doesn’t just happen because you setup a mailing list. (Gave example of thanking people who post anti-Django blog posts and asking what they didn’t like.) Don’t say anything that would get you kicked out of a bar.
     • Avoid monsters (trolls, vampires, etc.) Detect them early, and ignore them.
     • Spam can’t be an afterthought. Collaborative tools require spam filtering from Day 1. You’ll get spam. Lots of it. Google Groups is pretty good about cutting out spam.
  9. listen to the community. But smartly. Sometimes the vocal majority doesn’t represent the wishes of the whole community. Django’s magic-removal was a big risk, driven by the community. You also have to be willing to ask for help. Sometimes you don’t feel comfortable delegating tasks that you think suck — but not everyone has the same definition of “what sucks” — sometimes there’s someone who actually WANTS to do this task!
  10. Handling community contributions. You need a defined method for how you take contributions. It helped the Django project when they adopted a system for differentiating between patches that are controversial, and those that aren’t. (ie., simple bug fixes vs. design decisions.) A ticket reviewer makes this decision.
  11. Learn to be comfortable saying ‘no’ — there are plenty of Python web frameworks, and maybe someone’s needs are better handled by another framework. “If everyone can check in features, you have PHP.”

  From “The absolute minimum an open source developer must know about intellectual property”:

  • It’s a lawyer’s job to figure out what will go wrong with your plan. They are professional pessimists.
  • Only the “claims” in a patent are covered, not the stuff in the “specification.”
  • A header file is a “purely functional” expression, thus NOT-copyrightable.
  • If you don’t protect your Trademark, you lose it. This is why companies have to send cease and desist. The “get a first life” situation was important because Lindon explicitly granted them a license to use the Second Life trademark in the parody, thus they were able to demonstrate that they were protecting their mark.
  • If you tell someone how to do the work (ie., “work for hire”), then you own it.
  • An independent contractor owns their work unless the contract specifically assigns the rights to the company.
  • The person who made a patch owns the patch. By giving it to you, you get an applied license to use it, but because it’s implied, it’s fuzzy as to what you can do with it.

  From Robert M. Lefkowitz’s keynote:

  • Only 2% of the population can read source code. (And free software doesn’t matter if no one can read it!)
  • Proprietary software values function. Free software people value the building of the “community of learning” around the software, even if it has fewer features.
  • The traditional view is that computer literacy is about one’s ability to use applications, rather then to program. If this is right, then what’s the point? Computers might as well be printing presses.
  • In literature, you read the greats (ex., Shakespeare), then try to write like them. So in computer literacy, who are the greats? If we were going to make every high school students memorize a program, what would it be?
  • Great programmers break the rules elegantly. Bad programmers break the rules without realizing it.
   

• erik 3:10 pm on February 21, 2007 | 0 Permalink | Reply
  Tags: django, python (82), travel (25)

  Heading to PyCon 2007

  I’m off to PyCon 2007 (Dallas, TX) in the morning. I managaged to get into the Advanced Django tutorial (which I’m really looking forward to), so I’m heading up a day early. If you happen to be there, hopefully we’ll cross paths!

   

• erik 3:12 pm on February 16, 2007 | 3 Permalink | Reply
  Tags: code (25), django, technical (34), web (72)

  Triggering a browser’s “Save As…” dialog using a custom Content-Type header

  My previous post, “Passing JSON via the X-JSON HTTP header with Django and Prototype“, contained an example on writing custom HTTP headers from a Django-based web application. Continuing with that theme, here’s another header trick that I use in one of my apps to force the browser’s “Save As…” dialog box when viewing a particular URL.

  The feature that I wanted was the ability to generate an XML file based on an HTTP GET request, but to have the browser open a “Save As…” dialog instead of attempting to render it (as would normally happen with XML in a modern browser.) The solution is to exploit the web browser behavior of not handling unknown mime types. A sample implementation (written in Python for the Django Web Framework) follows:

  def save_as_xml(request):
      import datetime
  
      current_time = datetime.now()
  
      response = HttpResponse('PUT THE XML HERE')
      response['Content-Type'] = 'application/x-generated-xml-backup'
      response['Content-disposition'] = 'Attachment; filename=export.%s.xml' % (current_time.strftime("%Y-%m-%d"))
  
      return response
  

  Setting the Content-Type header to a made-up type ensures that the browser will not attempt to render the file. The Content-disposition header provides the mechanism for suggesting the filename of the content to be saved on the viewer’s system. In this case, I’m using the standard `datetime` module to insert the date into the suggested filename.

   

• erik 2:39 pm on February 16, 2007 | 0 Permalink | Reply
  Tags: code (25), django, technical (34), web (72)

  Passing JSON via the X-JSON HTTP header with Django and Prototype

  One of the demo sites I was working on this week needed to pass a small amount of JSON back with it’s page results. There are a few ways to do this (and I’d suggest this post, “Loading Content with JSON” as a starting point if you’re looking for ideas), but for simplicity, I decided to take advantage of the automatic X-JSON HTTP Header parsing feature in Prototype 1.5.0. (The Ajax.Request docs address this capability.)

  The sample code below demonstrates the use of the X-JSON header with an simple “sticky notes” web app. On the client-side, the JavaScript is quite simple. The second variable in the onSuccess callback handler will be automatically initialized using the data in the X-JSON header:

  function display_note(id) {
      new Ajax.Request('/api/note/' + id + '/', {
          method: 'get',
          onSuccess: function(transport, results) {
              alert("Note(" + results['id'] + ") `" + results['title'] + "`: " + results['body']);
          },
          }
      );
  }
  

  To handle this request, I’m using Django on the server with the following URL pattern:

  (r'^api/note/(?P\d+)/$', 'views.get_note')
  

  The `get_note` method implementation looks like this: [NOTE: For production use, you'll want some exception handling, but I removed the error handling to simplify the example.]

  def get_note(request, id):
      # Fetch the Note from the DB:
      note = Note.objects.get(pk=id)
      # Create the response object (with some dummy text for now):
      response = HttpResponse('Check the X-JSON header.')
      # Manually set the X-JSON header using the JSON generated from the Note record:
      response['X-JSON'] = cjson.encode(note.__dict__)
      # Return the response object:
      return response
  

  If you’d like to use this technique on your own sites, there are couple points to remember:

  1. You can’t return an empty HTTP Response regardless of there being an X-JSON header. If the response is empty, the browser will hang waiting for content to arrive.
  2. The X-JSON header should only be used for small payloads. Don’t stuff more then 8kb in your headers. If you’re sending more then that, move the JSON to the body of the response.
  3. The cjson and simplejson encoders don’t handle Django DateTime fields. For objects with DateTime fields, write an alternate method for converting the object into a dictionary before passing it to the json encoder.

  [Update: 2009/04/10]
  This post got some flak from a firewall vendor for demonstrating a technique of pushing JSON out that will likely bypass their security checks. Since I haven’t used their products in a long time, I’m not too bothered by this, but I do want to point out that you likely don’t want to use HTTP Headers to pass this kind of data in a production site anyway. You’ll find out quickly that there are character limits to how much you can put in the headers, and before long, the distinction between what data goes in the header and what goes in the body will blur. Once that happens you’ve got a big mess on your hands. Better would be to avoid this pattern all together. My post here simply demonstrates how to use the technique, should you be interested in doing so.

   

• erik 1:23 pm on December 30, 2006 | 8 Permalink | Reply
  Tags: blogging (26), code (25), django, python (82), technical (34), web (72)

  Moving my blog from WordPress to Django; Part 2: Migrating the data

  In Part 1 of this series, I described some of the motivation, and the components being used to build a new blog for myself. In this (lengthy) post, I’ll address the solution I used to move my content archives from WordPress to the new app.

  Installing new blog software is generally easy, but if you have legacy content that you need to preserve, the ability to move content between systems becomes of utmost importance. Fortunately, it’s quite common for popular software to provide import/export features; Having good tools to migrate content reduces switching costs, making it easy to try new software without fear of content lock-in. Unfortunately, with a home-grow blog platform, these tools need to be written from scratch.

  For my soon-to-be-launched Django-based blog, importing content from my WordPress installation was an early priority — there’s only so much testing you can do with lorem ipsum posts. In tackling this content migration, I considered the following four options:

  1. Support the legacy database schema.
  2. Export and Import at the database level (ie., SQL dump, some text file munching, and SQL imports.)
  3. Write an adapter layer to pull from the existing database and insert into the new database.
  4. Export the content into a neutral format, and import from that format.

  Regardless of the approach taken, I also added one important requirement: The import solution had to be so easy (and easily repeatable) that I would never hesitate to make a change to the database models when needed. Naturally, it’s nice to freeze the model once you have a stable release, but during development, even the database model should be open to agile iteration. I’ve worked on systems where every model change meant writing accompanying SQL scripts to alter the tables, and while effective, it wastes time, and I wanted the option to simply export, wipe the database clean, and re-import whenever needed. (And preferably by simply running a single script.)

  I finally settled on option #4, to export into a neutral format (XML), and write an importer for that format; However, I did briefly consider each of the above options:

  1) Supporting the legacy (WordPress) database schema sounds nice on the surface. This would allow the two systems to share the same database (thus eliminating the need to migrate content at all), while making it extremely easy to run the systems side-by-side (perhaps even balancing traffic between the two to test the deployment.) The downside though, is that the custom application would need to maintain the data relationships that WordPress was relying on. It’s certainly doable, but on further investigation, I found that I didn’t actually like everything about the WordPress schema; There was a bit too much de-normalized data that I didn’t want to keep around.

  2) Exporting and Importing at the database level would essentially involve a mysqldump, some sed/grep/perl magic, and a SQL import into a new database. This would get the job done, but could very well lead to endless hours of tweaking regex patterns; and the end result would basically be throw-away code.

  3) Writing an adapter layer was actually the most tempting at first. I knew that Django contained a tool for generating model definitions based on an existing database schema. If this worked for the WordPress database, then all I would need to do is write a thin layer to fetch content from one model and stick it into another. Sure enough, the `inspectdb` tool did do a good job, and I got so far as having routines for pulling posts and comments before realizing that this also wasn’t as reusable a solution as I wanted. Complicating matters was the need to do all this magic in a single database, since the Multiple Database Support branch of Django is still in development/testing.

  With the above options scratched off the list, I went in search of a means to export directly from WordPress into a neutral format. With a little googling, I found some posts about an export/import feature that might be “in development” in the WordPress tree, but I found no documentation on the feature. Fortunately, a few more searches turned up the “WordPress XML Export” plugin, which sounded like an effort to backport the exporting feature to early versions of WordPress. After first installing the XML Export plugin, I found that it didn’t actually work with the version of WordPress on my server, but a quick look through the source code revealed a hardcoded version check that was easy enough to modify. With that change made, the plugin has run like a champ ever since.

  The XML Export plugin outputs the full contents of a WordPress blog into a WXR file (WordPress eXtended RSS), which is an RSS 2.0 file, extended with a wordpress export namespace so that it can include extra metadata and comments.

  With the content archives now in a massive RSS file, the next task was to write an importer. To parse the XML, I decided to use ElementTree for it’s simplicity in getting the job done. Pulling the file into ElementTree is a one-liner (when wordpress_xml_file is a File object):

  tree = ET.parse(wordpress_xml_file)

  The entries can be easily iterated:

  for item in tree.findall("channel/item"):

  Extracting the basic elements was also straight-forward (which I stuck into a Dictionary):

  
results['link'] = item.find("link").text
results['pubDate'] = item.find("pubDate").text
results['summary'] = item.find("description").text
results['body'] = item.find("{http://purl.org/rss/1.0/modules/content/}encoded").text
results['post_date'] = item.find("{http://wordpress.org/export/1.0/}post_date").text
results['post_date_gmt'] = item.find("{http://wordpress.org/export/1.0/}post_date_gmt").text


  Extracting the Categories/Tags was only slightly more work:

  
  results['categories'] = []
  
  categories = item.findall("category")
  
  for c in categories:
      results['categories'].append(c.text)
  

  Pulling the comments was the only messy part of the process. The list of comments is easy enough to fetch…

  comments = item.findall("{http://wordpress.org/export/1.0/}comment")

  …but extracting the actual comment text is a little more work because some comments may contain child nodes. For example, a comment containing a hyperlink, bold tag, or any other HTML will be truncated if you simply use the `.text` attribute. To crawl the comment text and child tags, I used the `getiterator()` method, while concatinating `.text` attributes to assemble the full comment text. While doing this, I also decided to filter out any HTML tags from the comments, which made the process fairly simple:

  
  tmp_comment_list = []
  
  comment_tag = comment.find("{http://wordpress.org/export/1.0/}comment_content")
  
  for comment_tag_child in comment_tag.getiterator():
      tmp_comment_text = comment_tag_child.text
      if tmp_comment_text: tmp_comment_list.append(tmp_comment_text)
  
  the_comment['body'] = ' '.join(tmp_comment_list)
  
  results['comments'].append(the_comment)
  

  By writing an importer for the WXR/RSS 2.0 format, this not only solves the problem at hand, but also sets the groundwork for a reusable RSS importer. IMO, this potential reuse adds additional value to the solution (as opposed to one-off SQL munching or custom adaption layers), which makes it worth any additional work that might have gone into it. With a little re-factoring, the same system could also be extended to support the Movable Type Import Format, making the software very easy to setup and evaluate.

  In Part 3, I’ll skip some of the development details and jump into the server issues, with a focus on why the new blog hasn’t launched yet. The answer lies heavily in the challenge of running a Python-based application server in shared hosting environments. The common lack of mod_python, the RAM hit, etc., all add to the complexity in adopting Django.

   

• erik 6:41 pm on December 27, 2006 | 3 Permalink | Reply
  Tags: blogging (26), django, python (82), web (72)

  Moving my blog from WordPress to Django; Part 1: Assemble the wheel, don’t reinvent it

  I was hoping to write this post as an announcement for my new blogging solution, but instead (since I haven’t flipped the switch yet), I thought I’d start off with why I’m doing it, and what software I’ve pulled together to keep from reinventing the wheel. (In future posts I’ll address the development itself, the unique features, and the major obstacles in moving from a WordPress installation on a shared server, to a custom web app written using Django. This last bit, the actual hosting of a Django app, is a significant one, as it is the primary issue causing a delay in switching over).

  I moved my blog to WordPress software (from PyBlosxom, and a number of home-grown solutions before that) back in April 2005. I’ve been quite happy with WordPress, and would definitely recommend it for anyone who doesn’t enjoy coding (and maintaining) their own web apps. After writing a few custom plugins and a plain, but functional theme, my WordPress-based blog has been churning reliably for well over a year. However, after also using Django for over a year to build other web apps, it became too tempting not to use Django for my own site. (It really is a great framework to work with, particularly if you’re a fan of Python.)

  Building a custom app isn’t all roses and cherries. (I’m not sure what that means, but it sounds good.) With an established open source solution like WordPress, you have access to thousands of testers and hackers, all working to ensure that the software is reliable. You have access to good documentation, and plenty of bloggers who post solutions for custom integration problems. Furthermore, with PHP support being almost ubiquitous in shared hosting environments, you can have a WordPress installation up and running in a matter of minutes.

  With a home-grown system, you do ALL the heavy lifting in development, testing, and maintenance, and in that regard, you’re re-inventing the wheel in some areas, and leaving a community of support behind. Viewed in this light, it seems a little silly to build a custom solution when a proven, free system already exists. But custom apps can have their advantages if you can still leverage some open source communities while assembling a solution that is architected to address the specific needs you have. In my case, I tried to do as little custom, one-off engineering as possible (expect in the fun areas), while enabling a unique flexibility to re-think content interaction on my blog. I wanted the ability to prototype new feature ideas at the speed of Python + Django (which is to say, very fast), but not get bogged-down debugging ORM’s and template engines. (I’ve spent plenty of time doing that in the past.)

  Not wanting to write everything from scratch, my new solution is LAMP based (Linux [Ubuntu to be specific], Apache 2, MySQL, and Python), using the Django framework for it’s generated Admin CMS, object-relation mapping library, templating engine, URL mapping, etc. In other words, the only thing not leveraged from the open source community is the actual business logic of my app (which in a blog, can be quite simple.) I’m even leveraging external services like Akismet (for filtering comment spam), and del.icio.us, flickr, and Technorati for pulling in external content and metrics. I’m also using ElementTree (for the XML parsing in my content import system), Pygments (for syntax highlighting the code embedded in blog posts), simplejson (for generating JSON from Python objects), PyTechnorati (for accessing Technorati’s API’s), the Universal Feed Parser (for pulling in external RSS/ATOM feeds), and the Yahoo! Interface Library (for the CSS Fonts and Grids libraries.) During development, I’ve also relied heavily on Subversion and Bazaar for my revision control needs.

  With this arsenal of open source software, I was able to feature-match the bits I wanted from WordPress rather quickly, and then iterate on the presentation and interaction without the burden of implementing everything from scratch. Needless to say, I’m excited about the new site (it’s been running parallel to my WordPress blog for several months), and I’m eager to see what happens when I finally flip the switch and start routing traffic to it!