eriksmartt.com/blog

WP update b0rked my full-post feeds

I just noticed today that my Wordpress installation has switched itself over to partial/summary feeds instead of full-post feeds. I’m not a fan of partial feeds, but it seems that WP isn’t agreeing with me. Even after the 2.1.3 update, and installing the Full Text Feed plugin, it still doesn’t seem to be working. If you have any ideas, I’m all ears; otherwise, I’ll poke around a bit more and see if I can track down the cause.

Moving my blog from WordPress to Django; Part 2: Migrating the data

In Part 1 of this series, I described some of the motivation, and the components being used to build a new blog for myself. In this (lengthy) post, I’ll address the solution I used to move my content archives from WordPress to the new app.

Installing new blog software is generally easy, but if you have legacy content that you need to preserve, the ability to move content between systems becomes of utmost importance. Fortunately, it’s quite common for popular software to provide import/export features; Having good tools to migrate content reduces switching costs, making it easy to try new software without fear of content lock-in. Unfortunately, with a home-grow blog platform, these tools need to be written from scratch.

For my soon-to-be-launched Django-based blog, importing content from my WordPress installation was an early priority — there’s only so much testing you can do with lorem ipsum posts. In tackling this content migration, I considered the following four options:

1. Support the legacy database schema.
2. Export and Import at the database level (ie., SQL dump, some text file munching, and SQL imports.)
3. Write an adapter layer to pull from the existing database and insert into the new database.
4. Export the content into a neutral format, and import from that format.

Regardless of the approach taken, I also added one important requirement: The import solution had to be so easy (and easily repeatable) that I would never hesitate to make a change to the database models when needed. Naturally, it’s nice to freeze the model once you have a stable release, but during development, even the database model should be open to agile iteration. I’ve worked on systems where every model change meant writing accompanying SQL scripts to alter the tables, and while effective, it wastes time, and I wanted the option to simply export, wipe the database clean, and re-import whenever needed. (And preferably by simply running a single script.)

I finally settled on option #4, to export into a neutral format (XML), and write an importer for that format; However, I did briefly consider each of the above options:

1) Supporting the legacy (WordPress) database schema sounds nice on the surface. This would allow the two systems to share the same database (thus eliminating the need to migrate content at all), while making it extremely easy to run the systems side-by-side (perhaps even balancing traffic between the two to test the deployment.) The downside though, is that the custom application would need to maintain the data relationships that WordPress was relying on. It’s certainly doable, but on further investigation, I found that I didn’t actually like everything about the WordPress schema; There was a bit too much de-normalized data that I didn’t want to keep around.

2) Exporting and Importing at the database level would essentially involve a mysqldump, some sed/grep/perl magic, and a SQL import into a new database. This would get the job done, but could very well lead to endless hours of tweaking regex patterns; and the end result would basically be throw-away code.

3) Writing an adapter layer was actually the most tempting at first. I knew that Django contained a tool for generating model definitions based on an existing database schema. If this worked for the WordPress database, then all I would need to do is write a thin layer to fetch content from one model and stick it into another. Sure enough, the `inspectdb` tool did do a good job, and I got so far as having routines for pulling posts and comments before realizing that this also wasn’t as reusable a solution as I wanted. Complicating matters was the need to do all this magic in a single database, since the Multiple Database Support branch of Django is still in development/testing.

With the above options scratched off the list, I went in search of a means to export directly from WordPress into a neutral format. With a little googling, I found some posts about an export/import feature that might be “in development” in the WordPress tree, but I found no documentation on the feature. Fortunately, a few more searches turned up the “WordPress XML Export” plugin, which sounded like an effort to backport the exporting feature to early versions of WordPress. After first installing the XML Export plugin, I found that it didn’t actually work with the version of WordPress on my server, but a quick look through the source code revealed a hardcoded version check that was easy enough to modify. With that change made, the plugin has run like a champ ever since.

The XML Export plugin outputs the full contents of a WordPress blog into a WXR file (WordPress eXtended RSS), which is an RSS 2.0 file, extended with a wordpress export namespace so that it can include extra metadata and comments.

With the content archives now in a massive RSS file, the next task was to write an importer. To parse the XML, I decided to use ElementTree for it’s simplicity in getting the job done. Pulling the file into ElementTree is a one-liner (when wordpress_xml_file is a File object):

tree = ET.parse(wordpress_xml_file)

The entries can be easily iterated:

for item in tree.findall("channel/item"):

Extracting the basic elements was also straight-forward (which I stuck into a Dictionary):


results['link'] = item.find(”link”).text
results['pubDate'] = item.find(”pubDate”).text
results['summary'] = item.find(”description”).text
results['body'] = item.find(”{http://purl.org/rss/1.0/modules/content/}encoded”).text
results['post_date'] = item.find(”{http://wordpress.org/export/1.0/}post_date”).text
results['post_date_gmt'] = item.find(”{http://wordpress.org/export/1.0/}post_date_gmt”).text


Extracting the Categories/Tags was only slightly more work:

results['categories'] = []

categories = item.findall(”category”)

for c in categories:
    results['categories'].append(c.text)

Pulling the comments was the only messy part of the process. The list of comments is easy enough to fetch…

comments = item.findall("{http://wordpress.org/export/1.0/}comment")

…but extracting the actual comment text is a little more work because some comments may contain child nodes. For example, a comment containing a hyperlink, bold tag, or any other HTML will be truncated if you simply use the `.text` attribute. To crawl the comment text and child tags, I used the `getiterator()` method, while concatinating `.text` attributes to assemble the full comment text. While doing this, I also decided to filter out any HTML tags from the comments, which made the process fairly simple:

tmp_comment_list = []

comment_tag = comment.find(”{http://wordpress.org/export/1.0/}comment_content”)

for comment_tag_child in comment_tag.getiterator():
    tmp_comment_text = comment_tag_child.text
    if tmp_comment_text: tmp_comment_list.append(tmp_comment_text)

the_comment['body'] = ‘ ‘.join(tmp_comment_list)

results['comments'].append(the_comment)

By writing an importer for the WXR/RSS 2.0 format, this not only solves the problem at hand, but also sets the groundwork for a reusable RSS importer. IMO, this potential reuse adds additional value to the solution (as opposed to one-off SQL munching or custom adaption layers), which makes it worth any additional work that might have gone into it. With a little re-factoring, the same system could also be extended to support the Movable Type Import Format, making the software very easy to setup and evaluate.

In Part 3, I’ll skip some of the development details and jump into the server issues, with a focus on why the new blog hasn’t launched yet. The answer lies heavily in the challenge of running a Python-based application server in shared hosting environments. The common lack of mod_python, the RAM hit, etc., all add to the complexity in adopting Django.

Moving my blog from WordPress to Django; Part 1: Assemble the wheel, don’t reinvent it

I was hoping to write this post as an announcement for my new blogging solution, but instead (since I haven’t flipped the switch yet), I thought I’d start off with why I’m doing it, and what software I’ve pulled together to keep from reinventing the wheel. (In future posts I’ll address the development itself, the unique features, and the major obstacles in moving from a WordPress installation on a shared server, to a custom web app written using Django. This last bit, the actual hosting of a Django app, is a significant one, as it is the primary issue causing a delay in switching over).

I moved my blog to WordPress software (from PyBlosxom, and a number of home-grown solutions before that) back in April 2005. I’ve been quite happy with WordPress, and would definitely recommend it for anyone who doesn’t enjoy coding (and maintaining) their own web apps. After writing a few custom plugins and a plain, but functional theme, my WordPress-based blog has been churning reliably for well over a year. However, after also using Django for over a year to build other web apps, it became too tempting not to use Django for my own site. (It really is a great framework to work with, particularly if you’re a fan of Python.)

Building a custom app isn’t all roses and cherries. (I’m not sure what that means, but it sounds good.) With an established open source solution like WordPress, you have access to thousands of testers and hackers, all working to ensure that the software is reliable. You have access to good documentation, and plenty of bloggers who post solutions for custom integration problems. Furthermore, with PHP support being almost ubiquitous in shared hosting environments, you can have a WordPress installation up and running in a matter of minutes.

With a home-grown system, you do ALL the heavy lifting in development, testing, and maintenance, and in that regard, you’re re-inventing the wheel in some areas, and leaving a community of support behind. Viewed in this light, it seems a little silly to build a custom solution when a proven, free system already exists. But custom apps can have their advantages if you can still leverage some open source communities while assembling a solution that is architected to address the specific needs you have. In my case, I tried to do as little custom, one-off engineering as possible (expect in the fun areas), while enabling a unique flexibility to re-think content interaction on my blog. I wanted the ability to prototype new feature ideas at the speed of Python + Django (which is to say, very fast), but not get bogged-down debugging ORM’s and template engines. (I’ve spent plenty of time doing that in the past.)

Not wanting to write everything from scratch, my new solution is LAMP based (Linux [Ubuntu to be specific], Apache 2, MySQL, and Python), using the Django framework for it’s generated Admin CMS, object-relation mapping library, templating engine, URL mapping, etc. In other words, the only thing not leveraged from the open source community is the actual business logic of my app (which in a blog, can be quite simple.) I’m even leveraging external services like Akismet (for filtering comment spam), and del.icio.us, flickr, and Technorati for pulling in external content and metrics. I’m also using ElementTree (for the XML parsing in my content import system), Pygments (for syntax highlighting the code embedded in blog posts), simplejson (for generating JSON from Python objects), PyTechnorati (for accessing Technorati’s API’s), the Universal Feed Parser (for pulling in external RSS/ATOM feeds), and the Yahoo! Interface Library (for the CSS Fonts and Grids libraries.) During development, I’ve also relied heavily on Subversion and Bazaar for my revision control needs.

With this arsenal of open source software, I was able to feature-match the bits I wanted from WordPress rather quickly, and then iterate on the presentation and interaction without the burden of implementing everything from scratch. Needless to say, I’m excited about the new site (it’s been running parallel to my WordPress blog for several months), and I’m eager to see what happens when I finally flip the switch and start routing traffic to it!

Collecting Python for S60 success stories

Even though I find the title a little inappropriate (sorry, Kevin), there’s a question being posed over on the Forum Nokia blogs that’s worth responding to if you happen to use Python for S60:
“What’s Python good for, anyway?”

Kevin Sharp (the post’s author, and one of Nokia’s Technical Editors), is looking to help promote Python for S60, but first needs to collect some success stories.

I know there’s a few of you who drop by this blog from time to time that are doing amazing things with PyS60. If you can spare a few minutes to write a response, it would be a great help to the project. Thanks!

Switching to Google Reader

I’ve been a long-time NetNewsWire Pro (NNW) user, and still contend that it is one of the nicest desktop aggregators to read news in. With Bloglines and Newsgator integration, and some voodoo syncing features, NNW also makes it easy to roam between OS X machines. However, at home, I spend more time on my Linux box, and NNW’s Mac-only status means I need a different client. Furthermore, I complicate matters by occasionally wanting to check my feeds from my phone or Nokia 770. Realizing that a desktop client wasn’t going to solve my roaming needs, I’ve played with a number of web-based aggregators, but I’ve always found them lacking a bit. That is, until I tried the Google Reader redesign last week…

Moving from NNW to Google Reader was as easy as expected. It was simply a matter of exporting subscriptions to an OPML file, and uploading that file to Google Reader (under Settings :: Import/Export.)

Using Google Reader is quite pleasant. The big feature they nailed is the manner in which it can automatically flag posts as “read” once you’ve scrolled past them. This alone improves the interaction tremendously over other online readers I’ve tried. Furthermore, when you combine keyboard navigation, it’s easy to crank through news or scan headlines when you have a few minutes to spare. And finally, of course, roaming between machines is a breeze (though I haven’t tried it on the 770 just yet.)

If you haven’t read about the redesign work Google has done on Reader (and you’re interested in RSS/Atom aggregators, or slick AJAX apps in general), check out the Official Google Reader Blog for details.

Blogging with TextMate

I’ve been using TextMate to write my blog posts for quite some time now. I enjoy the convenience of MarsEdit, but TextMate lies in the center of my GTD process, and it’s such a nice text editor that I hate to leave it. Of course, actually making blog posts has required a little copy/paste action to my web CMS, but not anymore — TextMate now has a Blogging Bundle! Check out the screencast to see it in action. (And yes, this post was made directly from TextMate ;-)

Comment spam on the rise

FYI, I started getting comment-spammed out the wazzu last week — several hundred a day, and some of them started slipping past my spam filters (HashCash and Akismet.) Apparently I need to load up the armor a bit more, so until then, I turned moderation ‘on’ for all comments so that these pesky spammers don’t get through.

If you have any suggestions on other WordPress spam filters, please let me know!

Cheers,
Erik

Great corporate blogging: Why Adobe CS2 isn’t coming to MacIntel

Here’s a great example of using a corporate blog to explain an unfortunate product decision to customers, “Living Photoshop: Macintosh and the Intel switch“.

The blogger, Scott Byer, is explaining why Adobe isn’t going to release a MacIntel version of CS2 — the summary being that transitioning their entire development over to XCode wasn’t feasible for CS2. It’s what Adobe’s Mac customers need to hear (that there’s a significant technical hurdle which didn’t make sense for a product currently in the market), but it’s not the kind of thing that a press release can explain. In other words, it’s important information, but without bloggers, there’s no clear channel for how to communicate this stuff.

It’s also of note that the comments list (the conversation) stretches far beyond the length of the original post. The customer feedback is mixed — ranging from “You guys suck, I spent $1000 on this software!”, to “Yeah, I understand the challenge first hand. Thanks for the explanation.”