I’m happy to say that a few people have bought some of these books, and I’ve made enough referral money now to pick up a free book! I decide to buy “Innovation and Entrepreneurship” by Peter Drucker. Innovation, and the application of innovation as a competitive strategy, is a topic I’m particularly interested in. In addition, I really enjoyed Drucker’s “The Effective Executive“, so I’m looking forward to reading more of his work.

So thank you! Reading this next book will be even nicer knowing that my friends (and some strangers) were willing to share their “click-throughs” with me!

Installing new blog software is generally easy, but if you have legacy content that you need to preserve, the ability to move content between systems becomes of utmost importance. Fortunately, it’s quite common for popular software to provide import/export features; Having good tools to migrate content reduces switching costs, making it easy to try new software without fear of content lock-in. Unfortunately, with a home-grow blog platform, these tools need to be written from scratch.

For my soon-to-be-launched Django-based blog, importing content from my WordPress installation was an early priority — there’s only so much testing you can do with lorem ipsum posts. In tackling this content migration, I considered the following four options:

1. Support the legacy database schema.
2. Export and Import at the database level (ie., SQL dump, some text file munching, and SQL imports.)
3. Write an adapter layer to pull from the existing database and insert into the new database.
4. Export the content into a neutral format, and import from that format.

Regardless of the approach taken, I also added one important requirement: The import solution had to be so easy (and easily repeatable) that I would never hesitate to make a change to the database models when needed. Naturally, it’s nice to freeze the model once you have a stable release, but during development, even the database model should be open to agile iteration. I’ve worked on systems where every model change meant writing accompanying SQL scripts to alter the tables, and while effective, it wastes time, and I wanted the option to simply export, wipe the database clean, and re-import whenever needed. (And preferably by simply running a single script.)

I finally settled on option #4, to export into a neutral format (XML), and write an importer for that format; However, I did briefly consider each of the above options:

1) Supporting the legacy (WordPress) database schema sounds nice on the surface. This would allow the two systems to share the same database (thus eliminating the need to migrate content at all), while making it extremely easy to run the systems side-by-side (perhaps even balancing traffic between the two to test the deployment.) The downside though, is that the custom application would need to maintain the data relationships that WordPress was relying on. It’s certainly doable, but on further investigation, I found that I didn’t actually like everything about the WordPress schema; There was a bit too much de-normalized data that I didn’t want to keep around.

2) Exporting and Importing at the database level would essentially involve a mysqldump, some sed/grep/perl magic, and a SQL import into a new database. This would get the job done, but could very well lead to endless hours of tweaking regex patterns; and the end result would basically be throw-away code.

3) Writing an adapter layer was actually the most tempting at first. I knew that Django contained a tool for generating model definitions based on an existing database schema. If this worked for the WordPress database, then all I would need to do is write a thin layer to fetch content from one model and stick it into another. Sure enough, the `inspectdb` tool did do a good job, and I got so far as having routines for pulling posts and comments before realizing that this also wasn’t as reusable a solution as I wanted. Complicating matters was the need to do all this magic in a single database, since the Multiple Database Support branch of Django is still in development/testing.

With the above options scratched off the list, I went in search of a means to export directly from WordPress into a neutral format. With a little googling, I found some posts about an export/import feature that might be “in development” in the WordPress tree, but I found no documentation on the feature. Fortunately, a few more searches turned up the “WordPress XML Export” plugin, which sounded like an effort to backport the exporting feature to early versions of WordPress. After first installing the XML Export plugin, I found that it didn’t actually work with the version of WordPress on my server, but a quick look through the source code revealed a hardcoded version check that was easy enough to modify. With that change made, the plugin has run like a champ ever since.

The XML Export plugin outputs the full contents of a WordPress blog into a WXR file (WordPress eXtended RSS), which is an RSS 2.0 file, extended with a wordpress export namespace so that it can include extra metadata and comments.

With the content archives now in a massive RSS file, the next task was to write an importer. To parse the XML, I decided to use ElementTree for it’s simplicity in getting the job done. Pulling the file into ElementTree is a one-liner (when wordpress_xml_file is a File object):

tree = ET.parse(wordpress_xml_file)

The entries can be easily iterated:

for item in tree.findall("channel/item"):

Extracting the basic elements was also straight-forward (which I stuck into a Dictionary):


results['link'] = item.find("link").text
results['pubDate'] = item.find("pubDate").text
results['summary'] = item.find("description").text
results['body'] = item.find("{http://purl.org/rss/1.0/modules/content/}encoded").text
results['post_date'] = item.find("{http://wordpress.org/export/1.0/}post_date").text
results['post_date_gmt'] = item.find("{http://wordpress.org/export/1.0/}post_date_gmt").text


Extracting the Categories/Tags was only slightly more work:

results['categories'] = []

categories = item.findall("category")

for c in categories:
    results['categories'].append(c.text)

Pulling the comments was the only messy part of the process. The list of comments is easy enough to fetch…

comments = item.findall("{http://wordpress.org/export/1.0/}comment")

…but extracting the actual comment text is a little more work because some comments may contain child nodes. For example, a comment containing a hyperlink, bold tag, or any other HTML will be truncated if you simply use the `.text` attribute. To crawl the comment text and child tags, I used the `getiterator()` method, while concatinating `.text` attributes to assemble the full comment text. While doing this, I also decided to filter out any HTML tags from the comments, which made the process fairly simple:

tmp_comment_list = []

comment_tag = comment.find("{http://wordpress.org/export/1.0/}comment_content")

for comment_tag_child in comment_tag.getiterator():
    tmp_comment_text = comment_tag_child.text
    if tmp_comment_text: tmp_comment_list.append(tmp_comment_text)

the_comment['body'] = ' '.join(tmp_comment_list)

results['comments'].append(the_comment)

By writing an importer for the WXR/RSS 2.0 format, this not only solves the problem at hand, but also sets the groundwork for a reusable RSS importer. IMO, this potential reuse adds additional value to the solution (as opposed to one-off SQL munching or custom adaption layers), which makes it worth any additional work that might have gone into it. With a little re-factoring, the same system could also be extended to support the Movable Type Import Format, making the software very easy to setup and evaluate.

In Part 3, I’ll skip some of the development details and jump into the server issues, with a focus on why the new blog hasn’t launched yet. The answer lies heavily in the challenge of running a Python-based application server in shared hosting environments. The common lack of mod_python, the RAM hit, etc., all add to the complexity in adopting Django.

I moved my blog to WordPress software (from PyBlosxom, and a number of home-grown solutions before that) back in April 2005. I’ve been quite happy with WordPress, and would definitely recommend it for anyone who doesn’t enjoy coding (and maintaining) their own web apps. After writing a few custom plugins and a plain, but functional theme, my WordPress-based blog has been churning reliably for well over a year. However, after also using Django for over a year to build other web apps, it became too tempting not to use Django for my own site. (It really is a great framework to work with, particularly if you’re a fan of Python.)

Building a custom app isn’t all roses and cherries. (I’m not sure what that means, but it sounds good.) With an established open source solution like WordPress, you have access to thousands of testers and hackers, all working to ensure that the software is reliable. You have access to good documentation, and plenty of bloggers who post solutions for custom integration problems. Furthermore, with PHP support being almost ubiquitous in shared hosting environments, you can have a WordPress installation up and running in a matter of minutes.

With a home-grown system, you do ALL the heavy lifting in development, testing, and maintenance, and in that regard, you’re re-inventing the wheel in some areas, and leaving a community of support behind. Viewed in this light, it seems a little silly to build a custom solution when a proven, free system already exists. But custom apps can have their advantages if you can still leverage some open source communities while assembling a solution that is architected to address the specific needs you have. In my case, I tried to do as little custom, one-off engineering as possible (expect in the fun areas), while enabling a unique flexibility to re-think content interaction on my blog. I wanted the ability to prototype new feature ideas at the speed of Python + Django (which is to say, very fast), but not get bogged-down debugging ORM’s and template engines. (I’ve spent plenty of time doing that in the past.)

Not wanting to write everything from scratch, my new solution is LAMP based (Linux [Ubuntu to be specific], Apache 2, MySQL, and Python), using the Django framework for it’s generated Admin CMS, object-relation mapping library, templating engine, URL mapping, etc. In other words, the only thing not leveraged from the open source community is the actual business logic of my app (which in a blog, can be quite simple.) I’m even leveraging external services like Akismet (for filtering comment spam), and del.icio.us, flickr, and Technorati for pulling in external content and metrics. I’m also using ElementTree (for the XML parsing in my content import system), Pygments (for syntax highlighting the code embedded in blog posts), simplejson (for generating JSON from Python objects), PyTechnorati (for accessing Technorati’s API’s), the Universal Feed Parser (for pulling in external RSS/ATOM feeds), and the Yahoo! Interface Library (for the CSS Fonts and Grids libraries.) During development, I’ve also relied heavily on Subversion and Bazaar for my revision control needs.

With this arsenal of open source software, I was able to feature-match the bits I wanted from WordPress rather quickly, and then iterate on the presentation and interaction without the burden of implementing everything from scratch. Needless to say, I’m excited about the new site (it’s been running parallel to my WordPress blog for several months), and I’m eager to see what happens when I finally flip the switch and start routing traffic to it!

Kevin Sharp (the post’s author, and one of Nokia’s Technical Editors), is looking to help promote Python for S60, but first needs to collect some success stories.

I know there’s a few of you who drop by this blog from time to time that are doing amazing things with PyS60. If you can spare a few minutes to write a response, it would be a great help to the project. Thanks!

Moving from NNW to Google Reader was as easy as expected. It was simply a matter of exporting subscriptions to an OPML file, and uploading that file to Google Reader (under Settings :: Import/Export.)

Using Google Reader is quite pleasant. The big feature they nailed is the manner in which it can automatically flag posts as “read” once you’ve scrolled past them. This alone improves the interaction tremendously over other online readers I’ve tried. Furthermore, when you combine keyboard navigation, it’s easy to crank through news or scan headlines when you have a few minutes to spare. And finally, of course, roaming between machines is a breeze (though I haven’t tried it on the 770 just yet.)

If you haven’t read about the redesign work Google has done on Reader (and you’re interested in RSS/Atom aggregators, or slick AJAX apps in general), check out the Official Google Reader Blog for details.

If you have any suggestions on other WordPress spam filters, please let me know!

Cheers,
Erik

The blogger, Scott Byer, is explaining why Adobe isn’t going to release a MacIntel version of CS2 — the summary being that transitioning their entire development over to XCode wasn’t feasible for CS2. It’s what Adobe’s Mac customers need to hear (that there’s a significant technical hurdle which didn’t make sense for a product currently in the market), but it’s not the kind of thing that a press release can explain. In other words, it’s important information, but without bloggers, there’s no clear channel for how to communicate this stuff.

It’s also of note that the comments list (the conversation) stretches far beyond the length of the original post. The customer feedback is mixed — ranging from “You guys suck, I spent $1000 on this software!”, to “Yeah, I understand the challenge first hand. Thanks for the explanation.”

Since the ad revenue (*laugh*) doesn’t do anything to help operating costs, I’ve thought many times of taking them off; And this latest Adsense update just about made me pull the trigger. From the November Adsense updates comes this feature: “Gain new advertisers directly from your webpages.” The idea is pretty slick — the service allows an advertiser to directly buy AdWords advertising space on sites they’d like exposure on. Such purchases then pay a premium to the site owner. However, even though I allow dynamic adwords, I’m not interested in allowing direct advertising sales on my blog for ads I don’t get to approve. So, here’s the important bit:

“If you’d prefer not to participate in Onsite Advertiser Sign-up, you can opt out before it starts by visiting the My Account tab.”

Yeah… I’m a freak like that. I’m out.

• “See into S60” is a multi-author blog about S60.
• “Tommi’s S60 applications blog” covers Nokia and 3rd party applications for S60.
• “MRKTNGman” comes from an S60 marketing guy.

(Via http://www.blogs.s60.com/)

[Update: Ok. So maybe The Feature was actually Nokia's first dive into the blogging scene.]

Being BlogDay, I thought I’d share some URL’s you might have missed. I decided to pick a theme: Blogs by Nokia employees. There are a lot of photo-bloggers at Nokia (which makes sense), but not too many traditional bloggers. I like to keep an eye on these though:

• Lifeblog — Charlie Schick’s blog about “Mobile Lifestyle” and his work at Nokia.

• MyPhoneRocks.com — Paul Whitaker, blogs about mobile tech and business, along with the application of mobility in his life (which lately means taking lots of pictures of his broken Land Cruiser with his N90 ;-)

• christianlindholm.com — Christian Lindholm is definitely a Nokia employee to listen to. “He is the inventor of the Nokia Navi-key user interface, the father of the Series 60 user interface.”

• anti-mega — Chris Heathcote works in the Insight and Foresight group. I first met him at ETech earlier this year, where he at Matt Jones presented on Tangible Computing.

• Blackbeltjones — Matt Jones does “concept development and researching future mobile device user-experiences”; However, like Chris, Matt’s blog tends to be non-mobile focused.

(Update 2005-09-08: Well, Christian *was* with Nokia. Now he’s joining Yahoo! as VP of Global Mobile Product .)