Comments on: Moving my blog from WordPress to Django; Part 2: Migrating the data

By: erik

erik — Wed, 24 Jun 2009 01:24:47 +0000

Hmm.. I wonder if you might use a SAX parser instead, so that the whole XML document doesn’t have to be loaded into RAM.

By: Sajal

Sajal — Tue, 23 Jun 2009 15:43:20 +0000

Interesting post… Ive been planing for months to port my largest site to Django, but due to my laziness/other projects i havent started work yet.

I played a little earlier, and figured the XML dump route wouldnt work for me. It would eat up allt he ram in the server and freeze mysql. (my wordpress site has > 200,000 posts and > 500,000 tags … its a news site ) .. i tested this method long ago when i wanted to play with drupal..

The option i see is using XML-RPC make a python script which would one by one get posts and save it in Django …

or.. I had to write some own code which generates the XML(or any other dump format) pulling posts one by one.

If a script is running on my server which fetches posts one by one in XML-RPC it would take 30 mins + but not generate any load.

By: erik

erik — Fri, 11 Jul 2008 14:15:29 +0000

Thanks for the link Martey… and the point can’t be overstressed: The WXR that WordPress produces is an invalid mess! A strict parser will fail. If you want to work with it, you need a liberal, non-validating parser (and even still, you might need to touch up potential character encoding issues before you start.)

By: Martey

Martey — Fri, 11 Jul 2008 10:11:17 +0000

I tried doing this, but into serious issues because the WXR file produced was not well-formed XML.

Other people have had similar issues: http://lucumr.pocoo.org/cogitations/2008/02/18/how-not-to-do-xml/

By: erik

erik — Wed, 28 Feb 2007 17:39:23 +0000

Good point Jesse — I had a number of problems with the XML that came out of the plugin as well (and I also found out along the way that my MySQL tables were storing ‘latin-1′ instead of ‘utf8′… fun fun.) I ended up adding some entity encoding/translation to swap various entities and unicode characters into something more ascii-friendly. I hate doing this (I really wish it were easier to run everything as unicode), but that’s just where we’re at with our tools right now I guess.

The following are the characters I swapped:

“–” becomes “–”
“—” becomes “–”
“‘” becomes “‘”
“’” becomes “‘”
“‚” becomes “‘”
““” becomes ‘”‘
“”” becomes ‘”‘
“„” becomes ‘”‘
“…” becomes “…”
“″” becomes ‘”‘
“\u2019″ becomes “‘”
“\u2029″ becomes “?”

By: Jesse Legg

Jesse Legg — Wed, 28 Feb 2007 17:04:37 +0000

One additional note, in case anyone else reads this and has a similar problem.

The export plugin *might* generate some malformed XML, like unescaped or illegal characters. For example, I often use —, an em-dash. ElementTree seemed fine with it, but working with that data subsequently caused python to throw an UnicodeEncodeError. Eg. A simple ‘print results['body']‘ returns the following error:

>>> UnicodeEncodeError: ‘ascii’ codec can’t encode character u’\u2014′ in position 4
>>> 99: ordinal not in range(128)

I’m no XML expert. In fact, I know very little about character sets in general. So perhaps I need to change Python’s character set or something. I didn’t bother. Instead, I manually edited the RSS file and removed the offending characters.

By: erik

erik — Sat, 17 Feb 2007 19:45:39 +0000

Hi Jesse, thanks for the comment. I updated the URL. (And thanks for the tip on your blog about Wagamama coming to Boston. I’m in Boston more often then London lately.)

By: Jesse Legg

Jesse Legg — Sat, 17 Feb 2007 14:23:11 +0000

Awesome post, I'm working on this very same thing at the moment and your insights were helpful. I was having a hard time finding a decent export solution. You might want to include the full link to the WXR export plugin, instead of the Google search. That link being here.