Upgrading fun (well…)

When I selected Habari as a blogging platform, I knew I was in for an occasional rocky ride, with the software at version 0.5 and under very active development. Today I had planned to continue working on building my first plugin but instead my day turned out pretty rocky as once again I had to upgrade to the latest revision to get rid of a problem while a DDoS attack got in the way as well. But instead of growing my own plugin I learned a thing or two, so it still ended up being a good day, even though not all upgrade problems are solved yet.

Blogging tools

Several months ago I'd been looking at blogging tools, hoping to make blogging a little easier. I had (still have) a blog at My Opera, which (sort of) supports the MetaWeblog API and (sort of) other APIs as well. I tried a bunch of blogging tools, but had a host of problems: some just didn't install on Win2K, others installed but either didn't properly support the MetaWeblog or Movable Type API (or not at all), or had a problem with My Opera's somewhat experimental API implementation, or they retrieved only part of my posts, or cost a lot of $$$, or using the tool was in fact even more complicated than just using the native web interface. So I let that idea evaporate.


Then on Monday I somehow stumbled over ScribeFire, which is a blogging tool in the form of a Firefox extension. That sounded attractive, since it wouldn't be platform-dependent (I use both Windows and Mac OS X), let alone have to pay for two tools; instead, I could install it in portable Firefox (both for Windows and OS X) on my Nerd-on-a-stick and always have my blogging tool handy as long as I could find a computer with Windows or Mac OS X on it. (Doing it that way is more secure, too, since there is no need to type in a password: keyboard sniffers don't get a chance.) ScribeFire settings for My OperaAnd, of course, it's open source, with the project now hosted at Google Code! Installation went smooth enough but getting it to work with My Opera took some trial and error: it only works if you set that up manually and remember to end the API endpoint with a slash: without that it doesn't work. But then it actually looked better than any of the other tools I'd gotten to work (more or less). Naturally, the next question was: Does it work with Habari, too?

But Habari uses the Atom publishing protocol, which ScribeFire does not (yet) support (there is an open ticket for it). Luckily I was pointed to a MetaWeblog plugin for Habari, which I quickly installed. But whatever I tried, I could not get ScribeFire and the plugin to talk to each other. I tried poking the code a bit, but got exactly nowhere, and since it uses XML-RPC I had no idea how to even trace what was happening where. Rick Cockrum promised to look at it. Yesterday the "exciting" news was he had no news. This morning there was a message for me:

<JibbyBot> JavaWoman: (from rick_c, Wed, 10 Dec 08 04:34:08 +0000) to svn up. Hopefully her problems with metaweblog should be solved.

So I gave up all plans to work on my own plugin and proceeded to upgrade to the latest and greatest from SVN - fully expecting that would break a few other things before it got better…

DDoS intermezzo

The upgrade would have been smooth, if I hadn't been poking around in some of the code, and had to re-apply those changes. Fair enough, but in the middle of that my connection with the server suddenly went away; SSH kept disconnecting, and I couldn't log into to my control panel either. I filed an urgent ticket with my hoster, who got back to me fairly quickly explaining there had been a DDoS attack on another server — they had no idea how that could have impacted a different VPS (mine), but it had. But it gradually came back, and I could finish the upgrade. (Just while I'm writing this, things have gone extremely flaky again - no way I'll finish this post tonight.)

ScribeFire settings for my Habari blogWhen I then tried ScribeFire again, I found I could actually log in, it appeared in the list of blogs, and it had retrieved my complete list of tags — but none of my posts (it got a "malformed response" instead). Again, Rick had the solution: see if you have an ampersand in one of your posts, he said; Habari uses SimpleXML and that just will not work with a lone ampersand. (Fair enough, it's not valid in XML, but posts might have been converted before feeding them to SimpleXML). Sure enough, I found my very first post contained a lonely & and as soon as I had replaced that with &amp; ScribeFire had no more problems. And I suddenly realized you can use ScribeFire to import posts from one blog into another: load a post, switch to the other blog, and publish! Apart from the fact that I forgot to fix the date on it (back to the original) that worked brilliantly.ScribeFire as a separate window

So for tomorrow I'm planning to install ScribeFire into both my Windows and my Mac version of Portable Firefox on my Nerd-on-a-stick!

Summary again

I had set ScribeFire to publish as a draft by default and that went well. But I wanted to add a summary and tags to my post before publishing it (with the original timestamp). The first thing that was noticeable was that the tag tray had disappeared after the upgrade — fixed easily enough because that's now a plugin. I made a summary with what should be two paragraphs, counting on my latest tweaks to automatically format it for me, but for some reason that didn't work and I started experimenting to figure out why not. One of the things I tried was adding a heading so the formatter would have at least one real block to get its teeth in — which to my amazement completely disappeared again when I saved. When I added it directly in the database it appeared in the form, but when I edited it again and added a comment declaration for testing the formatter, both the heading and the comment disappeared. I also found that any Unicode characters I had in my summary were getting mangled when saved again.

At first I thought that somehow the very first version of the summary stayed put: was it saved only on first publish? But no, I was assured that action_publish_post() should work both for publishing and updating a post. After a lot of experimentation I found that the summary could be changed (I could add paragraph tags) but still the heading disappeared. whitelisted tags in inputfilter Suddenly I remembered inputfilter.php which I'd had to edit for the first installation of the GeSHi plugin. There it was: a lot of whitelisted HTML tags — with h1 through h6 conspicuously missing — everything not whitelisted would simply be removed. Adding the heading tags solved that problem:

  1. private static $whitelist_elements = array(
  2. // mk 20081210 added
  3. // http://www.w3.org/TR/html4/struct/global.html#h-7.5.5
  4. 'h1', 'h2', 'h3', 'h4', 'h5', 'h6',
enabling comments in inputfilterAnother small change would keep comments intact as well.

But I'd solved a symptom. The real problem was — again — that summary content was not treated the same way as post content (headings were not filtered from the actual post, only from the summary). I tried to trace through the code what happened when a post (and a summary) were saved, and when and where they were being treated differently, but I got hopelessly lost. Finally this got me on the right trail:

<JavaWoman> what I cannot find is WHERE it is determined that post entry isn't filtered while summary is
<rellem> JavaWoman: *any* $_POST, $_GET, $_COOKIE, or $_SERVER content is filtered, not just comments... that includes ALL form content
<rellem> JavaWoman: somewhere in the adminhandler it should use $_POST->raw(...) to get it
<JavaWoman> rellem but it isn't! all headings STAY in the post and are FILTERED OUT in the summary
<rellem> JavaWoman: that's exactly what i just said
<JavaWoman> you said it's all filtered

But it was only much later than that little conversation took place that I scrolled back and "adminhandler" jumped out at me and it finally sunk in: I had been looking in a lot of places but not there: I still get lost in this Habari code!

comparing adminhandler versions side-by-sideOnce I started comparing the new version of adminhandler.php side-by-side with my previous one, I found the mechanism: not $_POST->raw(…) but something else in the form_publish() function (see screenshot).

Sure enough, just adding a similar line to the Summary plugin had the desired effect; here's how my heavily tweaked version of Summary now builds its control for the Publish form:

  1. /**
  2. * Add fields to the publish page for Entries.
  3. *
  4. * @param FormUI $form The publish form
  5. * @param Post $post The post being published
  6. */
  7. public function action_form_publish($form, $post)
  8. {
  9. # mod mk 20081128 make the field a (resizable) text area instead of a text box
  10. # content textarea has a height of 330px; we'll make this 100.
  11. # mod mk 20081210 make control content "raw" to prevent filtering
  12. if ($post->content_type == Post::type('entry') ) {
  13. $form->insert( 'tags', 'textarea', 'summary', 'null:unused', _t('Summary') );
  14. $form->summary->template = 'admincontrol_textarea_low'; # added template!!
  15. $form->summary->class[] = 'resizable'; # added mk
  16. $form->summary->class[] = 'low'; # added mk
  17. $form->summary->raw = TRUE; # added mk: no filter!
  18. $form->summary->value = $post->info->summary;
  19. $form->summary->tabindex = 3;
  20. $form->tags->tabindex = 4;
  21. $form->buttons->save->tabindex = 5;
  22. }
  23. }
Lesson learned: It appears that setting a "raw" property on a form control with FormUI will prevent the contents of that control being filtered through inputfilter.php! And this won't be limited to Summary: any plugin adding a control to a form (using FormUI) can in this way determine whether or not its content should be filtered.

But I'm not entirely out of the woods yet: I still don't understand why the Summary content wasn't being properly formatted into paragraphs, or why Unicode characters are getting mangled. Maybe there's a conflict between the filtering and the formatting somehow? But those are problems for another day…

3 Responses to Upgrading fun (well…)

  1. 20 Habari Watch :: office Christmas party bulletin December 16, 2008 4:13pm

    ...Marjolein Katsma has had some fun (sort of) falling out of her hammock and diving head first into the Habari code and introducing herself to those welcoming, friendly developers on ...

  2. 23 No ScribeFire for now… oh, wait! ▪ Coadventures December 26, 2008 9:02pm

    ...In Upgrading fun (well…) I mentioned how glad I was having found...

  3. 3156 Silk Ties December 18, 2009 5:29pm

    Congrats on finishing your bumpy ride :) And you bumpy ride also helped me to solve a problem that i was having so thanks a lot!

Leave a Reply

Some HTML allowed (like a, em, strong, pre). If you want to embed a code fragment, its syntax will be highlighted if you surround it in pseudo-tags like this:
[geshi lang="php"]echo 'highlighted code!';[/geshi] (instead of using pre); specify language in the lang attribute. Do not enclose your code in tags like <?php … ?> as that will make it disappear.