notes from moving to blogger custom domains

I've been moving my blogs over to blogger's custom domains since blogger will be stopping support for ftp sometime soon (now later than they announced). Here's some of the good and the bad experiences I had during the move.

1. No way to customize the 'request access to this private blog page'.

Some of my blogs are private. I used to implement this with some php wrapper in the template that set a cookie based on a really simple question. My main goal was keeping out robots not real people. With custom domains the option I chose was to have a private invite only blog. That's all cool, but the page when you go to the blog not logged in, or logged in without permission provides no way at all to request permission to read the blog, or to contact the blog author. Or any way to customize any of the text on the page to tell people how to get access to the blog.

2. No way to set the favicon

Favicons are important to me. I have too many tabs open in Google Chrome to not  use favicons. The default favicon is the blogger favicon, this is annoying since it's the same as the blogger compose favicon too. I tried adding 'link rel="icon"' higher or lower down in the header and Chrome just ignored it.

In the end I copied some javascript that replaces the favicon. It's not perfect but it works most of the time. Webmaster Central still uses the blogger icon which is annoying.

3. Alternative/Feed information in the header

The Blogger feed page is super confusing. I can put my feedburner url in there but alas it never seems to show up in the source of the served page. Then people will subscribe to the blog via the blogger url which I may want to change later. The subscribe gadget/buttons do this too. plus it also puts links to rss versions of the feed in the header, which I don't have feedburner versions of. It's horrible that so many years after Google bought feedburner it's still not integrated well.

Also I have lots of subscribers who are subscribed to a url on the old site. I can't have the new custom domain replace the old site since I still need these links to work. Don't tell me about the failover site, because that doesn't work when you have a private blog!

In the end I wrote more javascript to strip out all the old feed information and replace it with my feed information. I also use my own subscribe buttons with my own urls in them.

I had to move all my blogs to custom domains which were different than the original blog urls. Now I need to wait for Google to index my public blogs even though the content didn't really change.

4. Those annoying screwdriver/wrench icons to edit gadgets! I turned off quicklinks.

I turned off quicklinks and yet blogger still insists on showing me the quick edit tools for all my gadgets, this is especially annoying on my private blogs where I must be signed in!

I ended up getting rid of these by hand editing the expanded template html a terrifying experience.

5. oauth feeds have drafts

As I documented before I fetch the feeds to my private blogs and make restricted versions of those feeds available semi-publicly. I was surprised to find out that feeds fetched via oauth had the draft blog posts in them too! I should have made a google account with only read access and fetched the blog feed using that (then getting my oauth stored credentials would have only given you access to read the blog, not post to it too!).

6 search gadgets for label or subscribe

The tools to search the layout gadgets are awful. I searched for label or labels and didn't get the main google provided label gadget (or it was hard to find). same with trying to find a subscribe gadget (which ended up not working for me anyway (see my complaint about feeds).

7. Redirecting the Old Blog to the New Blog.

O.K. enough complaining. I've listed how I fixed a few issues. Here's some other fixes I found useful:
for my public blogs it was easy to modify .htaccess files to redirect traffic to the new blog. Here's some rules that worked for me. note that archives used to end with .php but now end with .html It seems blogger keeps the file extensions for old posts so they still have .php at the end.

RedirectMatch 301 /scripts/(2[0-9]*.*) http://blog.wtwf.com/$1
RedirectMatch 301 /scripts/labels/(.*)\.php http://blog.wtwf.com/search/label/$1
RedirectMatch 301 /scripts/archive/(.*)\.php http://blog.wtwf.com/$1.html
RedirectMatch 301 /scripts$ http://blog.wtwf.com/
RedirectMatch 301 /scripts/$ http://blog.wtwf.com/
RedirectMatch 301 /scripts/index\.php$ http://blog.wtwf.com/

For my private blogs I wanted to put up an alert that the blog had moved but give a way to contact me. I did it with the following php that I added to the template so it appeared on every page:

<style>
.alert {                                                                        
  background-color: #f00;                                                       
  color: #fff;                                                                  
  padding: 50px 0px 50px 5%;                                                    
  margin: 50px 0px 50px 5%;                                                     
  border: double 3px #000;                                                      
  font-weight: bold;                                                            
  text-align: center;                                                           
}                                                                               
                                                                                
.alert a, .alert a:hover {                                                      
  color: #fff;                                                                  
}                    
</style>
<div class="alert">
This blog has moved!

The new location of this page is:
<?php echo $newloc ?>

If you are unable to read that page then please Email us using the Email
link on the left.
</div>

I added email addresses using javascript so they're not easily harvestable.

I do like that I can add javascript to my new blogger template by adding a html/javascript gadget, that worked pretty well for me.

Previous and Next links on posts

I use blogger to publish this site. my goal was to have a posting ui that wasn't on the site that somebody else looked after that could publish to my own servers. Blogger does this, but it's still lacking in a lot of features and I end up doing a bunch of hacky stuff with php to get it to work how I want. Case in point: it would be nice to have previous and next links at the top or bottom of every post. but blogger doesn't support that. At least not with the 'old' templates, the only ones available if you want to publish to your own site via sftp.

So here you go. prevnext.py goes around your blogger directory and makes little .prevnext files for every php file it finds that there is a previous or next one for. It' a little complicated by the fact you might post more than one post a day. It also makes a calendar in your archives directory called index.html that you can use CSS to style how you want it displayed.

Here is the css I use.

.code {
  overflow:auto; white-space: pre;
}

.prev {
  float: left;
}
.next {
 float: right;
}
.prevnext {
  clear: both;
}

.archive_calendar {}
.archive_calendar_start {}
.archive_calendar_year {
  color: #B8A80D;
  display: block;
}
.archive_calendar_month {
  padding-left: 3px;
}
.archive_calendar_end {
  display: block;
  margin:0px 0px 14px 0px;
}
.archive_calendar_half_year {
  display: block;
}

Here is how I include the .prevnext files in my templates

  $pn =  str_replace('.php', '.prevnext', $_SERVER['SCRIPT_FILENAME']);
  if (strpos( $_SERVER['SCRIPT_FILENAME'], 'index.php') === False &&
      file_exists($pn)) {
    include($pn);
  }
 }

and I run this from my crontab

1-59/5 * * * * cd /home/blogger/scripts; $HOME/bin/share/prevnext.py -b /scripts

works o.k. every time I try and play with templates and whatnot, I always feel like redoing the whole thing, but then decide it's a huge waste of time, I should concentrate on getting the info out there rather than prettying it up for the 3 people who might actually read this blog, and most of them read it in an rss reader anyway!

Shutup Jaiku

In my last post I bitched about the Jaiku IM gateway and how it was too chatty and all I wanted to do was post to Jaiku but not hear all the feed noise in my chat windows. Well a quick poke around and I found it was close to trivial to write a jabber robot. I used PyGtalkRobot which uses xmmppy (needed to install this from source) and pydns (the one from apt-get was fine for me).

Within an hour I had something working but it needed the Jaiku API key, not your Jaiku password. Some more hacking and I had a code snippet that given a Jaiku username and password would find the Jaiku API key. Since I'd found some useful snippets on snippets.dzone.com I uploaded my own useful snippet.

Meanwhile a helpful person suggested the ping.fm or imified bots but I couldn't work out what they did or didn't have a beta invite so I just kept plugging away at my thing. A little bit more time of cleanup and adding options and some minimal documentation and it was all good to go.

Normally I usually just release this stuff and let you run it. But this time I've decided to run it as a service and see how it goes. However beware! you'll be giving me your Jaiku user name and password. You really shouldn't do that, but you can trust me.

So just add shutupjaiku@wtwf.com as a talk contact and send it the same sign in you would to jaiku@jaiku.com

sign in username password

and then start sending messages and they'll show up on your Jaiku page. That address is ONLY used for chat, do not send it mail no one will read it.
Obviously there's lots of ways this little server could go, right now it does what I need.

I was delighted how easy PyGtalkRobot and the Jaiku API made everything. I was little irked there wasn't a nice way to get a Jaiku API key from a user name and password and also a few things irked me about PyGtalkRobot. Spitting out lots of output without using the logging api. Sending lines longer than 80 chars. And the crazy way you specify handlers, using the doc string to contain the regex that triggers the handler is just bizarre! Why not just register handlers like Python Httpd servers do?

If you'd like to download the source and run your own copy go for it! Hope you find it useful.

Jaiku

I've been playing around with Jaiku, Google's purchased Twitter clone/competitor. I also played with Twitter at the same time (actually revived an account I made ages ago). Twitter sucks, it's way way too slow to even approach usability so I gave up on it. Jaiku is zippy fast which is likely a function of the number of users of each service, but perhaps one scales better than the other.

I've had a need for something that's not quite blogging, but blogging, I try to keep my posts down, so that when they do show up they might be somewhat interesting, But I also have random blabbering, thoughts observations I'd like to note and perhaps aggregate into a post at some point. micro-blogging seemed ideal for this. But I also didn't want a high barrier to entry so Jaiku's IM robot seemed like a good idea. You just IM them a message and it shows up in your jaiku account. Perfect for easy Jaiku-ing. However the robot has an annoying habit of sending you messages about what everyone else is doing. I guess that's the point, but I find it annoying. Especially in gmail where it flashes the title when you have an unread IM, from Jaiku.

Grouping together posting and reading in the same IM robot is like making my blogger post page also be my google reader. They should be two separate activities that I can merge if I like.

Looks like I'm going to have to learn a Jaiku API and how to write my own jabber robot (in python!). Perhaps I'll host in on Google App Engine (I wonder if you can for things like this?).

As an aside, getting the IM set up was a nightmare since my Google Apps domain wasn't exporting SRV DNS records for jabber. The DNS hosting (chosen by Google) was down for editing so I had to deal with their support department (ugh!). Here is the magic command to make sure you have it set up right (replace example.com with your domain name):

dig srv _xmpp-server._tcp.example.com

Password protecting your site

I wanted to loosely password protect my website. I wanted to do this so that search engines wouldn't index the site and so that random people I didn't know wouldn't read it (without some work) but so that it was easy as pie for people I did know to get in and see what they want.

The solution I came up with was a form that asks for a password (there are no user names) and if you enter it correctly sets a cookie on your browser that should last for a year. Every time you visit the site your cookie life is extended. So as long as you come to the site at least once a year it should never ask you for a password again. I made the password ridiculously easy that anyone who knew me would know the answer to. I also made it possible to have a list of acceptable passwords so you could ask for the name of one of my cats and any correct name would get you in.

I didn't want to use basic http auth since there's not enough room to explain why you're asking for a password or to give hints about what might work and it also requires a username which further complicates matters, you can only tell people what to enter after they have failed once, and that's a bad user experience.

The devil's in the details of course, I usually run Cookie Safe which selectively allows me to allow sites to set cookies and I'm very frustrated when a page tries to set cookies, fails, but doesn't tell me. So I try and detect that scenario and report an error if it happens. If you're only allowing session cookies I try and set one of those too, but then you'll need to type a password next time you visit.

I also wanted it to be modular, so I could use the same code from many pages, so I made it a php include that you could use with only a small amount of code in the page you're protecting (your blogger template for example).

Here's how you use it

1. download restrict.php and save it as restrict.php somewhere on your server.

2. Create a page with the form users will enter the password on. A minimal example is included below:

<?php
if (isset($ARK_RESTRICT_ERROR)) {
  print "<h1>$ARK_RESTRICT_ERROR</h1>\n";
}
?>
<p>Access to this site is restricted!</p>

<p>Please enter my favorite color:<br />           
<form method="post" action="<?php echo CurrentPageUrl() ?>">        
<input type="text" name="answer" />                                       
<input type="submit" value="submit"/>                                     
</form>                                                                   

However, remember, this is the page search engines will see, so you might want to include a:

<meta name="ROBOTS" content="NOINDEX,NOFOLLOW" />

at the top and perhaps some better instructions or explaining why you do this. You might want to use the same template you use on the rest of your site?

3. Now for every page you want to protect you need to add this at the very very top of the page:

<php
  $ARK_RESTRICT_ANSWERS = array('red', 'no', 'blue', 'arghhh');
  $ARK_RESTRICT_FORM = '/www/html/form.php';
  include_once('/www/html/restrict.php');
?>

Make sure you add it at the very very top, since it sets some cookies and senders a Location: HTTP header if anything is output before it runs there will be errors.

Note how you provide the paths to the files on the web server machine (do not use urls).

Thats it, should just work now. Hope you find it useful.

Possible improvements:

TODO(ark) add a long error description variabletoo
TODO(ark) try and set cookies using javascript and report an error if there is one before the user even tries a password.

feedFixer

feedFixer takes an Atom feed and inserts as many elements as it can to indicate that this feed is private and should not be indexed by search engines. It also modifies old entries in the feed and 'expires' them, replacing the title and the text.

Download: feedFixer

It's design to be set up and run every hour (or minute) and only kicks into action when it notices one of the source feeds is newer than the destination feed.

It's configured by a simple python file ~/.feedFixerrc

# -*- Python -*-
global BASE_DIR, FEEDS

BASE_DIR = os.path.expanduser("/home/blogger")
FEEDS = (FeedFixer("html/atom.fromblogger.xml",
                   "html/atom.xml",
                   url="http://example.com/atom.xml"),
        )

or you could just modify the FEEDS global variable in the source file (but that will make upgrading harder).

I do this because Google Reader keeps posts around forever. If you subscribe to a feed you an then read everything that was ever posted to the feed. Having that much of my data in Google's control didn't make me feel comfortable so I wanted a way to take it back. Tombstoning works for that. It also means that other feed crawlers who might find my site will only get 1 months worth of posts to crawl at any one point and that should be a moving window with old posts replaced with tombstones when they crawl again.

I've written about feeds and search engines before but thought feedFixer deserved it's own page.

Implementation notes:
I started off using ElementTree to navigate the XML, it was so much easier, but their handling of namespaces on output rendered the output useless. I could work out how to name each namespace, but I wanted a default one with no prefix at all and couldn't do it. I moved over to using xml.minidom which worked pretty good and the output was lovely!

I want to change a few more things but I wanted to get this out there first:

TODO(ark) random text replace: Allow you to replace any text in the feed. Blogger feeds are full of the blog's id, and from this you can construct a feed url that gets all this data you're hiding direct from blogger. However doing this would make all the posts appear as new (new id's) so I didn't want to do it. Might be useful though.

TODO(ark) replace link too: when I tombstone a post I leave the link in there, but you might want to replace it with a standard link, or make it empty.

Final touches:
Now that you've expired all the old items in the current feed and verified that once Google Reader's crawler has picked it up, you should see your old items tombstoned in google reader. However if you keep going back you'll see your really old posts are still there. This is because they were too old to be in the current feed. You can make a new feed with all your items in if you're using blogger all you need is your blog's id number. It'll be up there at the top of the page when you start a post to your blog, or you can find it in the feed it looks like this: 10693130

wget 'http://www.blogger.com/feeds/1YOURBLOGNUMBER1/posts/default?start-index=1&max-results=200' -O fullfeed.xml

now fun feedFixer over that file and copy that to the place where your feed is for your blog. Allow a few hours or a day for Google Reader to pick it up and ALL your old posts should now be tombstoned! If you see one or two old posts that you posted but deleted there is a fix for that too: label each post in google reader with a tag (I use 'delete') then go into settings for google reader and under tags make your delete tag public. view the public page and copy the link for the feed for that page. Now do this on the command line:

wget 'PASTE_THE_FEED_URL_FROM_YOUR_LABEL_PAGE' -O - | xmllint --format - | fgrep original-id

Now you have a list of id's for those posts. Go back and make your label page private again. Get your feedFixed full feel and add a few more entries into it (xmllint --format makes this a pleasure to edit) and paste in those original-id's once google reader's feed fetcher picks them up they'll get tombstoned too.

protecting your blog:
In my next post I'll show you how to protect your blog from search engines while still letting your users in with a minimum amount of hassle.

Fix Blogger Post Greasemonkey Script

Over in my baby blog I've been posting a lot of images and text. Using blogger's include image option is really annoying so what I do is just drag images from picasaweb over into the compose window in blogger, it works great. However I have to do some slight adjustments to make the images how I want them. I want them left aligned with text flowing around them. I also want the text with each image to stay with that image. This involves adding stuff to the style attribute for my img elements and also wrapping each img with a div tag. I could do all this with the style sheets on my page but I also have my blog posts mailed out to a notify list and if I do it with style sheets the emails will look like ass. Previously I was doing this by keeping the snippets of HTML I needed to add in a Google Notebook and just copying them over when I needed to, it worked kinda o.k.

So I wrote a greasemonkey script to give me a new button at the bottom of the posting page that will fix the style of the current post. It adds the styles and divs I want and can also do other stuff. At some point I'm going to make it replace teh with the and wont with won't.

I borrowed some code from userscripts which gave me a new button at the bottom and then wrote the rest myself. It's not pretty but it works.

fixbloggerposts.user.js

Dragging images around the intertubes

Here's a top tip I've been using for a short while now that should make things a lot easier for you.

In GMail and Blogger when you're in the fancy rich text wysiwyg editor you can just drag images from elsewhere on the web and they'll be embedded perfectly for you. Just like that...

I find this ridiculously easy and intuitive but never really tried it until someone told me about it. It makes it trivial to embed thumbnails from Picasaweb into a blog post and allows you to show people what you mean (if you're talking about a product) without needing to attach an image.

Until recently I've had to have two browser windows open so that I can drag the image over, alternatively you can select the text around the image and hit copy and then move to the tab with your composition in it and hit paste and then remove the extra text. In firefox there didn't seem to be a way to copy the image (only the url). However I just found a neat way around that too. Google Notebook. It's a slick firefox extension to allow you to take snippets off websites. You can open notebook on one tab, drag down the image you want and then move to the new tab and drag it into your composition window, easy peasy. Note: you need to start typing in a new note to let it know you're going to enter something, enter one letter and then you can drag images over. You can even make notebooks of things you like to include, perhaps the Yahoo Emoticons? After all everyone knows they're SO much better than the lame Google Talk emoticons. I reformatted the layout to have the ones I like at the top and a nice two columns. Whle writing this I was delighted to find out they added a wave :-h it's been the one I've been missing for a while.

	:-c	call me		8->	daydreaming
	:)]	on the phone		~X(	at wits' end
	:-h	wave		:-t	time out
	:)	happy		:(	sad
	;)	winking		:D	big grin
	;;)	batting eyelashes		>:D<	big hug
	:-/	confused		:x	love struck
	:">	blushing		:P	tongue
	:-*	kiss		=((	broken heart
	:-O	surprise		X(	angry
	:>	smug		B-)	cool
	:-S	worried		#:-S	whew!
	>:)	devil		:((	crying
	:))	laughing		:|	straight face
	/:)	raised eyebrow		=))	rolling on the floor
	O:-)	angel		:-B	nerd
	=;	talk to the hand		I-)	sleepy
	8-|	rolling eyes		L-)	loser
	:-&	sick		:-$	don't tell anyone
	[-(	not talking		:O)	clown
	8-}	silly		<:-P	party
	(:|	yawn		=P~	drooling
	:-?	thinking		#-o	d'oh
	=D>	applause		:-SS	nailbiting
	@-)	hypnotized		:^o	liar
	:-w	waiting		:-<	sigh
	>:P	phbbbbt		<):)	cowboy

Warning: the image you include like this will still be on the server it always was, the owner of that server can then change the image to say "Stop sucking my bandwidth" or suchlike, so if you have a super popular blog, this might be a dickish thing to do.