Google Buzz Icon

I found this Google Buzz icon somewhere, but alas, I found it AFTER I'd already made a lame Google Buzz Icon for the apple script app that launches Chrome with a fake useragent to access the mobile version of Buzz.

Perhaps I'll fix up my lame icon to use this new one, otherwise you'll just have to make do with: buzz.icn. I didn't even make a nice mask, sorry.

To get this to show up for your app just replace applet.icns in Contents/Resources under your "Google Buzz.app" directory (You might need to click show bundle contents or something). Make sure you call it applet.icns. Finder is slow about updating icons, so you can quit it from activity monitor if you're impatient.

I used Icon Composer which I guess is part of XCode under /Developer/Applications/Utilities.

more private feeds

I've posted before about my little script to make atom feeds more private by expiring old posts by replacing the text, here and here. I've also posted about weak password protecting your sites. This was all that I did to have a blog that was private from the search engines but easy for people to read either in google reader or via email or in their browser. I did this using blogger.com's sftp support (I also ran a chrooted sftp server). Seems that my paranoia has fueled quite a few posts here in the past, eh! Well Blogger has decided to stop supporting sftp publishing so I've had t find another solution. Unlucky for me, I like the blogger posting UI and I think I found a good solution that will allow me to keep my content out of search engines and still make it accessible to folks that I want to read it. I'm not talking about this blog by the way, I'm talking about my daughter's blog and my personal blog. I'm going to have a blog that is invite only, you need to be specifically invited to read it, otherwise you end up at a rather unhelpful 'you need access' page with no way to contact me to ask for access. To make my blog readable in google reader all I needed to do was get the feed and make it available. I thought of a few ways to do this. First plan was to subscribe to the blog and then get some kind of email to rss gateway worked out. I wish mailman supported this but it doesn't. I did manage to get a patch to do it, but never worked on it because by the time I got the patch I also found my final solution. I wrote a Google AppEngine app that stored oauth credentials and fetched the feed from blogger, trimmed it and published it for all to see. I used code from the gdata python blogger oauth example and salmon protocol and ended up with my own 'Feed App' (Say it with a New Zealand accent) check out the source code at:

http://code.google.com/p/wtwf/source/browse/#svn/trunk/feeds/feedappwtwf

I plan to extend it to implement the expandRss functionality in a server that should be more reliable than my home network connection but for now it does the job and that makes me happy. I encountered a few problems while writing it. The oauth code required you to be logged in, but I wanted some of the urls to work without being logged in. I finally tracked it down and used users.create_login_url('/oauth/request_token') to make sure the user was logged in. The oauth.py code failed silently when the user wasn't logged in (very annoying!). I also found out that when you get the feed via oauth you end up getting all the draft blog posts too! so I had to write in a quick check to filter those out. I guess I really should make a google account that can only read the blogs then that might only get a feed with the real blog posts in it.

Now I'm off to work out how to point all my old blog posts at the new blog that's living in a subdomain and to play with the new fancy blogger templates, whee!

getting google buzz on your desktop

If you haven't yet got Google Buzz in your gmail, or like me you're using Google Apps For Your Domain (GAFYD) and you won't get it for a month or so there is a cunning trick you can use to get it on your desktop right now.

All you need to do is pretend to be a mobile phone, phones going to buzz.google.com see the phone version of the UI and it's serviceable for posting updates and managing friends. There's so much missing but it's a good start. Google Chrome accepts a --user-agent flag and the user agent for the Google Nexus One phone is Mozilla/5.0 (Linux; U; Android 2.1-update1; en-us; Nexus One). That's easy to add under Windows or Linux, on the Mac it's a little harder but the User Data Directory page gives you instructions on how to set up a different user-data-dir which is also a good idea. Here's the string I used.

do shell script "/Applications/Google\\ Chrome.app/Contents/MacOS/Google\\ Chrome --enable-udd-profiles --user-data-dir=/Users/$USER/Library/Application\\ Support/Google/Buzz --user-agent='Mozilla/5.0 (Linux; U; Android 2.1-update1; en-us; (Nexus One)'  > /dev/null 2>&1 &"

One day I'll work out a nice way to get the Buzz Icon in there, likely I'll spend days on it and finish the day before I get full featured buzz in my GAFYD mailbox!

my first google chrome extension

here is fix-style.crx tah dah. It makes the article text in Google Reader and Slashdot larger (and readable). Here is the source.

Since I wrote that I've also ported over some of my other greasemonkey scripts. I still need to polish them up a lot but they're working and that's cool. I'm looking forward to playing with settings pages and auto updates and what not.

I also made a Makefile (my first in about 7 years) to make the .crx file automatically using a python program I found on the intertubes (and modified slightly)

I'm mostly using the dev channel Chrome on Linux and Mac and they just added extensions support, it's really nice and is now my primary browser.

Mailman and Google Apps For Your Domain conflict

the recent addition of x-beenthere mail headers for Google Apps For Your Domain (gafyd)  groups stopped my mailman setup working. I thought I'd share it here in case anyone else is asking about it.

I have mailman running on server.example.com

I have example.com set up as google apps for your domain with gmail

I have a 'group' created to send mail from list@example.com to list@server.example.com

on my server I had list@server.example.com run the mailman binary with post to list whose email address is listed as list@exmaple.com

this was working perfectly however since Sep 17th 2009 gafyd is inserting x-beenthere: list@example.com which /var/lib/mailman/Mailman/Handlers/Approve.py detects as a loop and mailman mysteriously drops the message with a log in

/var/lib/mailman/logs/vette " (3902) Message discarded, msgid: ...."

the solution to me was to move the list to a non gafyd domain :(

iPhone

I finally got an iPhone. I wanted one the day they came out, but I'm allergic to waiting in line. I went for a bike ride instead. I showed up at the AT&T store 90 minutes after it was supposed to open, convinced apple couldn't screw up supplies as bad as they did for the 1st gen and there would be some left and I wouldn't have to wait in line. However this store was in a mall so didn't open at 8AM, they opened at 10AM and it was 9:30 and there was a line of 20 people in front. I decided to bail and after hearing about the activation issues, I'm really glad I did. When there was still a shortage a few days later I went to another AT&T store and put myself down for direct fulfilment, you pay, they ship it to the store. 2 weeks later I had my phone. No waiting in line, well worth it. I was a teeny bit annoyed, they said they'd email the day before and call day of, but I never got the email, then got the call when I didn't have my ID with me and wouldn't be going near that store for 2 more days, but I got it eventually.

I've had it a few days now and overall I can say I'm very happy with it. It's nice to use, the screen is awesome, it's liberating to be able to get to the 'net anytime I want.

However there were quite a few issues I wanted to document in hopes that they help someone else and to highlight that Apple doesn't always get it right and it doesn't always 'just work'.

1. voice mail setup was a huge a pain. When my buddy Mark got his iPhone, I would call him and end up in voice mail, but I couldn't leave a message since he hadn't 'set it up'. WTF. He was previously an AT&T customer but he had to set it up all over again. He finally did and I could leave long ranting messages about how he sucked since he had an iPhone and I didn't. When I got mine I was eager to not make the same mistake. I tried for a long time to find out how the hell to set this up, but couldn't work it out. In the end I asked Mark and the secret is, you need to call your own number from the mobile phone and then it will go into voice mail setup menus.
   
2. Google link tabs infinite redirect. One of the very first things I did was click on Safari and click bookmarks and click on Google. A nice page shows up, I click on some tabs at the top like gmail, or reader and it goes into an infinite redirect and is completely useless. What a horrible first experience using Google on an iPhone.
   
3. Impossible to find gmail/talk/calendar for Google apps for your domain pages. After you get the gmail page to show up logging in doesn't work since it's regular gmail and I want Google apps for your domain mail sign in. It's very difficult to find the right urls for mail/talk/calendar for your domain. Your best bet is to get the Google iPhone application and use the links off that.
   
4. There's no indication on the screen that the ringer is off when you slide the little switch on the side to turn the ringer off an icon flashes on the screen to show you that you turned the ringer off, but pick it up later and there's no indicator.
   
5. closing a safari window is difficult; you need to show all windows then click red-x on the window. Then it always seems to go join a window you didn't want to join.
   
6. I'm still disjointed about exiting applications - must hit home button. It seems only one app is running at any one time, switch between apps and the old app closes down and the new one starts up again. This makes it frustrating to go back to google talk (a web app) since it must reload.
   
7. No way to tell how far into a song you are in iPod
8. No easy way to fast forward in a song; you hold down the next song button but you can't tell how much it's fast forwarding and how much of the song you have left.
   
9. I'm still getting used to typing on the on screen keyboard, but it's much better than a 12 button t9 type text entry, I'm less filled with dread at having to type something.
10. No cell phone networking at first, reboot solved it (hold down the home button)
11. icons for gmail/calendar/reader look good, but switch to https and they don't show anymore.
12. It doesn't work with my car stereo! It seems my old iPod is too old and this new iPod is too new! crap!

Here's some good points I especially like:

1. iPod fades between songs. My old iPod (photo 80G) didn't do this, iTunes did, my shuffle didn't. Now my iPhone does, I like it. keeps the beats flowing.
   
2. iPod rating songs very easy. This is a big use case for me so I'm delighted it's so easy. I like it a lot.
   
3. replaces 3 devices for me (shuffle, iPod, phone). When I ride my bike into work I was having to take three things with me. My shuffle to listen to (quietly) when I rode. My phone of course and my iPod so that I could rate songs while I rode the bus back home at the end of the day. I had even made a special bike water bottle with foam padding to hold all the crap I didn't need in my pocket. Now I just slip my iPhone in my pocket and we're good to go!
4. When you get a call and you're listening to the iPod it fades out and then rings, clicking the headphone microphone thing picked up the call, an awesome experience.

I don't have any favorite apps yet, I guess the Google one helps find the right URLs but I rarely use it now. I'm looking for a good secure crypto notebook type thing, I'm trying a few.

Moving to google code

I'm slowly moving all my code over to being hosted Google Code. This will allow me to learn how to use subversion and also allow me to have RSS Feeds for my checkins. I can also have a wiki page per project and then let this blog be a blog about updates to each of the projects (or more likely new projects, I mean, really, who wants to maintain their old code).

I've already checked in some stuff, some of it hasn't ever been released before! Go check it out at: http://wtwf.googlecode.com/

If you want to follow the feed of stuff I check in the feed url is: http://code.google.com/feeds/p/wtwf/svnchanges/basic

If you use google reader, you can get a preview of what the source code changes feed looks like in google reader.

So far I'm liking it a lot. You can't beat the price!

Perhaps one day I'll even have people submit fixes (yay!), or file bugs (boo!) using the google code interface.

Download Picasa Web Album

I recently needed to download some friend's Picasa Web Albums and did a quick search and didn't find anything readable or that worked. I had planned to use the fancy pants gData API but then saw that the solutions I found just used the RSS feed that each album has, this was much simpler to use.

download: downloadpicasawebalbum

Sadly you'll need ElementTree which is kinda easy to install, but still a pain, it's easy XML parsing though so it's what I use (even if it does screw everything up when you output stuff with namespaces in it).

Here's my use case. We all go to some event, friends start posting their album links, I download all the pictures, add it to iPhoto and then rate and upload my own album of pictures I like that I can then reference from my blogs etc. This way I don't have to worry about people removing pictures and me losing them (from my posts and just losing them in general).

This was the quickest piece of code I've knocked together and put on this site, took less than 2 hours.

Search Results

I added search to this blog. If you're viewing at http://wtwf.com/scripts/ you can see it over there on the right.

Awesome!

feedFixer

feedFixer takes an Atom feed and inserts as many elements as it can to indicate that this feed is private and should not be indexed by search engines. It also modifies old entries in the feed and 'expires' them, replacing the title and the text.

Download: feedFixer

It's design to be set up and run every hour (or minute) and only kicks into action when it notices one of the source feeds is newer than the destination feed.

It's configured by a simple python file ~/.feedFixerrc

# -*- Python -*-
global BASE_DIR, FEEDS

BASE_DIR = os.path.expanduser("/home/blogger")
FEEDS = (FeedFixer("html/atom.fromblogger.xml",
                   "html/atom.xml",
                   url="http://example.com/atom.xml"),
        )

or you could just modify the FEEDS global variable in the source file (but that will make upgrading harder).

I do this because Google Reader keeps posts around forever. If you subscribe to a feed you an then read everything that was ever posted to the feed. Having that much of my data in Google's control didn't make me feel comfortable so I wanted a way to take it back. Tombstoning works for that. It also means that other feed crawlers who might find my site will only get 1 months worth of posts to crawl at any one point and that should be a moving window with old posts replaced with tombstones when they crawl again.

I've written about feeds and search engines before but thought feedFixer deserved it's own page.

Implementation notes:
I started off using ElementTree to navigate the XML, it was so much easier, but their handling of namespaces on output rendered the output useless. I could work out how to name each namespace, but I wanted a default one with no prefix at all and couldn't do it. I moved over to using xml.minidom which worked pretty good and the output was lovely!

I want to change a few more things but I wanted to get this out there first:

TODO(ark) random text replace: Allow you to replace any text in the feed. Blogger feeds are full of the blog's id, and from this you can construct a feed url that gets all this data you're hiding direct from blogger. However doing this would make all the posts appear as new (new id's) so I didn't want to do it. Might be useful though.

TODO(ark) replace link too: when I tombstone a post I leave the link in there, but you might want to replace it with a standard link, or make it empty.

Final touches:
Now that you've expired all the old items in the current feed and verified that once Google Reader's crawler has picked it up, you should see your old items tombstoned in google reader. However if you keep going back you'll see your really old posts are still there. This is because they were too old to be in the current feed. You can make a new feed with all your items in if you're using blogger all you need is your blog's id number. It'll be up there at the top of the page when you start a post to your blog, or you can find it in the feed it looks like this: 10693130

wget 'http://www.blogger.com/feeds/1YOURBLOGNUMBER1/posts/default?start-index=1&max-results=200' -O fullfeed.xml

now fun feedFixer over that file and copy that to the place where your feed is for your blog. Allow a few hours or a day for Google Reader to pick it up and ALL your old posts should now be tombstoned! If you see one or two old posts that you posted but deleted there is a fix for that too: label each post in google reader with a tag (I use 'delete') then go into settings for google reader and under tags make your delete tag public. view the public page and copy the link for the feed for that page. Now do this on the command line:

wget 'PASTE_THE_FEED_URL_FROM_YOUR_LABEL_PAGE' -O - | xmllint --format - | fgrep original-id

Now you have a list of id's for those posts. Go back and make your label page private again. Get your feedFixed full feel and add a few more entries into it (xmllint --format makes this a pleasure to edit) and paste in those original-id's once google reader's feed fetcher picks them up they'll get tombstoned too.

protecting your blog:
In my next post I'll show you how to protect your blog from search engines while still letting your users in with a minimum amount of hassle.

Archving things on the web

I often seem images as I'm browsing around that make me laugh a lot myconfinedspace.com has been an especially good source recently . I'd like to keep them, but it needs to be as easy and seamless as possible. Here's what I do.

If I can I select the image, open up a google notebook and click on 'clip'
now it's in my google notebook.
However notebook used to serve a cached copy of the image, but it no longer does that so I have one more step to make sure I don't lose the image.
I have a cron job that runs every week that backs up my public notebook url and all the images referenced from it. It's easy to do with wget, here's the command line I use:

wget -t 1 -T 15 -N -E -H -k -K -p http://google.com/notebook/public/NOTEBOOK_PATH

That makes a whole bunch of directories for each website and has all the images on the local disk. If an image vanishes off the net it should stay around in the directories and I'll still have it.

After it's loaded you can run my wwwis script over it to fix all the image width and heights too.

picasaweb album size and density

I was getting pretty close to my 1 gig limit in my picasaweb album and I'm too stingy to buy more storage from Google. So I needed to work out which albums to trim down or remove. When I started using picasaweb I would upload the highest quality pictures and later on I went for fastest upload (lower quality/filesize) so I knew I couldn't count on the number of pictures in the album alone. Picasaweb does show you the size of the album, but only on each album's page so that would take a long time to work out. So I fired up the picasaweb gdata api and got some data, it was super easy to do. I used my bookmarks.py script to do the authentication (which I still don't really understand) and then got the XML for all my albums. Then I collected data and sorted them and displayed them in worst offender order.

The only really niggly point was that the XML used namespaces and I was using ElementTree to parse the XML and findall required some weird arguments to find the elements. A Bit of trial and error and skimming the ElementTree docs I found the solution to that problem. Then I polished up my turd and here it is:

albumsize.py

You can run it with just an picasaweb username (with --user) and it will show you all the public albums. provide a google account address (with --email) and it will prompt you for a password and log in and show you all albums.

It will then print out your albums in worst to least offenders
Each line lists the following data:

• album name
• pics: the number of pictures
• size: the size of the album (possibly in KB?)
• density: how many pictures per MB
• size#: position of this album in the albums if they were ordered by size
• density#: position of this album in the albums if they were ordered by density
• target: sum of size and density (results are ordered on this column)

When I was first writing this I thought I could turn this into a nice web service for y'all. I could make my script really simple and have it return the data via JSON and then process it in a nice HTML table that you could sort the columns on. I could even make links to the albums (including auth keys). But then why not just send the XML back and just make my script a proxy for fetching picasaweb api XML. And then I'd have to parse the XML in JavaScript and I kinda already had everything wanted (in lovely Python) so I kinda gave up on that idea as not interesting enough.

Public Google Bookmarks Feeds

I used to use del.icio.us for my bookmarks. There were two things I really liked about it, private bookmarks and public bookmark feeds. There were many things I didn't like about it, it was slow, foxylicious was kinda funky to show my bookmarks in my browser and adding bookmarks was really slow as the browser popped up a whole new window and it was a little too public. Plus I always, always had to pause and think really hard about where to add those damn periods when I was typing in the url. However the public feeds were great, I could subscribe to them in Google Reader and then embed them on my web pages and even do some simple jiggery pokery so they'd look just like regular lists and just like part of my template. That was the "Recent Links" you'd see on the side, go ahead and look, it's back now and that's really the point of this post....

When Google Toolbar started to integrate Google Bookmarks I decided to make the switch. I used Mihai's script to move over all my del.icio.us bookmarks. It was going great, I was easily adding bookmarks by clicking on the star in my toolbar (I really only use toolbar for that now and the very fancy 'UP' button (a feature so trivial it never seems to get mentioned on any google toolbar pages)). However I forgot all about my links feeds for my guests! They were going stale for over a year, eventually I just removed that section from my website but I missed it and wanted it back. Google Bookmarks have rss feeds, but they're protected behind some authentication. I finally worked out how to authenticate to Google Services (without sending my google password over basic auth (sheesh!)), You need to get an SID cookie value and then pass that in with future requests. The GData API provided something that was pretty close to what I wanted, and I finally hacked something together. It runs every hour and gets feeds and html for some queries. I implemented private bookmarks by having a label called _private that I remove from all results.

It also runs as a monitor every day from a crontab to make sure it's still running.

bookmarks.py and .bookmarksrc

Hope you find it useful.

Changing email address

I just changed my email address. The start of a new year seemed as good a time as any to do it. My new address is on a whole new domain which I'll only use for email and chat and calendar. There will be no web presence associated with it. I'm pretty happy with the new setup so thought I would blog about it here.

I've been a happy Gmail user since 2004 and was delighted when they allowed me to send mail as another account. I'd log into Gmail and be able to send @wtwf.com email which was cool. BUT it still had my Gmail address tied into the email headers and outlook would display all this "on behalf of" bullshit which was just plain crappy. I'd still get people replying just to my Gmail address and not to the wtwf.com address too which was a pain and my flaky net connection and power problems where my server was located meant that sometimes mail couldn't get through to me. Also it was annoying that when people wanted to chat with me they should use the Gmail address, but all anyone knew me as was the wtwf.com. Likewise for calendar invites etc... it was all too confusing.

The new mail is on a domain I own, that's privacy protected by a proxy registrar (gotta love that) and it's all handled by Google still. So I get the slick Google Gmail and calendar and talk all for free and if in the future I don't like it I can just move everything elsewhere since I own the domain. I got this all through Google's apps for your domain (GAFYD) anyone can sign up at http://google.com/a/ and if you don't have a domain they'll let you buy one right there and then (using Google checkout).
That part was easy, it's the moving everything over that was a little though. Here's the steps I took. I found it much easier to have Firefox loaded with my new account and then have safari (or any other browser) loaded up with my old account, then I didn't have to keep logging in and out.

• Gmail contacts - easy - just export to Gmail csv and then import in the new account
• Google talk buddies - hard - I just re-requested to talk to all my friends again (very annoying)
• Gmail filters - I just entered them all in again. I did notice that if I copied and pasted from the show filters page in my old account lots of weird zero width characters were added.
  
• old Gmail mail - I've downloaded all my Gmail from my old account via pop into Mozilla Thunderbird. I'm not sure how/if I'm going to upload it all into my new account it's likely I'll lose the dates of my mails. If only Google allowed POP-IN on GAFYD but they only just rolled it out for regular Gmail users. Hopefully that will get turned on soon and I can just do that. Alternatively I could use something like fetchmail to bounce it all.
  
• new Gmail mail - It's easy to set up mail forwarding from your old Gmail account to your new GAFYD one. You can even set up a vacation message to alert people to the address change, I opted to just have a small message in my signature for my new account.
  
• calendars - medium - I just added my new account as an admin on all my existing calendars. The only one I need to work on was *my* calendar which has very little in it, so I just rebuilt it.

I also wanted to make sure that I only had one identity with Google. So that meant creating a 'Google Account' for my new spangly email address and moving all my Google services over to using this new login. I never wanted to log in using my Gmail user name again. Here's the success I had with various Google services..

• blogger - I just added my new account as admin on all my blogs
• search history - I just can't work out how to move my stuff over
• bookmarks - I can't seem to move this over either
• checkout - I just entered everything again and left all my purchase history behind.
• docs & spreadsheets - like blogger, I just added my new account as a collaborator on my docs
• notebook - see blogger and docs & spreadsheets
• reader - this was easy, export all subscriptions as OPML and then import them in the new account. Painless, worked great! I unsubscribed from everything in my old account so that I wouldn't accidentally start reading stuff there.
• webmaster tools - I just added all my sites again
  

So it's not exactly been painless and they could have made it a whole lot easier but I think it's going to be worth it in the long run. I'll have one address that everyone can use for email, invites and the like. It'll be run by Google so should have much better up time than before and I still forward all mail back to my server so I have a copy on my servers (just in case).

Inject gaim logs into gmail

I love the new chat feature in gmail, and I doubly love the fact that my chat history is nicely searchable. For a while now I've wanted to import all my old chats into gmail so I can search over them too. This weekend I sat down and wrote gaim logs mailer. Gaim is an open source multi protocol chat client with god awful almost non-existent insecure password management. It takes your logs and sends them to an email address of your choice. It's driven by a config file in ~/.gaim/logsmailerrc and keeps track of which logs it's mailed so it wont mail them again. You can run this every night if you want. Here's an example logsmailerrc.

I still have three todo items for it:
- make it parse yahoo messenger client logs (which are lamely obfuscated)- make it keep a timestamp/md5sum of when the file was sent so that if more stuff gets added to the file it's resent.
- make it upload adium logs too

There is one problem:
- even though the date is correct in the email, gmail seems to index based on the date received. So you can't do before: or after: searches and find your messages.

If you use this to email your own account, and then fetch that via POP3 it might have better luck with the dates. I also heard if I send it directly to gmail using IMAP and use IMAP APPEND then gmail should preserve the timestamps:

You want to use the optional INTERNALDATE argument in order to set the
"received time" to the time of the message, assuming that's the date
problem you were having. Otherwise, you need a Date: header that is
correct for the "message time".