mythtv and mythvideo

A while ago my main hard drive died. Of course I had lots of backups, but nothing that I could boot so I reinstalled the OS. Since I was reinstalling I took the opportunity to install the latest ubuntu Karmic Koala. It all went well and I slowly rebuilt my machine setting things up how I like. First I got mail (postfix) and my own version of Mailman set up. Then it was on to the websites and then I took a break. These last three days I've been battling trying to get all my movies back to a viewable state. I was using XBMC before. I was hampered by some of the magic tricks I had learned in the past to get tv out working on my cheap radeon card. Here's what I learned:

• you need to dpkg --purge gdm to stop any kind of X starting when the machine boots
• you no longer need fglrx - don't even think of trying it, it's a red herring
• you need to add vga=789 to /etc/default/grub:GRUB_CMDLINE_LINUX_DEFAULT to get /dev/fb0
• you need to usermod -a -G audio $USER and then log out an back in to be able to use the soundcard
• aplay -l lists all your soundcards
• lspci and lsusb are really useful too
• ONLY gnome desktop appears to activate the TV out properly, any other window manager just seems to hang startx
•  /usr/bin/xrandr --output S-video --set load_detection 1 --size 800x600 is also needed after the X server has started (10 seconds) to make TV out happen
• xbmc is  slow useless monkey and dead to me
• mythtv and mythvideo are the new hotness and my new best friend
• mythtv and mythvideo just work and are awesome
• you can run mythfrontend over X11 for setting up metadata but not play movies like that
• when getting metadata hit 'w' to get it
• if that fails enter some of the title
• you can easily download coverart from amazon and upload it scp $(\ls -t | head -1)  server:/var/lib/mythtv/coverart/uploaded/
• I have /Movies be only directories and symlinks to the real data files living on lots of mounted volumes.
• If you move a symlink around all your metadata is lost. see below
• I am way too anal about getting all my metadata set up - at least good titles and coverart
• You can access the database directly and mess around with values
• backup your database!
• mysqldump --defaults-file=~/lib/mysql/.my.cnf.mythtv --opt mythconverg  >mtv.$(date +%F).sql
• I must find all my old mame roms for the myth mame plugin to play things!
• it works with lirc but you need a ~/.lircrc and then you symlink to that from ~/.mythtv/
• you can run dvdshrink under wine and it actually works!

So I had most of my metadata set up but my directory structure was very flat. Moving symlinks around would loose my metadata so I wrote a quick program to allow me to move links around and keep the database up to date. Of course this little script may totally bork YOUR setup, so be careful. This is super pre-alpha at this point, I might polish it up later.

Have a look at mythvideomv.py you probably want to click on 'raw file' to get the actual file.

battling fglrx

There were new dns exploits around and when I ran apt-get dist-upgrade it had some bind and ssl packages in the list so I thought it would be important to do the upgrade. It also upgraded my kernel which I think is where the problems started.

Then my xbmc started being really slow. Like 1 frame a second slow. Of course vlc was just fine. fglrxinfo reported the Mesa drivers which meant I wasn't getting hardware video acceleration like I should be. What followed was a long list of frustration and I'm going to document what's working in the hope that when this happens again I'll be able to get it working quicker. Let it be known. I barely understand what half of these commands do but in the end I got it working.

$dpkg -l '*fglrx*'
# then remove all of them
$rmmod fglrx intel_agp agpgart 
$depmod -a
$ls -l  /usr/lib/dri/fglrx_dri.so 
# remove it if it's there
$apt-get install build-essential cdbs fakeroot dh-make debhelper debconf libstdc++5 dkms linux-headers-$(uname -r) 
$apt-get install xorg-driver-fglrx-envy
$Xorg
$glxinfo | fgrep -i dir
direct rendering: Yes
$fglrxinfo 
display: :0.0  screen: 0
OpenGL vendor string: ATI Technologies Inc.
OpenGL renderer string: Radeon X300/X550/X1050 Series
OpenGL version string: 2.1.7659 Release
$glxgears 
7921 frames in 5.0 seconds = 1584.096 FPS
7855 frames in 5.0 seconds = 1570.389 FPS
7796 frames in 5.0 seconds = 1559.176 FPS
$fgl_glxgears 
Using GLX_SGIX_pbuffer
1871 frames in 5.0 seconds = 374.200 FPS
1971 frames in 5.0 seconds = 394.200 FPS
2018 frames in 5.0 seconds = 403.600 FPS
2005 frames in 5.0 seconds = 401.000 FPS

I also found out that my mceusb2 lirc remote receiver can look for more than one remote at a time. I found the TIVO_Series_1 worked great and now I can move on to stage 2 of my control. I plan to have a small python script looking at the output of irw and when it detects the xbmc remote it will switch the receiver on and change inputs on the video switch. When it sees the tivo remote it'll switch everything back. I need to work out the secret url's to hit to control the destiny networks d3k (which controls the receiver and video switch) so I'll have to fire up a windows machine to do that (it's internet explorer only). If I get the lirc dongle to emit IR then I can avoid much of the d3k which will suit me just fine.

I also got spdif out working so now I have dolby surround sound in the movies I watch. I was expecting trouble, but it really was as easy as connecting the cable and changing audio output to be digital. Awesome! We're really close to a pretty good working system. After all that, Mame is next on the list!

Filling up a 1.5 TB hard drive

My 1.5TB drive arrived and I threw it into the server. Then I had to fill it up. I have two external 750 gig drives filled with the raw video_ts files from all my DVD's that I ripped myself. It took under about 18 hours to copy all the data over, with a gap in the middle while I was sleeping and then switched over the drives. I'm pretty happy with rsync but wished I'd used the -h flag to report the sizes in a human readable form. Since I'm running some system status software I have a nice graph of the hard drive filling up:


Things are going pretty well with the xbmc setup. I got my Mediagate GP-IR02BK remote and that appeared to work first time with lirc and when I restarted xbmc it was controlled by the remote! One problem was that the DVD Menu button didn't appear to show the DVD Menu which is kind of critical.

I've worked around it for now by adding this to ~/.xbmc/userdata/Lircmap.xml

<hash>DVD</hash>

and also adding this ~/.xbmc/userdata/Keymap.xml

<keymap>
<FullScreenVideo>
 <remote>
    <hash>ShowVideoMenu</hash>
 </remote>
</FullScreenVideo>
</keymap>

Notice how that's Keymap.xml not keymap.xml the xbmc documentation is obviously written by windows developers who don't care about the Case of FiLeNaMEs. I'm also irked by the camel casing in entity names in their xml too and their non-consistent use of different cases for file names (like Keymap.xml and advancedsettings.xml)

I also found that the log output to xbmc is in /var/tmp/$LOGNAME-xbmc.log

Since I was changing config files a lot I ran xbmc like this:

while true; do echo "Starting XBMC"; xbmc; sleep 1; done

This allowed me to hit control-C in the window it's launched from and it would restart.
I couldn't find a way to get it to restart and to also tail -f the log file (since it starts a new log file every time it starts). I guess I could pkill tail but that seemed a bit drastic.

Other things I learned and wouldn't mind remembering: mkfs -t ext3 /dev/sdb1
and to remove a file that starts with a - sign: rm -r ./-Funky\ Filename

TV Out working with a cheap ATI X1050 card

I got TV out working on my cheap $20 ATI X1050 PCIE graphics card under Linux. It worked first time after following these instructions. I had to modify it slightly so there was only one display (the TV). I even got it to work with x11vnc which meant I could do most of my setup remotely and not have to have the TV on.

However after following some XBMC instructions I noticed I was running the Mesa GL library and not the ATI hardware one.

DISPLAY=:0.0 glxinfo | fgrep direct

was reporting direct rendering: No

DISPLAY=:0.0 fglrxinfo

was showing Mesa.

Even though I was getting pretty impressive frame rate with Mesa I upgraded to the downloadable (from ATI) driver and that screwed everything up. Even though it was now reporting direct rendering the performance was sucky. I also noticed that as soon as I killed the first X Client, X windows would exit too. not too sure what that's about. I uninstalled the downloaded drivers and went back to the gflrx drivers from ubuntu and that gave me direct rendering and pretty good frame rates. Sadly x11vnc no longer works, even though I downloaded the most recent version and compiled it myself. It keeps getting ShMem errors.

With downloaded ATI drivers
# fgl_glxgears
Using GLX_SGIX_pbuffer
1423 frames in 5.0 seconds = 284.600 FPS
1780 frames in 5.0 seconds = 356.000 FPS
1741 frames in 5.0 seconds = 348.200 FPS
# glxgears
1837 frames in 5.0 seconds = 366.721 FPS
1927 frames in 5.0 seconds = 385.396 FPS
1904 frames in 5.0 seconds = 380.799 FPS

With ubuntu fglrx drivers
# fgl_glxgears
Using GLX_SGIX_pbuffer
2568 frames in 5.0 seconds = 513.600 FPS
3032 frames in 5.0 seconds = 606.400 FPS
3009 frames in 5.0 seconds = 601.800 FPS
# glxgears
12657 frames in 5.0 seconds = 2531.301 FPS
12632 frames in 5.0 seconds = 2526.393 FPS
12531 frames in 5.0 seconds = 2506.009 FPS

Once I had video set up Audio was a doddle, installed alsa and alsamixer and alsaplayer and immediately sound came out! I still need to spend some time to get digital optical audio out, but I'll leave that for some other day.

XBMC has screen resolution and calibration tools built in. I also used [aticonfig --tv-info] and [aticonfig --tv-geometry=40x90+0+2] before I found the calibration menu and got my screen size set up.

Controlling XBMC is still a problem. I'm using the web ui (turn on in settings), synergy from a laptop, the iPhone web ui is too limited (Couldn't find any transport controls), But the $3 iPhone App seems workable for now. I had hoped to be able to use my really old X10 MouseRemote but it appears to have died. I tried it with two different serial ports and neither got any data out of it. The little LED didn't flash when I pressed buttons. So I looked up some remotes online and ended up ordering a Mediagate MCE Remote which should work just fine. I'll keep you posted on that.

Lastly I fear that my current home automation controller. A Destiny Networks D3K is close to it's end of life. The company's gone bust so I don't get any updates anymore. The control software only runs under Windows and my last machine that had it installed died. I'm trying to muddle along with what I have set up right now, so to get the HTPC/XBMC you need to press VCR and there's no way to get to VCR now (unless you go in and press buttons (the horror!).
Perhaps with this new remote I'll be able to control some stuff. Seems relays and IR emitters are expensive (at least from Global Cache, so I might have to work in some arduino boards or something. That's a long time in the future though.

Sooner than that I'm going to get a 1.5TB drive (or two) and put more of my movies on it. And I need to work out how to Rip some of these Disney DVD's...

zoneminder, bt878 based video capture card and ubuntu hardy

I use zoneminder for my security cameras and with the new server (and new OS) I had to set up zoneminder again. I have a bt878 based card which worked great in the old setup.

Here's some tips to get things going:

xawtv -hwscan

Should be able to find your hardware, if not you're not going anywhere.

xawtv -device /dev/video0

or whatever hwscan told you, should show images, if not you need to look into that.

Once you have that all working the secret sauce is to add the user that runs your apache cgi scripts to the video group, so they can access /dev/video0

usermod  -a -G video www-data

now zoneminder should show your cameras.

My cameras are 320x240 NTSC

Other useful stuff:

lsmod | fgrep bt

bt878                  11992  0
bttv                  176180  2 bt878
ir_common              36100  1 bttv
compat_ioctl32          2304  1 bttv
i2c_algo_bit            7300  1 bttv
videobuf_dma_sg        15108  1 bttv
videobuf_core          18820  2 bttv,videobuf_dma_sg
btcx_risc               5896  1 bttv
tveeprom               16528  1 bttv
i2c_core               24832  4 bttv,i2c_algo_bit,i2c_piix4,tveeprom
videodev               29312  2 bttv
v4l2_common            18304  2 bttv,videodev
v4l1_compat            15492  2 bttv,videodev

My card was $20 from geeks.com, It has 4 inputs and I'm really happy with it.

another new server

The old new server didn't work out at all. It had a weird glitch that when the memory usage went up to 100% and it started to want to try and use the swap the machine would lock up. Sometimes I could preempt this with rebooting, but other times it was stuck until I got home to hard power cycle it. Luckily I got it from someone at work and after we had tried different memory modules he took it back. It was lucky really since I'd been looking around and really wanted something a little better and was more into the idea of actually having a Home Theater PC (HTPC) in the rack. Especially since my daughter is starting to want to watch movies with us and I don't want to mess around with opening DVD's and stuff.

I did a little research and with some help from the egg heads at work came up with this:

• PHENOM(9XXX)QUAD,AM2+/AM2,780G,4DDR2,ATX
• Athlon 64 X2 Dc 5200+ AM2 2.7GHZ 1MB 65NM 65W 2000MHZ Pib
• CORSAIR XMS2 4GB ( 2 X 2GB ) PC2-6400 800MHz 240-pin DDR2 CL5 Dual Channel Desktop Memory Kit - TWIN2X4096-6400C5
• Coolmax 14616 V-400 ATX Power Supply

Pretty sweet for under $200! When it arrived I started pulling apart my old 4u case to put this new stuff in and while I was halfway through, I thought, you know... I really should just smoke test this new stuff before installing it. I dismantled more stuff while that thought was still nagging in my head and finally stopped and set it up on the bench. Sure enough it didn't work! I tried lots of things and couldn't get it to work. Filed a support ticket with the board manufacturer and put my old server back together. The next day I bought a new motherboard and power supply from Frys to test some theories. When I got them home they too didn't work and I finally hit jackpot when I tried each of the memory modules separately. One was bad! I ordered a whole new set and will mail one set back once I find two that work and then took the other stuff back to Frys.

Now my server was working!

This motherboard had so many awesome ports I was delighted, especially with the optical audio out, external SATA and the hdmi connector. I was sad it didn't have svideo and didn't have a com port either (I have an old X10 remote that has a serial port and also a weather station I plan to hook up at some point). However graphics cards with svideo are available for $20 (ATI X1050) and you can get usb to serial port adapters for under $10 so that's not a big deal. Once I read the manual I found out there were lots of headers for more ports so I got another 4 USB, 1 firewire and more audio ports for a $15 floppy disk sized header and ordered the official com port header (I thought about making my own, but it was just too much trouble).

Another thing that pleased me was that I could adjust the bios settings over the DVI port, which means I can mess with my system in the den, rather than dragging it into the bedroom where the only VGA display in the house is. This allowed me to find the second easter egg. Even though every bios ever made has allowed you to set what happens when the power comes back on, my old motherboard didn't. I was delighted I could make my server come back on automatically after a power cut with the new board. Yipee. Of course once I put the X1050 graphics card in there I couldn't access the bios at all on either DVI port, but at least once the system booted I could see the Linux console on the X1050 DVI port. Good signs I'll be able to get svideo working at some point.

So now I'm setting up the new machine and I might make one or two posts about that as time goes on. I will leave you with this quick (and dirty) performance number. The old server reported just under 1,800 BogoMips and the new system with 2 core's clocks in at 10,800 (that's 5,400 per core). Now to find something to do with all that power.

a new server

I'm moving to a new server at home sometime soon, so if things go screwy, that's why. I dropping the huge 4U rack mounted beast I've posted about before. In its place is going to be a teeny shuttle case with one PCI slot which is really all I need for my security cam capture cards. I'm going through the stages of setting up ubuntu 8 (hardy) on it right now. I got the PC for $120 which I thought was a deal. I ordered a 1TB SATA drive ($100!) to throw in there and it'll just have one drive for now. It has room for another drive, so at some point I might add another 1.5TB or so. If I need it.

Installing ubuntu was a doddle, I installed the server version since that has the longest support lifetime. To get it usable I'm listing the packages I installed incase I need to do it again:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install gcc python-dns binutils-doc autoconf automake1.9 bison flex gcc-doc gcc-multilib gdb libtool make manpages-dev gcc-4.2-doc gcc-4.2-locales gcc-4.2-multilib libgcc1-dbg libgomp1-dbg libmudflap0-4.2-dev libmudflap0-dbg libc6-dev libc-dev
apt-get install gcc python-dns binutils-doc autoconf automake1.9 bison flex gcc-doc gcc-multilib gdb libtool make manpages-dev gcc-4.2-doc gcc-4.2-locales gcc-4.2-multilib libgcc1-dbg libgomp1-dbg libmudflap0-4.2-dev libmudflap0-dbg
sudo apt-get install mailman
sudo apt-get install smokeping curl libauthen-radius-perl libio-socket-ssl-perl libnet-dns-perl libnet-ldap-perl libnet-telnet-perl libsocket6-perl libio-socket-inet6-perl echoping
sudo apt-get install apache2 ssh libapache2-mod-python php5 libapache2-mod-perl2 bind9 mailman screen tkdiff xauth mailx screen cvs python-cheetah python-mysqldb pychecker x11-apps resolvconf libapache2-mod-python-doc python-egenix-mxdatetime python-mysqldb-dbg tclreadline mesa-utils pdksh libbsd-resource-perl xterm

spec's are: Celeron 420, 1 GB of DDR2 RAM, Shuttle KPC K45 SFF

Anonymous mailman lists

I have a need for an anonymous mailman list and here is how I set it up.

1. create the list (for me I had to run /usr/lib/mailman/bin/newlist the web ui didn't work for me). In this example I'm going to use the name "privatelist".
   
2. In the web ui, on the general options page, under "Hide the sender of a message, replacing it with the list address (Removes From, Sender and Reply-To fields)" select Yes
3. In the web ui under "Archiving Options" just say NO to "Archive Messages?
4. In the web ui under "Digest options" select NO to "Can list members choose to receive list traffic bunched in digests?"
5. modify /usr/lib/mailman/Mailman/Handlers/Cleanse.py to add the following underneath if mlist.anonymous_list:

   
   del msg['received-spf']
   del msg['authentication-results']
   del msg['domainkey-signature']
   del msg['dkim-signature']

6. rm Cleanse.py[oc]
7. python -O -c 'import py_compile; py_compile.compile("Cleanse.py")'
8. python -c 'import py_compile; py_compile.compile("Cleanse.py")'
   
9. cd /var/lib/mailman/lists/privatelist; rm digest.mbox; ln -s /dev/null digest.mbox
10. remove /var/lib/mailman/archices/*/privatelist
11. replace /var/lib/mailman/archives/private/privatelist.mbox/privatelist.mbox with a link to /dev/null
12. restart mailman
    

That's what I did, post to your list with a different email address as a member and then view the message in your recipient's account. Make sure to view all headers and see if there is anything in there that might indicate the original sender. There may be something I missed, if so let me know!

There go my nines

I'm pretty sure no one noticed, but my server was down for quite some time this weekend...

Short Version:

Before	After
Linux Server Pentium III 600Mgz 256Meg Ram Beige Dell Box on a shelf Windows Server AMD Athlon 1Ghz 512 Meg ram Sweet 4U rack mount case Running Security Cam Software	Linux Server AMD Athlon 1Ghz 512 Meg ram Sweet 4U rack mount case Running webserver, email, Security cam Software, stuff

Long Version:
But I think it's back up for a while now and it's better than ever!
Someone at work was selling an old Motherboard, processor, 0.5Gig memory and a fancy graphics card for cheap, so I made a low offer, told them I hoped they got a more reasonable offer and then ended up 'winning' it. It was caked in some horrible brown gunk which I suspect was due to someone smoking in the house. I powered down my server (a really old dell Pentium 3 600Mhz), threw in the new motherboard and powered it up. I had heard about dell power supplies being wired weird but thought my system predated that silliness. It turns out it didn't. So now I needed a new case and power supply (of course a standard power supply wouldn't fit in a dell case!). I figured I should finally do it right and get a rack mount so the computer could actually fit in the 17" rack I have in my AV closet (where my current server just sits on a shelf like a dork). Boy those things are expensive I was looking at paying over $150 for a not particularly nice one that was likely going to be a total hassle (getting cards to fit in it) since I didn't have room for another 4U case in the rack. You see I already have a 4U server in the rack, it was my old home theater PC and has since been re purposed to run the security software than records my security cameras. This machine had been down for a little while ever since I needed a windows PC to fix the dead disk in my TiVo. A TiVo backup that was too old to update itself caused me to have to buy instant cake which I begrudgingly did, but it ended up working so well I should have done that right from the start, I did feel like a failed TiVo hacker for not being able to do it myself though. But since the machine was down and out, I plonked the new board on top, hooked up the ATX power supply and booted it up and sure enough it worked fine. Even better the fancy graphics card sent signal to the DVI connector from the start s I could set bios settings with it connected to my apple display (DVI only) in the den. Before that I'd have to lug my machine into the bedroom, balance it on a wooden stand and connect it to the LCD TV in there (the only display in the house to accept VGA input!). I emailed who I bought the motherboard from and told them it worked fine. I decided to setup my new server in parallel with my old one (minimum downtime!) and installed ubuntu on a 200 gig drive I had, I couldn't get the damn thing to boot, I tried everything, other drives, swapping master/slave, all sorts of bios jiggery pokery, nothing worked, also the bios power on cycle took a really long time which made it more frustrating. In the end I gave up, emailed the seller with the bad news and fortunately they agreed to take it back. I still had the idea of putting my linux server in a rack case, and it always bothered me that I ran a windows server at home, The only reason I ran it was because I had convinced myself no Linux drivers could possibly exist for my 4 input video capture card. A quick Google search and I was proved wrong, quickly negating the need for a windows server, I started moving the drives over to my 4U case and they kept having disk errors so I reinstalled Ubuntu on a spare 200 gig drive and got ssh working and then started copying over all the files I'd need to get my server back up and running. Finally it's all working great! I like the new security software since it's free and open source and I can tweak it if I need to. I really like that I have one less sever and a windows one at that, so my AV Closet is much tidier now!

Everything feels right about this.

Here's the Ubuntu packages I had to install to make it usable for me:

apache2 ssh libapache2-mod-python2.4 php5 libapache2-mod-perl2 bind9 mailman screen tkdiff xauth mailx screen xlogo cvs python-cheetah python-mysqldb pychecker

TODO(ark) find xv somewhere

Here's the configuration files I had to copy over, I really should make sure these are backed up:

• /etc/apt/sources.list
• /etc/apache2
• /etc/bind
• /etc/postfix
• /etc/init.d/postfix
• /etc/mailman
• /var/lib/mailman
• /usr/lib/mailman