Beryl stopped working

I am running xfce4 and beryl 0.2.0 SVN on Archlinux (on my T42). Beryl is quite buggy at the best of times. The latest problem I had was berly not starting at all. I seemed to be stuck in xfce4. A quick look at the arch forums and someone suggested making sure the 'Window Decorations' plugin is selected (It's under 'Visual Effects'). It was on my setup, but I tried deselected it and selecting it again, and magically it's all working again. Weird.

Thinkpad 600x 2.6.19 kernel

I thought I'd try the 2.6.19.2 Archlinux kernel on my 600X. There seems to be lots of big changes in the kernel recently. Makes me far more hesitant to upgrade. For the 600X, I had been using apm suspend to RAM for quite some time on it with older kernels. It seemed to 'just work'. It stopped working with 2.6.19. I had to edit the /etc/apmd_proxy to do a pccardctl eject prior to suspend and then a pccardctl insert after it woke up. I have a realtek 8139 pcmcia card in the machine.

I have actually almost had ACPI work on this machine too. So far I can get it to suspend and wake up once. But the 2nd time it never wakes up.

h264 encoding

Having played with divx encoding and tv recording on linux, I thought I'd try out this h.264 encoding. The vague features that I had read about were:

• Smaller file sizes
  
• Better quality
  
• I think the ipod video uses it
  
• Quicktime uses it
  

Having read a good article in Linux journal about using h.264 encoding I set about trying it myself.

Tools
I use Debian Sarge stable (this is my main server ... and I'm happy to keep Debian on it) so chances are I have to compile everything from scratch:

• MPlayer 1.0pre8
  
• faac 1.25
  
• MP4Box (part of gpac 0.4.2)
  
• x264 0.53.583 (I think is was the latest daily snapshot)
  

FAAC was a prick to compile as all the source files seem to have CR.LF at the end of every line (obviously from a Windows box). Make sure you read the Wiki for FAAC and there is a tiny bit of shell code (see here) to simplify stripping out the carriage returns. I ran the configure step for faac as :

'./configure --with-mp4v2'.

I can't remember the specifics of compiling x264 or gpac. You might have to do a bit of trawling to find the answers to get them compiled.

The technique from that Linux Journal article for converting video was the following:

• Dump the audio out as raw PCM audio
  
• Convert the PCM audio to AAC audio using FAAC B
  
• Use mencoder to dump out the source video in YUV 4:2:0 and using a fifo pipe that into x264 to convert the video part of the file to h.264
  
• Use MP4Box to combine the h.264 video and AAC audio into an MP4 container
  

A simple example is:

# dump pcm audio
mencoder -ao pcm -vc null -vo null source.mpg

# convert to AAC
faac --mpeg-version 4 audiodump.wav

# Convert the video to h264
mkfifo /tmp/myfifo
mencoder -vf format=i420 -nosound -ovc raw -of rawvideo \
-ofps 25 -o /tmp/myfifo source.mpg >/dev/null 2>&1 &
x264 -o source.h264 --fps 25 --crf 26 --progress \
/tmp/myfifo 720x576
rm /tmp/myfifo

# Encapsulate in MP4 container
MP4Box -add source.h264 -add audiodump.aac -ftps 25 source.mp4

In my case the source.mpg was a 720x576 PAL MPEG2 capture from a Hauppage PVR card. You'd obviously need to change the resolution settings and fps settings depending on your source material.

That crf setting appears to be very clever. It stands for Constant Rate Factor and it seems good at maintaining a set level of quality in a video regardless of whether there are low motion or high motion scenes throughout. There is a bitrate setting that you can use with x264, but I find that this is ignored when you start using the crf setting. The Linux Journal article suggests using a crf setting of between 18 and 26. With lower values producing better quality but with consequently larged file sizes.

With any kind of video encoding, there are often a bunch of bizarrely named parameters that you can tweak on your way to finding a good compromise between file size and quality. x264 has heaps of them. The example above used a single pass encode, but you can also do two pass or three pass (and possibly more passes) encodes to improve quality. In writing this article, I have only played around with a few of these settings. Some of my observations so far:

• I've somehow been able to convert a 23 minute show (typical half hour show minus commercials) into an 80MB mp4 file. While the quality wasn't marvellous it was a lot better than a divx of that file size.
• You need a lot of CPU power for h264 encoding. I've recently upgraded to a Core 2 Duo CPU ... and you need it ... unless you like waiting. x264 has a '--threads' option which is great for taking advantage of multi-core CPUs. Even with this CPU grunt, x264 tends to plod along at between 17fps and 32fps depending on my settings and the type of source material.
• Quicktime seems to very fickle about what kinds of h264 encoded stuff you throw at it. I rarely use Windows these days, and mplayer and XBMC seem to be happy playing back what I've encoded. One thing I did notice early on was that a 720x576 H264 video won't play back very well on XBMC, giving me the impression that the XBox does not have enough CPU power. Since then I have downscaled the video while I convert it and XBMC is fine (so long as there isn't lots of onscreen motion)

There seems to be a reasonably active community of windows users using these tools... albeit with some GUI frontend to disguise it. Check out the Doom 9 forums for MPEG-4 Encoder GUIs especially the ones that relate to MeGUI. MeGUI seems to have a bunch of 'profiles' which are just XML files that describe the settings to use with x264. I haven't quite worked out how to convert one of these XML profiles to x264 parameters ... as the parameter naming is slightly different but they are definately a good start once you get the hang of it.

Encoding examples

NB: I have no idea how well these will fair if you try to play them back in Windows.... especially using Quicktime. You could try using the Windows ports of mplayer or even VLC as a better option than quicktime.

I also soon discovered that the MP4Box command appends to an mp4 file if it already exists, so make sure the mp4 file is deleted before MP4Box runs. Another useful hint is to t use the '-tmp' option with MP4Box as the default behaviiour of MP4Box is to create rather large temp files un /tmp. If you're running short on disk space, MP4Box will screw up ... and most likely not tell you.

In all the examples, I have previously encoded the sound like:

# You always get a 'your computer is too slow' message. Don't
# worry about it.
mplayer -ao pcm:fast:file=source.wav -vc null -vo null source.mpg
# Create source.aac
faac --mpeg-vers 4 source.wav

Resize 720x576 source to 720x304. Convert in one pass

FIFO=/tmp/myfifo
mkfifo $FIFO
mencoder -vf scale=720:304,format=i420 -nosound -ovc raw -of rawvideo -ofps 25 -o $FIFO source.mpg 2>&1 >/dev/null &
## Found a note somehwere that file output here needs to end in .264 or .h264
x264 -o source.h264 --crf 23 --sar 281:500 --threads 2 --fps 25 \
--bframes 1 --b-rdo --bime --weightb --direct auto --subme 6 \
--trellis 1 --analyse p8x8,b8x8,i4x4,p4x4 --me umh --progress \
--no-psnr $FIFO 720x304

MP4Box -tmp /somedir -add source.h264:par=281:500 -add source.aac -fps 25 source.mp4

The sar/par params above require some explanation. The video source is 720x576 PAL in 4:3 aspect ratio. Since the video is downscaled to 720x304 some aspect ratio info needs to go back into the video so that the player knows how to play it back (I think Quicktime ignores these settings). The calculation looks like:

DAR (Display aspect ratio) = 4/3 = 1.3333

the SAR and PAR = 1.3333 * 304 / 720 = 0.562 = 562/1000 = 281 / 500

So the SAR/PAR = 281:500
ie. Lowest fraction
For a 16x9 source video you could try a SAR/PAR of 3:4

Resize 720x576 source to 512:416. Convert in one pass . Less quality

FIFO=/tmp/myfifo
mkfifo $FIFO
mencoder -vf scale=512:416,format=i420 -nosound -ovc raw -of rawvideo -ofps 25 -o $FIFO source.mpg 2>&1 >/dev/null &
## Found a note somehwere that file output here needs to end in .264 or .h264
x264 -o source.h264 --crf 26 --sar 541:500 --threads 2 --fps 25 \
--bframes 1 --b-rdo --bime --weightb --direct auto --subme 6 \
--trellis 1 --analyse p8x8,b8x8,i4x4,p4x4 --me umh \
--progress --no-psnr $FIFO 512x416

MP4Box -tmp /somedir -add source.h264:par=541:500 -add source.aac -fps 25 source.mp4

NB: The scaling factor here 512x416 was specifically chosen as a size that approximates 4:3 aspect ratio so that it would hopefully work with Quicktime ... but I don't think it plays smoothly on quicktime this way.


Resize 720x576 source to 512:416. Convert in two passes. Low quaility, but small file

FIFO=/tmp/myfifo
mkfifo $FIFO
mencoder -vf scale=512:416,format=i420 -nosound -ovc raw \
-of rawvideo -ofps 25 -o $FIFO source.mpg 2>&1 >/dev/null &
## Found a note somehwere that file output here needs to end in .264 or .h264
x264 --bitrate 300 --crf 26 -o /dev/null --pass 1 --stats test.stats \
--sar 541:500 --threads 2 --fps 25 --bframes 2 --subme 6 --progress \
--analyse p8x8,b8x8,i4x4,p4x4 --no-psnr $FIFO 512x416
# 2nd pass
mencoder -vf scale=512:416,format=i420 -nosound -ovc raw \
-of rawvideo -ofps 25 -o $FIFO source.mpg 2>&1 >/dev/null &
## Found a note somehwere that file output here needs to end in .264 or .h264
x264 --bitrate 300 -o source.h264 --pass 2 --stats test.stats \
--sar 541:500 --threads 2 --fps 25 --bframes 2 --subme 6 \
--progress --analyse p8x8,b8x8,i4x4,p4x4 --no-psnr $FIFO 512x416

MP4Box -add source.h264:par=541:500 -add source.aac.aac \
-fps 25 source.mp4

More Archlinux

Well I've been using Arch on an old Thinkpad 600X and a newer T42 for about a month or so now.

In the past I've used slackware on these systems. I occasionally try other distros ... to see what I'm supposedly missing out on. But I generally keep coming back to slackware. Generally, I like that slackware starts up quickly and doesn't start a bunch of stuff that you don't really want. I also like that it's aimed at experienced Unix users. It's generally been a good setup for compiling programs as well.

However, it's 2007 now and more and more Open Source stuff seems to have a huge array of dependencies ... that make compiling from scratch very time consuming and prone to failure.

Now, there is one other distro I have used for quite some time; Debian. I run it on a server here and use the stable branch (Sarge at the moment). I like debian because apt-get pretty much 'just works' and it's not a 'latest and greatest' type of distro ... which is generally what you want for a stable server. Having said that, none of the new 'cool stuff' for linux is available on Sarge. By this I mean the XGL/AIGLX/Compiz/Beryl stuff.

I wanted to try AIGLX plus Beryl on my Thinkpad T42. Sure, flashy desktops are not really my thing, but I wanted to at least try it out. AIGLX generally means using Xorg 7.1 or newer. Sarge still uses Xfree86 4.3. Slackware 11.0 uses Xorg 6.9. Sure there are notes and packages for putting Xorg 7.1 into Slackware (on the net), but it doesn't look pretty.

So I tried Debian Etch. I actually put this on my Thinkpad 600X (P3 500MHz, 576MB ram) first. I know, I know it'll never run Beryl ... but I just wanted to see how Etch installed and ran before I put it on my T42. The most noticeable thing was how slow it felt. The 600X had been running Slackware 10.2 and 11 for quite sometime and generally felt quite responsive. With Etch it just felt laggy. Some research led me to various posts that Slackware is compiled for i486 and optimised for i686 ... whereas I assume Debian Etch is not. This is all with a custom kernel (ie. compiled by me with specific support for the P3) and a lightweight window manager (icewm)

Archlinux
As per the previous post, I now have this installed on a 600X and a T42. All going reasonably well. Being laptops it often takes an eternity to get ACPI or APM working 'just right', and the fact that Arch seems to upgrade kernels to the latest and greatest all the time was annoying, hence my preference to set 'IgnorePkg = kernel26' in the /etc/pacman.conf file and then compile my own kernel.

Speedwise Arch is as fast or faster than Slackware (in terms of general responsiveness)

AIGLX and beryl
Setting up AIGLX and Beryl was one of the main aims of installing Arch in the first place. Anyway, there are some very nice short and to the point entries in the Archlinux wiki about how to setup aiglx and beryl. I ended up getting it running on top of XFCE4. It wasn't too hard. My graphics chip in the T42 (ATI radeon 7500) is too old for the fglrx driver, so I have to use the open source one in Xorg (which actually works pretty well).

So now I could do my rotating cube mumbo jumbo, Expose effect etc. Now, I'm not too interested in wobbly windows etc, but I am interested in the speed with which I can do stuff in linux. I use linux for work, and in the past I've been annoyed with how (even with lightweight wm's) linux can be slow in switching between apps ... especially when you compare it to windows running on the same machine. Sometimes this is just the lagginess of screen redrawing or the delay of alt-tab'ing. When you have multiple apps open including mutliple xterms, the speed with which you can 'find an app' is quite important. So I was pleasantly surprised that AIGLX + beryl + xfce4 was really really responsive and fast. All that hardware rendering is very fast and you can flip around apps quite quickly. In my opinion you can probably 'navigate' faster than when using a desktop in regular 2D mode. And this is on an older 3D chip.

However, one thing that doesn't work well is multiple monitors. I have a 17 inch 1280x1024 monitor plugged into the VGA port of my T42. In the past I've used the MergedFB2 setup (google for it) to make one very wide desktop which can handle the different screen resolutions together (the T42 just has a 1024x768 screen). That works fine without Beryl. With Beryl the screen is all screwed up. The only way I can get the two monitors to mostly work is to have the 17" monitor running at 1024x768. The closest I got to it all working was to set a 'Virtual 2048 768' in the Screen section of my xorg.conf, but that kind of gives you a mangled 2048 pixel wide screen. Supposedly there is some kind of 2048 pixel wide limit.... but I'm not too sure.

Good things about Arch linux 

• The wiki is good
  
• The startup mechanism is simple, and not much is started by default
• There are enormous numbers of precompiled packages available in the various repositories. Its very easy to build the packages yourself. For example, I wanted to try the Linuxant hsfmodem driver. If I googled for 'archlinux hsfmodem' I see there is a package for it in the Arch AUR repository. Howerver, thats for a proper arch kernel, not my own one. So I just downloaded the tar.gz files for hsfmodem.tar.gz (and eventually hsfmodem-utils.tar.gz), extracted them, then ran 'makepkg' in each directory, and then I had a hsfmodem-utils hsfmodem package compiled for my kernel which I could then install with pacman -U
• The fonts seem reasonable (once you add a few ttf fonts)
• Everything is compiled for i686 (at least a Pentium II)
• You don't have to use Gnome or KDE

Archlinux annoyances

• It is a 'latest and greatest' type of distro. Its meant to be some kind of rolling distro without major releases (eg. My 0.7.2 install magically became a 0.8 setup during a pacman package update). So far in the month or two I've been running Arch, this has led to two annoying bugs creeping in. All these appeared after doing a pacman distro update. First, thunderbird decided that drag n drop should not work, and second, Beryl decided to paint large sections of the screen white. In each case I had to trawl through the Archlinux forums to find other people had the problem and the suggested fix. In each case I had to downgrade a package that had recently been updated. Arch keeps all the packages it installs under /var somewhere so it was simply a case of finding the correct one and running pacman -U against it (for reference, I had to downgrade to gtk2-2.10.6-1 and libgl-dri-6.5.1-3 ). The fix was easy once I found out what to do ... but it was annoying that such bugs could creep in.

Arch Linux notes (Dec 2006)

I am a long term Slackware user, but thought I'd try out this Archlinux distro. Supposedly Arch is a bit like slackware but with a package manager similar to more mainstream distros. Notes below relate to my install on an old Thinkpad 600X (and also a T42)

I downloaded the 0.7.2 base CD (about 150MB iso) and the larger 550MB+ iso
I actually just did a base install in all cases so the bigger CD is probably a
waste.

That base install gives you a 500MB root fs that has a 2.6.16 kernel and
gcc but no Xorg. I mainly used the guide here:

http://wiki.archlinux.org/index.php/Gavins_quickguide_for_dummies_like_me

Key things during the install:

• The Thinkpad 600x required me to do a 'modprobe yenta_socket' and 'modprobe 8139too' to get the lan card recognised (I have some Belkin rtl8139 based lan card). The t42 just required a 'modprobe e1000'


• I didn't create a filesystem in swap


• I just used ext3 for root

I just chose the base bundle and preselected everything in it
/etc/rc.conf changes involved setting hostname, ip address etc, and removing the exclamation in front of gateway

Now that guide suggested doing a pacman -Syu straight after install. That does
a full update of all packages including the kernel. I made sure I added the following
to /etc/pacman.conf:

IgnorePkg = kernel26

That means it won't autoupdate your kernel when you eventually do upgrade (NB: I have since tried to use the latest and greatest 2.6 kernels that Arch provides ... 
but generally things break since I'm using it on two laptops. My preference is to compile my
own kernels and sort out the power management issues).

Once it was up I added xorg

pacman -S xorg

Then added the module for the video card:

tp600x : xf86-video-neomagic
t42: xf86-video-ati

Then ran xorgconfig

Then added twm in and xterm

I could never work out what the xclock package was.

I added icewm in later and set up my .xinitrc to run it.

Some other stuff I've added

alsa-lib alsa-utils
mesa
gconf
xmms
xfce4
ttf-dejavu ttf-bitstream-vera
openssh
mrxvt
opera
mozilla-firefox
mozilla-thunderbird
tightvnc
iptables
mplayer
laptop-mode-tools
libstdc++
skype
remind
portmap (for NFS mounts)
beryl (T42)
beryl-plugins-svn (T42)
wpa_supplicant (T42)

I waited until I was ready to download 100MB and then did the pacman -Syu
It asks some questions about mkinitcpio etc. I think its best to say Yes to these.

Some of the ones above are part of other repositiories (see pacman.conf). I had to enabl
e the unstable one (for beryl) and the community one (skype and remind)

There is a good page on setting up beryl here:

http://wiki.archlinux.org/index.php/Beryl

I tried the 'without a window manager' way but ended up using xfce4 and got it
all going.

NB: the ipw2200 wireless stuff on the t42 requires the firmware files in /lib/firmware

I measured the LCD screen on teh T42 and put in the dimensions (in mm) into xorg.conf
and that improved the fonts on thunderbird

Putting in a different kernel

(NB: This is not the right way to do it in Arch) 
There are some notes about using something called ABS. No idea what that is.
I just made backups of /boot/vmlinuz26 and /boot/initrd26.img, then extracted
linux-2.6.17 under /usr/src. Created the symlink for /usr/src/linux. Copied the .config
in the old source directory (and yep its a cut down source .. the old one that is).

cd /usr/sr/linux (ie. my new one)
make oldconfig and answer questions
make menuconfig
make bzImage
make modules
make modules_install

Now do a mkinitrd kernel_version=2.6.17-ARCH
Copy the bzimage into /boot/vmlinuz26. The new initrd will be called
/boot/initrd-2.6.17-ARCH.img

NB: You can play with /etc/mkinitrd.conf to leave out the SATA and SCSI modules in the
initrd.

I also think mkinitcpio is used from kernel 2.6.18 or 19

Put something like this in /boot/grub/menu.lst

title Arch Linux - New
root (hd0,6)
kernel /boot/vmlinuz26 root=/dev/hda7 ro acpi_sleep=s3_bios video=radeonfb radeon_force
_sleep=1
initrd /boot/initrd-2.6.17-ARCH.img

NB: I also patched the ibm_acpi module to allow level control. I'll need
to patch the radeonfb module as well.

HAL

hal is some weird hw abstraction thingy that looks like its used by lots of
desktops (gnome/kde) for better detection of devices when you plug them
in.

You have to add it to the DAEMONS list in /etc/rc.conf. It should go before 'network':

DAEMONS=(syslog-ng !hotplug !pcmcia hal network netfs crond acpid alsa laptop-mode sshd
paul)