A little while back I added the code for generating symlinks automatically and although that is much faster, 1 fast file will always be better than 20 fast files. So taking a bit of code from garbetjie and a bunch of mods things are getting fast.
Here is a before and after (or after and before) of what the page speed plug-in for Firefox has to say. Few more tweaks and it will be well into the 90's.
My Firefox is playing up (like always) so checking in Chrome, the page load times are less than 1 second. That is pretty damn good considering my vps is in the USA and I am in Europe.
Here is what the source looks like after the changes, doing just about everything google pagespeed promised. Just need to figure out the images now :)