Grub already has the ability to remember which kernel to run the next time the system is booted. You simply need to use the ``savedefault'' command in the entries you are interested to in ``menu.lst'' and to add a ``default saved'' at the beginning of the same file.
However, let's suppose we just installed a brand new kernel. We could just add a new entry in ``menu.lst'' and set the default to the number of the new kernel.
The next time we reboot we could just hope that everything will be fine and wait for ssh to come back to life.
But, what would it happen if we were able to put a ``savedefault number_of_old_kernel'' in the new kernel entry? During the first reboot, the old kernel would be saved as the default one. Can you see the advantage? I still have troubles... now, let's say you use ``default saved'' instead of ``default new_kernel'', and let's say you run the grub shell from your linux prompt and give a ``savedefault new_kernel''. What about it now? The first time you reboot, the new kernel will be booted (it is the default entry). While the new kernel is booted, ``savedefault number_of_old_kernel'' will ask grub to run the old kernel next time the system is booted. Now if something goes wrong, we could just reboot the system to have the server back to life with the old kernel. This could be enough for most people: in most data centers, you just need to phone some nice guy to have your server rebooted.
However, the linux kernel gives us a nice hand: if we specify something like ``panic=10'' to the kernel boot line, linux will reboot in 10 seconds if a panic shows up, and a panic is something you will probably get if the compiler was sick while compiling the kernel or if you forgot some patch or some important driver.
During the first successful boot, we could even add a script in init.d to tell grub to remember to boot the new kernel instead of the old one, and to remove the lines that provided us some more reliability.