products | company | support | training | contact us
|
mail us
|
mail this page products | company | support | training | contact us |
This is a survival guide to keeping FreeBSD up-to-date. Including upgrading FreeBSD (in this case from 5.3 to 6.2 but the principles are similar with most releases), keeping it up-to-date using the binary freebsd-update utility, maintaining the ports collection and using rsync for backup. While the FreeBSD Handbook is generally a terrific resource - always check it first - it can either make assumptions about knowlege or it keeps things as simple as possible to minimize errors.
As with all things FreeBSD there are about 100,000 million ways of doing things which can leave folk confused when they read three different HOWTOs all describing different ways to accomplish the same goal. In many cases it's just simple habit. The author of the HOWTO has always done it that way. There is no attempt to analyze the reasons - it just works. But one is left with that vaguely uncomfortable feeling that if anything goes wrong the only solution will be a panicky google search and a lot of hope.
FreeBSD provides many way of doing everything. Here are your choices for upgrade.
Our upgrade probably represents pretty much a worst case scenario since we have on our target system:
To summarize the upgrade:
uname -a
Source upgrade via cvsup (/usr/ports/net/cvsup-without-gui) or csup (/usr/ports/net/csup).
Non-standard locations for the source and object trees. By default the source is assumed to be in /usr/src and object files plus build output are placed in /usr/obj. We have to place them in /var/src and /var/obj because of space constraints in a poor installation done many years ago and which we have never found the time to change.
Note: If you are using standard source/object paths we note the normal commands as well.
The handbook says it, every HOWTO in the world says it so we're gonna say it too. Back up the system before you start. See rsync and backup. As a minimum you should back up /etc and if it's a web server /usr/local/www (or wherever you keep your webs) and then anything else that is unique to this system (check-list).
Most folks get kinda nervous when it comes time for an upgrade and postpone the fateful day as long as possible. This is a big mistake since the FreeBSD team have done a great job in making it relatively simple and painless. This section is for those who want to know when they are in danger and when they are not!
Step 1 Sync source. Totally painless using cvsup or csup. No impact on the rest of the system other than using about ~20m of disk space. Production work can run concurrently. do it as often as you want.
Step 2 make buildworld. Totally painless but can take a long time. Compiles and builds the new infrastructure, libraries, utilities, etc. in the /obj directories but installs NOTHING. You can stop at this point for as long as you want and continue to run production systems. Generally anything with the word build in front, like buildworld is painless since it only builds and does NOT install anything. If there is no build or it has the word install then it does, surprisingly, install stuff.
Step 3 make kernel. Reasonably painless. Compiles, builds and installs a new kernel - but copies the old one to /boot/kernel.old first. If your system crashes after this step is complete the reboot will load the new kernel by default. You can force it to reload the old one (the process is described in the section). Otherwise you can continue production and stay in this state as long as you want.
Step 4 boot to single user mode. Not risky. The new kernel may not work but since recovery is trivial and nothing has changed there is almost no risk. At this point you are running a new kernel while everything else (utilities etc.) are the old versions. You are in single user mode and cannot run any production tasks. Minimize use of commands to the essentials. Typically top, ps and various other system commands either don't work or give strange results.
Step 5 mergemaster and installworld. Risky. At this point the processes will update/merge configuration files in /etc and install the upgraded utilities and libraries. Crashes here can vary from slightly to extremely painful.
Step 6 boot to normal mode. Happiness and joy. Normally. Depending on the nature of the upgrade you should be back running production systems but which may not be optimized for the new version so continue to the next step and soon as practical.
Step 7 fix ports. Updating the ports is a pretty solid test of the system so if it works you are home free and should have all your snazzy new, optimized applications working.
Step Back If it all goes horribly wrong. Some, non-exhaustive, notes about recovery.
This step involves updating the source to the version you want to install using the cvsup utility (or the newer csup utility). In our case we will use the RELEASE version of FreeBSD 6.2 (see notes. If you do not have cvsup installed (/usr/ports/net/cvsup-without-gui) then it may be worth installing csup (usr/ports/net/csup) instead which takes much less time to install and run.
By default, the build process assumes /usr/src for the source files and places compiled files and applications in /usr/obj. We will be using non-standard paths so need we to do more work. If you are using the standard paths ignore commands marked with NON_STD_PATH_ONLY comment below. Edit /usr/share/examples/cvsup/stable-supfile with you favorite editor (don't they all suck) as follows:
# shows only the lines that need to be changed # leave all others unmodified *default host=cvsupxx.FreeBSD.org # where cvsupxx is the required host or mirror site # from this list # NON_STD_PATH_ONLY *default prefix=/var # if you are using standard paths leave this parameter untouched # normally *default prefix=/usr *default release=cvs tag=RELENG_6 # this updates to 6.x-STABLE # binary updates NOT suported # OR use *default release=cvs tag=RELENG_6_2 # to update to 6.2-RELEASE-px (see important notes below) # binary updates are supported # run cvsup cvsup -g -L 2 /usr/share/examples/cvsup/stable-supfile # OR csup -L 2 /usr/share/examples/cvsup/stable-supfile
The full list of CVS BRANCH TAGS is available. The documentation mentions -CURRENT and -RELEASE but rarely mentions -STABLE in relation to cvs tags. There are, in fact, three major tag types:
| -CURRENT | Pure development branch - for hairy chested developers, or if you are really in need of some new feature and are prepared to take some risks. CVS tag name is HEAD. Binary updates (freebsd-update) are NOT supported for this tag. |
| -STABLE | Development branch for current release and is used as the base from which specific releases are generated. Typically contains the latest security updates at the point at which you down loaded but it is changing all the time. Some level of risk is involved, offset against being udated for the lastest security fixes, and significantly lower risk than using -CURRENT. CVS tag name is typically RELENG_X (currently RELENG_6) and has a major (X), but no minor, version number. Binary updates (freebsd-update) are NOT supported for this tag. If you use this tag you must use source updating for all subsequent security fixes. |
| -RELEASE | A specific, frozen-in-time, release of FreeBSD, for example, 6.2. CVS tag name is typically RELENG_X_Y (currently RELENG_6_2) and has a major (X) and a minor (Y), version number. Binary updates (freebsd-update) are supported for this tag type only since this constitutes the so called security branch after installation. |
You can find the information about what version you will have by looking at the script /usr/src/sys/conf/newvers.sh (or in our non-standard case /var/src/sys/conf/newvers.sh) once the cvsup (or csup) run is complete. Look for the variables REVISION and BRANCH - these will also be output when you load the new kernel and issue either uname -r or uname -a.
When this step is complete you should read the file /usr/src/UPDATING (or in our non-standard case /var/src/UPDATING) for specific notes about the release.
This step involves building all the various tools/utilities required to run FreeBSD. They are built and saved in the defined directores and are NOT INSTALLED (remember if it says build at the front it does NOT install). This process is carried out in normal production mode and has no impact (other than CPU and disk utilization) on production jobs. We usually run it overnight. This process consumed ~440m of disc space on our system.
By default, the build process assumes /usr/src for the source files and places compiled files and applications in /usr/obj. We will be using non-standard paths so we need to do more work. If you are using the standard paths ignore commands marked with NON_STD_PATH_ONLY comment below:
# check to make sure you have enough space on the target # partitions df -m # NON_STD_PATH_ONLY # set environmental variable to change OBJ path MAKEOBJDIRPREFIX=/var/obj export MAKEOBJDIRPREFIX # do not issue commands if you are using /usr/obj # NON_STD_PATH_ONLY cd /var/src # if you are using the standard build paths issue cd /usr/src # in all cases make buildworld # this will run for some time
At the end of this step your base system is unchanged (all the new stuff is in the /src and /obj directories) and you can remain in this state for as long as you want.
This step involves building a basic kernel to run the new version of FreeBSD. You will see two methods of doing this in the literature. The commands below show both and note the differences. This process is carried out in normal production mode and has no impact (other than CPU and disk utilization) on production jobs. This process consumed ~290m on our target system.
We show building a kernel using the GENERIC file which is the least risk path and the only one you can use if you want to use binary security updates. Once the upgrade process is entirely complete you can go back and build a new custom kernel if required.
By default the kernel build process assumes /usr/src for the source files and places compiled files and applications in /usr/obj. We will be using non-standard paths so need to do more work. If you are using the standard paths ignore commands marked with NON_STD_PATH_ONLY comment below:
# check to make sure you # have enough space df -m # NON_STD_PATH_ONLY # set environmental variable to change OBJ path # only if you have reloaded since the previous step # or confirm with printenv MAKEOBJDIRPREFIX=/var/obj export MAKEOBJDIRPREFIX # do not issue the above line if you are using /usr/obj # NON_STD_PATH_ONLY cd /var/src # if you are using the standard path issue cd /usr/src # in all cases make kernel KERNCONF=GENERIC # this makes and installs the new kernel # but copies the current kernel to /boot/kernel.old # before doing so # you will also see the following make buildkernel KERNCONF=GENERIC make installkernel KERNCONF=GENERIC # the above two command have exactly the same effect as # make kernel KERNCONF=GENERIC # if you like typing two lines use the above method
At the end of this step you will have in /boot/kernel the new version of a GENERIC FreeBSD kernel for your selected release and in /boot/kernel.old your previous version (in our case 5.3) of the kernel. If your system crashes or you reboot and do nothing the default will be to load the new version kernel. You can however force load the old kernel at boot time.
This step involves booting the new kernel since all the subsequent work will be done under the new version of FreeBSD and in single user mode. This is not a risky step since if the new kernel crashes you can revert to the previous one at boot time. Issue the following commands:
# shutdown current system cleanly shutdown -r now # OR reboot # at the beastie prompt select 4 (boot to single user mode) # or enter boot -s if its just a prompt # when the shell prompt appears accept it [ENTER] # then do fsck -p # fsck not strictly necessary if you shut down cleanly # but does no harm # change root from read-only mount -u / # mount all fstab entries (load all file systems) mount -a # if your BIOS clock is at wall-clock time (local time) # issue this command, if it is at UTC time do not # if you have no idea then issue the command anyway adjkerntz -i # your system should now be functional and ready for the next step # confirm the kernel version number uname -a # which should reflect the new version you expect # if not STOP and figure out why before moving on
If the new kernel crashes or fails to load in single user mode then take as much diagnostic information as you can (re-run the boot if necessary) and then load the previous kernel version as follows:
# at the beastie screen select option 6 if present # then at the prompt enter boot /boot/kernel.old # and your previous kernel will load # to restore your previous kernel as default while you # figure out what went wrong cd /boot # save broken kernel in case you need more info mv kernel kernel.bad # restore old kernel to default mv kernel.old kernel
If everything has gone well you should now be loaded on a new OS but with all the tools and infrastructure using the old version. Time now to change that.
This is the biggest risk step since it involves upgrading the tools/utilities and configuration files to the new version and can be extremely lengthy. First we run mergemaster to fix up our /etc configuration files using the -p option:
# NON_STD_PATH_ONLY # you need to define where the source directory is mergemaster -p -m /var/src # do not issue the above line if you are using default /usr/src # if you are using the default (/usr/src) just issue: mergemaster -p # you will be prompted to reconcile or ignore changed files # the following changes are important # 1. password file # if you are using a dhcp client 6.2 requires # a new _dhcp user account # 2. group file # 6.x introduces a new audit group # in most other cases keep your own files (use the d option) # when merging the group and/or password (or any other files) # select the m option which displays each line of the file # with current on left and new on right # the prompt (%) allows you to select: # l - use left version # r - use right version # e l - edit then use left version # e r - edit then use right version # e b - edit then use left and right concatented # e - edit new version
At the end of the first mergemaster command we have versions of configuration files that have changed (normally in /var/tmp/temproot/etc) ready to replace all our /etc files. These files have NOT been installed so you can revert to your old system at anytime. Now we upgrade the base system and utilties to the new version by running make installworld which we built light-years ago with the make buildworld.
By default, the build process assumes /usr/src for the source files and places compiled files and applications in /usr/obj. We will be using non-standard paths so need to do more work. If you are using the standard paths ignore commands marked with NON_STD_PATH_ONLY comment below:
# NON_STD_PATH_ONLY cd /var/src # if you are using the standard paths then cd /usr/src # NON_STD_PATH_ONLY # we need to define that our obj path is non standard # edit Makefile and Makefile.inc1 in each file # find line starting with MAKEOBJDIRPREFIX?= /usr/obj # change to MAKEOBJDIRPREFIX?= /var/obj # we expect (but did not try) this to work instead of editing # the two makefiles MAKEOBJDIRPREFIX=/var/obj export MAKEOBJDIRPREFIX # in all cases make installworld # this runs for 5 - 10 minutes
When we first ran the make installworld it failed with the following message:
touch: not found *** Error code 127
After some panicky googling - we enjoy the panics, makes us feel alive - we discovered various mail archives that indicated the clock was wrong. Having followed the instructions make installworld continued to fail. Our suspicions are now that the clock problem fix is extremely misleading advice. Remember at this point the make installworld is trying to use the tools in the /obj directories to perform the actions not those in the normal locations such as /usr/local/sbin etc.. This is done to ensure that the new kernel is matched with the new utilities to minimise any problems and to allow fairly serious changes to take place in userland.
If you get the Error Code 127 message we would recommend that you first confirm that the new version of the utility which has not been found, actually exists (was built correctly). In the case of touch (our problem) you will find this in (default) /usr/obj/var/src/usr.bin/touch (in our non-standard case /var/obj/var/src/usr.bin/touch). This is a directory and should contain at least touch.o and a touch executable. If these are not present, something mis-fired and you should back-track to step 1 and repeat the process checking each step of the way for good results. In our case the 127 error was a consequence of our non-standard paths which was fixed using the makefile edits described above.
Finally we run mergemaster again to bring the /etc directories and others up-to-date (again if you are using standard paths ignore the NON_STD_PATH_ONLY commented lines):
# NON_STD_PATH_ONLY # supply the location of the /obj directory mergemaster -i -m /var/obj # if running with standard paths use this line mergemaster -i # this can run for a considerable time
At the end of this step you have a system which has all the base libraries, tools and utilities ugraded to the new release (but not the ports). Time to test it.
# you can just do this exit # but we would suggest that you do a full reboot shutdown -r now # OR reboot # at the beastie prompt hit ENTER (loads default) or wait for # timer to expire which then loads the default # and watch your system spring into life
If you need a custom kernel you can do it anytime from here onwards using your favorite procedure.
Finally in this section we need to do a bit of housekeeping by cleaning the /obj directory (unless you will be building a custom kernel in which case do it afterwards). To recover some of that space you consumed:
# the old classic method # NON_STD_PATH_ONLY cd /var/obj chflags -R noschg * rm -rf * # if you are using the standard paths cd /usr/obj chflags -R noschg * rm -rf * # OR # NON_STD_PATH_ONLY cd /var/src make cleanworld # if you are using the standard paths cd /usr/src make cleanworld # in all cases to clean temproot # used by mergemaster cd /var/tmp chflags -R noschg temproot/* rm -r temproot
Your production work should now be running but perhaps non-optimally since it may (most likely is) running via a compatability layer. Now it's time to fix the ports and bring everything up to date.
This is a low risk task but should be done as soon as practical. The current port tree reflects those of the previous release and may have substantially changed. Certainly the ABI of the installed ports reflects the previous version so they need to be updated in any case. You will find that csup was magically installed during the 6.x upgrade (now part of the base install) so we will use that utility instead of cvsup.
First we update the entire ports collection (you can add/edit the /var/db/sup/refuse file to minimise the work here). So we edit /usr/share/examples/cvsup/ports-supfile as shown (csup uses the same file format as cvsup):
# select your favorite host *default host=cvsupxx.FreeBSD.org # make sure that ports-all is defined not # one of the sub-collections # to run the update using csup use: csup -L 2 /usr/share/example/cvsup/ports-supfile # or using cvsup use: cvsup -g -L 2 /usr/share/example/cvsup/ports-supfile
When this is complete you should read /usr/ports/UPDATING. The endless joys.
Note: The refuse file seems to one of those closely hidden secrets and is not installed by default. Its purpose is to minimise the work done with a ports update by eliminating those items you do not want. It should be created before running the update above - so if you are following serially - it will help next time. To create a refuse file do the following:
cd /var/db/sup touch refuse vi refuse (or whatever editor you use) # add all required lines such as # to inhibit ports ports/games ports/japanese # to inhibit language documentation doc/jp_* doc/jp # etc, etc, etc
Now run the following commands to clean up the ports collection:
# if you don't have portupgrade installed you should have, so cd /usr/ports/ports-mgmt/portupgrade make install clean # update the ports index cd /usr/ports make fetchindex # update the ports database pkgdb -u # fix the ports database looking for stale or duplicate entries pkgdb -F # BEWARE: can ask lots of questions # now upgrade your all your ports (will take a long time) portupgrade -af # or you can do them selectively, for example portupgrade apache13
Welcome to the bright shiny new world. Take a couple of days off.
If things appear to have gone wrong then these notes, together with a good dose of panic may help. However if you are reading this section for real and you do not have a backup, start thinking about ritual suicide.
If it is any comfort the first time we ever did this we ran all the step 5 processes at least a couple of times before we were successful and lived to tell the tale.
Up until Step 4 nothing serious has happened - recovery is trivial by restarting the entire process. You do however want to free up the space you may already have consumed by doing the following:
# the old classic method # NON_STD_PATH_ONLY cd /var/obj chflags -R noschg * rm -rf * # if you are using the standard paths cd /usr/obj chflags -R noschg * rm -rf * # OR # NON_STD_PATH_ONLY cd /var/src make cleanworld # if you are using the standard paths cd /usr/src make cleanworld
Then start the entire process again from Step 1.
If you fail in step 5 doing the first mergemaster run nothing has happened. Boot into your old kernel and try and figure what went wrong. Or you can rerun and watch more closely for clues - the utility provides lots of options for starting again - try them all - but methodically. If you are using non-standard paths make sure the -m option is correct for your /obj directory. You can jump backwards and forwards between the old and new kernels on a reboot as often as you like. You may even become addicted to the process. Look at the various files in /var/tmp/temproot and try and figure how far you got.
If you fail in step 5 doing the make installworld you may end up with a partially updated system, however it typically fails very early in the run when make attempts the first access. In this case nothing has yet happened and you can most likely reboot into the old kernel without any problems. However it's best to stay in the single user mode if you can and continue the debug.
If it fails later in the run try and figure the exact place that it failed and the reasons (use google - from another system, since yours is unusable - snigger, snigger) and in this case stay in the single user new kernel for as long as you can to repair the system. You can rerun the make installworld (well it can't get any worse can it) or use make installworld > /tmp/installworld to capture the output and then use an editor to plow through the file. Again if you are using non-standard paths make sure the MAKEOBJDIRPREFIX variable is set and correct or that you have correctly edited the Makefile and Makefile.inc1 files.
If you have no ideas and it's a re-image and re-install or nothing, then boot into either kernel (try both) using full multi-user mode and see what happens. You may get some more diagnostic ideas. It may even work.
If you fail in step 5 doing the second mergemaster you may end up with a partially updated system - but the commands and base libraries have been updated. You should be able to boot into the new system. However try and figure the exact place that it failed and the reasons and stay in the single user new kernel for as long as you can to repair the system. You can rerun mergemaster using mergemaster -i > /tmp/merge2 to capture the output and then use an editor to plow through the file looking for the errors. Again, if you are using non-standard paths make sure you use the -m option and that it points to the correct path.
While the utility freebsd-update has apparently been part of the ports collection for a number of years (so we are told) as of 6.2 it is now part of the base system. It is a method of keeping your system updated with kernel patches and security fixes. To use freebsd-update you must meet the following criteria:
# to check your release uname -r # anything post 6 with -RELEASE is supported
This a great service innovation. What was a painful process is now trivial in the extreme. We currently wait for a Security Advisory (freebsd-announce mailing list) and then do a manual update. The author of freebsd-update, Colin Percival, encourages use of the daily cron service since the "no update" overhead is trivial. Find a way to use this service - it really is that good.
Freebsd-update will take one or more of the following parameters:
# command format freebsd-update [options] # where one or more options may be present from below: # fetch - download (not install) required updates # install - installed updated downloaded by fetch # rollback - uninstall the most recent install # cron - required parameter when running from cron (see below) # fetch then install updates freebsd-update fetch install # running from cron # daily cron driven automatic update # add the following line to /etc/crontab 0 3 * * * root /usr/sbin/freebsd-update cron fetch install # IMPORTANT: If your machine is set to UTC use a random # value other than 3 to avoid a deluge of requests to the # update servers or use this to run every n days at 4 am or other time 0 4 */n * * /usr/sbin/freebsd-update cron fetch install
Note: Corrected version of this note. For whatever reason during the update procedure we ended up with two versions of freebsd-update and no /etc/freebsd-update.conf. The latest version is installed in /usr/sbin and assuming a normal path is found and execute before any older freebsd-update (in /usr/local/sbin). The missing freebsd-update.conf may be found at /usr/src/etc/freebsd-update.conf (or in our case /var/src/etc/freebsd-update.conf) and is shown below. Finally there is a version of freebsd-update on the ports tree which cannot be installed under FreeBSD 6.2 and in any case is the OLD version which uses a different parameter set. If you have incorrectly deleted the new version you can restore it from http://www.freebsd.org/cgi/cvsweb.cgi/src/usr.sbin/freebsd-update/freebsd-update.sh download version 1.8, place in /usr/sbin and rename to freebsd-update (remove the .sh suffix) and chown to 0644.
# delete the old version if installed rm /usr/local/sbin/freebsd-update # check and delete any freebsd-update.conf[.sample] # in /usr/local/etc
As stated earlier after the upgrade we did not have a copy of the configuration file in /etc/freebsd-upgrade.conf after the source upgrade process. You can find it in /usr/src/etc/freebsd-update.conf and copy unmodified to /etc. In case this method fails we reproduce the standard distribution of this file below. Note: The parameter format of this file has changed from previous versions of freebsd-update hence our advice to delete any old freebsd-update.conf files from /usr/local/etc to avoid confusion.
RSS Feed
