qtp-newmodel


History

The original upgrade.sh script was written by  Jake Vickers.

 Erik Espinoza's current-download-script and parts of the current-install-script written by  Nick Hemmesch were added to it.

Backup and restore of configuration files was added by  Devendra Meena.

Thanks go to Jake, Nick, Erik, and Devendra who contributed to the previous upgrade.sh script. I could not have written the new script without their previous work as a starting point. -ES

On July 13, 2006 Eric 'shubes' announced to the  email list the release of new upgrade scripts. The new scripts are a substantial change to the former process, and thus go by new names, qtp-newmodel and various qtp-* sub-scripts. They create a new model of QmailToaster with minimal effort and downtime.

The scripts have undergone substantial testing and use, and are now considered to be Production status. If you have any problems and/or suggestions, please follow the support guidelines.


Overview

After selecting which packages to upgrade and downloading them, qtp-newmodel creates a sandbox then builds and installs the new source packages in the sandbox. If there are any errors, the only thing that is broken is the sandbox, not the production environment. Once all of the packages are built and applied in the sandbox environment and there are no errors, it tells you the build was clean. Then it asks if you want to apply it to the production area. At that point, when you say yes, it stops qmail, updates the production packages using the binary rpms that were built in the sandbox, then starts qmail again.

This process achieves 2 things. First, it safely builds the sources into the sandbox without jeopardizing the production environment. Second, it makes the downtime very short (less than a minute) while it applies the sandbox RPMs to the production environment. If you have a lot of packages to update, the source builds can take a while. For example, it can take about 45 minutes to build and apply the sources in the sandbox, but only 45 seconds to update into production. Thus the production QMT is only down for 45 seconds.

Here's an overview of what the scripts do (in sequence):

  • Determine the DISTRO and ARCH variables - the qtp-whatami script interrogates /etc/*-release to see which distro your machine is, and sets variables accordingly. If your distro isn't supported, the newmodel script will not continue.
  • invokes qtp-config to set common environment variables
  • Downloads current package list from the  official QmailToaster web site
  • zlib and djbdns are removed from the list (neither is a *-toaster package, and qtp-newmodel does not support them)
  • Checks to see which package updates are not installed, and lets you select which ones to process. Normally, you'll want to select all that are available. You may also elect to rebuild/reinstall any installed packages.
  • Downloads source packages from the  official QmailToaster web site.
  • Builds the sandbox by invoking either the qtp-mount-sandbox script (for unionfs type sandbox) or the qtp-build-sandbox script (for linked and copied type sandboxes).
  • chroot to the sandbox and invokes qtp-build-rpms, which:
    • invokes qtp-remove-pkgs to remove certain packages from the sandbox
    • builds and installs each package in the sandbox
  • copies the binary rpms out of the sandbox
  • backs up control and configuration files
  • stops qmail
  • invokes qtp-remove-pkgs to remove certain packages that need to be removed
  • upgrades the packages with a single rpm -Uvh command
  • invokes qtp-convert to do conversion processing:
    • mrtg file changes
    • mysql password location change
  • invokes queue_repair.py to clean up any queue anomalies
  • starts qmail
  • runs spamassassin debug configuration
  • cleans up source rpms, binary rpms and the sandbox

That's it in a nutshell.


Aspects

Some aspects of the script:

  • all questions asked (except the first one) have a default value. If you're not sure what to do, take the default.
  • all directories are created automatically by the script.
  • the script has one command line option: -b which will run the script without prompting. This is convenient if you don't want to babysit it, but should not be used until you're comfortable with the process. To run it in background, probably overnight, you might do something like:
    # qtp-newmodel -b >qtp-newmodel.log 2>&1 &
    
  • the compile output is sent to a log file instead of the screen. If there is a compile error, the last 10 lines of the log are displayed on the terminal. Other pertinent output is sent to both the terminal and the log file. This can be useful for problem determination.
  • the script can be interrupted (preferrably by using quit when asked) and rerun at a later time, and all previously completed work is saved. This means you can run it at any time to download and build the rpms (the most time consuming), then run it again later when you're ready to actually do the upgrade. During development testing, the upgrade portion for all packages took less than 10 minutes (most of which was generating domain keys) on a P-II/266. Timings with modern hardware are considerably faster.

The Sandbox

The sandbox is an isolated image of your current system (aka chroot jail). The qtp-newmodel process uses one of three types of sandboxes: Copied, Linked, or Unioned. Just before the sandbox is built, you will be asked which type you would like to use.

A Copied sandbox is a normal copy of most of the system files. This takes a large amount of disk space (2.4G or so), and quite a bit of time (20 cpu minutes or so on a P-II/266) to create. A Copied sandbox is the least preferable type, but it works under any circumstance. It was the first type developed.

The Linked type of sandbox was developed to reduce disk space requirements. The script will attempt to use hard links (directory entries) instead of making actual copies of a majority of the files. All toaster and rpm files are still copied though, so they are not changed in the sandbox. While the Linked type is a big improvement over a Copied sandbox as far as disk space is concerned (down to as little as 100M), creating it still takes a good deal of time, although it is a bit faster to create than a Copied sandbox. In addition, these savings can only be achieved when the sandbox and other major system directories are maintained on the same partition as the root (/) filesystem, as hard links cannot cross partition boundaries. The script will automatically copy files which are on separate partitions when trying to build a linked sandbox. Any partition scheme should work, although a linked sandbox provides the greatest savings when the system is largely on a single partition.

The Unioned type of sandbox was later developed for qtp-newmodel by  Justice London. This is also known as an Overlay file system. It's the same thing that's used by "Live" CD distros. In essence, the existing file system is used as a base for all reads. Whenever a file is to be changed, a copy of the existing file is automatically made to a separate area, and the changes are made to the copy. New files are also added in the separate area. The existing/primary file system is never modified.

The Unioned type of sandbox is preferred. It's by far the fastest, and it uses a minimal amount of disk space. The union filesystem used by qtp-newmodel uses the Filesystem in USEr space (FUSE) architecture provided by 2.6.14+ kernels (and 2.6.9+ kernels with the additional dkms package), which means that it does not need to be compiled against any specific kernel, and can be distributed as typical RPM package. The required binary packages are available in QTP yum repository, and are installed automatically for you using yum.

The only drawback to a Unioned sandbox is that it only works with kernel versions 2.6.9 or greater. If your toaster runs on an older kernel, qtp-newmodel will not offer you the option of a Unioned sandbox.

The sandbox can be placed on any mounted partition. If the sandbox is on a partition other than root (/), the linked option will not be available.

Note regarding reuse: If any substantial packages (i.e. qmailtoaster-plus, OS libs) have been upgraded since the sandbox was built, you should create a new sandbox (the script gives you this option). Otherwise, the sandbox can safely be reused. There's little point in reusing a unionfs sandbox though, so removing a unionfs sandbox is recommended.


Instructions

The scripts are available as part of the  QmailToaster-Plus package. They are not supported independently. Please follow the Installation Instructions in order to obtain them.

To run the script(s), at the command prompt simply enter:

# qtp-newmodel

You may also invoke it from qtp-menu.

Simply answer the questions at the prompts, and that's all there is to it!


Tailoring

qtp-config

There are three variables set by the qtp-config script that you may care to change, although it is strongly recommended that you change them only if you have a good reason to do so and you know what you are doing:

  • UPGRADE_DIR=/usr/src/qtp-upgrade is a general working directory. Subdirectories are SRPMS, RPMS, backups, old-rpms and log (pretty much self explanatory).
  • SANDBOX=/opt/qtp-sandbox is a directory where the sandbox is built. See The Sandbox above for considerations. This should point to a directory that does not yet exist, and is used exclusively for the sandbox.
  • RPMBUILD_OPTIONS is a variable the contents of which are included in the rpmbuild command for all packages. This can be set to any valid rpmbuild parameter, such as RPMBUILD_OPTIONS="-v" to turn on verbose mode, or RPMBUILD_OPTIONS="--quiet" for quiet mode.

rpmbuild

If you need to tailor the rpmbuild command for a specific package, you can do so by creating a file in the /opt/qmailtoaster-plus/etc/rpmbuild directory containing the command line option(s) to be used for building the package. For example, if you want to use the "--define 'spambox 1'" option for the qmailadmin-toaster package, you simply create a file named qmailadmin-toaster in the rpmbuild directory that contains "--define 'spambox 1'" (without the double quotes). There is a qmailadmin-toaster.sample file included in the  QmailToaster-Plus package for reference.