Summary

The Rebuild Queue plugin is designed to let you shift the rebuilding operations typically done by Movable Type to one or more "offline" processes.

With this plugin, you can fine-tune your building operations so they are distributed and distribute the built pages across any number of servers.

Requirements

Installation

To install, place the "RebuildQueue" directory in this distribution underneath your Movable Type "plugins" directory. Once the files are physically installed, accessing Movable Type will trigger an upgrade procedure which will install the plugin's table into your database.

The plugin files, once installed should match that shown below:

MT_DIR/

mt-static/

RebuildQueue/

rebuildq.gif

plugins/

RebuildQueue/

lib/

RebuildQueue/

Daemon.pm File.pm Plugin.pm Publisher.pm

RebuildQueue.pl

All files should have permission that make them readable by the web server and/or account being used to invoke the offline RebuildQueue.pl script (644 permissions); the RebuildQueue.pl script should be executable (755 permissions).

Once the installation is complete, you are ready to configure the plugin.

Configuration

Global Configuration Options

Navigate to the Plugin listing from the MT System Overview menu. There you will find an entry for the Rebuild Queue plugin. Click the "Show Settings" link to reveal the global configuration settings.

"Number of Workers": This setting lets you define how many rebuild "worker" processes you plan to run. The default number is 1, but you can increase this number if you wish. When the Rebuild Queue schedules a page to be rebuilt, it can randomly assign the task to one of the worker processes. This helps distribute the rebuilding load across multiple processes or servers.

Weblog Configuration Options

To enable the Rebuild Queue, you must go to the weblog(s) you wish it to manage and enable it for each of them. From the Weblog Settings page, go to the "Plugins" tab and click the "Show Settings" link for the Rebuild Queue entry listed there. The following settings are available at the weblog level:

"Enable for this Weblog?": Check this option to enable the Rebuild Queue to manage rebuilding for that weblog.

"Force Building to Worker": If you have configured (at the global level) 2 or more worker processes, you will see this option. It lets you assign any queued build operations for this weblog to a particular worker. This gives you control over which worker process builds the weblog instead of it being randomly distributed.

Cron Jobs and Daemon Processes

Since rebuilding is done separately, it's important to set up the Rebuild Queue processes to do the actual rebuilding operations and mirroring you may require.

The RebuildQueue.pl script is used to run these processes. Running RebuildQueue.pl without any options will show the command usage.

A typical cron job for the RebuildQueue itself might be something like this:

*/15 * * * * cd /path/to/RebuildQueue; ./RebuildQueue.pl

This will invoke the Rebuild Queue every 15 minutes to build any pages that have been modified since the last execution. Of course, you can invoke it more often than that if you prefer.

*/15 * * * * cd /path/to/RebuildQueue; ./RebuildQueue.pl -sync -to user@hostname:

This cron job will handle the rsync job to copy content from the local server to another.

*/15 * * * * cd /path/to/RebuildQueue; ./RebuildQueue.pl -daemonize

This will invoke the Rebuild Queue process as a daemon; that is, it will remain running until the process has been halted. This may be preferable since it will see pending rebuild requests almost immediately and handle them sooner. It only allows one daemon to run, so setting it up on a cron job is a way to make sure the daemon process is always running, even if it is killed for one reason or another.

*/15 * * * * cd /path/to/RebuildQueue; ./RebuildQueue.pl -daemonize -worker 2

This sets up a second worker daemon. I said only one daemon can run, but that only applies by worker id (the default worker id is 1). So you can't have multiple daemons running with the same worker id, but you may run as many workers as you want.

As long as the database is shared (or you can replicate to multiple databases), the Rebuild Queue daemons can be running on totally different servers and then syncing to the others in your web farm.

You may also want to 'nice' these processes so they don't overpower the server when they wake up to do some work.

Throttling

You may also throttle the rebuilding of certain templates or types of templates. That is, you may want to limit the rebuilding of category archives to once per day.

./RebuildQueue.pl -throttle category=8640

Throttle times are given in seconds.

Or perhaps you want to only build a given template once every hour, regardless of how many times it is requested to rebuild through the MT interface or comment system. The template's id is 10.

./RebuildQueue.pl -throttle 10=360

You may repeat the -throttle switch as many times as you require to specify your rules.

Example

Lets say you have a few weblogs that are used to publish a set of media web sites. The pages published are fairly heavy with content and/or comments and take time to process, so you want to shift that work to the Rebuild Queue instead. You also have 3 web servers that handle visitor traffic and 1 that is used by your bloggers to publish the content.

mt.zine.com (MT installation) www, www2, www3.zine.com (web farm for visitors; load balanced)

On mt.zine.com, you'd have your typical MT installation plus the Rebuild Queue plugin.

You set up a cron job that runs the Rebuild Queue daemon.

*/15 * * * * cd /path/to/RebuildQueue; ./RebuildQueue.pl -daemonize

You also set up a cron job that handles syncing to the web servers:

10,25,40,55 * * * * /path/to/RebuildSync.sh

RebuildSync?.sh is a custom script, containing:

#!/bin/sh cd /path/to/RebuildQueue RSYNC_RSH=ssh ./RebuildQueue.pl -sync -to user@www: -to user@www2: -to user@www3:

The sync command takes all files that the Rebuild Queue daemon has rebuilt and rsyncs them to the servers you specify. Note that the physical file paths as they exist on the "mt.zine.com" server must be the same on the other servers to match up.

Authors

Brad Choate and Jay Allen of Six Apart

Questions

I'd like to to understand which files get rebuilt when with RebuildQueue, and how the "Built w/Indexes" settings affects this. We typically want things like the main index, main rss feed, and a couple of custom index templates rebuilt on each post, but then have the rest of the templates rebuilt with cron. Does the "Built w/Indexes" setting accomplish this or have any affect on what happens with RebuildQueue?