root/trunk/RebuildQueue/README.txt @ 171

Rebuild Queue
Version 1.04


Summary
=======

The Rebuild Queue plugin is designed to let you shift the rebuilding
operations typically done by Movable Type to one or more "offline"
processes.

With this plugin, you can fine-tune your building operations so they
are distributed and distribute the built pages across any number of
servers.


Requirements
============

Movable Type version 3.3 or later (or Movable Type Enterprise).

The 'rsync' command if using remote server synchronization. (see: http://samba.anu.edu.au/rsync/)

Time::HiRes (see: http://search.cpan.org/dist/Time-HiRes/)

Installation
============

To install, place the "RebuildQueue" directory in this distribution
underneath your Movable Type "plugins" directory. Once the files are
physically installed, accessing Movable Type will trigger an upgrade
procedure which will install the plugin's table into your database.

The plugin files, once installed should match that shown below:

    MT_DIR/
           mt-static/
                     RebuildQueue/
                                  rebuildq.gif
           plugins/
                   RebuildQueue/
                                lib/
                                    RebuildQueue/
                                                 Daemon.pm
                                                 File.pm
                                                 Plugin.pm
                                                 Publisher.pm
                                RebuildQueue.pl


All files should have permission that make them readable by the
web server and/or account being used to invoke the offline
RebuildQueue.pl script (644 permissions); the RebuildQueue.pl
script should be executable (755 permissions).

Once the installation is complete, you are ready to configure the
plugin.


Configuration
=============

Global Configuration Options
----------------------------

Navigate to the Plugin listing from the MT System Overview menu. There you
will find an entry for the Rebuild Queue plugin. Click the "Show Settings"
link to reveal the global configuration settings.

"Number of Workers": This setting lets you define how many rebuild "worker"
processes you plan to run. The default number is 1, but you can increase
this number if you wish. When the Rebuild Queue schedules a page to be
rebuilt, it can randomly assign the task to one of the worker processes. This
helps distribute the rebuilding load across multiple processes or servers.

Weblog Configuration Options
----------------------------

To enable the Rebuild Queue, you must go to the weblog(s) you wish it to
manage and enable it for each of them. From the Weblog Settings page, go
to the "Plugins" tab and click the "Show Settings" link for the Rebuild Queue
entry listed there. The following settings are available at the weblog level:

"Enable for this Weblog?": Check this option to enable the Rebuild Queue to
manage rebuilding for that weblog.

"Force Building to Worker": If you have configured (at the global level) 2
or more worker processes, you will see this option. It lets you assign any
queued build operations for this weblog to a particular worker. This gives
you control over which worker process builds the weblog instead of it being
randomly distributed.

Cron Jobs and Daemon Processes
------------------------------

Since rebuilding is done separately, it's important to set up the
Rebuild Queue processes to do the actual rebuilding operations and
mirroring you may require.

The RebuildQueue.pl script is used to run these processes. Running
RebuildQueue.pl without any options will show the command usage.

A typical cron job for the RebuildQueue itself might be something like this:

*/15 * * * * cd /path/to/RebuildQueue; ./RebuildQueue.pl

This will invoke the Rebuild Queue every 15 minutes to build any pages that
have been modified since the last execution. Of course, you can invoke it
more often than that if you prefer.

*/15 * * * * cd /path/to/RebuildQueue; ./RebuildQueue.pl -sync -to user@hostname:
 
This cron job will handle the rsync job to copy content from the local
server to another.

*/15 * * * * cd /path/to/RebuildQueue; ./RebuildQueue.pl -daemonize

This will invoke the Rebuild Queue process as a daemon; that is, it will
remain running until the process has been halted. This may be preferable
since it will see pending rebuild requests almost immediately and handle
them sooner. It only allows one daemon to run, so setting it up on a cron
job is a way to make sure the daemon process is always running, even if
it is killed for one reason or another.

*/15 * * * * cd /path/to/RebuildQueue; ./RebuildQueue.pl -daemonize -worker 2

This sets up a second worker daemon. I said only one daemon can run, but
that only applies by worker id (the default worker id is 1). So you can't
have multiple daemons running with the same worker id, but you may run as
many workers as you want.

As long as the database is shared (or you can replicate to multiple
databases), the Rebuild Queue daemons can be running on totally different
servers and then syncing to the others in your web farm.

You may also want to 'nice' these processes so they don't overpower the
server when they wake up to do some work.

Throttling
----------

You may also throttle the rebuilding of certain templates or types of
templates. That is, you may want to limit the rebuilding of category
archives to once per day.

    ./RebuildQueue.pl -throttle category=8640

Throttle times are given in seconds.

Or perhaps you want to only build a given template once every hour,
regardless of how many times it is requested to rebuild through the MT
interface or comment system. The template's id is 10.

    ./RebuildQueue.pl -throttle 10=360

You may repeat the -throttle switch as many times as you require to
specify your rules.


Example
=======

Lets say you have a few weblogs that are used to publish a set of media
web sites. The pages published are fairly heavy with content and/or
comments and take time to process, so you want to shift that work to
the Rebuild Queue instead. You also have 3 web servers that handle visitor
traffic and 1 that is used by your bloggers to publish the content.

    mt.zine.com (MT installation)
    www, www2, www3.zine.com (web farm for visitors; load balanced)

On mt.zine.com, you'd have your typical MT installation plus the Rebuild
Queue plugin.

You set up a cron job that runs the Rebuild Queue daemon.

*/15 * * * * cd /path/to/RebuildQueue; ./RebuildQueue.pl -daemonize

You also set up a cron job that handles syncing to the web servers:

10,25,40,55 * * * * /path/to/RebuildSync.sh

RebuildSync.sh is a custom script, containing:

    #!/bin/sh
    cd /path/to/RebuildQueue
    RSYNC_RSH=ssh
    ./RebuildQueue.pl -sync -to user@www: -to user@www2: -to user@www3:

The sync command takes all files that the Rebuild Queue daemon has rebuilt
and rsyncs them to the servers you specify. Note that the physical file
paths as they exist on the "mt.zine.com" server must be the same on the
other servers to match up.


Availability
============

The latest release of this plugin can be found at this address:

    http://code.sixapart.com/


License
=======

This plugin is released under the Artistic License.


Authors
=======

Brad Choate and Jay Allen of Six Apart