Development Environment for Drupal Sites

Mar 04 16:44

As you may know, GeologyRocks uses drupal, a rather excellent CMS. This brings about a few headaches when trying to develop your own code for it as the content (i.e. the stuff you need to test out your new code) is in the database, not as a series of HTML files as for a static websites. To help me develop new features for GeologyRocks, I've set up a development environment for drupal-based sites. This is how I did it...

Method

The aim is to create three versions of the website:

A mirror
A development version (sandbox)
A stable checkout version which contains files, etc ready to upload to live server

Each site needs to function the same as the live server (i.e. minimal database and file changes). In addition we need a scripts to keep the mirror up-to-date (both files and database), to re-create the sandbox version at will from the CVS repository (easy!) and to update the sandbox from the mirror version (the database and files in case we screw up). Note that it is always prefereable to update the working directory by CVS, rather than the mirror in the case of a terminal programming error. However, the working directory will need updating from the mirror periodically as the changes accumulate on the live site. This enviroment was set up on a linux server using apache and a few standard linux tools, but could be done on windows. It also assumes the live site is on a server running CPANEL, a standard interface to most shared hosting packages.

The general strategy is:

create working dir and stable dir
create website, ready for live server
once ready, place code on live server
initiate mirroring procedure
develop on working dir, every so often updating the database from the mirror. The files may also need updating, depending on how active the site is.This allows development on an environment similar (or identical) to the live site.

Set-up

All three versions should be available from the local server via canonical domains, i.e. blah.example.com

Directory structure:

/home/html/
    mirror/SITE
    stable/SITE
    working/SITE

Apache conf:
The set up relies heavily dependent on apache Virtual Hosts. Each site needs the following httpd.conf changes:

# set up mirror
<VirtualHost *>
    ServerName mirror.geologyrocks.co.uk
    DocumentRoot /home/html/mirror/geologyrocks/public_html/
    DirectoryIndex index.php
    <Directory "/">
        Options FollowSymLinks 
        AllowOverride All
        DirectoryIndex index.php
   </Directory>
</VirtualHost>
# set up sandbox
<VirtualHost *>
    ServerName test.geologyrocks.co.uk
    DocumentRoot /home/html/working/geologyrocks/public_html/
    DirectoryIndex index.php
    <Directory "/">
        Options FollowSymLinks        
        AllowOverride All
        DirectoryIndex index.php
   </Directory>
</VirtualHost>
# set up stable version
<VirtualHost *>
    ServerName stable.geologyrocks.co.uk
    DocumentRoot /home/html/stable/geologyrocks/public_html/
    DirectoryIndex index.php
    <Directory "/">
        Options FollowSymLinks        
        AllowOverride All
        DirectoryIndex index.php
    </Directory>
</VirtualHost>

Host set-up:
/etc/hosts need changing to add pseudo-domains on all machine you want to access the development environment from:

192.168.0.3 test.geologyrocks.co.uk
192.168.0.3 mirror.geologyrocks.co.uk
192.168.0.3 stable.geologyrocks.co.uk

on Windows machine, change:
c:\WINDOWS\SYSTEM32\drivers\etc\hosts

The entries need adding on ALL computers that need access. Note that these sub-domains cannot then be used on the site proper from those machine.

Mirroring:

Two scripts are needed: one to replicate the database, one to mirror the files.
The database replication script takes a backup of the SQL database downloaded from the CPANEL backups, extracts the SQL file and uploads it to the local database.
Backups and mirroring is done in a directory inside /root
To get the backup from the remote database, cron the following script (replace drupal with dbase name, minus any hosting prefix):

#!/bin/bash
x=$(/bin/date +%d-%m-%y)
FILE="/home/geologyrocks/Backups/sql-${x}.gz"
wget -q --http-user="USER" --http-passwd="PASS" \ 
-O $FILE http://HOST:PORT/getsqlbackup/drupal.gz
# replicate database into local mirror
perl /root/dbrep.pl ${FILE} USER PASS geology_drupal
# get any new files
# first get the public_html folder
perl ftpsync.pl -g /home/html/mirror/geologyrocks/public_html \
   ftpserver=ftp.SERVER ftpdir=public_html \
   ftpuser=USER ftppasswd=PASS
# then get the files folder
perl ftpsync.pl -g /home/html/mirror/geologyrocks/drupal \
ftpserver=ftp.SERVER ftpdir=drupal ftpuser=USER \
ftppasswd=PASS

ftpsync is a small perl script to synchronise an ftp server with a local directory.

dbrep.pl is another small perl script which drops a mysql database, re-creates it and imports data from a sql.gzip file (as downloaded from CPANEL).

This script can be adapted to download a file backup also, therefore doubling up as a backup script. This only needs to be run periodically as we have, what is in essence, a full backup with the ftp synch:

#!/bin/bash
FILE="/home/geologyrocks/Backups/full-${x}.tar.gz"
wget -q --http-user="USER" --http-passwd="PASS" \
-O $FILE http://www.geologyrocks.co.uk:2082/getbackup/backup-geologyrocks.co.uk-${x}.tar.gz

The script can then be added to the root crontab:

export EDITOR=emacs
crontab -e
# GeologyRocks-autoBackup, every day at 8
0 20 * * *      /root/grbackup.sct
# GeologyRocks-autoBackup, every sunday at 8
0 20 * * 7      /root/grbackup-full.sct

While editing the crontab, it is worth placing the appropriate drupal entries for the site in question (once per hour should suffice), e.g.:

# devel drupal
00 * * * * wget -O - -q http://localhost/working/drupal-dev/cron.php

CVS

CVSROOT is located at /home/cvs. Simple commands like checkin, checkout and import should be easily found on the web. Using:

DEFAULT (date; cat; (sleep 2; cd /home/html/stable/; cvs -q update -d) &) >> $CVSROOT/CVSROOT/updatelog 2>&1

in the CVSROOT loginfo file, automatically gives us the stable checkout. whenever we check something in, it is checked out to the stable version. Note that the stable and working versions use the same database, but different files.

To add a website to the CVS repository:

cd /home/html/working/
cd geologyrocks/
cvs -d /home/cvs import geologyrocks jonhill version3

Then mv existing directory, checkout module and test. When happy rm mv'd directory
The module also needs checking out into the stable directory in order for the automatic checkout to work.

Updating working version

To update both files and db of working version to match the mirror version, execute the script update_working.sct:

./update_working.sct /home/html/working/geologyrocks/ \
/home/html/mirror/geologyrocks/ geologyrocks_working geology_drupal

This should be done periodically to ensure the working version is using up-to-date data. Do a ci before running this. A ci is done automatically by the script after completion.

Appendices

dbrep.pl

#!/usr/bin/perl -w
# script to untar a db backup download
# and put it in a local database
# copyright 2007, Jon Hill

use strict;
use DBI;
use Mysql;
use IO::Uncompress::AnyInflate qw(anyinflate $AnyInflateError);

# set up temp file to write gz output to
my $temp_file = "/root/tempfile.sql";

# get cl arguments
# argument 0: gzip sql file
my $input_archive = $ARGV[0];
# argument 1: user
my $user = $ARGV[1];
# argument 2: password
my $password = $ARGV[2];
# argument 3: database to use
my $database = $ARGV[3];

my $hostname = "localhost";

print "Reading from $input_archive...\n";

my $status = anyinflate $input_archive => $temp_file
or die "anyinflate failed: $AnyInflateError\n";

# connect to database and drop current database
my $dsn = "DBI:mysql:database=$database;host=$hostname;";

my $dbh = DBI->connect($dsn, $user, $password);
my $rc = $dbh->func("dropdb", $database, 'admin');

# Create a new database. This must not fail, thus we don't
# catch errors.
$dbh = Mysql->connect($hostname, undef, $user, $password);
$rc = $dbh->createdb($database);

# now execute sql script on new database
exec ("mysql $database -u $user -p$password < $temp_file");

update_working.sct

#!/bin/bash
#*******************************************
#          
# update_working.sct               
#                                            
# written by Jon Hill, GeologyRocks.co.uk 
# Copyright, Jon Hill, March, 2007      
#                                
# Objective:                               
#   Update a working HTML directory from a mirror.
#   Updates both files (new or newer) and database. 
#
#********************************************

#**
# Argument list:
#   working_dir
#   mirror_dir
#   working_db
#   mirror_db
#**

# E.G.
#./update_working.sct
#/home/html/working/geologyrocks/
#/home/html/mirror/geologyrocks/ 
#geologyrocks_working geology_drupal

# set up some standard variables
MYSQL_USER="USER"
MYSQL_PASS="PASS"
MYSQL_HOST="localhost"
CVSROOT=/home/cvs

# get command line variables
E_BADARGS=65

if [ ! -n "$1" ]
then
  echo "Usage: `basename $0` argument1 argument2 etc."
  exit $E_BADARGS
fi

# grab command line arguments
WORKING_DIR=$1
MIRROR_DIR=$2
WORKING_DB=$3
MIRROR_DB=$4

# first update files, using linux cp command
# this copies new and newer files, maintaining symbolic links
cp -udrp $MIRROR_DIR $WORKING_DIR//

# now update database
# delete
mysqladmin -f -h $MYSQL_HOST -u $MYSQL_USER -p$MYSQL_PASS \
           drop $WORKING_DB
# create
mysqladmin -h $MYSQL_HOST -u $MYSQL_USER -p$MYSQL_PASS \
           create $WORKING_DB
#dump mirror and pipe to mysql to put in working db
mysqldump -h $MYSQL_HOST -u $MYSQL_USER -p$MYSQL_PASS \
           $MIRROR_DB | \
           mysql -h $MYSQL_HOST -u $MYSQL_USER -p$MYSQL_PASS \
           $WORKING_DB

# cvs ci 
cd $WORKING_DIR
cvs -d $CVSROOT ci -m "Update from mirror"

# cvs ci 
cd $WORKING_DIR
cvs -d $CVSROOT ci -m "Update from mirror"

# now we have to change the drupal settings to
# match working db
cd $WORKING_DIR
cd public_html/sites/default
sed -e s/"$MIRROR_DB"\'/"$WORKING_DB"\'/g settings.php > settings1.php
mv settings1.php settings.php

Tags:

Jon's blog
Login or register to post comments
Send to friend

GeologyRocks