Best Practices

In the spirit of open-source software, I'm going to post here the tips and tricks I've learned about how to manage Web sites.  I hope you find them useful!

Moving a site to another hosting company

Alas, a good hosting company is hard to find.  All too often, we find that the proverbial grass is greener in another pasture, and it's time to move.  But a modern Web site is a dynamic thing, being edited by lots of people.  How do you make sure that the most recent version gets copied?  Here are the steps I've developed:

  1. Set up the control panel on the new site to match the old site.  Start with the domain names, subdomains, SSL certificates, and databases, leaving the other steps for later.  This is a good time to ask the client (or your own users) for any passwords to email accounts and maling lists, since people can sometimes be slow to get you that information.
  2. Make a preliminary copy of all the files and databases, for testing purposes.  The easiest way to do this is from the command line:
    1. (on the old site) tar -czf files.tar.gz public_html
    2. mysqldump -u username -p --all_databases > databases.sql
    3. ftp newIPaddress
    4. put files.tar.gz
    5. put databases.sql
    6. (on the new site) tar -xzf files.tar.gz
    7. mysql -u username -p < databases.sql
    If you don't have access to the command line,use an FTP program like FireFTP to download the entire public_html directory to your computer, and then use phpMyAdmin to export each of the databases to files on your computer.  Then reverse the steps to put the files and databases on the new server.
  3. Download a copy of the files.tar.gz file (from step 2.1 above) to your own computer, and unpack it.  You will use this in a later step.  If you don't have access to the command line, use an FTP program like FireFTP to download the entire public_html directory to your computer.  (This may take some time.)
  4. Change the hosts file on your own computer so that the domain name points to the new server's IP address.  This helps greatly in testing, because you can use the domain name in your browser instead of an IP address.  Some software's clean URLs and other features will not work properly when accessed with an IP address.
  5. Change the configuration files on the new site to accommodate the new database names and usernames, since these usually differ from one host to the next.
  6. When you are satisfied that the site is working properly, notify the client (and/or your own users) of when you intend to move.  Try to pick a low-traffic time of day or a weekend -- study your access statistics.  Allow at least 3 hours of uninterrupted time to make the move final.  This is also a good time to notify the client (and/or users) of the Webmail addresses where they can get their mail on the old and new servers -- this means IP addresses, since the domain name will point to only one of the two places.
  7. Before the time arrives, finish copying any control panel settings you haven't already copied.  This includes email accounts, forwarding addresses, and mailing list settings (including their membership rosters), but not the archives of mailing lists -- generally you don't have access to these and have to ask support.  This is a good time to ask.
  8. At the appointed time, lock down the site from further editing.  In Drupal this is quite easy -- just go to Admin -> Site Maintenance and set it to Off-line.  In PmWiki it is more difficult to do... probably the easiest way is to edit the template or SiteHeader page and make a prominent note saying not to make any changes, and just rely on people to do the right thing.
  9. Use a good FTP program with a synchronization feature, like FireFTP, to connect to the old server and synchronize your copy of the files, downloading only the files that are new or have changed.  Then connect to the new server and synchronize again, this time uploading only the files that are new or have changed.  This is much quicker than downloading another complete copy of the site.
  10. Export and import the databases again, either using the command line steps above or phpMyAdmin.
  11. Repeat step 5 above to correct the usernames in any configuration files.
  12. Change the DNS (domain name server) entries for the domain name(s), to point to the new host.  Be aware that the change will take some time to propagate across the Internet.
  13. Unlock the site on the new server, undoing any changes you made in step 8.
  14. Remove the lines from your hosts file (from step 4) so that you see what others will see.

You may ask, why copy the site twice instead of just once?  Because something unexpected will almost certainly go wrong, usually at step 5, and you want it to happen before you have set a deadline for the move, not after.

Why Drupal?

Much has been written already about why to use Drupal, and a little about why not to:

Here are my own two cents.  I taught myself the PHP programming language during slow times at work in 2002-04, and used it to build some database front-ends.  In 2005, I wrote my own content-management system (CMS) from scratch, without having researched the fact that there were already such programs available.  Then I was hired by PSDUUA, and they wanted me to use PmWiki, so for all of 2006-07 I was using PmWiki almost exclusively -- whenever a client wanted a site, I gave them a wiki.  It's a very solid CMS and a powerful programming platform, as far as it goes, and I could teach anyone to use it in under an hour.  I wrote six modules (called "recipes") and contributed them to the project.

The trouble with PmWiki was, clients kept outgrowing it.  They wanted features like blogs and forums and media galleries and shopping carts that PmWiki is not really designed for and doesn't do very well.  The developer community behind PmWiki, while fairly active, is small (maybe a hundred programmers), so if someone drops the ball on a particular recipe, no one may ever pick it up again.  Up until this month (January 2009) the core PmWiki program itself depended on one person, who was unavailable for most of 2007 and 2008.  And I can't really criticize, because I myself became unavailable in 2008 and stopped maintaining the recipes I'd contributed.  In any case, by the end of 2007 I was feeling disenchantd with PmWiki, and I'd heard great things about Drupal, and all I needed was an excuse to learn it.

In 2008 I worked for Smartphone & Pocket PC magazine, and they paid me to learn Drupal and sent me to DrupalCon.  It was just the opportunity I was looking for.  DrupalCon was a blast and just confirmed my decision to devote myself to Drupal: it's a going concern.  There are thousands of programmers around the world working to make Drupal and its modules better every day.  That's more programmers than work on Microsoft Office, I believe.  What that means is that any security problems are fixed within hours, and whatever new trendy features a client may want are usually available before they ask.

There are disadvantages, of course. 

On the whole, though, in my opinion Drupal is a good foundation for any new Web site, because even if the site doesn't need it now, it is likely to later.  It's better to start with a strong foundation than rebuild later.  And since I'm now proficient with Drupal, I can install and configure it in the same amount of time as I formerly took to install and configure PmWiki, or build static HTML pages, or write PHP code.

OK, but why not Joomla! or Wordpress?  Drupal is currently the 3rd most popular open-source CMS, after these two behemouths.  Why am I championing the (relative) underdog?  Well, partly because I have so much invested in it already.  But also because, after seeing the community at work at DrupalCon 2008, and grasping where the project is headed, I fully expect Drupal 7 (due later this year) to surpass all other CMSes in capabilities and user-friendliness, and then in popularity.  The easiest way to prepare a site for Drupal 7 now is to embrace Drupal 6.

Always assume multisites!

(Note: this article is about Drupal, but the same philosophy applies to other content-management systems that have multisite capability, such as PmWiki's WikiFarms feature.  In order to use multisites, you will need to have a hosting account that allows you to keep more than one domain or subdomain in the same account, so that they can access the same files.  Generally speaking, PHP cannot run files that are in another account.)

There are differing schools of thought about whether it's better to have a separate Drupal installation for each site (for example, example.com and cart.example.com), or to use its built-in multisite capability.  In my experience it is better to assume from the start that you may serve multiple sites from a single installation, because even if you never do, you will at least keep your site's files cleanly separate from Drupal's files, which makes upgrading or moving between hosting accounts much easier.  And if you do wind up hosting multiple sites, you will save a ton of time doing upgrades in one place instead of several, and all your sites' performance will improve from server-side caching of the PHP files.

Let me explain what I mean.  When you do a fresh install of Drupal 6, a settings.php file is created for you at drupal/sites/default/settings.php, and a files directory is created at drupal/sites/all/files .  (In Drupal 5, you had to create the file and directory manually, which meant that you might well put the files directory at drupal/files -- outside the sites directory -- but fortunately this has been corrected in more recent versions.)  Like this:

  • drupal
    • sites
      • all
        • files
        • modules
        • themes
      • default
        • settings.php

The default directory is for when no more specific settings.php file is available for the domain name you're using, and the all directory is available to all sites regardless of what domain name you're using... in other words, if you decide to use the same Drupal installation to serve cart.example.com, you would put its settings.php file at drupal/sites/cart.example.com/settings.php, and the example.com site would continue to use the one in the default directory, like so:

  • drupal
    • sites
      • all
        • files
        • modules
        • themes
      • cart.example.com
        • settings.php
      • default
        • settings.php

Both sites would be able to access the same files, which can be handy, but presents problems if you're not sharing users between the sites, because two users with the same ID number could potentially overwrite each other's files!

But the main problem with this setup is that if you later decide to serve a site with another domain name such as demonstration.com, it's no longer clear what the "default" means.  Is "default" example.com or demonstration.com?  Well, it's whatever isn't otherwise specified.  You can move the files into a subdirectory, but you will have to run a series of queries on the database -- potentially on four or five tables -- to change where Drupal looks for all the files.  You may also have to change your theme templates, if you referenced images that were in the files directory.

Instead, I highly recommend creating a directory for the domain name from the start: before you run Drupal the first time, simply create a directory at drupal/sites/example.com.  Drupal will find it and put the settings.php and files directories there instead of in its default locations.  All files specific to example.com will be kept in this one location.  (You're on your own to make sure that themes and modules specific to the site are kept in the right directory.) 

  • drupal
    • sites
      • all
        • modules
      • example.com
        • settings.php
        • files
        • themes
      • cart.example.com
        • settings.php
        • files

If you decide to upgrade this site separately from the others in your account (for example, because another site relies on modules that aren't yet available for the new version of Drupal), you can just move that one directory from the one drupal directory to the other, without worrying about leaving any files behind. Of course you'll also have to copy any modules that it may need from the drupal/sites/all/modules directory ... there are too many advantages to sharing modules among sites to go putting them in the site-specific directories.

  • drupal-5
    • sites
      • all
        • modules
      • cart.example.com
        • settings.php
        • files
  • drupal-6
    • sites
      • all
        • modules
      • example.com
        • settings.php
        • files
        • themes

Similarly, if you need to move a site to a different hosting account, you can move its files without affecting any of the other sites.  This is very handy if you happen to be a Web developer for multiple clients.

Installing and Upgrading Drupal and its Modules from the Command Line

The Drupal project has fond hopes of one day producing the most user-friendly content-management system in the world.  However, so far it's written by and for programmers, so often the easiest way to install and upgrade it is from the UNIX command line!  Here's how to do it:

To install Drupal on the command line:

  1. in a Web browser, go to http://drupal.org/project/Drupal+project , right-click the Download link for the version you want, and copy the link location.
  2. In an SSH (or Telnet) session to your host account, type
    cd www
    to go to your document root.  (Note: your document root may be called something else, like public_html.)
  3. Type
    wget 
    and paste the link location from step 1, then press Enter.  The file will download to your account, for example drupal-6.9.tar.gz.  "wget" means "get from the Web."
  4. Type
    tar -xzf drupal-6.9.tar.gz
    (or whatever the file name was from step 3) to expand the file into a directory, in this case drupal-6.9.  Think of "-xzf" as standing for "extract zip file."  Think of "tar" as what you use to stick a bunch of files together!
  5. At this point many people would rename the directory to something nicer, like drupal.  I recommend instead making a symbolic link (shortcut) to it, like so:
    ln -s drupal-6.9 drupal
    so that you can keep multiple versions in your account at once without getting confused as to which is which.  "ln -s" stands for "link symbolically"
  6. It is now safe to delete the file you downloaded, like so:
    rm drupal-6.9.tar.gz
    "rm" stands for "remove."
  7. Go to your new URL in a Web browser and follow the directions there.

To upgrade Drupal on the command line:

  1. Log onto your Drupal site with the admin account.  You will need to be admin in order to run the update.php script below.
  2. Follow steps 1-4 above to get the most recent version and expand it.
  3. Copy the sites directory from your existing installation into your new one:
    cp -R drupal-6.8/sites drupal-6.9
    assuming 6.8 is your existing installation and 6.9 is the new one.  "cp -R" stands for "copy recursively," that is, including the contents of all subdirectories.  If you have not put all of your site's files in the sites directory, you may have to repeat this step for files, modules, themes, etc.  It is a very good practice to always put all of your site's files in the sites directory for this reason.
  4. Back up your database to a file, like so:
    mysql -u username -p databasename > drupal-6.8.sql
    where username and databasename are your database name and username, and 6.8 is your existing version.  This will take the contents of databasename and save them to a file called drupal-6.8.sql.
  5. Redirect the symbolic link to point to the new version, like so:
    rm drupal; ln -s drupal-6.9 drupal
    assuming drupal is the symbolic link you created in step 5 above and drupal-6.9 is the new directory you want it to point to.  The semicolon between the two commands prevents any delay between removing the old link and creating the new one.
  6. Run the update.php script, which will be at http://www.yoursite.com/drupal/update.php (where yoursite.com is your domain name).
  7. Test your site to make sure it is running correctly.  If you don't see any problems in a day or two, proceed.
  8. Make a compressed backup of your old installation, like so:
    tar -czf drupal-6.8.tar.gz drupal-6.8
    (where drupal-6.8 is your old version)  Think of "-czf" as standing for "create zip file."
  9. Delete your old installation.  Because some of the files are write-protected, this takes two commands:
    chmod -R u+w drupal-6.8
    rm -R drupal-6.8
    "chmod u+w" means "change the modifiers so that the user can write."  "-R" in both case means "recursive," that is, including all subdirectories.

To install a module or theme on the command line:

  1. in a Web browser, go to the module's page at drupal.org , right-click the Download link for the version you want, and copy the link location.
  2. In an SSH (or Telnet) session to your host account, type
    cd www/drupal/sites/all/modules
    to go to your modules directory.  (Note: do not put modules in the drupal/modules directory -- that's only for core modules!)  Themes go in www/drupal/sites/all/themes instead.
  3. Type
    wget 
    and paste the link location from step 1, then press Enter.  The file will download to your account, for example views-6.x-2.2.tar.gz .
  4. Type
    tar -xzf views-6.x-2.2.tar.gz
    (or whatever the file name was from step 3) to expand the file into a directory, in this case views.
  5. It is now safe to delete the file you downloaded, like so:
    rm views-6.x-2.2.tar.gz
  6. Go to your admin/build/modules page in a Web browser and enable the module.  Or admin/build/themes for a theme.

To upgrade a module or theme on the command line:

  1. Follow steps 1-3 above to download the file to the proper directory.
  2. Make a backup of the existing module, like so:
    tar -czf views-6.x-2.1.tar.gz views
    where views is the module and 6.x-2.1 is the version you are currently using.  You may also want to back up your database by following step 4 of the "upgrade Drupal" instructions above.
  3. Type
    tar -xzf views-6.x-2.2.tar.gz --overwrite
    (or whatever the file name of the file was) to expand the file into the existing module or theme's directory, in this case views, overwriting any files that already exist.
  4. It is now safe to delete the file you downloaded, like so:
    rm views-6.x-2.2.tar.gz
  5. Run the update.php script, which will be at http://www.yoursite.com/drupal/update.php (where yoursite.com is your domain name).
  6. Go to admin/by-module on your site and check the settings for the module you just upgraded, in case they have changed.

Use Google Maps to map Drupal content

Suppose you have some content in your Web site that is geographical in nature, like photos from a trip, or users, or carpools to an event, or a portfolio of clients.  How do you map that information?  In Drupal, it's pretty easy to set up, and very easy to use once it's set up.  Here are the steps:

Setup tasks

  1. Download and install the Location and GMap modules in your /drupal/sites/all/modules directory.
  2. In your site's admin/build/modules page, enable Location, Gmap, and Gmap Location.
  3. Go to admin/settings/location, Enable the Display of Locations, and check Use a Google Map.
  4. Go to http://code.google.com/apis/maps/signup.html and get an API key for your URL.
  5. Go to admin/settings/gmap, paste in your API key, scroll to the bottom of the form, and save the configuration.  (You must do this before the other configuration options will work.)
  6. On the same page, set your Default Map Settings -- these affect the map you see when setting a location for a node.  If you don't understand a setting, ignore it.
  7. Go to admin/settings/gmap_location and adjust the settings for your user and node maps.  If you're not going to use one or the other, just ignore it.
  8. Go to admin/user/permissions and set the permissions for the gmap_location and location modules.  In particular, "show node map" and/or "show user map" must be enabled for a user to see the maps in question.  "user location" means that when a point on the user map is clicked, info about that user will show up; if you leave this unchecked, only locations will be shown without personal info.
  9. Go to admin/content/types, edit each content type that you want to map, and set the Locative Information settings.
  10. Go to admin/build/menu and edit the Navigation menu items for Node Locations and User Locations.

Mapping your content

  1. Edit a node, or create a new one, of a type that you're mapping.
  2. Under Location, click the map to add a marker to it.  Pick up and drop the marker repeatedly to zoom in so you can place it more precisely.  (This is a lot quicker than zooming and panning.)
  3. Save your changes, and go to map/node to survey your handiwork.

Block spam with Mollom

Before I installed Mollom on my sites, spam was a continual threat.  Not only were email forms at risk of being exploited, flooding their recipients with hundreds of unwanted messages a day, but comment spam could take over a site, covering useful pages in garbage or actually coopting the browser with iframes and other dangerous HTML tags.  I fought back as best I could with captchas and input filters, but it was an arms race I wasn't likely to win.

Then at DrupalCon I heard about Mollom, and I put it into practice, and since then I have been able to open up comments to unregistered users and relax input filters with no worries at all. Mollom doesn't show a captcha unless you have already called your humanity into question, so the majority of users never see a captcha.

Let me give you an idea what volume of spam I'm dealing with.  Here are the Mollom statistics for smartphonemag.com for the past 32 days:

Mollom spam-blocking statistics for smartphonemag.com

If the graphic above were as interactive as the actual stats (which are Flash), you could mouse over the graph to see each data point.  The maximum number of "ham" (valid) posts during this month was 35, the most I see on any of my sites.  Mollom will process up to 100 valid posts a day for free.  So this is the free service we're using here, and it blocked nearly 2900 spam messages in a single day, from this one site alone.  That is an unbelievable value! Even at the paid rate of 1 Euro per day, it would still be an unbelievable value.

The one downside of the free service is that if the Mollom server goes down, you have to make a choice: stop receiving any posts on your site, or let them all through.  We have chosen to let them through, and the server did go down once, and about 500 spam comments made it onto the site.  But before Mollom we were tackling spam every single day.  In the time we've used it, we've had a total of 3 false negatives (i.e. spam messages passing as valid).  We deleted them and reported them to Mollom, and no messages like them have made it through since.

I highly recommend Mollom to anyone with a Web site that accepts any kind of public feedback.

Drupal 7: the road ahead

A number of you are my friends on Facebook and may have wondered why I got so excited a few days ago about DrupalCon Paris (which I'm not attending -- it's happening right now) and the accompanying "code freeze" of Drupal 7.  Without getting too technical, here are some of the recent developments and what they mean for you as Drupal users.

  1. Only two versions of Drupal are maintained at a time.  Currently that's 5 and 6, but as soon as Drupal 7 is released, Drupal 5 will be discontinued.  All the sites I work on are already running Drupal 6 except for SmartphoneMag.com and its Ubercart site... in the upcoming months I'll be working hard to move them to Drupal 6 so that they won't be running unmaintained software.  (The danger of running unmaintained software is if a bug is found, it may not get fixed.)  The timeline of the Drupal 7 release is detailed in a recent blog post by founder Dries Buytaert.
  2. Drupal 7 is the first version to emphasize user-friendliness.  Seriously, it hasn't been a priority until now.  But two years ago, the then-new Drupal 6 was subjected to a merciless bout of usability testing at three major universities, and it failed very embarrassingly, and the Drupal core developers resolved that that wouldn't happen again.  I'm itching to get my hands on Drupal 7 to see how much more friendly it is.  Technical details on the changes in Drupal 7 can be found in my notes from a presentation in March.
  3. Dries's keynote speech from Paris hasn't been released on the Web yet, but according to his blog he announced that his for-profit company, Acquia, will soon offer Drupal hosting services.  For small sites (like most of yours), they will offer a Drupal-7-based service called Drupal Gardens, and for large sites (like iPhoneLife.com) they will offer a high-performance, "cloud-based" (distributed) architecture called Acquia Hosting.  Since hosting is a perpetual concern for any site, I'll keep a close eye on these services so I can give you professional advice.

I'll be among the first to download Drupal 7 when it goes alpha, and I'll keep you posted!