Using MindTouch on Amazon EC2

1- Setting-up an Amazon EC2 account

If you don't already have an account for using Amazon Web Services, you will need to create one by connecting to : http://aws.amazon.com

Then you will have to sign-up for the service "Amazon Elastic Compute Cloud (EC2) - beta".

Read the information provided on the main page and then suscribe by clicking on :

now it's time to configure your client for using EC2

2- Amazon EC2 client configuration

Use the "Getting Started" guide to EC2 to configure your client.

Basically, you will need to :

- generate then download your X.509 certificate and private key from the AWS website.

- install the "Amazon EC2 command-line tools"

- setup your environment variables

- generate an ssh keypair to allow public key authentication with your instances.

For managing your EC2 instances, you might also consider using the EC2UI extension for Firefox.

3- Launching the MindTouch AMI

The MindTouch AMI is listed on the Amazon AWS website. All you need to launch it is the "instance-id".

to launch the instance from your client, run the command :

$ ec2-run-instances ami-xxxxxxxx -k your-keypair -t m1.small 

 and wait 2 minutes for the instance to boot. You can check the current state of your instance using the command :

$ ec2-describe-instances
RESERVATION     r-626d9c0b      366391106850    default
INSTANCE        i-ba6d94d3      ami-xxxxxxxx    ec2-67-202-63-xxx.compute-1.amazonaws.com  ... running ...

now you may want to allow connections to your instances, for example ports 22 (SSH) and 80 (HTTP)

$ ec2-authorize default -p 22
$ ec2-authorize default -p 80

You can now connect to http://ec2-67-202-63-xxx.compute-1.amazonaws.com/ to install your MindTouch.

You can also SSH your instance using public key authentication :

$  ssh -i your-keypair root@ec2-67-202-63-xxx.compute-1.amazonaws.com

Notice : when you want to stop your instance, use the "ec2-terminate-instances" command.

4- Updating your Amazon EC2 MindTouch installation

to update your MindTouch installation to the latest stable release, you can use the "updateWiki.sh" script.

$  ssh -i your-keypair root@ec2-67-202-63-xxx.compute-1.amazonaws.com
       
[root@domU-xx-xx:~] yum update dekiwiki

Notice : older AMIs are not package based so updateWiki.sh should be used. This script will stop the MindTouch service during the update and will generate a logfile under "/var/log/dekiwiki". 

5- Using "dataWiki.sh" to backup and restore your data

The main constraint using Amazon EC2 is the lack of "non-volatile" storage.  For that purpose you can use the "dataWiki.sh" script.

• In order to backup your data, SSH your instance and run :

[root@domU-xx-xx:~] dataWiki.sh -b
-- Dekiwiki data backup --
-- stopping services --
-- saving database --
-- saving configuration --
-- creating /mnt/backupWiki-2008-03-07.tar.gz --
-- backup completed --
don't forget to copy /mnt/backupWiki-2008-03-07.tar.gz to a safe place before shutting your EC2 instance down
-- starting services --

Note that the script will stop the MindTouch service during the backup and will generate a logfile under "/var/log/dekiwiki". This script is not interactive and can be run by cron or scripted.

• To restore your MindTouch data on a fresh instance :

- SSH your instance

- Copy your backup tarball to your instance (SCP, FTP, etc.)

- run :

[root@domU-xx-xx:~] dataWiki.sh -r /path/to/backupWiki.tar.gz 
-- Dekiwiki data restore from backupWiki-2008-02-27.tar.gz --
-- stopping services --
-- restoring Dekiwiki database --
-- restoring Dekiwiki configuration --
-- restore completed --
-- starting services --

Note that the script will stop the MindTouch service during the restore and will generate a logfile under "/var/log/dekiwiki".

Your MindTouch site is now restored.

EDIT : another way is to use the Amazon EBS feature which allows persistent storage on EC2 instances.

6- Transferring data between EC2 and S3.

You can transfer data between EC2 and your local network without problem as you have all the standard tools available (SCP, FTP, etc.). Moreover, SCP (or PSCP) is already configured to use public key authentication which makes it even easier.

For using Amazon S3, the MindTouch image provides the "s3cmd" and "s3sync" commands.

- Set your environment :

export AWS_ACCESS_KEY_ID=your_access_key_id
export AWS_SECRET_ACCESS_KEY=your_secret_access_key

- How to use "s3cmd" command :

[root@domU-xx-xx:~] s3cmd -h
s3cmd [options] <command> [arg(s)]              version 1.0.6
  --help    -h        --verbose     -v     --dryrun    -n
  --ssl     -s        --debug       -d     --progress

Commands:
s3cmd  listbuckets  [headers]
s3cmd  createbucket|deletebucket  <bucket>  [headers]
s3cmd  list  <bucket>[:prefix]  [max/page]  [delimiter]  [headers]
s3cmd  delete  <bucket>:key  [headers]
s3cmd  deleteall  <bucket>[:prefix]  [headers]
s3cmd  get|put  <bucket>:key  <file>  [headers]

- How to use "s3sync" command :

[root@domU-xx-xx:~] s3sync -h
s3sync [options] <source> <destination>         version 1.1.4
  --help    -h          --verbose     -v     --dryrun    -n
  --ssl     -s          --recursive   -r     --delete
  --public-read -p      --expires="<exp>"    --cache-control="<cc>"
  --exclude="<regexp>"  --progress           --debug   -d           
One of <source> or <destination> must be of S3 format, the other a local path.

Examples: (using S3 bucket 'bucket' and prefix 'pre')
  Put the local etc directory itself into S3
        s3sync  -r  /etc  bucket:pre
        (This will yield S3 keys named  pre/etc/...)
  Put the contents of the local /etc dir into S3, rename dir:
        s3sync  -r  /etc/  bucket:pre/etcbackup
        (This will yield S3 keys named  pre/etcbackup/...)
  Put contents of S3 "directory" etc into local dir
        s3sync  -r  bucket:pre/etc/  /root/etcrestore
        (This will yield local files at  /root/etcrestore/...)
  Put the contents of S3 "directory" etc into a local dir named etc
        s3sync  -r  bucket:pre/etc  /root
        (This will yield local files at  /root/etc/...)

Note that data transfers from EC2 to S3 are free of charge. For managing your S3 storage, you might also consider using the S3fox extension for Firefox.

Links :

- Official Amazon EC2 documentation

- Official Amazon S3 documentation