Migrating from MediaWiki to MindTouch

Running the Converter

1. Log into the MindTouch VM.

2.  cd /opt/deki/dist

3. Open mindtouch.deki.mwconverter.xml in a text editor and change it to match your installation:

• Change <startup-xml> element to point to the location of the mindtouch.deki.startup.xml file.  On a VM installation, use /etc/dekiwiki/mindtouch.deki.startup.xml.
• Change <uri.converter> element to point to the IP address or FQDN of the MediaWiki-Bridge.  To leverage MindTouch's public MediaWiki-Bridge, use http://mwc.mindtouch.com.  If your wiki contains private data that should not be sent over the internet, you will need to install a local copy of the MediaWiki-Bridge.
• Change <db-server> to the IP address or FQDN of your MySQL server containing the MediaWiki database.
• Change <db-port> to the port of your MySQL server containing the MediaWiki database
• Change <db-catalog> to MySQL catalog name for the MediaWiki database.  
• Change <db-user> to an account in your MySQL Database that can read the MediaWiki database
• Change <db-password> to the password of a MySQL account that can read the MediaWiki database
• Change <mwrootpage> to the path of a MediaWiki page that should be treated as the MindTouch home page. 
• Change <db-userprefix> and <db-prefix> to your MediaWiki database  prefix (if any), or blank if none.
• Change the <image-dir> to the location of your MediaWiki file attachment directory.  This directory must be accessible to your VM (copied to a local directory or via a file share mount).
   

Below is a sample mindtouch.deki.mwconverter.xml configured for a VM:

<?xml version="1.0" encoding="utf-8" ?>
<config>
  <deki>
    <!-- Path to mindtouch.deki.startup.xml -->
    <startup-xml>/etc/dekiwiki/mindtouch.deki.startup.xml</startup-xml>
  </deki>

  <!-- MediaWiki configuration settings -->
  <mediawiki>

    <!-- MediaWiki URL ex.  http://deki-hayes/config/mw/index.php -->
    <uri.converter>http://mwc.mindtouch.com</uri.converter>
    
    <!-- MediaWiki database connection information -->
    <db-server>localhost</db-server>
    <db-port>3306</db-port>
    <db-catalog>mediawikidb</db-catalog>
    <db-user>root</db-user>
    <db-password hidden="true">password</db-password>
    <db-options>pooling=true; Connection Timeout=5; Protocol=socket; Min Pool Size=2; Max Pool Size=50; Connection Reset=false;character set=latin1;ProcedureCacheSize=25;Use Procedure Bodies=true;</db-options>
    
    <!-- MediaWiki user table prefix.  Leave blank if the MediaWiki database tables do not use a prefix -->
    <db-userprefix></db-userprefix>
    <sites> 
      <site>

        <!-- MediaWiki table prefix.  Leave blank if the MediaWiki database tables do not use a prefix -->
        <db-prefix></db-prefix>
        
        <!-- Path of the MediaWiki home page.  All other pages will be placed under this page -->
        <mwrootpage>Main_Page</mwrootpage> 

        <!-- Path to MediaWiki images folder -->
        <imagedir>/mediawiki/files</imagedir>

      </site>
    </sites>
  </mediawiki>
</config>

4. Set execute permissions:  chmod +x mindtouch.deki.mwconverter.exe

5.  Stop  MindTouch:  /etc/init.d/dekiwiki stop

6.  Launch the converter:  ./mindtouch.deki.mwconverter.exe. 

WARNING:   Running the converter will replace your existing MindTouch data.

7.  Restart MindTouch:   /etc/init.d/dekiwiki restart

8.  Rebuild the search index

NOTE: Since MediaWiki does not support the notion of a page heirarchy, you may have a large number of pages under the home page. This flat structure can greatly increase the MindTouch navigation panel load time. To avoid this delay, you can either disable the MindTouch navigation panel or reorganize your site to have a page heirarchy.

To disable the navigation panel, edit /var/www/dekiwiki/LocalSettings.php and set the following: 

$wgNavPaneEnabled = false; 
$wgNavPaneCallEnabled=false; 

To reorganize your site to have a page heirarchy, use the Desktop Connector.

Installing the MediaWiki-Bridge

NOTE:  this procedure is only necessary if you cannot use http://mwc.mindtouch.com due to privacy concerns.

To install a private copy of the MediaWiki-Bridge:

• Log into the MindTouch VM.
• cd /var/www/deki-hayes/config
• svn co https://svn.mindtouch.com/source/pub...-bridge/trunk/ mw

Later, if you need to update the MediaWiki-Bridge files:

• Log into the MindTouch VM.
• cd /var/www/deki-hayes/config/mw
• svn up

Once the files for the MediaWiki-Bridge have been installed, follow the usual mediawiki installation procedure:

• Navigate to http://your-vm-ip/config/mw/config
• Fill out the form (call the new database "mediawikidb")
• Move the generated LocalSettings.php file from /var/www/deki-hayes/config/mw/config to /var/www/deki-hayes/config/mw.

To configure the <uri.converter> element in mindtouch.deki.mwconverter.xml, use http://your-vm-ip/config/mw/index.php.  Set your-vm-ip to either the hostname or IP address of your MindTouch VM; do not use 127.0.0.1 since it will cause subsequent conversion errors.

Running the Converter

C:\> cd \Documents and Settings\%USER%
C:\Documents and Settings\user> mkdir src
C:\Documents and Settings\user> mkdir src\dekiwiki 
C:\Documents and Settings\user> mkdir src\dekiwiki\trunk

<?xml version="1.0" encoding="utf-8" ?>
<config>
  <deki>
    <!-- Path to mindtouch.deki.startup.xml -->
    <startup-xml>C:\dekiwiki\mindtouch.deki.startup.xml</startup-xml>
  </deki>

  <!-- MediaWiki configuration settings -->
  <mediawiki>

    <!-- MediaWiki URL ex.  http://deki-hayes/config/mw/index.php -->
    <uri.converter>http://mwc.mindtouch.com</uri.converter>
    
    <!-- MediaWiki database connection information -->
    <db-server>localhost</db-server>
    <db-port>3306</db-port>
    <db-catalog>mediawikidb</db-catalog>
    <db-user>root</db-user>
    <db-password hidden="true">password</db-password>
    <db-options>pooling=true; Connection Timeout=5; Protocol=socket; Min Pool Size=2; Max Pool Size=50; Connection Reset=false;character set=latin1;ProcedureCacheSize=25;Use Procedure Bodies=true;</db-options>
    <sites> 
      <site>

        <!-- Path of the MediaWiki home page.  All other pages will be placed under this page -->
        <mwrootpage>Main_Page</mwrootpage> 

        <!-- Path to MediaWiki images folder -->
        <imagedir>C:\Temp\wiki-images\en</imagedir>

      </site>
    </sites>
  </mediawiki>
</config>

            Plug service = Plug.New(host.Uri.AtAbsolutePath("converter"), TimeSpan.MaxValue);
            DreamMessage m = service.PostAsync().Wait();
            if (!m.IsSuccessful)
                Console.WriteLine(m);
            else
                Console.WriteLine("Success!");
            Console.WriteLine("Press Enter to continue...");
            Console.ReadLine();
        }
    }
}

Installing the MediaWiki-Bridge

NOTE:  this procedure is only necessary if you cannot use http://mwc.mindtouch.com due to privacy concerns.

Create a directory where you want the modified MediaWiki files to be installed and fetch the source code from the Mindtouch Subversion server.  From a command prompt enter the following command, to create directory where you want the code installed (something along the lines of C:\Documents and Settings\%USER%\src\mediawiki-bridge is a reasonable choice, assuming %USER% is your Window user name) and fetch the source code:

C:\> cd \Documents and Settings\%USER%
C:\Documents and Settings\user> mkdir src\mediawiki-bridge

Open Windows Explorer, navigate to the C:\Documents and Settings\{user}\src\ directory, right click  on the mediawiki-bridge directory and select "SVN Checkout..." from the context menu.  When the TortiseSVN checkout dialog opens, enter the URL https://svn.mindtouch.com/source/public/mediawiki-bridge in the "URL of Repository" text box and click "OK".  Note:  Once you have performed this step the first time, the TortiseSVN content menu changes to "SVN Update" and you no longer have to enter the URL.

You will need to copy all of the modified MediaWiki PHP files into your MindTouch environment.  The default location is in a mw subdirectory in the MindTouch config directory.  The example below assumes you installed your local MindTouch in C:\var\www\dekiwiki.

C:\> cd \Documents and Settings\%USER%\src\mediawiki-bridge\
C:\Documents and Settings\%USER%\src\mediawiki-bridge\trunk> xcopy /E trunk c:\var\www\dekiwiki\config\mw
Does C:\var\www\dekiwiki\config\mw specify a file name
or directory name on the target
(F = file, D = directory)? d

You will need a LocalSettings.php file in C:\var\www\dekiwiki\config\mw (or wherever you installed your modified MediaWiki pages).  If you visit http://{IP address}/config/mw/config/index.php where {IP address} is the IP address of the conversion box (Note:  on my configuration, I had to use the IP address since localhost would redirect the traffic to the Dream server), you'll be presented with the MediaWiki setup page.  Enter the same values as your production MediaWiki installation, with the exception of the Database server (the database is not used during the conversion, so you can use your local copy on the MindTouch conversion box or create a new one).  Copy the resultant LocalSettings.php file from C:\var\www\dekiwiki\config\mw\config to C:\var\www\dekiwiki\config\mw (or wherever you installed your modified MediaWiki pages).

Open a browser and visit http://{IP address}/config/mw/index.php.  You should see "Hellow World!".  If you don't you'll need to fix your HTTP configuration. 

Note:  If you have an older MediaWiki installation, you may get a database error that complains about tables not found.  You can either return to step 1 (Prepare the MediaWiki Environment) and upgrade MediaWiki, or determine which tables are missing and manually create them now.  If you chose to determine which tables are missing, open the file C:\var\www\dekiwiki\config\mw\maintenance\tables.sql to see which table definitions the modified MediaWiki expects.  Open a MySQL prompt and execute

mysql> use {wikidb}; show tables;

(where {wikidb} is the name of your MediaWiki database instance).  Compare each table in the tables.sql file to the list of tables in MySQL.  Likely candidates will (probably) be at the bottom of the file.  If any are missing, copy and paste the table defenition command from the tables.sql file into a new file mediawiki_upgrade.sql.  Once you are finished, run the mediawiki_upgrades.sql file in MySQL with the following command:

mysql> source mediawiki_upgrade.sql

Mediawiki Templates

MediaWiki frequently incorporates cascading style sheets to provide a uniform layout to a set of pages.  Frequently they tend to appear in MediaWiki templates.  The MindTouch converter will convert the template, but may not handle the content as cleanly as you may like. 

First, the converter will add calls to the the mediawiki.args template.  This is a service provided by mindtouch.mediawiki.dll,.  You must build and install the mediawiki extension.

The remainder of this section catalogs some cleanup required when converting a fairly simple Mediawiki template.  It uses a fairly simple example that exposes a couple of things to watch out for.  In this example, the following MediaWiki template is used to format the contents of a file (grabbed from some other MediaWiki page back in 2005 (I've long since forgotten where):

{| WIDTH="75%" style="background-color: _cke_saved_style="background-color: _fckstyle="background-color: #f2fff2; border: solid 1px #bfffbf;" 
| style="background-color: #c1ffc1; border: solid 1px #a0ffa0" | <font size="-1">Code: {{{1}}}</font> 
|- 
|
{{{2}}}
|}

It expects two arguments:  the name of a file, and the file contents.  It can be invoked with the following Mediawiki syntax (this example shows an edit to be made to the /etc/conf.d/ntpd file:

 {{Box File|<b>/etc/conf.d/ntpd</b>|
 <nowiki># /etc/conf.d/ntpd
 
 # Options to pass to the ntpd process
 # Most people should leave this line alone ...
 # however, if you know what you're doing, feel free to tweak
 NTPD_OPTS="-u ntp:ntp"</nowiki>}}

and produces output that looks like the following:

The converter does a very good job at converting the template and properly created a template in MindTouch with the same colors and borders, however the sytax that calls the template may have errors.  For the  example above, the following output appeared after running the Mediawiki converter (for every revision of this article):

line 1, column 100: invalid Primary

The converter created the following:

{{ wiki.template('Box_File', mediawiki.args(['<b>/etc/conf.d/ntpd</b>', 
'<span class=\"plain\">' .. 
# /etc/conf.d/ntpd

# Options to pass to the ntpd process 
# Most people should leave this line alone ... 
# however, if you know what you're doing, feel free to tweak 
NTPD_OPTS="-u ntp:ntp" .. 
'</span>'])) }}

Note:  Line spacing has been added for clarity, the converter produced the output on two (very long) lines.  The content of the /etc/conf.d/ntpd file contains characters that were being interpreted by MindTouch.  To fix that, I had to enclose the argument in single quotes (the second argument falls between the two ellipsis, "..", and in the example above runs from the 3rd to the 8th line).  Unfortunately the content of the argument actually contains a single quote in the form of an apostrophe in the word "you're" (which is one reason it was selected as an example).  I modified it to the following:

{{ wiki.template('Box_File', mediawiki.args(['<b>/etc/conf.d/ntpd</b>', 
'<span class=\"plain\">' .. 
'# /etc/conf.d/ntpd\n
\n
# Options to pass to the ntpd process \n
# Most people should leave this line alone ... \n
# however, if you know what you\'re doing, feel free to tweak\n
NTPD_OPTS="-u ntp:ntp"' .. 
'</span>'])) }}

Note:  Line spacing has been added for clarity, as soon as you save the changes and edit it again, the arguments will be run together.  The following steps fix the error:

1. Manually enclose the argument (between the ".." ellipsis) in single quotes to prevent MindTouch from interpreting the template contents (the <nowiki> tag does the same thing in Mediawiki)
2. Escape any appostrophe characters or single quotes that appear in the argument's content
3. Replace newlines with "\n". 

This is a reasonable translation of the look and feel of the Mediawiki content.  There are some issues you need to be aware of, however:

• You may need to manually visit a subset of the pages that call a template, not just the template itself:
  • In many cases newlines are critical to the content of the template.  Mediawiki preserves newlines in template arguments, while in MindTouch you need to explicitly use '\n' syntax in the MindTouch tempalte arguments.  Unfortunately the mediawiki converter replaces newlines with spaces.  You can imagine how difficult it will be if the content is a capture of the Linux "make menuconfig" output cataloging all the options in a Linux kernel configuration, all run together on one line (yes, I have some of those).
  • MindTouch may try to interpret characters in the arguments to the template, causing an error message.  You may need to quote the arguments, and escape any apostrophe characters that appear in the argument (by preceeding them with a backslash ('\') character.
• Even after editing the page, prior revisions (from before the Mediawiki conversion) will no longer look right.
   

This is a very simple use of Mediawiki templates.  They can easily get much more complex. 

Tags

Before troubleshooting tags, make sure you rebuild the search index.  Login to your newly converted MindTouch site as your administrator, and visit "Control Panel" --> "Site Settings" --> "Rebuild Site Index >>".  This may take a few minutes to complete.

After running the converter and viewing the converted pages, ensure that each has the tags expected listed at the bottom of the page. Click on the tag names to ensure that all the pages have the desired tag.  In my case, however, MindTouch frequently showed fewer page titles than expected or none at all (the message "There are no pages that are tagged xxx in this language."  was displayed).

What made it stranger was that if I created a MindTouch tag definition page (user the tag "define:xxx"), the full set of pages showed up under "Related Pages for 'xxx'", it was only on the "Site Tags" page where results were omitted.

After munging around in the MySQL database, I found that a number of pages ended up being owned by user ID 0 (page_user_id = 0), which is non-existant.  Here's a list of the MindTouch users table after the migration:

mysql> select user_id, user_name from users;
+---------+-----------+
| user_id | user_name |
+---------+-----------+
|       3 | Anonymous |
|       2 | Moderator |
|       1 | Admin     |
+---------+-----------+

Only those pages with page_user_id = 0 didn't show up in the Site Tags list (http://{FQDN}/Special:Tags?tag=xxx) when you clicked on a tag name.  Selecting a new owner fixed the problem:

mysql>  UPDATE pages SET page_user_id = 2 WHERE page_user_id = 0;

Note:  I picked page_user_id = 2 because that was a moderator on my Mediawiki site.

After restarting my Dream server, all my tags were available.  Woo Hoo!!!