How To Setup MogileFS

refer: http://durrett.net/mogilefs_setup.html

How To Setup MogileFS

Version: 0.04, October 7, 2006

Author: Brett G. Durrett (first name at last name dot net)

Source: http://durrett.net/mogilefs_setup.html

Note: These directions were written for MogileFS < 2.0 – I have not done a 2.0 install but I have been told by very good sources that 2.0 is the way to go now as it is backward compatible (making sure you don't update the schema if you are already running < 2.0 in a mixed environment) and it is in production by LiveJournal (oh, and it is better code with better features).

Overview

This document explains how to setup and configure a basic MogileFS installation. It is intended for the novice systems administrator and should enable anybody with the skills to install a basic Linux distro to get MogileFS up and running on it.

I am going to assume you have three roles for this setup. A machine should be able to handle more than one role. I will use hosts mogiledb.yourdomain.com, mogiletracker.yourdomain.com and mogilestorage.yourdomain.com – rename these to meet your needs.

Getting MogileFS

If you look around you can find MogileFS directly on the Danga website, in an abandoned CVS repository and in the current SVN repository. If you pull from the website you will not have admin tools to make setup easier. If you pull from CVS you will have a version from October, 2004. If you pull from SVN you will have a shiny new version that works best with this documentation.

Using svn (http://subversion.tigris.org/ ), checkout the latest and greatest version (the example below pulls from the trunk).

$ mkdir mogilefs-src

$ cd mogilefs-src

$ svn checkout http://code.sixapart.com/svn/mogilefs/trunk

Installation

Creating a Database

Setting up a MySQL server is beyond the scope of this document – there are packages available for most distributions, find one that suits your needs. Once you have a MySQL server up and running on host mogiledb.yourdomain.com, create a table and user for MogileFS. Some libraries don't play well with new MySQL passwords – if you use these, set the password using the “OLD_PASSWORD” function. Make sure you change the password to something better than the example.

# mysql

mysql> CREATE DATABASE mogilefs;

mysql> GRANT ALL ON mogilefs.* TO 'mogile'@'%';

mysql> SET PASSWORD FOR 'mogile'@'%' = OLD_PASSWORD( 'sekrit' );

mysql> FLUSH PRIVILEGES;

mysql> quit

You will also need to create the schema – that is covered later in this document.

Setting up the Trackers and Storage Servers

You need to obtain and install the MogileFS-specific perl modules and their dependencies before MogileFS will install successfully. Hopefully you pulled the MogileFS source from the SVN repository – if so, the server components will be located in the trunk/server directory. You probably need to install dependent modules first (see below) but once all dependencies have been met, you can install Mogile with the following commands (starting in the top of the SVN directory you pulled):

# cd trunk/server/

# perl Makefile.PL

# make

# make test

# make install

If you get any errors during this process it will probably be errors telling you that a dependent module is missing. If during the 'make test' step you get the error, "t/00-startup....DBI connect('mysql','root',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at t/lib/mogtestlib.pl line 16" it can probably be ignored – it should not be necessary for MySQL to be running on any host other than mogiledb.yourdomain.com.

You probably want to install some helpful utilities on each tracker or storage server as well (these will be needed for later configuration). These are located in the trunk/utils directory and can be installed with the following commands (starting in the top of the SVN directory you pulled):

# cd trunk/utils/

# perl Makefile.PL

# make

# make test

# make install

You also want the API – the utilities will require this. These are located in the trunk/api/perl directory and can be installed with the following commands (starting in the top of the SVN directory you pulled):

# cd trunk/api/perl

# perl Makefile.PL

# make

# make test

# make install

Installing Perl Module Dependencies

As for Perl modules, you need (at least) Perlbal and Danga-Socket and any dependencies. If you are not sure how to locate the dependent modules, consider using CPAN ( http://cpan.org/ ) as it can install a module and all of the dependencies. If you already have a reasonable Perl installation you probably have most of the non-MogileFS modules already.

The basic process for installing most perl modules is:

extract the module:

# tar -xvzf Linux-AIO-1.9.tar.gz

Create the makefile:

# cd Linux-AIO-1.9

# perl Makefile.PL 

Build the code:

# make 

Test the code (some modules may not have this step but it is okay to run anyway):

# make test 

Install it:

# make install

If you see any errors stating that a module dependency is missing, obtain and install the missing module first and try again.

Setup

Database Configuration

The database is empty and will need a schema applied. The ' trunk/server' directory has a utility named 'mogdbsetup' to make this process simple. By default it assumes the database is located on localhost so if you are running it from a different host you will need to provide the host name on the command line.

# ./mogdbsetup --dbhost=mogiledb.yourdomain.com --dbname=mogilefs --dbuser=mogile --dbpassword=sekrit 

Again, make sure you replace the host and password so that they match you database configuration from above.

The mogdbsetup utility does not specify a table type by default so your tables will match the defaults for your database. In many cases this will mean that you end up with MyISAM tables. If you prefer InnoDB tables you will either need to make sure your database defaults to InnoDB or you can manually convert the tables (both of these are outside of the scope of this document but there are plenty of examples out there).

Tracker Configuration

On each tracker server (mogiletracker.yourdomain.com), create a configuration file at /etc/mogilefs/mogilefsd.conf with the following:

db_dsn DBI:mysql:mogilefs:mogiledb.yourdomain.com

db_user mogile

db_pass sekrit

conf_port 6001

listener_jobs 5

db_dsn points to your database instance. If you are running the database on the same machine as the storage server you can omit ":mogiledb.yourdomain.com: and it will use the local machine. db_user and db_pass should match the user and password you configured when setting up your database.

The program 'mogilefsd' will not run as root so you will need to run this as a non-root user. To create a user for this, enter the following command and follow the prompts to create the "mogile" user:

# adduser mogile

In order to use the tools to setup the storage servers you will need to have the trackers running. Refer to "Starting Trackers", below.

Storage Server Configuration

On each storage server, create the storage directory (make sure it has access permissions for the user you will use to run mogstored):

# mkdir /var/mogdata

Configure it:

On each storage server, create a configuration file at /etc/mogilefs/mogstored.conf with the following:

httplisten=0.0.0.0:7500

mgmtlisten=0.0.0.0:7501

docroot=/var/mogdata

Use 'mogadm' to add each storage server to the database. This requires that the trackers are already running so if you have not already started them, refer to " Starting Trackers ", below. You need to supply the Perl lib path which has the 'MogileFS.pm' perl module installed – this was installed if you installed the API in the "Setting up the Trackers and Storage Servers" section above. The following example would add the host mogilestorage.yourdomain.com as a storage server, assuming that mogilestorage.yourdomain.com had an IP address of 192.168.42.3 (listening on port 7500) and your tracker had an IP address of 192.168.42.1 (listening on port 6001):

# mogadm --lib=/usr/local/share/perl/5.8.4 --trackers=192.168.42.1:6001 host add mogilestorage --ip=192.168.42.3 --port=7500 --status=alive

You can confirm that your host(s) were added with the following command;

# mogadm --lib=/usr/local/share/perl/5.8.4 --trackers=192.168.42.1:6001 host list

You also need to add devices for each storage host. If you don't provide a device id mogadm is supposed to do it for you... in practice I find it dies with an error, so you will need to manually add a unique device id after the host:

# mogadm --lib=/usr/local/share/perl/5.8.4 --trackers=192.168.42.1:6001 device add mogilestorage 1

Finally, add the correctly-name device (folder) to each storage host. I have been unable to get the tools to handle this well, so I am probably doing something wrong. As a workaround, I used the modadm device list command to see what device names were assigned and then I added the folders to my storage hosts. Run the following command:

# mogadm --lib=/usr/local/share/perl/5.8.4 --trackers=192.168.42.1:6001 device list

It will list each host and the device name followed by its status and storage available. Here is example output:

mogilestorage [1]: alive

used(G) free(G) total(G)

dev1: alive 0.892 67.772 68.664

This means "mogilestorage" has a host id of "1" and it has one device named "dev1" on it and that device is in the "alive" state (your other statistics will probably be zeros). Using the example output above, you would simply create the directory on mogilestorage.yourdomain.com:

# mkdir -p /var/mogdata/dev1

Finally, confirm your devices are configured:

# mogadm --lib=/usr/local/share/perl/5.8.4 --trackers=192.168.42.1:6001 device list

Running MogileFS

Starting Storage Servers

Start each storage server (mogilestorage.yourdomain.com) by running the following command as root:

# mogstored --daemon

Starting Trackers

Trackers will not run as root so you will need to run them as another user. If you created the "mogile" user when seetingup the trackers, the following commands will work (assumes you start logged in to mogiletracker.yourdomain.com as root):

# su mogile

$ mogilefsd -c /etc/mogilefs/mogilefsd.conf --daemon

$ exit

You can confirm that the trackers are running with the following command:

# ps aux | grep mogilefsd

If you don't get a list of running processes the trackers are not running.

Try It

Do a Quick Sanity Test

The 'mogadm' tool can be used to make sure your trackers are functioning. You need to supply the Perl lib path which has the 'MogileFS.pm' perl module installed – this was installed if you installed the API in the "Setting up the Trackers and Storage Servers " section above. The following example would check all mogile components using the trackers at IP address 192.168.42.1 and 192.168.42.2, both listening on port 6001:

# mogadm --lib=/usr/local/share/perl/5.8.4 --trackers=192.168.42.1:6001,192.168.42.2:6001 check

Try it with Real Data

Create a domain:

# mogadm --lib=/usr/local/share/perl/5.8.4 --trackers=192.168.42.1:6001,192.168.42.2:6001 domain add testdomain

Add a class to the domain:

# mogadm --lib=/usr/local/share/perl/5.8.4 --trackers=192.168.42.1:6001,192.168.42.2:6001 class add testdomain testclass

Troubleshooting

This section is still very incomplete. Let me know if you have other common problems that need to be added.

When I run mogadm I get "Unable to retrieve host information from tracker(s)":

mogadm requires the tarckers to be running before it is run.

When starting the storage daemon I get "ERROR: Directory not found for service mogstored":

You did not create a storage directory or you are starting the mogstored as a user that does not have access to the directory.

Problems Connecting:

Make sure your firewall is open. Using the examples in this document, port 7500 and 7501 needed on storage servers, 6001 on trackers.

While testing I get “MogileFS::Backend: couldn't connect to mogilefsd backend at /usr/local/share/perl/5.8.4/MogileFS.pm line 56”:

Make sure your tracker connects to the database:

# su mogile

$ mogilefsd -c /etc/mogilefs/mogilefsd.conf

I get a "REQUEST FAILURE" on "Checking devices..." when doing a mogadm check:

Confirm the devices (folders) exist in the /var/mogdata directory and that the use running the mogstored process has full permissions to these directories. If the device does not exist, add it – it will take a few seconds for mogadm check to reflect the fixed directory. For example, if mogilestorage.yourdomain.com had the device "dev1" on it, you would add the directory:

# mkdir -p /var/mogdata/dev1

Document Revision History

2006-10-07 Version 0.04 Brett G. Durrett: changed two typos of /vag to /var, added plug for 2.0

2006-07-03 Version 0.03 Brett G. Durrett: highlighted need to start trackers before using mogadm (and added troubleshooting reference), formatting changes.
2006-06-26 Version 0.02 Brett G. Durrett: minor typo corrections and fixed characters trashed in converting document to html
2006-06-25 Version 0.01 Brett G. Durrett: created initial document