Cluster PVFS2
From Bootable Cluster CD
Setting up a PVFS2 cluster on the BCCD is very similar in many ways to setting up a PVFS2 server on a single host and also closely follows the quickstart guide.
There are two types of servers necessary in a PVFS2 cluster: metadata servers and I/O servers. A particular node in the cluster can be one or both types of servers. A metadata server keeps up with metadata (such as permissions and time stamps) for the file system. An I/O server actually stores a portion of the PVFS2 file data. It makes sense to make all BCCD nodes I/O servers, and depending on the size of the cluster, at least a few also as metadata servers.
A machine can also be a client. A client machine is a node that can read and write PVFS2 files. It also makes sense to make all BCCD nodes clients, so that all of them may access the file system.
Here's the steps to building your very own PVFS2 cluster on the BCCD:
1. Boot up the BCCD
Do this in all the machines you would like to use in your cluster (see Booting up the CD). In the below example, I'm setting up four nodes: costarican, guatemalan, aulait, and mocha (yes, they are all coffee names). I'm setting all of them up as I/O servers, and I'm setting up costarican and mocha as metadata servers.
2. Log in as root
The root password should be letmein or on the login screen.
3. Allow the user root ssh access on all nodes
For an explanation of how to do this, go to Customization Tips and Tricks: Allowing SSH Access to the User Root.
4. Generate the server configuration files on one machine
[root@costarican:~]#pvfs2-genconfig /etc/pvfs2-fs.conf /etc/pvfs2-server.conf
**********************************************************************
Welcome to the PVFS2 Configuration Generator:
This interactive script will generate configuration files suitable
for use with a new PVFS2 file system. Please see the PVFS2 quickstart
guide for details.
**********************************************************************
You must first select the network protocol that your file system will use. The only currently supported options are "tcp", "gm", and "ib".
* Enter protocol type [Default is tcp]:
Choose a TCP/IP port for the servers to listen on. Note that this script assumes that all servers will use the same port number.
* Enter port number [Default is 3334]:
Next you must list the hostnames of the machines that will act as I/O servers. Acceptable syntax is "node1, node2, ..." or "node{#-#,#,#}".
* Enter hostnames [Default is localhost]: costarican, guatemalan, aulait, mocha
Now list the hostnames of the machines that will act as Metadata servers. This list may or may not overlap with the I/O server list.
* Enter hostnames [Default is localhost]: costarican, mocha Configured a total of 4 servers: 4 of them are I/O servers. 2 of them are Metadata servers. * Would you like to verify server list (y/n) [Default is n]?
Choose a file for each server to write log messages to.
* Enter log file location [Default is /tmp/pvfs2-server.log]:
Choose a directory for each server to store data in.
* Enter directory name: [Default is /pvfs2-storage-space]: /mnt/rw/pvfs2-metastorage Writing fs config file... Done. Writing 4 server config file(s)... Done. [root@costarican:~]#
5. Copy the relevant pvfs2 config files to all of the server nodes
One of the easiest ways to do this is with scp (secure copy). Each node must have at least the generalized pvfs2-fs.conf file and the pvfs2-server.conf file that corresponds to the node's hostname. The generalized commands to do this are below:
scp /etc/pvfs2-fs.conf root@[node]:/etc/. scp /etc/pvfs2-server.conf-[node] root@[node]:/etc/.
where [node] is the hostname of the destination server. The idea is to copy the two relavent configuration files from the node they all were generated on to the etc directory on the destination node. For example, below is the actual command to copy the pvfs2-fs.conf and pvfs2-server.conf-mocha files from the node costarican to the node mocha:
[root@costarican:~]# scp /etc/pvfs2-fs.conf root@mocha:/etc/. [root@costarican:~]# scp /etc/pvfs2-server.conf-mocha root@mocha:/etc/.
Remember, each node must have the pvfs2-fs.conf and its own pvfs2-server.conf file!
6. Generate a /etc/pvfs2tab on each machine
Create it to reflect the current configuration. Open a file called /etc/pvfs2tab with your favorite editor, and put as the single and only line:
tcp://[node]:3334/pvfs2-fs /mnt/rw/pvfs2 pvfs2 default 0 0
where [node] is the hostname of the current machine. For example, the node costarican should have this line in its /etc/pvfs2tab file:
tcp://costarican:3334/pvfs2-fs /mnt/rw/pvfs2 pvfs2 default 0 0
Remember, each node must have its own /etc/pvfs2tab file!
7. Start the servers.
The generalized command to do this would be:
pvfs2-server /etc/pvfs2-fs.conf /etc/pvfs2-server.conf-[node] -f pvfs2-server /etc/pvfs2-fs.conf /etc/pvfs2-server.conf-[node]
where [node] is the hostname of the node you are on. For example, to start the servers (both metadata and I/O) on the node costarican, you would issue the following commands:
[root@costarican:~]# pvfs2-server /etc/pvfs2-fs.conf /etc/pvfs2-server.conf-costarican -f [root@costarican:~]# pvfs2-server /etc/pvfs2-fs.conf /etc/pvfs2-server.conf-costarican
Remember, these two server-specific commands must be issued on all nodes with servers (either metadata, I/O, or both)!
8. Test out the configuration with pvfs2-ping
Script started on Mon Jul 26 14:47:57 2004 [root@costarican:~]# pvfs2-ping -m /mnt/rw/pvfs2 (1) Parsing tab file... (2) Initializing system interface... (3) Initializing each file system found in tab file: /etc/pvfs2tab... /mnt/rw/pvfs2: Ok (4) Searching for /mnt/rw/pvfs2 in pvfstab... PVFS2 server: tcp://costarican:3334 Storage name: pvfs2-fs Local mount point: /mnt/rw/pvfs2 meta servers: tcp://costarican:3334 tcp://mocha:3334 data servers: tcp://costarican:3334 tcp://mocha:3334 tcp://aulait:3334 tcp://guatemalan:3334 (5) Verifying that all servers are responding... meta servers: tcp://costarican:3334 Ok tcp://mocha:3334 Ok data servers: tcp://costarican:3334 Ok tcp://mocha:3334 Ok tcp://aulait:3334 Ok tcp://guatemalan:3334 Ok (6) Verifying that fsid 1335610011 is acceptable to all servers... Ok; all servers understand fs_id 1335610011 (7) Verifying that root handle is owned by one server... Root handle: 1048576 Ok; root handle is owned by exactly one server. ============================================================= The PVFS filesystem at /mnt/rw/pvfs2 appears to be correctly configured. [root@costarican:~]#
9. Configure the client services on each node
Remember, the following steps must be performed on each node that is to be a client.
Load the pvfs2 module:
[root@costarican:~]# modprobe -a pvfs2 [root@costarican:~]# lsmod Module Size Used by Not tainted pvfs2 50308 0 (unused) ide-floppy 12956 0 (autoclean) af_packet 10120 1 (autoclean) pcnet32 14304 1 mii 2176 0 [pcnet32] crc32 2880 0 [pcnet32] pcmcia_core 45472 0 nls_iso8859-1 2780 1 (autoclean) nls_cp437 4284 1 (autoclean) loop 9208 3 (autoclean) unix 14312 22 (autoclean) cloop 5616 1 [root@costarican:~]#
Prepare the mount point:
[root@costarican:~]# mkdir /mnt/rw/pvfs2
Start the client processes:
[root@costarican:~]# pvfs2-client pvfs2-client-core [root@costarican:~]# ps aux | grep pvfs2-client root 1255 0.2 0.2 2072 428 ? Ss 13:18 0:00 pvfs2-client pvfs2-client-core root 1256 2.6 14.6 27572 23816 ? S 13:18 0:00 [pvfs2-client-co] root 1259 0.0 14.6 27572 23816 ? S 13:18 0:00 [pvfs2-client-co] [root@costarican:~]#
Finally, mount the pvfs2 filesystem:
[root@costarican:~]# mount.pvfs2 tcp://host130:3334/pvfs2-fs /mnt/rw/pvfs2 [root@costarican:~]# mount | grep pvfs2 pvfs2 on /mnt/rw/pvfs2 type pvfs2 (rw) [root@costarican:~]#
10. Test out the system with pvfs2-ls and pvfs2-cp
Test to see if the server target space is initialized (mainly just for the curiosity side of things):
[root@costarican:~]# ls /mnt/rw/pvfs2-metastorage 5bd959a7 collections.db storage_attributes.db
See what's there:
[root@costarican:~]# pvfs2-ls /mnt/rw/pvfs2
Yes, there's nothing there...we didn't populate it! Populate the target filesystem and check to see that it shows up where it should:
[root@costarican:~]# pvfs2-cp -t /lib/libc.so.6 /mnt/rw/pvfs2 Wrote 1070180 bytes in 0.040103 seconds. 25.449556 MB/seconds [root@costarican:~]# pvfs2-ls /mnt/rw/pvfs2 libc.so.6 [root@costarican:~]# ls /mnt/rw/pvfs2 libc.so.6 [root@costarican:~]#
And now you've got a full PVFS2 cluster with metadata servers, I/O servers, and clients running off the BCCD! Gravy!
