ClusterSetupGuide
This page captures some information about setting up bigdata over a Fedora 10 minimum install and presumes that you have root privileges and will install bigdata to run as root (the latter is not necessary, but that is what is shown here). See the ClusterGuide for more general information on a bigdata cluster install.
Install the following packages. Some of these are optional (telnet, emacs, nfs-utils, ntp).
yum install man # man page support.
yum install mlocate #(0:0.21.1-1); so I could find procmail's lockfile, which is at /usr/bin/lockfile
yum install emacs # or the editor of your choice.
yum install screen # optional job control utility.
yum install telnet # optional (useful for testing services and firewall settings)
# yum install rpcbind # used by NFS, but I believe this is part of the core distro.
yum install nfs-utils # iff you will use NFS for the shared volume.
yum install sysstat # used to collect performance counters from the OS and services.
yum install ntp # optional, but highly recommended.
yum install subversion # used to checkout bigdata from its SVN repository.
yum install ant # used to build bigdata from the source code.
Linux, like many other operating systems, has a very aggressive posture towards free memory. By default, Linux will allow your applications to occupy no more than 1/2 of the available RAM before it begins to swap things out. You can fix this by turning down the swappiness parameter to ZERO.
sysctl -w vm.swappiness=0
You MUST be able to resolve the hostnames in the cluster using DNS.
Normally someone is administering DNS and you don't have to worry about
this. If that is not true, then the easy fix is to edit /etc/hosts to
make sure each host in the cluster knows the name and IP associated with
all the hosts in the cluster.
Here is a sample /etc/hosts file. Your file must reflect the IP addresses and host names in your cluster.
127.0.0.1 localhost localhost.localdomain
x.y.z.129 BigData0
x.y.z.130 BigData1
x.y.z.131 BigData2
VNC can be used to remotely login to the X-Windows desktop on the machines in the cluster. This can be very useful and it can be done securely using an ssh tunnel. This installs X-Windows, the KDE desktop, and the VNC server. See 1 for more information.
# install X and KDE
yum -y install xorg*
yum -y install xfce*
yum update # required to get around kdebase-wallpapers conflict for fc10.
yum -y install kde*
yum install vnc-server #(0:4.1.3-1.fc10)
-------------------- edit ~/.vnc/xstart --------------------
# Uncomment the following two lines for normal desktop:
unset SESSION_MANAGER
exec /etc/X11/xinit/xinitrc
# -------------------- edit edit --------------------
# -------------------- edit /etc/sysconfig/vncservers --------------------
# Define at least one vncserver here. Choose your own display
# resolution. Use the "-localhost" option to restrict connections to
# SSH tunnels. The remote machine should port forward local 5901 to
# remote localhost:5901 and then connect using "localhost:1".
VNCSERVERS="1:root"
VNCSERVERARGS[1]="-geometry 1280x1024 -nolisten tcp -nohttpd -localhost"
# -------------------- edit edit --------------------
# Set the vnc password.
vncpasswd
# Start vncserver.
/etc/init.d/vncserver start
# Configure the vncserver runlevels.
chkconfig vncserver on
