Research and Development Network

h1. Greenqloud AFS

This is a project to make it easy to add more storage to an AFS cell (in this case dev.ru.is) using the "greenqloud":https://my.greenqloud.com/ service.  Idea inception by Joe Foley <foley@ru.is> on Friday Feb 8 at the "greenqloud-RU hackathon":http://hackathon.greenqloud.com/

h2. References/Links

* DevNet AFS Server page. Needs DevNet developer access:  [[devnet:AFS Server]]

* Greenqloud https://my.greenqloud.com

* Hackathon wiki http://hackathonwiki.greenqloud.com/index.php/Main_Page

* OpenAFS forum post on adding fileserver https://lists.openafs.org/pipermail/openafs-info/2006-September/023495.html

* DNS and Kerberos issues http://www.faqs.org/faqs/kerberos-faq/general/section-47.html

h2. Installation

h3. Server basics

# Create a greenqloud instance.  Minimum suggested settings: 

## Image: Ubuntu Server 12.04.1 LTS

## Size: Milli (1 CPU, 1024MB memory)

## Additional disk: Nano (10GB)

## Reserved IP Address

## Security group: (if you've already setup the AFS ports in a group, use them.  Otherwise default)

## Name: afsfs<NUM>

# setup the ports if you didn't have a security group setup

## TCP: 22, 88, 464, 749

## UDP: 88, 464, 749, 4444, 9878, 7000-7007

# Attach some additional storage for the AFS partition.  This can be whatever you need.  Remember what the partition name is.

# Login to the server and update the packages

<pre>apt-get -y update; apt-get -y upgrade</pre>

h3. DNS

We'll be creating a convenience DNS name @afsfs6.dev.ru.is@ for the server.  This is optional, but very helpful.  The rest of the instructions assume that you have completed this step.

# Login to the DevNet DNS server: @ns1.dev.ru.is@

# Become root

<pre>ksu</pre>

# Add a CNAME pointer at the bottom of the file to the server e.g.:

<pre>afsfs6         in        a     46.149.19.79   ; greenqloud afstest2</pre>

# Restart the named

<pre>service named restart</pre>

h3. Kerberos

You can skip this part if you're not going to login to the machine much, but it's rather useful.  The rest of these instructions assume you have completed this step.  If you haven't, you will need to replace @ksu@ with @sudo su -@

# Become root

<pre>sudo su -</pre>

# Install the kerberos packages (clients and user utilities)

<pre>apt-get -y install krb5-user krb5-clients</pre>

## When it asks, put your Kerberos realm.  This is @DEV.RU.IS@ for our system.

# Copy over the kerberos config from one of the DevNet servers.  This can be downloaded using wget from our AFS web gateway.

<pre>cd /etc

rm krb5.conf

wget http://afs.dev.ru.is/project/devnet/Public/Workstation/etc/krb5.conf</pre>

# Now get srvtabs for the internal and external names of the machine

<pre>

ifconfig | grep inet | head -1

          inet addr:10.1.1.205  Bcast:10.1.1.255  Mask:255.255.255.0

host 10.1.1.205

205.1.1.10.in-addr.arpa domain name pointer i-88917dc6.greenqloud.internal.

kadmin

Authenticating as principal foley/admin@DEV.RU.IS with password.

Password for foley/admin@DEV.RU.IS: 

kadmin:  ank -policy host -randkey host/i-46-149-19-79.compute.is-1.greenqloud.com

Principal "host/i-46-149-19-79.compute.is-1.greenqloud.com@DEV.RU.IS" created.

kadmin:  ktadd host/i-46-149-19-79.compute.is-1.greenqloud.com

Entry for principal host/i-46-149-19-79.compute.is-1.greenqloud.com with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.

Entry for principal host/i-46-149-19-79.compute.is-1.greenqloud.com with kvno 2, encryption type arcfour-hmac added to keytab FILE:/etc/krb5.keytab.

Entry for principal host/i-46-149-19-79.compute.is-1.greenqloud.com with kvno 2, encryption type des3-cbc-sha1 added to keytab FILE:/etc/krb5.keytab.

Entry for principal host/i-46-149-19-79.compute.is-1.greenqloud.com with kvno 2, encryption type des-cbc-crc added to keytab FILE:/etc/krb5.keytab.

kadmin:  ank -policy host -randkey host/i-88917dc6.greenqloud.internal

Principal "host/i-88917dc6.greenqloud.internal@DEV.RU.IS" created.

kadmin:  ktadd host/i-88917dc6.greenqloud.internal

Entry for principal host/i-88917dc6.greenqloud.internal with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.

Entry for principal host/i-88917dc6.greenqloud.internal with kvno 2, encryption type arcfour-hmac added to keytab FILE:/etc/krb5.keytab.

Entry for principal host/i-88917dc6.greenqloud.internal with kvno 2, encryption type des3-cbc-sha1 added to keytab FILE:/etc/krb5.keytab.

Entry for principal host/i-88917dc6.greenqloud.internal with kvno 2, encryption type des-cbc-crc added to keytab FILE:/etc/krb5.keytab.

kadmin: quit

</pre>

# Give your user permission to @ksu@ as root.  Edit @/root/.k5login@ and add your kerberos principal, e.g. foley@DEV.RU.IS.

# Now test @ksu@

<pre>ksu

Authenticated foley@DEV.RU.IS

Account root: authorization for foley@DEV.RU.IS successful

Changing uid to root (0)</pre>

# Note, there is some weird issue with getting GSSAPI ssh to work from external connections.  I'm looking into it.

h3. AFS Server

# Login to the server and become root

<pre>ksu</pre>

# Install packages (debian/ubuntu).  This may take a while because DKMS needs to compile the openafs kernel module.

<pre>apt-get install -y openafs-krb5 openafs-fileserver</pre>

## The cell for DevNet is @dev.ru.is@  *Note!  This is lower-case!*

## Accept the default cache size on the server

# Start the AFS client on the machine.  It will start on reboot also

<pre>service openafs-client start</pre>

# Check that you can see the dev.ru.is cell

<pre>ls /afs/dev.ru.is

activity  dept  org      reference   service   system

course    host  project  robots.txt  software  user

</pre>

# Partition our data partition.  Just make a standard linux primary partition

<pre> fdisk /dev/vdb</pre>

# Format the partition.  Ext2 or Ext3 work well.  Ext4 apparently has issues.

<pre>mkfs.ext3 -m 0 -L vicepa /dev/vdb1</pre>

# Make the vicepa directory.  AFS uses any /vice?? directories available to store volume data.

<pre>mkdir /vicepa</pre>

# Add a line to @/etc/fstab@ that will automount it

<pre>echo "/dev/vdb1 /vicepa ext3 defaults 0 3" >> /etc/fstab</pre>

# Test the automount

<pre>mount -a

ls /vicepa

lost+found

</pre>

# Copy over the files needed to properly configure the AFS server

<pre>cp -r /afs/dev.ru.is/project/devnet/Public/AFS-server/etc/openafs/* /etc/openafs/.</pre>

# Copy over the key (encrypted)

<pre>scp root@afsdb1.dev.ru.is:/etc/openafs/server/KeyFile /etc/openafs/server/.</pre>

# Check that the key is installed properly

<pre>setkey list

kvno    5: key is: 3476928054b676e3

All done.

</pre>

# Install a fileserver instance using the @bos@ utility

<pre>bos create localhost fs fs -cmd /usr/lib/openafs/fileserver -cmd /usr/lib/openafs/volserver -cmd /usr/lib/openafs/salvager -localauth</pre>

# Make sure that the instance appears happy

<pre> bos status localhost -localauth

Instance fs, currently running normally.

    Auxiliary status is: file server running.</pre>

# Also look into the vicepa for the new data files

<pre>ls /vicepa

AFSIDat  Lock  lost+found

</pre>

h3. Creating and mounting volumes

# Become an AFS administrator and get tokens

<pre>kinit foley/afsadm@DEV.RU.IS; aklog</pre>

# Create a volume of size 3GB(3000000KB) on partition a (/vicepa) of server afsfs6

<pre>vos create afsfs6.dev.ru.is a tmp.greenqloud 3000000</pre>

# Mount the volume somewhere sensible

<pre>fs mkmount /afs/dev.ru.is/service/greenqloud tmp.greenqloud</pre>

# Put someone on the access list for the new volume

<pre>fs setacl  /afs/dev.ru.is/service/greenqloud foley all </pre>

h2. Later stuff learned from the first try

* Make sure that your superusers are in /etc/openafs/server/UserList e.g.

<pre>root.admin

foley.afsadm</pre>