Project

General

Profile

Actions
History

Wiki » Documentation »

Greenqloud AFS

This is a project to make it easy to add more storage to an AFS cell (in this case dev.ru.is) using the greenqloud service. Idea inception by Joe Foley <foley@ru.is> on Friday Feb 8 at the greenqloud-RU hackathon

References/Links

Installation

Server basics

  1. Create a greenqloud instance. Minimum suggested settings:
    1. Image: Ubuntu Server 12.04.1 LTS
    2. Size: Milli (1 CPU, 1024MB memory)
    3. Additional disk: Nano (10GB)
    4. Reserved IP Address
    5. Security group: (if you've already setup the AFS ports in a group, use them. Otherwise default)
    6. Name: afsfs<NUM>
  2. setup the ports if you didn't have a security group setup
    1. TCP: 22, 88, 464, 749
    2. UDP: 88, 464, 749, 4444, 9878, 7000-7007
  3. Attach some additional storage for the AFS partition. This can be whatever you need. Remember what the partition name is.
  4. Login to the server and update the packages
    apt-get -y update; apt-get -y upgrade

DNS

We'll be creating a convenience DNS name afsfs6.dev.ru.is for the server. This is optional, but very helpful. The rest of the instructions assume that you have completed this step.

  1. Login to the DevNet DNS server: ns1.dev.ru.is
  2. Become root
    ksu
  3. Add a CNAME pointer at the bottom of the file to the server e.g.:
    afsfs6         in        a     46.149.19.79   ; greenqloud afstest2
  4. Restart the named
    service named restart

Kerberos

You can skip this part if you're not going to login to the machine much, but it's rather useful. The rest of these instructions assume you have completed this step. If you haven't, you will need to replace ksu with sudo su -
  1. Become root
    sudo su -
  2. Install the kerberos packages (clients and user utilities)
    apt-get -y install krb5-user krb5-clients
    1. When it asks, put your Kerberos realm. This is DEV.RU.IS for our system.
  3. Copy over the kerberos config from one of the DevNet servers. This can be downloaded using wget from our AFS web gateway.
    cd /etc
rm krb5.conf
wget http://afs.dev.ru.is/project/devnet/Public/Workstation/etc/krb5.conf
  4. Now get srvtabs for the internal and external names of the machine
    ifconfig | grep inet | head -1
          inet addr:10.1.1.205  Bcast:10.1.1.255  Mask:255.255.255.0
host 10.1.1.205
205.1.1.10.in-addr.arpa domain name pointer i-88917dc6.greenqloud.internal.
kadmin
Authenticating as principal foley/admin@DEV.RU.IS with password.
Password for foley/admin@DEV.RU.IS: 
kadmin:  ank -policy host -randkey host/i-46-149-19-79.compute.is-1.greenqloud.com
Principal "host/i-46-149-19-79.compute.is-1.greenqloud.com@DEV.RU.IS" created.
kadmin:  ktadd host/i-46-149-19-79.compute.is-1.greenqloud.com
Entry for principal host/i-46-149-19-79.compute.is-1.greenqloud.com with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/i-46-149-19-79.compute.is-1.greenqloud.com with kvno 2, encryption type arcfour-hmac added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/i-46-149-19-79.compute.is-1.greenqloud.com with kvno 2, encryption type des3-cbc-sha1 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/i-46-149-19-79.compute.is-1.greenqloud.com with kvno 2, encryption type des-cbc-crc added to keytab FILE:/etc/krb5.keytab.
kadmin:  ank -policy host -randkey host/i-88917dc6.greenqloud.internal
Principal "host/i-88917dc6.greenqloud.internal@DEV.RU.IS" created.
kadmin:  ktadd host/i-88917dc6.greenqloud.internal
Entry for principal host/i-88917dc6.greenqloud.internal with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/i-88917dc6.greenqloud.internal with kvno 2, encryption type arcfour-hmac added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/i-88917dc6.greenqloud.internal with kvno 2, encryption type des3-cbc-sha1 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/i-88917dc6.greenqloud.internal with kvno 2, encryption type des-cbc-crc added to keytab FILE:/etc/krb5.keytab.
kadmin: quit
  5. Give your user permission to ksu as root. Edit /root/.k5login and add your kerberos principal, e.g. .
  6. Now test ksu
    ksu
Authenticated foley@DEV.RU.IS
Account root: authorization for foley@DEV.RU.IS successful
Changing uid to root (0)
  1. Note, there is some weird issue with getting GSSAPI ssh to work from external connections. I'm looking into it.

AFS Server

  1. Login to the server and become root
    ksu
  2. Install packages (debian/ubuntu). This may take a while because DKMS needs to compile the openafs kernel module.
    apt-get install -y openafs-krb5 openafs-fileserver
    1. The cell for DevNet is dev.ru.is Note! This is lower-case!
    2. Accept the default cache size on the server
  3. Start the AFS client on the machine. It will start on reboot also
    service openafs-client start
  4. Check that you can see the dev.ru.is cell
    ls /afs/dev.ru.is
activity  dept  org      reference   service   system
course    host  project  robots.txt  software  user
  5. Partition our data partition. Just make a standard linux primary partition
     fdisk /dev/vdb
  6. Format the partition. Ext2 or Ext3 work well. Ext4 apparently has issues.
    mkfs.ext3 -m 0 -L vicepa /dev/vdb1
  7. Make the vicepa directory. AFS uses any /vice?? directories available to store volume data.
    mkdir /vicepa
  8. Add a line to /etc/fstab that will automount it
    echo "/dev/vdb1 /vicepa ext3 defaults 0 3" >> /etc/fstab
  9. Test the automount
    mount -a
ls /vicepa
lost+found
  10. Copy over the files needed to properly configure the AFS server
    cp -r /afs/dev.ru.is/project/devnet/Public/AFS-server/etc/openafs/* /etc/openafs/.
  11. Copy over the key (encrypted)
    scp root@afsdb1.dev.ru.is:/etc/openafs/server/KeyFile /etc/openafs/server/.
  12. Check that the key is installed properly
    setkey list
kvno    5: key is: 3476928054b676e3
All done.
  13. Install a fileserver instance using the bos utility
    bos create localhost fs fs -cmd /usr/lib/openafs/fileserver -cmd /usr/lib/openafs/volserver -cmd /usr/lib/openafs/salvager -localauth
  14. Make sure that the instance appears happy
     bos status localhost -localauth
Instance fs, currently running normally.
    Auxiliary status is: file server running.
  15. Also look into the vicepa for the new data files
    ls /vicepa
AFSIDat  Lock  lost+found

Creating and mounting volumes

  1. Become an AFS administrator and get tokens
    kinit foley/afsadm@DEV.RU.IS; aklog
  2. Create a volume of size 3GB(3000000KB) on partition a (/vicepa) of server afsfs6
    vos create afsfs6.dev.ru.is a tmp.greenqloud 3000000
  3. Mount the volume somewhere sensible
    fs mkmount /afs/dev.ru.is/service/greenqloud tmp.greenqloud
  4. Put someone on the access list for the new volume
    fs setacl  /afs/dev.ru.is/service/greenqloud foley all

Later stuff learned from the first try

  • Make sure that your superusers are in /etc/openafs/server/UserList e.g.
    root.admin
foley.afsadm

Updated by Joseph Foley over 9 years ago · 1 revisions