Research and Development Network

h1. AFS Server

These directions were originally on the Centipede project on https://projects.cs.ru.is

Will be running [[OpenLDAP]], Kerberos5 server, and AFS.  We will be using 3 of the IBM servers on top of the Cisco Catalyst.

{{toc}}

h2. Tasks

* Joe: 

** Interface with IT (*done*)

** install Kerberos server (*done*)

** install AFS db server gryla (*done*)

** install AFS cell server 0 gryla

*** vicepa is installed 8GB

** install AFS cell server 1 (stekkjarstaur)

** install AFS db server 2 stekkjarstaur (*mostly done, need db sync*)

*** vicepa is installed 500GB USB

** INstall AFS db server 3 giljagaur

** install AFS cell server 2 (samvinna.ru.is)

*** vicepa is installed 100GB (lvm)

* Stephan: [[install]] Ubuntu 10.4.3 server on the machines (*done*)

* Ute: Look into proper imaging under linux

h2. Desired Hardware

* Drobo (automated raid system with iSCSI) http://www.drobo.com/products/business/b1200i/index.php#!prettyPhoto

h2. Instructions

* [[devnet-documentation:AFS_Client_Installation]]

* [[How To Add Users]]

h3. Guides/links

* OpenAFS Official Documentation http://docs.openafs.org/

* OpenAFS Detailed Guide http://techpubs.spinlocksolutions.com/dklar/afs.html

* slapd documentation https://help.ubuntu.com/community/OpenLDAPServer

* Kerberos/LDAP on Ubuntu (outdated) http://bobcares.com/blog/?p=435

* OpenAFS on Ubuntu (outdated) http://bobcares.com/blog/?p=501

* IBM AFS 3.6 Documentation http://www-01.ibm.com/software/stormgmt/afs/manuals/Library/unix/en_US/HTML/index.htm

* Interrealm AFS http://www.cs.cmu.edu/~help/afs/cross_realm.html

* AFS on Linux Presentation http://www.dia.uniroma3.it/~afscon09/docs/wiesand.pdf

* mod_waklog (apache integration with AFS) [[mod_waklog]]

* Kerberos and DNS http://www.faqs.org/faqs/kerberos-faq/general/section-47.html

* Adding another fileserver https://lists.openafs.org/pipermail/openafs-info/2006-September/023495.html

* Kerberos and AFS tutorial (Secure endpoints) http://www.secure-endpoints.com/talks/Kerberos_Tutorial_BPW2007.pdf

* Key security issues

** http://openafs.org/pages/security/OPENAFS-SA-2013-003.txt

** http://www.openafs.org/pages/security/install-rxkad-k5-1.6.txt

h3. Getting started

* Install Ubuntu 12.04LTS

* Install Kerberos KDC (directions at [[Kerberos]])

* Setup DNS autodiscovery

<pre>

dev.ru.is.		IN	AFSDB	1 afsdb1.dev.ru.is.

</pre>

* Ubuntu

** client configuration is in @/etc/openafs@

** server configuration is in @/etc/openafs/server@

* CentOS

** client configuration is in @/var/vice/etc@ (the traditional)

** server configuration is in @/usr/afs/etc@

h3. Client

*Important!  Do this first!*

see [[AFS_Client_Installation]]

h3. Server

# Packages

<pre>sudo apt-get install openafs-krb5 openafs-{fileserver,dbserver}</pre>

h3. Keys and accounts

# Add entries to @/etc/openafs/CellServDB@ and @/etc/openafs/server/CellServDB@

## Note that the AFS cell *MUST* be lowercase

<pre>

>dev.ru.is			# Reykjavik University

130.208.209.37			#afsdb1.dev.objid.net 

130.208.209.39			#afsdb2.dev.objid.net 

130.208.209.40			#afsdb3.dev.objid.net 

</pre>

# Edit your @/etc/krb5.conf@

<pre>       

[libdefaults]

dns_lookup_kdc = true

dns_lookup_realm = true

[realms]

DEV.RU.IS = {

                kdc = kerberos.dev.ru.is

                kdc = kerberos-1.dev.ru.is

                kdc = kerberos-2.dev.ru.is

                admin_server = kerberos.dev.ru.is

</pre>

# Add principal “afs/dev.ru.is” and import the key to /etc/openafs/afs.keytab.  This is also a good time to setup the normal keytab.  Replace HOSTNAME with the reverse resolvable hostname.  If this is on greenqloud, you will need to make both the external name and the internal and add it to the keytab.

## WARNING:  If you run the kadmin, it will increment the kerberos version number, which will not allow the new servers to talk to the old.  Instead, copy the afs.keytab using scp to all of the machines!

<pre>

kadmin.local

kadmin: addprinc -policy service -randkey -e des-cbc-crc:normal afs/dev.ru.is

kadmin: ktadd -k /etc/openafs/afs.keytab -e des-cbc-crc:normal afs/dev.ru.is

kadmin: ank -policy host -randkey host/HOSTNAME

kadmin: ktadd host/HOSTNAME

kadmin: ank -policy host -randkey host/HOSTNAME

kadmin: ktadd host/INTERNALHOSTNAME

</pre>

# Remember the KVNO (key version number)

<pre>klist -ke /etc/openafs/afs.keytab</pre>

# Import the secret key into the AFS system.  Replace KVNO with the version number

<pre>asetkey add KVNO /etc/openafs/afs.keytab afs/dev.ru.is

</pre>

#Now test with bos (afs-fileserver must be running, possibly restarted)

<pre>sudo service openafs-fileserver restart

sudo bos listkeys afsdb1 -localauth

#key 3 has cksum 2586520638

#Keys last changed on Fri Mar 30 02:10:25 2012.

#All done.

</pre> 

# Setup kerberized root shell access.  Replace my principal with yours or add it to the list.

<pre>

sudo vi /root/.krb5login

foley@DEV.RU.IS

</pre>

# Test

<pre>ksu</pre>

h3. Partitions for vice (AFS cell)

# Make the partitions in your filesystem (as an image)

<pre>cd /home

sudo dd if=/dev/zero of=vicepa.img bs=100M count=80   # (8 GB partition)

sudo mkfs.ext4 vicepa.img

sudo sh -c "echo '/home/vicepa.img /vicepa ext4 defaults,loop 0 2' >> /etc/fstab"

sudo tune2fs -c 0 -i 0 -m 0 vicepa.img

</pre>

# Now we mount it

<pre>sudo mkdir -p /vicepa

sudo mount /vicepa

</pre>

To add more disks, see [[AFS Server#Adding-More-disks|Adding More disks]]

h3. Firewall settings

# Need to poke holes in the firewall also (http://security.fnal.gov/cookbook/KerberosPorts.html)

## Login to the firewall  @bridge.objid.net@

## Open these ports in @/etc/shorewall/rules@

<pre>## AFS and kerberos 

## From http://security.fnal.gov/cookbook/KerberosPorts.html

ACCEPT all    net:130.208.209.37-130.208.209.40 tcp,udp 88 #krb

ACCEPT all    net:130.208.209.37-130.208.209.40 tcp 749

ACCEPT all    net:130.208.209.37-130.208.209.40 tcp,udp 464

ACCEPT all    net:130.208.209.37-130.208.209.40 udp 749,4444

ACCEPT all    net:130.208.209.37-130.208.209.40 udp 9878

ACCEPT all    net:130.208.209.37-130.208.209.40 udp 7000:7007

</pre>

h3. Make the new cell

# If you get issues with the case on the afs cell (all UPPERCASE is bad)

## sudo dpkg-reconfigure openafs-client

## sudo dpkg-reconfigure openafs-fileserver

# Make the Cell!

<pre>sudo afs-newcell</pre>

## Yes, we meet the requirements

##Principal: root/admin

## If you have problems with the ip address in the CellServDB, make sure it matches in @/etc/hosts@!

## If you see issues about network connections, you probably have an orphan process or something running on that port.

## if you have to clear out the user database, it is in @/etc/openafs/server/ThisUser@

h3. Testing out the cell and making the root volume

# Test out kinit and tokens

<pre>sudo su # (We want to switch to the root user)

kinit root/admin

Password for root/admin@SPINLOCK.HR: PASSWORD

aklog

</pre>

## if you get errors, it means that weak crypto is not enabled

# Check out your tokens with @kinit -5f@ and @tokens@

# Now create the root volume

<pre>afs-rootvol

...

4) The AFS client must be running pointed at the new cell.

Do you meet these conditions? (y/n) y

You will need to select a server (hostname) and AFS partition on which to

create the root volumes.

What AFS Server should volumes be placed on? gryla.dev.objid.net

What partition? [a] a

</pre>

# Everything should now be happy!

# Add the update server to distribute config files over encrypted channel

<pre>sudo bos create localhost upserveretc simple  "/usr/lib/openafs/upserver  -crypt /etc/openafs" -localauth </pre>

# Add the backup server (database servers need this process)

<pre>sudo bos create localhost buserver simple "/usr/lib/openafs/buserver" -localauth</pre>

h3. Standard partitions

# Now we create some of the "standard" partitions.  This is based upon the MIT configuration.  Note that these all start with very small quota! and are their own "class".  You will need to schedule backup partitions for them individually.  Note that you will need to go to the read-write of the root which is /afs/.dev.ru.is

<pre>

cd /afs/.dev.ru.is

vos create afsdb1 a activity

vos create afsdb1 a course

vos create afsdb1 a project

vos create afsdb1 a software

vos create afsdb1 a system

vos create afsdb1 a dept

vos create afsdb1 a org

vos create afsdb1 a reference

</pre>

# Now we mount them in the root area

<pre>

fs mkmount activity activity

fs mkmount course course

fs mkmount project project

fs mkmount software software

fs mkmount system system

fs mkmount dept dept

fs mkmount org org

fs mkmount reference reference

</pre>

# Finally, since this is a read-only volume, we have to "release" it

<pre>vos release root.cell</pre>

# And check to make sure the new directories show up in /afs/dev.ru.is

h3. Administrative users

* Login to the kerberos server as root and create an afsadm user

<pre>kadmin.local

addprinc addprinc -policy user <user>/afsadm

quit</pre>

* If you need super-super user capability, you will need to run this command with root/admin

<pre>kinit root/admin

aklog

bos adduser afsdb1 <user>/afsadm

</pre>

* Now you are a super-super user and can make the afs server dance.  Sometimes it takes a few minutes for the protection database to update, so you might have to wait.  Unfortunately, you are not done, you still need to add them to the group system:administrators so they can do other useful operations.

* Create an equivalent user to the Kerberos user

<pre>pts createuser <user>.afsadm</pre>

* Add them to the group system:administrators.  You may need to wait for the ptserver to sync up

<pre>pts adduser <user>.afsadm system:administrators</pre>

h3. Backup partitions

* Create all backup partitions everywhere!

<pre> vos backupsys </pre>

* Automate backup of user partitions at 1:00 and the temp partition at 0:01

<pre>bos create afsdb1 backupusers cron -cmd "/usr/bin/vos backupsys -prefix user -localauth" "1:00"

bos create afsdb1 backuptemp cron -cmd "/usr/bin/vos backupsys -prefix temp -localauth" "0:01"

</pre>

* now go create Oldfiles in the appropriate places

<pre>cd /afs/dev.ru.is/user/f/fo/foley

fs mkmount Oldfiles user.foley.backup

</pre>

h3. Adding More disks

http://docs.openafs.org/AdminGuide/ch03s08.html

# Get administrator tickets and tokens/var/lib/openafs/local

<pre> kinit <something>/afsadm; aklog

</pre>

# format and partition them with ext4 and give them a useful name for later.  For example

## figure out the next /vicep?? name, in this case it is /vicepb

<pre>fdisk /dev/sdb1

mkfs.ext4 -L grylavicepb /dev/sdb1</pre>

# Add the appropriate entry to /etc/fstab

<pre>echo "/dev/sdb1 /vicepb ext4 defaults 0 2" >> /etc/fstab</pre>

# See if it gets mounted properly

<pre>sudo mount -a </pre>

# Kick the fs service to get it to rescan and notice the disk

<pre>bos restart <machine name>  fs</pre>

# it should now show up  on the list of partitions

<pre>vos listpart afsdb1</pre>

# now you can make new volumes

h3. Multihomed servers

It can be a problem if one of the network interfaces is on a private network (like gryla).  You can restrict which network interfaces the server interfaces by putting NetInfo and NetRestrict files in @/var/lib/openafs/local@

More info at http://docs.openafs.org/AdminGuide/ch03s09.html

h3. Installing another server

These instructions were from examining the afs-newcell script by Sam Hartmans. Make sure you are logged in as root.

*Also, make sure that you do not have the hostname as 127.0.0.1 in /etc/hosts!*

# Stop the client

<pre>service openafs-client stop</pre>

# Stop the server

<pre>service openafs-fileserver stop</pre>

# Copy the afs.keytab from a running server

<pre>scp root@gryla.dev.ru.is:/etc/openafs/server/afs.keytab /etc/openafs/server/afs.keytab</pre>

# Install the key

<pre>asetkey add 3 /etc/openafs/server/afs.keytab afs/dev.ru.is</pre>

# Edit the CellServDB in @/etc/openafs/server/CellServDB@ and append it to the end of @/etc/openafs/CelServDB@ if it isn't already in there.

# Secure, but possibly database corrupting:

## STart up the server

<pre> service openafs-fileserver start </pre>

## Add the admin user through bos

<pre>bos adduser afscell1 root.admin -localauth</pre>

## Create initial protection database by hand.  Bit of a hack

<pre>pt_util -p /var/lib/openafs/db/prdb.DB0 -w</pre>

### start typing in these commands exactly.  Note the space before the last line!

<pre>root.admin 128/20 1 -204 -204

system:administrators 130/20 -204 -204 -204

 root.admin 1</pre>

## start up the ptserver and vlserver

<pre>bos create afscell1 ptserver simple /usr/lib/openafs/ptserver -localauth

bos create afscell1 vlserver simple /usr/lib/openafs/vlserver -localauth

</pre>

# Less secure, but more traditional

## Shutdown the server

<pre> service openafs-fileserver stop </pre>

## Startup in noauth mode

<pre>/usr/sbin/bosserver -noauth</pre>

## Add the admin user through bos

<pre>bos adduser localhost root.admin -noauth</pre>

## Protection database and vlserver

<pre>bos create localhost ptserver simple /usr/lib/openafs/ptserver -noauth

bos create localhost vlserver simple /usr/lib/openafs/vlserver -noauth

</pre>

## Make sure that systems:administrators has root.admin in it

<pre>pts membership system:administrators -noauth</pre>

## Kill the unauthenticated bosserver

<pre>pkill bosserver</pre>

## Start the bos server

<pre> service openafs-fileserver start </pre>

# start up the fileserver

<pre>bos create localhost fs fs -cmd /usr/lib/openafs/fileserver -cmd /usr/lib/openafs/volserver -cmd /usr/lib/openafs/salvager -localauth</pre>

# setup the backup server

<pre>sudo bos create localhost buserver simple "/usr/lib/openafs/buserver" -localauth</pre>

# setup the update clients to make /etc/openafs sync correctly every 2 hours

<pre>sudo bos create -server localhost -instance upclientetc -type simple -cmd "/usr/lib/openafs/upclient afsdb1.dev.ru.is -crypt -t 120 /etc/openafs"</pre>

# Wait for a while (30 sec) for the ubik to sync up

# Start up the client

<pre>service openafs-client start</pre>

# Check to make sure the servers are still running

<pre>bos status localhost</pre>

# You are ready to make volumes in the partition and mount them!

h3. Replication

It is a good idea to replicate files that are changed infrequently, particularly the root.afs and other bottom level directories.  Remember that you will need to only change things in the read-write /afs/.dev.ru.is then do a <pre>vos release <id></pre>.

foley has created a script example in @/afs/dev.ru.is/project/devnet/Scripts/afs-replicate-vols.sh@

# Create the replications on the various servers

<pre>vos addsite afsdb2.dev.ru.is a root.afs

vos addsite samvinna.dev.ru.is a root.afs

</pre>

# Now setup a bos job to release them on a regular schedule http://docs.openafs.org/AdminGuide/ch04s05.html

<pre>

bos create afsdb1.dev.ru.is releaserootafs cron "/usr/bin/vos release root.afs -local" 0:00

bos create afsdb1.dev.ru.is releaserootcell cron "/usr/bin/vos release root.cell -local" 0:00

</pre>

h2.  Shared keys and multiple Kerberos realms

h3.  Shared keys

Log into the servers and create matching krbtgt principals with matching KVNO.  (Make sure the kvno matches or it will not work.)

Copies of the magic passwords are on the servers at @/etc/krb5kdc/sharedpw.txt@

Commands for crossrealm between CS.RU.IS and DEV.RU.IS

<pre>kadmin.local  -e "des3-hmac-sha1:normal des-cbc-crc:v4"

addprinc -requires_preauth krbtgt/DEV.RU.IS@CS.RU.IS

addprinc -requires_preauth krbtgt/CS.RU.IS@DEV.RU.IS

</pre>

You will then need to modify the @/etc/krb5.conf@ files on the clients.  (or setup DNS)

<pre>[capath]

DEV.RU.IS = {

          CS.RU.IS = .

}

CS.RU.IS = {

         DEV.RU.IS = .

}</pre>

To test it, get keys in one realm and see if you can kvno in the other

<pre>kinit foley@CS.RU.IS

kvno foley@DEV.RU.IS</pre>

Once it is properly working, you can get service keys (such as AFS) in the other realm.

<pre>kinit foley@CS.RU.IS

aklog dev.ru.is

 klist

Ticket cache: FILE:/tmp/krb5cc_7812_A14864

Default principal: foley@CS.RU.IS

Valid starting    Expires           Service principal

17/04/2013 02:09  18/04/2013 02:09  krbtgt/CS.RU.IS@CS.RU.IS

17/04/2013 02:09  18/04/2013 02:09  krbtgt/DEV.RU.IS@CS.RU.IS

17/04/2013 02:09  18/04/2013 02:09  afs/dev.ru.is@DEV.RU.IS

</pre>

h3. Multiple realms

# put a space separated list of the valid kerberos realms to map into @/etc/openafs/server/krb.conf@ then restart.

** If you're feeling lazy, login to the DB servers and run:

<pre>cp /afs/dev.ru.is/project/devnet/SVN/Machines/AFSDB1/etc/openafs/server/krb.conf /etc/openafs/server/.

bos restart localhost -localauth -all

</pre>

# then add the afs service keys into the KeyList

h1. CentOs setup

* https://www.sit.auckland.ac.nz/Installing_OpenAFS_on_Red_Hat_distributions_including_Fedora

h2. AFS Client

# add this to /usr/vice/etc/CellServDB

<pre>>ru.is          #Reykjavik University

130.208.209.47  #njord.dev.ru.is

</pre>

h2. AFS Server 

# sudo yum install -y openafs-server

# edit /usr/afs/etc/CellServDB

<pre>ru.is #Reykjavik University

njord.dev.ru.is   130.208.209.47

</pre>

h1. Active Directory

Go here for the [[ActiveDirectory KDC]] setup

h1. Status

* Machines number 1, 3 and 5 on the stack above the catalyst (counted from the top) have ubuntu 10.04 installed and sshd running.

* root password: 100feet

* Number 2 and 4 are broken. 2 does not boot and 4 has errors in RAM and does not detect a disk.

* Network is set up and Ubuntu installed.

** gryla (130.208.209.37)  kerberos server, kadmin server, AFSDB server (dev.ru.is), AFSCELL server (dev.ru.is a and b)

** stekkjastaur (130.208.209.39)

** giljagaur (130.208.209.40).

* No proper DNS for now, so they are in Joe's dev.objid.net domain (e.g., gryla.dev.objid.net).

h1. Frequently Asked Questions

h2. I set it up, but the bos commands aren't working and the filserver isn't starting.  The FileLog says something about "Couldn't get CPS for AnyUser"

Chances are good that your keys aren't properly setup.  https://lists.openafs.org/pipermail/openafs-info/2001-December/002736.html

Make sure that the kvno is correct for all of the AFS keys installed on all of the servers.