Backup with Amanda: tape, NAS, Amazon S3

From MyWiki

Revision as of 15:36, 28 February 2015 by Admin (Talk | contribs)
Jump to: navigation, search

Contents

Amanda

Preparing the server

Syslog settings

We need to configure Syslog to send messages out to Logstash through Redis in RELP format.

root@amdbkp-01:~# apt-get update

root@amdbkp-01:~# apt-get install python-software-properties

root@amdbkp-01:~# add-apt-repository ppa:adiscon/v8-stable
You are about to add the following PPA to your system:
 Contains the latest RSyslog V8-Stable packages and dependencies.
 More info: https://launchpad.net/~adiscon/+archive/ubuntu/v8-stable
Press [ENTER] to continue or ctrl-c to cancel adding it

gpg: keyring `/tmp/tmpLRDNaP/secring.gpg' created
gpg: keyring `/tmp/tmpLRDNaP/pubring.gpg' created
gpg: requesting key 5234BF2B from hkp server keyserver.ubuntu.com
gpg: /tmp/tmpLRDNaP/trustdb.gpg: trustdb created
gpg: key 5234BF2B: public key "Launchpad PPA for Adiscon" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
OK

Another apt-get update

root@amdbkp-01:~# apt-get update

Install RELP library and rsyslog

root@amdbkp-01:~# apt-get install librelp0 rsyslog rsyslog-relp

Create configuration file to send messages to Redis in RELP format

root@amdbkp-01:~# vi /etc/rsyslog.d/sendsyslog2logstash.conf

Add this line into the file, save and exit.

action(type="omrelp" target="redis-m.production" port="2514")

Add RELP configuration to /etc/rsyslog.conf file (add the below lines into Modules section):

#RELP Output
module(load="omrelp")

Turn off repeated message reduction, so you have every message instead of those pesky "message repeated N times" lines.

# Filter duplicated messages
$RepeatedMsgReduction off

Restart Syslog

root@amdbkp-01:~# service rsyslog restart

Sysctl settings

Some of the estimates from clients may run for some time (tens of minutes) and TCP connection might be killed due to inactivity by Fortigate firewall. Turns out Fortigates have a default setting of five minutes (see the article). Refer to Connection reset by peer article for more details.

We need to reduce net.ipv4.tcp_keepalive_time from default 7200 to 180 to survive that.

Put the below into /etc/sysctl.conf:

# Amanda related, please, do not remove!
# Ref http://wiki.zmanda.com/index.php/Mesg_read:_Connection_reset_by_peer
net.ipv4.tcp_keepalive_time = 180

Change it while the server is still running:

root@amdbkp-01:~# echo 180 > /proc/sys/net/ipv4/tcp_keepalive_time

Installing the software

  • WARNING* Restore test revealed problem with the Amanda 3.3.0 included by default with Ubuntu 12.04. Ended up installing the latest version of Amanda 3.3.5 downloaded directly from Amanda.

Thanks to this thread for the hint!

First we pull packages from Amanda site and remove default Ubuntu 12.04 LTS Amanda packages (if installed).

root@amdbkp-01:~# cd /tmp
root@amdbkp-01:~# wget -t0 -c http://www.zmanda.com/downloads/community/Amanda/3.3.5/Ubuntu-12.04/amanda-backup-server_3.3.5-1Ubuntu1204_amd64.deb .
root@amdbkp-01:~# wget -t0 -c http://www.zmanda.com/downloads/community/Amanda/3.3.5/Ubuntu-12.04/amanda-backup-client_3.3.5-1Ubuntu1204_amd64.deb . 
    
root@amdbkp-01:~# apt-get remove amanda-server amanda-client amanda-common

This packages depend on xinetd and get text been installed. Since we are not using xinetd, we will ignore it, but install gettext

Inetd way

If you want to stay with inetd. Mind you, it will create problem with 'apt-get install' later.

root@amdbkp-01:~# apt-get will be throwing this error later
root@amdbkp-01:~# apt-get install gettext
root@amdbkp-01:~# dpkg -i --ignore-depends=xinetd ./amanda-backup-server_3.3.5-1Ubuntu1204_amd64.deb ./amanda-backup-client_3.3.5-1Ubuntu1204_amd64.deb

Put back the line to inetd.conf. You don't really need this until you want to stick with inetd.

root@amdbkp-01:~# tail -1 /etc/inetd.conf 
amanda stream tcp nowait amandabackup /usr/libexec/amanda/amandad amandad -auth=bsdtcp amdump amindexd amidxtaped

Kick inetd to reload its config

root@amdbkp-01:~# ps -ef | grep inetd
root@amdbkp-01:~# kill -HUP 28822

Check that inetd picked up the config

root@amdbkp-01:~# netstat -anp | grep 1008
tcp        0      0 0.0.0.0:10080           0.0.0.0:*               LISTEN      28822/inetd  
Xinetd way

The packages built by Amanda team are dependant on xinet and unless you like some pain, you better stick to the defaults

        
root@amdbkp-01:~# apt-get install gettext xinetd
root@amdbkp-01:~# dpkg -i ./amanda-backup-server_3.3.5-1Ubuntu1204_amd64.deb ./amanda-backup-client_3.3.5-1Ubuntu1204_amd64.deb 

Fix permissions on the directories that were owned by Ubuntu's own Amanda package backup:backup

root@amdbkp-01:~# chown -R amandabackup:disk /var/lib/amanda
root@amdbkp-01:~# chown -R amandabackup:disk /etc/amanda
root@amdbkp-01:~# chown -R amandabackup:disk /tmp/amanda
root@amdbkp-01:~# chown -R amandabackup:disk /amanda

Amanda 3.3.5 .amandahosts is not /etc ---> /var/lib/amanda/.amandahosts

amandabackup@amdbkp-01:~$ ls -l /var/lib/amanda/.amandahosts
-rw------- 1 amandabackup disk 250 Jul  1 01:14 /var/lib/amanda/.amandahosts

amandabackup@amdbkp-01:~$ cat /var/lib/amanda/.amandahosts
localhost root amindexd amidxtaped
localhost.localdomain root amindexd amidxtaped
localhost amandabackup amdump
localhost.localdomain amandabackup amdump
amdbkp-01.production  amandabackup amdump
amdbkp-01 amandabackup amdump

Time

Fix /etc/ntp.conf to point to internal NTP servers 192.168.0.10 and 192.168.1.10

driftfile /var/lib/ntp/ntp.drift
statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable
server 192.168.1.10
server 192.168.0.10
server ntp.ubuntu.com
restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery
restrict 127.0.0.1
restrict ::1
ntpq> pe
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*192.168.1.10      193.1.219.116    2 u   13   64   37    0.169   10.502   1.580
 192.168.0.10      91.189.94.4      3 u   11   64   37    0.119    5.908   1.268
 91.189.94.4     193.79.237.14    2 u   13   64   37   13.113    5.955   1.247

Holding space

Create logical volume and filesystem:

root@amdbkp-01:~# lvcreate -n amandalv -L100G amdbkp-01-vg
root@amdbkp-01:~# mkfs.ext4 /dev/amdbkp-01-vg/amandalv 
root@amdbkp-01:~# mount /dev/amdbkp-01-vg/amandalv /amanda
root@amdbkp-01:~# chown amandabackup:disk /amanda
root@amdbkp-01:~# chmod g+w /amanda 

Add the new filesystem to /etc/fstab

root@amdbkp-01:~# grep amanda /etc/fstab
/dev/mapper/amdbkp--01--vg-amandalv /amanda            ext4    defaults        0       2

Postfix configuration

We only need to send emails out, not to receive any, so the configuration is as slim as it can be:

root@amdbkp-01:~# cat /etc/postfix/main.cf 
relayhost = smtp.production 
transport_maps = hash:/etc/postfix/transport

root@amdbkp-01:~# cat /etc/postfix/transport
production	smtp:smtp.production 
corporate	smtp:smtp.production 

root@amdbkp-01:~# cat /etc/aliases
# See man 5 aliases for format
postmaster:    root
root:	alex.markelov @corporate

Start Postfix

root@amdbkp-01:~# service postfix start

Configuring encryption

Create encryption key

root@amdbkp-01:~# pwgen -s 45 -N1

Save the key in /var/lib/amanda/.am_passphrase

root@amdbkp-01:~# vi /var/lib/amanda/.am_passphrase
root@amdbkp-01:~# chown amandabackup:disk /var/lib/amanda/.am_passphrase
root@amdbkp-01:~# chmod 0600 /var/lib/amanda/.am_passphrase
root@amdbkp-01:~# ls -l ~amandabackup/.am_passphrase 
-rw------- 1 amandabackup disk 77 Jul  1 00:46 /var/lib/amanda/.am_passphrase

Dumptype tar-client-encrypt is responsible for encrypted backups. An example below makes full backups every time.

define dumptype tar-client-encrypt {
        always-full-tar
        comment "partitions dumped with tar + encrypted"
        encrypt client
        client_encrypt "/usr/sbin/amcrypt-ossl"
        client_decrypt_option "-d"
}

Wrapper script to report backup status to Logstash/Kibana

Below is the script that engages amdump for given configuration and then sends report in JSON format to Redis.

#
# Wrapper script to run dump and pump report out to Logstash/Kibana
#
# Pre-requisites: mlogger, RELP configuration in rsyslog
#

PATH=/usr/sbin:/usr/bin:/bin
LOG_PRG_NAME=amanda_report

if [ $# -lt 1 ]
then
	echo "Usage: `basename $0` <backup_configuration_name>"
	exit -1
fi

# Perform the backup
amdump $1 

# send out JSON formatted report to Logstash
amreport --format=json, $1 | sed -e 's/^$/ /g' | tr -d '\n' | mlogger -t ${LOG_PRG_NAME}

When you create cron job for backing up, use the script (example below)

# Data Integration backups
0 20 * * *      /etc/amanda/amdump_and_report.sh prod-1week-vtape-RAIT

Tape library

Changer is available as /dev/sg9 device:

root@amdbkp-01:~# mtx -f /dev/sg9 status
  Storage Changer /dev/sg9:1 Drives, 16 Slots ( 0 Import/Export )
Data Transfer Element 0:Empty
      Storage Element 1:Full 
      Storage Element 2:Full 
      Storage Element 3:Full 
      Storage Element 4:Full 
      Storage Element 5:Empty
      Storage Element 6:Empty
      Storage Element 7:Empty
      Storage Element 8:Empty
      Storage Element 9:Empty
      Storage Element 10:Empty
      Storage Element 11:Empty
      Storage Element 12:Empty
      Storage Element 13:Empty
      Storage Element 14:Empty
      Storage Element 15:Empty
      Storage Element 16:Empty

You can load particular tape using mtx command:

root@amdbkp-01:~# mtx -f /dev/sg9 load 1
Loading media from Storage Element 1 into drive 0...done

root@amdbkp-01:~# mtx -f /dev/sg9 status
  Storage Changer /dev/sg9:1 Drives, 16 Slots ( 0 Import/Export )
Data Transfer Element 0:Full (Storage Element 1 Loaded)
      Storage Element 1:Empty
      Storage Element 2:Full 
      Storage Element 3:Full 
      Storage Element 4:Full 
      Storage Element 5:Empty
      Storage Element 6:Empty
      Storage Element 7:Empty
      Storage Element 8:Empty
      Storage Element 9:Empty
      Storage Element 10:Empty
      Storage Element 11:Empty
      Storage Element 12:Empty
      Storage Element 13:Empty
      Storage Element 14:Empty
      Storage Element 15:Empty
      Storage Element 16:Empty

Checking tape status using mt command:

root@amdbkp-01:~# mt -f  /dev/nst0 status
drive type = 114
drive status = 1140850688
sense key error = 0
residue count = 0
file number = 0
block number = 0

Preparing tapes

Test backup

Test restore

Test encryption

Amazon S3

Reference http://wiki.zmanda.com/index.php/How_To:Backup_to_Amazon_S3

Levente created Access KeyID and Secret Access Key for me

Logged in to S3 console and created bucket called infra there.

File:Infra bucket.png

In the bucket created folder called Amanda where the vTapes will reside.

File:Amanda folder.png

Add the server itself to allowed hosts

root@amdbkp-01:~# cat /etc/amandahosts 
localhost amandabackup
localhost root amindexd amidxtaped
amdbkp-01 amandabackup
amdbkp-01.production  amandabackup

Create production-S3 config

root@amdbkp-01:~# su - amandabackup
amandabackup@amdbkp-01:~$ mkdir /etc/amanda/production-S3 
amandabackup@amdbkp-01:~$ mkdir /etc/amanda/production-S3/log
amandabackup@amdbkp-01:~$ mkdir /etc/amanda/production-S3/index

Below is content of configuration file /etc/amanda/production-S3/amanda.conf

  • WARNING* S3 secret key is not the real one in the config.
org " production"     # your organization name for reports
mailto "root alex.markelov @corporate"  # space separated list of operators at your site
dumpuser "amandabackup"               # the user to run dumps under

inparallel 4            # maximum dumpers that will run in parallel (max 63)
                        # this maximum can be increased at compile-time,
                        # modifying MAX_DUMPERS in server-src/driverio.h
dumporder "sssS"        # specify the priority order of each dumper
                        #   s -> smallest size
                        #   S -> biggest size
                        #   t -> smallest time
                        #   T -> biggest time
                        #   b -> smallest bandwitdh
                        #   B -> biggest bandwitdh
                        # try "BTBTBTBTBTBT" if you are not holding
                        # disk constrained

taperalgo first         # The algorithm used to choose which dump image to send
                        # to the taper.

                        # Possible values:
                        #   [first|firstfit|largest|largestfit|smallest|last]
                        # Default: first.

                        # first         First in - first out.
                        # firstfit      The first dump image that will fit on
                        #               the current tape.
                        # largest       The largest dump image.
                        # largestfit    The largest dump image that will fit on
                        #                the current tape.
                        # smallest      The smallest dump image.
                        # last          Last in - first out.

displayunit "k"         # Possible values: "k|m|g|t"
                        # Default: k.
                        # The unit used to print many numbers.
                        # k=kilo, m=mega, g=giga, t=tera

netusage  600 Kbps      # maximum net bandwidth for Amanda, in KB per sec

dumpcycle 1 weeks       # the number of days in the normal dump cycle
runspercycle 5          # the number of amdump runs in dumpcycle days
                        # (4 weeks * 5 amdump runs per week -- just weekdays)
tapecycle 15 tapes      # the number of tapes in rotation
                        # 4 weeks (dumpcycle) times 5 tapes per week (just
                        # the weekdays) plus a few to handle errors that
                        # need amflush and so we do not overwrite the full
                        # backups performed at the beginning of the previous
                        # cycle

usetimestamps	yes	# Default: Yes. This option allows Amanda to track 
			# multiple runs per calendar day. The only reason one 
			# might disable it is that Amanda versions before 2.5.1 
			# can't read logfiles written when this option was 
			# enabled.

bumpsize 20 Mb          # minimum savings (threshold) to bump level 1 -> 2
bumppercent 20          # minimum savings (threshold) to bump level 1 -> 2
bumpdays 1              # minimum days at each level
bumpmult 4              # threshold = bumpsize * bumpmult^(level-1)

runtapes 1              # number of tapes to be used in a single run of amdump

# amazonaws S3
device_property "S3_BUCKET_LOCATION" "eu-west-1"        # Your S3 bucket location
device_property "S3_ACCESS_KEY" "XXX"			# Your S3 Access Key
device_property "S3_SECRET_KEY" "XXX"			# Your S3 Secret Key
device_property "S3_SSL" "YES"      # Curl needs to have S3 Certification Authority (Verisign today) 
                                    # in its CA list. If connection fails, try setting this no NO 

tpchanger "chg-multi:s3:infra/Amanda/production-S3/slot-{01,02,03,04,05,06,07,08,09,10}" # Number of tapes in your "tapecycle"
changerfile  "s3-statefile"                                           # Amanda will create this file
tapetype S3

define tapetype S3 {
    comment "S3 Bucket"
    length 30 gigabytes # Bucket size 30GB
}

maxdumpsize -1          # Maximum number of bytes the planner will schedule
                        # for a run (default: runtapes * tape_length).

labelstr "^PRODUCTION-[0-9][0-9]*$"     # label constraint regex: all tapes must match

holdingdisk hd1 {
    comment "main holding disk"
    directory "/amanda" # where the holding disk is
    use -100 Mb         # how much space can we use on it
                        # a non-positive value means:
                        #        use all space but that value
    chunksize 1Gb       # size of chunk if you want big dump to be
                        # dumped on multiple files on holding disks
                        #  N Kb/Mb/Gb split images in chunks of size N
                        #             The maximum value should be
                        #             (MAX_FILE_SIZE - 1Mb)
                        #  0          same as INT_MAX bytes
    }

autoflush yes     #
        # if autoflush is set to yes, then amdump will schedule all dump on
        # holding disks to be flush to tape during the run.

infofile "/etc/amanda/production-S3/curinfo"       # database DIRECTORY
logdir   "/etc/amanda/production-S3/log"           # log directory
indexdir "/etc/amanda/production-S3/index"         # index directory


define dumptype global {
    comment "Global definitions"
    # This is quite useful for setting global parameters, so you don't have
    # to type them everywhere.  All dumptype definitions in this sample file
    # do include these definitions, either directly or indirectly.
    # There's nothing special about the name `global'; if you create any
    # dumptype that does not contain the word `global' or the name of any
    # other dumptype that contains it, these definitions won't apply.
    # Note that these definitions may be overridden in other
    # dumptypes, if the redefinitions appear *after* the `global'
    # dumptype name.
    # You may want to use this for globally enabling or disabling
    # indexing, recording, etc.  Some examples:
    # index yes
    # record no
    # split_diskbuffer "/raid/amanda"
    # fallback_splitsize 64m
}

define dumptype always-full {
    global
    comment "Full dump of this filesystem always"
    compress none
    priority high
    dumpcycle 0
    exclude list ".exclude.lst"
}

define dumptype always-full-tar {
    global
    program "GNUTAR"
    comment "Full backup with tar of this filesystem always"
    compress none
    priority high
    index
    dumpcycle 0
    holdingdisk required
    exclude list ".exclude.lst"
}

define dumptype tar-client-encrypt {
	always-full-tar
	comment "partitions dumped with tar + encrypted"
	encrypt client
	client_encrypt "/usr/sbin/amcrypt-ossl"
	client_decrypt_option "-d"
}

define interface local {
    comment "a local disk"
    use 1000 kbps
}

define interface le0 {
    comment "10 Mbps ethernet"
    use 400 kbps
}

Creating vTapes

Create first 9 vTapes in S3 and label it

amandabackup@amdbkp-01:~$ for i in 1 2 3 4 5 6 7 8 9; do amlabel production-S3 PRODUCTION-00$i slot $i; done; 

Check slot-09 vTape

amandabackup@amdbkp-01:~$ amdevcheck production-S3 s3:infra/Amanda/production-S3/slot-09 

The command should return SUCCESS if everything is OK

Test backup

Add /etc to disklist for test purpose:

amandabackup@amdbkp-01:~$ cat /etc/amanda/production-S3/disklist 
amdbkp-01    /etc   tar-client-encrypt

Back it up:

amandabackup@amdbkp-01:~$ amdump production-S3 &
[1] 28522
backup@amdbkp-01:~$ amstatus production-S3
Using /etc/amanda/production-S3/log/amdump.1
From Mon Jun 30 23:58:09 IST 2014

amdbkp-01:/etc 0      3251k finished (23:58:11)

SUMMARY          part      real  estimated
                           size       size
partition       :   1
estimated       :   1                 3240k
flush           :   0         0k
failed          :   0                    0k           (  0.00%)
wait for dumping:   0                    0k           (  0.00%)
dumping to tape :   0                    0k           (  0.00%)
dumping         :   0         0k         0k (  0.00%) (  0.00%)
dumped          :   1      3251k      3240k (100.34%) (100.34%)
wait for writing:   0         0k         0k (  0.00%) (  0.00%)
wait to flush   :   0         0k         0k (100.00%) (  0.00%)
writing to tape :   0         0k         0k (  0.00%) (  0.00%)
failed to tape  :   0         0k         0k (  0.00%) (  0.00%)
taped           :   1      3251k      3240k (100.34%) (100.34%)
  tape 1        :   1      3251k      3240k (  0.01%) PRODUCTION-001 (2 chunks)
4 dumpers idle  : no-dumpers
taper status: Idle
taper qlen: 0
network free kps:       600
holding space   :  97673216k (100.00%)
chunker0 busy   :  0:00:00  ( 14.23%)
 dumper0 busy   :  0:00:00  ( 13.83%)
   taper busy   :  0:00:01  ( 42.72%)
 0 dumpers busy :  0:00:02  ( 86.13%)            not-idle:  0:00:01  ( 50.26%)
                                               no-dumpers:  0:00:01  ( 49.74%)
 1 dumper busy  :  0:00:00  ( 13.87%)
[1]+  Done                    amdump production-S3

Check S3 web UI to see that the tape PRODUCTION-001 was used

File:Production-S3 vtapes.png

Test restore

You need to make sure your /etc/amanda/amanda-client.conf doesn't have tapedev defined. It led to error for S3 when I was trying to set it to what I believed was correct (tapedev "s3:infra/Amanda" or tapedev "s3:infra/Amanda/production-S3").

Error I was getting "Amanda header not found -- unlabeled volume?"

root@amdbkp-01:~# amrecover production-S3
AMRECOVER Version 3.3.5. Contacting server on localhost ...
220 amdbkp-01 AMANDA index server (3.3.5) ready.
Setting restore date to today (2014-07-01)
200 Working date set to 2014-07-01.
200 Config set to production-S3.
200 Dump host set to amdbkp-01.
Use the setdisk command to choose dump disk to recover
amrecover> setdisk /etc
200 Disk set to /etc.
amrecover> lcd /tmp
amrecover> add inetd.conf
Added file /inetd.conf
amrecover> extract

Extracting files using tape drive s3:infra/Amanda on host localhost.
The following tapes are needed: PRODUCTION-002

Extracting files using tape drive s3:infra/Amanda on host localhost.
Load tape PRODUCTION-002 now
Continue [?/Y/n/s/d]? y
Amanda header not found -- unlabeled volume?
Load tape PRODUCTION-002 now
Continue [?/Y/n/d]? n
Got no header and data from server, check in amidxtaped.*.debug and amandad.*.debug files on server
amrecover> quit
200 Good bye.

Commenting out tapedev in amanda-client.conf solved the issue. Ref: https://forums.zmanda.com/showthread.php?2811-Problem-amrecover-from-S3

root@amdbkp-01:/tmp/1# cat /etc/amanda/amanda-client.conf 
#
# amanda-client.conf - sample Amanda client configuration file.
#
# This file normally goes in /etc/amanda/amanda-client.conf.
#

conf "DailySet1"		# your config name

index_server "localhost"	# your amindexd server
tape_server  "localhost"	# your amidxtaped server
#tapedev      "s3:infra/Amanda"
#tapedev      "tape:/dev/YOUR-TAPE-DEVICE-HERE"	# your tape device
			# if not set, Use configure or ask server.
			# if set to empty string "", ask server
			# amrecover will use the changer if set to the value
			# of 'amrecover_changer' in the server amanda.conf.

#   auth	- authentication scheme to use between server and client.
#		  Valid values are "bsd", "bsdudp", "bsdtcp", "krb5", "local",
#		  "rsh" and "ssh".  
#		  Default: [auth "bsdtcp"]
auth "bsdtcp"

ssh_keys ""			# your ssh keys file if you use ssh auth

Successful restore after fixing /etc/amanda/amanda-client.conf. We will create /tmp/1 to restore file into.

root@amdbkp-01:/tmp# mkdir /tmp/1
root@amdbkp-01:/tmp# amrecover production-S3
AMRECOVER Version 3.3.5. Contacting server on localhost ...
220 amdbkp-01 AMANDA index server (3.3.5) ready.
Setting restore date to today (2014-07-01)
200 Working date set to 2014-07-01.
200 Config set to production-S3.
200 Dump host set to amdbkp-01.
Use the setdisk command to choose dump disk to recover
amrecover> lcd /tmp/1
amrecover> setdisk /etc
200 Disk set to /etc.
amrecover> add inetd.conf
Added file /inetd.conf
amrecover> extract

Extracting files using tape drive chg-multi:s3:infra/Amanda/production-S3/slot-{01,02,03,04,05,06,07,08,09} on host localhost.
The following tapes are needed: PRODUCTION-002

Extracting files using tape drive chg-multi:s3:infra/Amanda/production-S3/slot-{01,02,03,04,05,06,07,08,09} on host localhost.
Load tape PRODUCTION-002 now
Continue [?/Y/n/s/d]? y
Restoring files into directory /tmp/1
All existing files in /tmp/1 can be deleted
Continue [?/Y/n]? y

./inetd.conf
amrecover> quit
200 Good bye.

Now checking the file we just restored:

root@amdbkp-01:/tmp# cd 1
root@amdbkp-01:/tmp/1# ls -al
total 12
drwxr-xr-x 2 root root 4096 Jul  1 01:45 .
drwxrwxrwt 6 root root 4096 Jul  1 01:45 ..
-rw-r--r-- 1 root root 1170 Jul  1 01:12 inetd.conf

Test encryption

  • NOTE* You can use the same method to download vTape from S3 and restore locally without using Amanda

To prove that we have the vTapes encrypted at S3, first, download vTape from S3 using web UI

File:Sample tape download.png

Alexs-MacBook:Downloads alex$ ls -l slot-02f00000001-b0000000000000000.data 
-rw-r--r--@ 1 alex  staff  3512368  1 Jul 01:16 slot-02f00000001-b0000000000000000.data

Copy the file over to Amanda server

Alexs-MacBook:Downloads alex$ scp slot-02f00000001-b0000000000000000.data amdbkp-01:/tmp/
slot-02f00000001-b0000000000000000.data                                      100% 3430KB   3.4MB/s   00:01 

Check the vTape with and without decryption

root@amdbkp-01:/tmp# dd if=slot-02f00000001-b0000000000000000.data bs=32k |/usr/sbin/amcrypt-ossl -d|tar tvf -
drwxr-xr-x root/root      1865 2014-07-01 01:13 ./
drwxr-xr-x root/root         6 2014-04-24 15:25 ./X11/
drwxr-xr-x root/root         1 2012-07-03 07:21 ./X11/xkb/
drwxr-xr-x root/root        22 2014-04-24 15:32 ./acpi/
drwxr-xr-x root/root        11 2014-04-24 15:32 ./acpi/events/
drwxr-xr-x root/root       952 2014-06-11 06:36 ./alternatives/
drwxrwx--- amandabackup/disk 53 2014-07-01 00:46 ./amanda/ 

<more output skipped>

Now, try without decryption part of the command (/usr/sbin/amcrypt-ossl -d) and see how tar can't do anything with the tape:

root@amdbkp-01:/tmp# dd if=slot-02f00000001-b0000000000000000.data bs=32k |tar tvf -
tar: This does not look like a tar archive
tar: Skipping to next header
107+1 records in
107+1 records out
tar: 3512368 bytes (3.5 MB) copiedExiting with failure status due to previous errors, 0.00662878 s, 530 MB/s

NAS based vTapes

The below based on vTape setup in Amanda wiki.

We need NFS client

root@amdbkp-01:~# apt-get install nfs-common

Mount the test NFS share and prep directory for vTapes

root@amdbkp-01:~# mount.nfs dc-nas-01:/test /vtapes
root@amdbkp-01:~# mkdir /vtapes/amanda.vtapes
root@amdbkp-01:~# chown amandabackup:disk /vtapes/amanda.vtapes
root@amdbkp-01:~# ls -dl /vtapes/amanda.vtapes
drwxrwx---+ 2 amandabackup disk 4096 Jul  7 10:50 /vtapes/amanda.vtapes

Creating vTapes

Switch to amanda user

root@amdbkp-01:~# su - amandabackup

Prepare configuration directory. We will take S3 directory and files as the starting point:

amandabackup@amdbkp-01:~$ cd /etc/amanda/ 
amandabackup@amdbkp-01:~$ cp -rp production-S3 production-vtape 
amandabackup@amdbkp-01:~$  cp -rp production-S3 production-vtape
amandabackup@amdbkp-01:~$  cd production-vtape/
amandabackup@amdbkp-01:~$  rm -rf tapelist* disklist s3-statefile curinfo index/* log/*

Define tape type VTAPE and make the size the same as S3 bucket, because we are going to do RAIT with S3 later.

tapetype VTAPE

define tapetype VTAPE {
    comment "NFS vTape share"
    length 30 gigabytes # Bucket size 30GB
}

Define tpchanger

tpchanger "chg-disk:/vtapes/amanda.vtapes"
changerfile  "vtapes-statefile"                                           # Amanda will create this file

Create the directory structure and label the first 9 tapes

amandabackup@amdbkp-01:~$ for i in 1 2 3 4 5 6 7 8 9; do mkdir /vtapes/amanda.vtapes/slot${i} && amlabel production-vtape PRODUCTION-00$i slot $i; done;

amdevcheck failed until the first backup run. Must be a bug.

amandabackup@amdbkp-01:~$ amdevcheck production-vtape file:/vtapes/amanda.vtapes/slot1
MESSAGE Error checking directory /vtapes/amanda.vtapes/slot1/data/: No such file or directory
DEVICE_ERROR

Test backup

Check which tape is next in the backup cycle

amandabackup@amdbkp-01:/etc/amanda/production-vtape$ amadmin production-vtape tape next

Start the backup (note: we had copied the configuration from S3 configuration directory, so it's already has disklist file with /etc in there) and check its status:

amandabackup@amdbkp-01:/etc/amanda/production-vtape$ amdump production-vtape &
[1] 10546
amandabackup@amdbkp-01:/etc/amanda/production-vtape$ amstatus production-vtape
Using /etc/amanda/production-vtape/log/amdump.1
From Mon Jul  7 12:26:13 IST 2014

amdbkp-01:/etc 0      3361k finished (12:26:14)

SUMMARY          part      real  estimated
                           size       size
partition       :   1
estimated       :   1                 3350k
flush           :   0         0k
failed          :   0                    0k           (  0.00%)
wait for dumping:   0                    0k           (  0.00%)
dumping to tape :   0                    0k           (  0.00%)
dumping         :   0         0k         0k (  0.00%) (  0.00%)
dumped          :   1      3361k      3350k (100.33%) (100.33%)
wait for writing:   0         0k         0k (  0.00%) (  0.00%)
wait to flush   :   0         0k         0k (100.00%) (  0.00%)
writing to tape :   0         0k         0k (  0.00%) (  0.00%)
failed to tape  :   0         0k         0k (  0.00%) (  0.00%)
taped           :   1      3361k      3350k (100.33%) (100.33%)
  tape 1        :   1      3360k      3360k (  0.01%) PRODUCTION-001 (1 chunks)
4 dumpers idle  : 0
taper status: Idle
taper qlen: 0
network free kps:       600
holding space   :  97673216k (100.00%)
chunker0 busy   :  0:00:00  ( 10.25%)
 dumper0 busy   :  0:00:00  (  9.59%)
   taper busy   :  0:00:00  (  8.04%)
 0 dumpers busy :  0:00:01  ( 90.41%)                   0:  0:00:01  (100.00%)
 1 dumper busy  :  0:00:00  (  9.59%)
[1]+  Done                    amdump production-vtape

Now when you do amdevcheck on the slot1 it fails, but succeeds on vtape directory above, because that's where the data file is. Strange!

amandabackup@amdbkp-01:/etc/amanda/production-vtape$ amdevcheck production-vtape file:/vtapes/amanda.vtapes/slot1
MESSAGE Error checking directory /vtapes/amanda.vtapes/slot1/data/: No such file or directory
DEVICE_ERROR
amandabackup@amdbkp-01:/etc/amanda/production-vtape$ amdevcheck production-vtape file:/vtapes/amanda.vtapes
SUCCESS


Test restore

Prepare destination directory where we restore into for the test:

root@amdbkp-01:~# mkdir /tmp/1 && cd /tmp

Run the test restore

root@amdbkp-01:/tmp# amrecover production-vtape
AMRECOVER Version 3.3.5. Contacting server on localhost ...
220 amdbkp-01 AMANDA index server (3.3.5) ready.
Setting restore date to today (2014-07-07)
200 Working date set to 2014-07-07.
200 Config set to production-vtape.
200 Dump host set to amdbkp-01.
Use the setdisk command to choose dump disk to recover
amrecover> lcd /tmp/1
amrecover> setdisk /etc
200 Disk set to /etc.
amrecover> add inetd.conf
Added file /inetd.conf
amrecover> extract

Extracting files using tape drive chg-disk:/vtapes/amanda.vtapes on host localhost.
The following tapes are needed: PRODUCTION-001

Extracting files using tape drive chg-disk:/vtapes/amanda.vtapes on host localhost.
Load tape PRODUCTION-001 now
Continue [?/Y/n/s/d]? 
Restoring files into directory /tmp/1
All existing files in /tmp/1 can be deleted
Continue [?/Y/n]? 

./inetd.conf
amrecover> quit
200 Good bye.

And now check the file we just restored:

root@amdbkp-01:/tmp# vi 1/inetd.conf 

Test encryption

Locate the vTape file. This is 00001.amdbkp-01._etc.0 in our case:

root@amdbkp-01:/tmp# ls /vtapes/amanda.vtapes/slot1/
00000.PRODUCTION-001  00001.amdbkp-01._etc.0
root@amdbkp-01:/tmp# ls -l /vtapes/amanda.vtapes/slot1/
total 3432
-rw-rw----+ 1 amandabackup disk   32768 Jul  7 12:26 00000.PRODUCTION-001
-rw-rw----+ 1 amandabackup disk 3473520 Jul  7 12:26 00001.amdbkp-01._etc.0

Use dd to inspect the header of the vTape (note: we need only the header hence bs=720)

root@amdbkp-01:/tmp# dd if=/vtapes/amanda.vtapes/slot1/00001.amdbkp-01._etc.0 bs=720 count=1
AMANDA: SPLIT_FILE 20140707122613 amdbkp-01 /etc  part 1/-1  lev 0 comp N program /bin/tar crypt enc client_encrypt /usr/sbin/amcrypt-ossl client_decrypt_option -d
ORIGSIZE=3360
DLE=<<ENDDLE
<dle>
  <program>GNUTAR</program>
  <disk>/etc</disk>
  <level>0</level>
  <auth>BSDTCP</auth>
  <encrypt>CUSTOM<custom-encrypt-program>/usr/sbin/amcrypt-ossl</custom-encrypt-program>
    <decrypt-option>-d</decrypt-option>
  </encrypt>
  <record>YES</record>
  <index>YES</index>
  <datapath>AMANDA</datapath>
  <exclude>
    <list>.exclude.lst</list>
  </exclude>
</dle>
ENDDLE
To restore, position tape at start of file and run:
	dd if=<tape> bs=32k skip=1 | /usr/sbin/amcrypt-ossl -d | /bin/tar -xpGf - ... 


1+0 records in
1+0 records out
720 bytes (720 B) copied, 0.000256799 s, 2.8 MB/s

Test the vTape by extracting the list of files inside:

root@amdbkp-01:/tmp# dd if=/vtapes/amanda.vtapes/slot1/00001.amdbkp-01._etc.0 bs=32k skip=1 | /usr/sbin/amcrypt-ossl -d | /bin/tar -tvf -
drwxr-xr-x root/root      1920 2014-07-07 10:50 ./
drwxr-xr-x root/root         6 2014-04-24 15:25 ./X11/
drwxr-xr-x root/root         1 2012-07-03 07:21 ./X11/xkb/
drwxr-xr-x root/root        22 2014-04-24 15:32 ./acpi/
drwxr-xr-x root/root        11 2014-04-24 15:32 ./acpi/events/
drwxr-xr-x root/root       952 2014-06-11 06:36 ./alternatives/
drwxrwx--- amandabackup/disk 71 2014-07-07 11:04 ./amanda/

<… more output skipped…>

Doing the same without decryption (/usr/sbin/amcrypt-ossl -d) will result in failure:

root@amdbkp-01:/tmp# dd if=/vtapes/amanda.vtapes/slot1/00001.amdbkp-01._etc.0 bs=32k skip=1 | /bin/tar -tvf -
/bin/tar: This does not look like a tar archive
/bin/tar: Skipping to next header
105+1 records in
105+1 records out
/bin/tar: 3440752 bytes (3.4 MB) copiedExiting with failure status due to previous errors
, 0.0067797 s, 508 MB/s

RAIT (NAS + S3)

Here is the part of Amnda configuration file amanda.conf, which defines the RAIT.

  • WARNING* tapetype should not be inside each changer definition! I took S3 tapetype as the one here, but it can be called whatever.
tpchanger "chg-rait:{nas,s3}"
tapetype S3

define changer nas {
 # vTapes
 tpchanger "chg-disk:/vtapes/amanda.vtapes"
 changerfile  "vtapes-statefile"                                           # Amanda will create this file
}

define changer s3 {
 # amazonaws S3
 device_property "S3_BUCKET_LOCATION" "eu-west-1"       # Your S3 bucket location
 device_property "S3_ACCESS_KEY" "XXX"			# Your S3 Access Key
 device_property "S3_SECRET_KEY" "XXX"			# Your S3 Secret Key
 device_property "S3_SSL" "YES"      # Curl needs to have S3 Certification Authority (Verisign today) 
                                     # in its CA list. If connection fails, try setting this no NO 

 tpchanger "chg-multi:s3:infra/Amanda/production-S3/slot-{01,02,03,04,05,06,07,08,09}" # Number of tapes in your "tapecycle"
 changerfile  "s3-statefile"                                           # Amanda will create this file
}

define tapetype S3 {
    comment "S3 Bucket"
    length 30 gigabytes # Bucket size 30GB
    blocksize 1 megabytes
}


tapelist we take from S3 configuration, because RAIT driver won't create tapes for us in S3. Took the list from S3 configuration directory. Mind the blocksize parameter, it's set in amanda.conf file.

0 PRODUCTION-009 reuse BLOCKSIZE:10240
0 PRODUCTION-008 reuse BLOCKSIZE:10240
0 PRODUCTION-007 reuse BLOCKSIZE:10240
0 PRODUCTION-006 reuse BLOCKSIZE:10240
0 PRODUCTION-005 reuse BLOCKSIZE:10240
0 PRODUCTION-004 reuse BLOCKSIZE:10240
0 PRODUCTION-003 reuse BLOCKSIZE:10240
0 PRODUCTION-002 reuse BLOCKSIZE:10240
0 PRODUCTION-001 reuse BLOCKSIZE:10240

Creating vTapes

Because we had tested vTape and S3 configurations by now, we have tapes in the tow locations having different timestamps and data in it, so RAIT check will be failing. To fix this we need to clean up the both tape locations and re-create tapes.

Removing and re-creating vTapes

amandabackup@amdbkp-01:~$ rm -rf /vtapes/amanda.vtapes/*
amandabackup@amdbkp-01:~$ cd /etc/amanda/production-vtape
amandabackup@amdbkp-01:~$ rm -rf tapelist* vtapes-statefile curinfo index/* log/*
amandabackup@amdbkp-01:~$ for i in 1 2 3 4 5 6 7 8 9; do mkdir /vtapes/amanda.vtapes/slot${i} && amlabel production-vtape PRODUCTION-00$i slot $i; done;

Removing and re-creating vTapes in S3

amandabackup@amdbkp-01:~$ cd /etc/amanda/production-S3
amandabackup@amdbkp-01:~$ rm -rf tapelist* s3-statefile curinfo index/* log/*

Login to Amazon S3 management console and remove all tape files from infra/Amanda/production-S3 bucket, then re-create the tapes from the backup server:

amandabackup@amdbkp-01:~$ for i in 1 2 3 4 5 6 7 8 9; do amlabel production-S3 PRODUCTION-00$i slot $i; done;

Copy production-vtape/vtapes-statefile and production-S3/s3-statefile to /etc/amanda/production-S3-vtape-RAIT

amandabackup@amdbkp-01:~$ cat s3-statefile 
$STATE = {
           'current_slot_by_config' => {
                                         'production-S3-vtape-RAIT' => 's3:infra/Amanda/production-S3/slot-04'
                                       },
           'slots' => {
                        's3:infra/Amanda/production-S3/slot-09' => {
                                                                        'label' => 'PRODUCTION-009',
                                                                        'device_status' => '0',
                                                                        'f_type' => '1',
                                                                        'state' => 1,
                                                                        'device_error' => undef
                                                                      },
                        's3:infra/Amanda/production-S3/slot-03' => {
                                                                        'label' => 'PRODUCTION-003',
                                                                        'device_status' => '0',
                                                                        'f_type' => '1',
                                                                        'state' => 1,
                                                                        'device_error' => undef
                                                                      },
                        's3:infra/Amanda/production-S3/slot-04' => {
                                                                        'device_status' => '0',
                                                                        'label' => 'PRODUCTION-004',
                                                                        'f_type' => '1',
                                                                        'device_error' => undef,
                                                                        'state' => 1
                                                                      },
                        's3:infra/Amanda/production-S3/slot-07' => {
                                                                        'label' => 'PRODUCTION-007',
                                                                        'device_status' => '0',
                                                                        'f_type' => '1',
                                                                        'state' => 1,
                                                                        'device_error' => undef
                                                                      },
                        's3:infra/Amanda/production-S3/slot-01' => {
                                                                        'label' => 'PRODUCTION-001',
                                                                        'device_status' => '0',
                                                                        'f_type' => '1',
                                                                        'state' => 1,
                                                                        'device_error' => undef
                                                                      },
                        's3:infra/Amanda/production-S3/slot-06' => {
                                                                        'device_status' => '0',
                                                                        'label' => 'PRODUCTION-006',
                                                                        'f_type' => '1',
                                                                        'device_error' => undef,
                                                                        'state' => 1
                                                                      },
                        's3:infra/Amanda/production-S3/slot-08' => {
                                                                        'device_status' => '0',
                                                                        'label' => 'PRODUCTION-008',
                                                                        'f_type' => '1',
                                                                        'device_error' => undef,
                                                                        'state' => 1
                                                                      },
                        's3:infra/Amanda/production-S3/slot-05' => {
                                                                        'label' => 'PRODUCTION-005',
                                                                        'device_status' => '0',
                                                                        'f_type' => '1',
                                                                        'state' => 1,
                                                                        'device_error' => undef
                                                                      },
                        's3:infra/Amanda/production-S3/slot-02' => {
                                                                        'device_status' => '0',
                                                                        'label' => 'PRODUCTION-002',
                                                                        'f_type' => '1',
                                                                        'device_error' => undef,
                                                                        'state' => 1
                                                                      }
                      }
         };
amandabackup@amdbkp-01:~$ cat vtapes-statefile 
$STATE = {
           'drives' => {
                         '/vtapes/amanda.vtapes/drive0' => {}
                       }
         };

If we need more tapes in the RAIT, this article describes how to add more tapes. This requires fiddling with configuration file.


Test backup

Run test backup and check its status:

amandabackup@amdbkp-01:~$ amdump production-S3-vtape-RAIT &
[1] 11922
amandabackup@amdbkp-01:~$ amstatus production-S3-vtape-RAIT
Using /etc/amanda/production-S3-vtape-RAIT/log/amdump.1
From Mon Jul  7 19:55:30 IST 2014

amdbkp-01:/etc 0      3421k finished (19:55:31)

SUMMARY          part      real  estimated
                           size       size
partition       :   1
estimated       :   1                 3420k
flush           :   0         0k
failed          :   0                    0k           (  0.00%)
wait for dumping:   0                    0k           (  0.00%)
dumping to tape :   0                    0k           (  0.00%)
dumping         :   0         0k         0k (  0.00%) (  0.00%)
dumped          :   1      3421k      3420k (100.03%) (100.03%)
wait for writing:   0         0k         0k (  0.00%) (  0.00%)
wait to flush   :   0         0k         0k (100.00%) (  0.00%)
writing to tape :   0         0k         0k (  0.00%) (  0.00%)
failed to tape  :   0         0k         0k (  0.00%) (  0.00%)
taped           :   1      3421k      3420k (100.03%) (100.03%)
  tape 1        :   1      3420k      3420k (  0.01%) PRODUCTION-002 (1 chunks)
4 dumpers idle  : 0
taper status: Idle
taper qlen: 0
network free kps:       600
holding space   :  97673216k (100.00%)
chunker0 busy   :  0:00:00  ( 11.74%)
 dumper0 busy   :  0:00:00  ( 10.84%)
   taper busy   :  0:00:00  ( 54.68%)
 0 dumpers busy :  0:00:01  ( 89.16%)                   0:  0:00:01  (100.00%)
 1 dumper busy  :  0:00:00  ( 10.84%)

Now check that both locations were used. Check S3 and NAS and see that tape 02 was used.

S3:

File:RAIT tape 02 on S3 used.png

NAS:

File:RAIT tape 02 on NAS used.png

Test restore

Prepare test restore destination directory and restore a file into it:

root@amdbkp-01:~# mkdir /tmp/1
root@amdbkp-01:~# cd /tmp/1

root@amdbkp-01:/tmp/1# amrecover production-S3-vtape-RAIT
AMRECOVER Version 3.3.5. Contacting server on localhost ...
220 amdbkp-01 AMANDA index server (3.3.5) ready.
Setting restore date to today (2014-07-07)
200 Working date set to 2014-07-07.
200 Config set to production-S3-vtape-RAIT.
200 Dump host set to amdbkp-01.
Use the setdisk command to choose dump disk to recover
amrecover> setdisk /etc
200 Disk set to /etc.
amrecover> lcd /tmp/1
amrecover> add resolv.conf
Added file /resolv.conf
amrecover> extract

Extracting files using tape drive chg-rait:{nas,s3} on host localhost.
The following tapes are needed: PRODUCTION-002

Extracting files using tape drive chg-rait:{nas,s3} on host localhost.
Load tape PRODUCTION-002 now
Continue [?/Y/n/s/d]? 
Restoring files into directory /tmp/1
All existing files in /tmp/1 can be deleted
Continue [?/Y/n]? 

./resolv.conf
amrecover> quit
200 Good bye.

Check what we have restored:

root@amdbkp-01:/tmp/1# ls -l
total 0
lrwxrwxrwx 1 root root 29 Apr 24 15:25 resolv.conf -> ../run/resolvconf/resolv.conf

Test encryption

It's the same process as what we did for S3 and NAS respectively above. Fetch the vTape file from S3 and use dd to inspect it and you can do the same with NAS based vTape file.


RAIT (NAS Datacenter + NAS off-site)

We will create a RAIT over two NAS appliances:

  • Office comms room office-nas-01
  • Datacenter dc-nas-01

From the previous configuration (NAS+S3) we already have vTapes share mounted off the NAS in Datacenter.

root@amdbkp-01:~# df -k /vtapes
Filesystem                                    1K-blocks       Used   Available Use% Mounted on
dc-nas-01:/test                          29056019968 9221493632 19834002048  32% /vtapes

Now we mount NFS share off office based (off-site) NAS:

root@amdbkp-01:~# mkdir /vtapes-offsite

root@amdbkp-01:~# mount office-nas-01:/AMANDA /vtapes-offsite

root@amdbkp-01:~# df -k | grep vtapes
Filesystem                                    1K-blocks       Used   Available Use% Mounted on
dc-nas-01:/test                          29056019968 9222531968 19832963712  32% /vtapes
office-nas-01:/AMANDA                          5763950016 4348221728  1415204000  76% /vtapes-offsite

Added entries to stab

root@amdbkp-01:~#  tail -2 /etc/fstab
dc-nas-01:/test /vtapes	nfs	defaults	2	2
office-nas-01:/AMANDA	/vtapes-offsite nfs	defaults	2	2

Fixing ownership and permissions on the mounted share:

root@amdbkp-01:~# chown -R amandabackup:disk /vtapes-offsite
root@amdbkp-01:~# ls -ld /vtapes*
drwxrwxrwx+ 6 root         root 4096 Jul 14 20:28 /vtapes
drwxrwxrwx  2 amandabackup disk 4096 Jul 16 16:35 /vtapes-offsite


root@amdbkp-01:~# ls -ld /vtapes*
drwxrwxrwx+ 6 root         root 4096 Jul 14 20:28 /vtapes
drwxrwxrwx  2 amandabackup disk 4096 Jul 16 16:35 /vtapes-offsite

root@amdbkp-01:~# chmod o-w /vtapes-offsite

root@amdbkp-01:~# ls -ld /vtapes*
drwxrwxrwx+ 6 root         root 4096 Jul 14 20:28 /vtapes
drwxrwxr-x  2 amandabackup disk 4096 Jul 16 16:35 /vtapes-offsite

Create new Amanda configuration. S3+NAS RAIT is the basis for this one.

root@amdbkp-01:~# su - amandabackup 
amandabackup@amdbkp-01:~$ cd /etc/amanda
amandabackup@amdbkp-01:/etc/amanda$ ls
amanda-client.conf  production-S3  production-S3-vtape-RAIT  production-tape  production-vtape

amandabackup@amdbkp-01:/etc/amanda$ cp -rp production-S3-vtape-RAIT production-vtape-RAIT
amandabackup@amdbkp-01:/etc/amanda$ cd production-vtape-RAIT/
amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ ls -l
total 21
-rw-rw-r-- 1 amandabackup disk 7732 Jul  7 20:17 amanda.conf
-rw------- 1 amandabackup disk  135 Jul  7 17:01 changer
drwxr-xr-x 3 amandabackup disk 1024 Jul  7 19:48 curinfo
-rw-r--r-- 1 amandabackup disk   45 Jul  7 12:11 disklist
drwxrwxr-x 3 amandabackup disk 1024 Jul  7 19:48 index
drwxrwxr-x 3 amandabackup disk 1024 Jul  7 20:57 log
-rw------- 1 amandabackup disk 5784 Jul  7 22:44 s3-statefile
-rw------- 1 amandabackup disk  472 Jul  7 20:57 tapelist
-rw------- 1 amandabackup disk    0 Jul  7 19:48 tapelist.lock
-rw------- 1 amandabackup disk  135 Jul  7 22:44 vtapes-statefile

amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ mkdir /vtapes/amanda.vtapes-RAIT
amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ ls -ld /vtapes/amanda.vtapes-RAIT
drwxrwx---+ 2 amandabackup disk 4096 Jul 16 16:42 /vtapes/amanda.vtapes-RAIT

Clean up configuration directory off S3+NAS files

amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ rm -rf tapelist* vtapes-statefile curinfo index/* log/* s3-statefile changer

amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ ls -l
total 11
-rw-rw-r-- 1 amandabackup disk 7732 Jul  7 20:17 amanda.conf
-rw-r--r-- 1 amandabackup disk   45 Jul  7 12:11 disklist
drwxrwxr-x 2 amandabackup disk 1024 Jul 16 16:43 index
drwxrwxr-x 2 amandabackup disk 1024 Jul 16 16:43 log

Edit Amanda file. We define the two changers and the RAIT changer based on the two:

tpchanger "chg-rait:{nas-tc,nas-gqh}"
tapetype VTAPE

define changer nas-tc {
 tpchanger "chg-disk:/vtapes/amanda.vtapes-RAIT"
 changerfile  "vtapes-statefile-tc"                                           # Amanda will create this file
}

define changer nas-gqh {
 tpchanger "chg-disk:/vtapes-offsite"
 changerfile  "vtapes-statefile-gqh"                                           # Amanda will create this file
}

define tapetype VTAPE {
    comment "NFS vTape share"
    length 30 gigabytes # Bucket size 30GB
}

Creating vTapes

Create vTapes and label it. The good thing about two NAS location is that you can do this in one go for both (not like with S3 and NAS, where you had to create vTapes for each configuration separately)

amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ for i in 1 2 3 4 5 6 7 8 9; do mkdir /vtapes/amanda.vtapes-RAIT/slot${i} /vtapes-offsite/slot${i} && amlabel production-vtape-RAIT PRODUCTION-00$i slot $i; done;
Reading label...
Found an empty tape.
Writing label 'PRODUCTION-001'...
Checking label...
Success!
Reading label...
Found an empty tape.
Writing label 'PRODUCTION-002'...
Checking label...
Success!

<...skipped similar output for the rest of the vTapes...>

Check the results in both NAS units.

amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ find /vtapes-offsite/
/vtapes-offsite/
/vtapes-offsite/slot1
/vtapes-offsite/slot1/00000.PRODUCTION-001
/vtapes-offsite/slot8
/vtapes-offsite/slot8/00000.PRODUCTION-008
/vtapes-offsite/slot4
/vtapes-offsite/slot4/00000.PRODUCTION-004
/vtapes-offsite/slot3
/vtapes-offsite/slot3/00000.PRODUCTION-003
/vtapes-offsite/slot6
/vtapes-offsite/slot6/00000.PRODUCTION-006
/vtapes-offsite/slot2
/vtapes-offsite/slot2/00000.PRODUCTION-002
/vtapes-offsite/slot7
/vtapes-offsite/slot7/00000.PRODUCTION-007
/vtapes-offsite/slot5
/vtapes-offsite/slot5/00000.PRODUCTION-005
/vtapes-offsite/slot9
/vtapes-offsite/slot9/00000.PRODUCTION-009
amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ find /vtapes/amanda.vtapes-RAIT/
/vtapes/amanda.vtapes-RAIT/
/vtapes/amanda.vtapes-RAIT/slot2
/vtapes/amanda.vtapes-RAIT/slot2/00000.PRODUCTION-002
/vtapes/amanda.vtapes-RAIT/slot5
/vtapes/amanda.vtapes-RAIT/slot5/00000.PRODUCTION-005
/vtapes/amanda.vtapes-RAIT/slot1
/vtapes/amanda.vtapes-RAIT/slot1/00000.PRODUCTION-001
/vtapes/amanda.vtapes-RAIT/slot4
/vtapes/amanda.vtapes-RAIT/slot4/00000.PRODUCTION-004
/vtapes/amanda.vtapes-RAIT/slot8
/vtapes/amanda.vtapes-RAIT/slot8/00000.PRODUCTION-008
/vtapes/amanda.vtapes-RAIT/slot7
/vtapes/amanda.vtapes-RAIT/slot7/00000.PRODUCTION-007
/vtapes/amanda.vtapes-RAIT/slot9
/vtapes/amanda.vtapes-RAIT/slot9/00000.PRODUCTION-009
/vtapes/amanda.vtapes-RAIT/slot6
/vtapes/amanda.vtapes-RAIT/slot6/00000.PRODUCTION-006
/vtapes/amanda.vtapes-RAIT/slot3
/vtapes/amanda.vtapes-RAIT/slot3/00000.PRODUCTION-003

Checking configuration. Ignore the error about /etc/.exclude.lst missing, we don't have it yet (and may not even need it)

amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ amcheck production-vtape-RAIT
Amanda Tape Server Host Check
-----------------------------
Holding disk /amanda: 97775616 KB disk space available, using 97673216 KB
slot {1,1}: volume 'PRODUCTION-001'
Will write to volume 'PRODUCTION-001' in slot {1,1}.
NOTE: skipping tape-writable test
Server check took 0.196 seconds

Amanda Backup Client Hosts Check
--------------------------------
ERROR: amdbkp-01: [Can't open exclude file /etc/.exclude.lst (No such file or directory)]
Client check: 1 host checked in 2.066 seconds.  1 problem found.

(brought to you by Amanda 3.3.5)

Test backup

amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ amdump production-vtape-RAIT & 
[1] 11946
amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ amstatus  production-vtape-RAIT 
Using /etc/amanda/production-S3-vtape-RAIT/log/amdump.1
From Wed Jul 16 16:54:46 IST 2014

amdbkp-01:/etc 0      3561k finished (16:54:48)

SUMMARY          part      real  estimated
                           size       size
partition       :   1
estimated       :   1                 3560k
flush           :   0         0k
failed          :   0                    0k           (  0.00%)
wait for dumping:   0                    0k           (  0.00%)
dumping to tape :   0                    0k           (  0.00%)
dumping         :   0         0k         0k (  0.00%) (  0.00%)
dumped          :   1      3561k      3560k (100.03%) (100.03%)
wait for writing:   0         0k         0k (  0.00%) (  0.00%)
wait to flush   :   0         0k         0k (100.00%) (  0.00%)
writing to tape :   0         0k         0k (  0.00%) (  0.00%)
failed to tape  :   0         0k         0k (  0.00%) (  0.00%)
taped           :   1      3561k      3560k (100.03%) (100.03%)
  tape 1        :   1      3560k      3560k (  0.01%) PRODUCTION-001 (1 chunks)
4 dumpers idle  : 0
taper status: Idle
taper qlen: 0
network free kps:       600
holding space   :  97673216k (100.00%)
chunker0 busy   :  0:00:00  ( 23.85%)
 dumper0 busy   :  0:00:00  ( 23.28%)
   taper busy   :  0:00:00  ( 12.77%)
 0 dumpers busy :  0:00:01  ( 76.63%)                   0:  0:00:01  (100.00%)
 1 dumper busy  :  0:00:00  ( 23.28%)
[1]+  Done                    amdump production-vtape-RAIT

Check the result (new tape file in slot1) in both NAS.

amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ find /vtapes-offsite/
/vtapes-offsite/
/vtapes-offsite/slot1
/vtapes-offsite/slot1/00001.amdbkp-01._etc.0
/vtapes-offsite/slot1/00000.PRODUCTION-001
/vtapes-offsite/data
/vtapes-offsite/slot8
/vtapes-offsite/slot8/00000.PRODUCTION-008
/vtapes-offsite/slot4
/vtapes-offsite/slot4/00000.PRODUCTION-004
/vtapes-offsite/slot3
/vtapes-offsite/slot3/00000.PRODUCTION-003
/vtapes-offsite/slot6
/vtapes-offsite/slot6/00000.PRODUCTION-006
/vtapes-offsite/slot2
/vtapes-offsite/slot2/00000.PRODUCTION-002
/vtapes-offsite/slot7
/vtapes-offsite/slot7/00000.PRODUCTION-007
/vtapes-offsite/slot5
/vtapes-offsite/slot5/00000.PRODUCTION-005
/vtapes-offsite/slot9
/vtapes-offsite/slot9/00000.PRODUCTION-009
amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ find /vtapes/amanda.vtapes-RAIT/
/vtapes/amanda.vtapes-RAIT/
/vtapes/amanda.vtapes-RAIT/slot2
/vtapes/amanda.vtapes-RAIT/slot2/00000.PRODUCTION-002
/vtapes/amanda.vtapes-RAIT/slot5
/vtapes/amanda.vtapes-RAIT/slot5/00000.PRODUCTION-005
/vtapes/amanda.vtapes-RAIT/slot1
/vtapes/amanda.vtapes-RAIT/slot1/00001.amdbkp-01._etc.0
/vtapes/amanda.vtapes-RAIT/slot1/00000.PRODUCTION-001
/vtapes/amanda.vtapes-RAIT/slot4
/vtapes/amanda.vtapes-RAIT/slot4/00000.PRODUCTION-004
/vtapes/amanda.vtapes-RAIT/slot8
/vtapes/amanda.vtapes-RAIT/slot8/00000.PRODUCTION-008
/vtapes/amanda.vtapes-RAIT/slot7
/vtapes/amanda.vtapes-RAIT/slot7/00000.PRODUCTION-007
/vtapes/amanda.vtapes-RAIT/slot9
/vtapes/amanda.vtapes-RAIT/slot9/00000.PRODUCTION-009
/vtapes/amanda.vtapes-RAIT/slot6
/vtapes/amanda.vtapes-RAIT/slot6/00000.PRODUCTION-006
/vtapes/amanda.vtapes-RAIT/data
/vtapes/amanda.vtapes-RAIT/slot3
/vtapes/amanda.vtapes-RAIT/slot3/00000.PRODUCTION-003

Let's get some bigger files to test backup speed. There are some big TSM files in nas-02:/test, let's grub those.

root@amdbkp-01:~# mount nas-02:/test /mnt

We will need a separate filesystem for the test. Check free space left in the VG.

root@amdbkp-01:~# vgdisplay 
  --- Volume group ---
  VG Name               amdbkp-01-vg
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  12
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                7
  Open LV               7
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               272.98 GiB
  PE Size               4.00 MiB
  Total PE              69883
  Alloc PE / Size       36243 / 141.57 GiB
  Free  PE / Size       33640 / 131.41 GiB
  VG UUID               ENArtj-VebI-x5WG-Lpc0-zN5X-cLpL-FRsT9W

Create new LV and filesystem.

   
root@amdbkp-01:~# lvcreate -n testlv -L10G amdbkp-01-vg
  Logical volume "testlv" created

root@amdbkp-01:~# mkfs.ext4 /dev/amdbkp-01-vg/testlv 

root@amdbkp-01:~# mkdir /test

root@amdbkp-01:~# mount /dev/amdbkp-01-vg/testlv /test

root@amdbkp-01:~# chmod 0755 /test

root@amdbkp-01:~# cp /mnt/TSM71/{TSM_7.1_WIN_*,TSM_7.1_LIN86_AGT_ML.bin} /test/
root@amdbkp-01:~# df -k /tmp
Filesystem                                    1K-blocks        Used   Available Use% Mounted on
/dev/mapper/amdbkp--01--vg-testlv              10190136     4868312     4797536  51% /test

Add /test to disklist.

amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ cat /etc/amanda/production-vtape-RAIT/disklist 
#amdbkp-01    /etc   tar-client-encrypt
amdbkp-01    /test   tar-client-encrypt

Check configuration for errors.

amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ amcheck production-vtape-RAIT
Amanda Tape Server Host Check
-----------------------------
Holding disk /amanda: 97775616 KB disk space available, using 97673216 KB
slot {1,1}: volume 'PRODUCTION-001' is still active and cannot be overwritten
slot {2,2}: volume 'PRODUCTION-002'
Will write to volume 'PRODUCTION-002' in slot {2,2}.
NOTE: skipping tape-writable test
NOTE: info dir /etc/amanda/production-S3-vtape-RAIT/curinfo/amdbkp-01/_test does not exist
NOTE: it will be created on the next run.
NOTE: index dir /etc/amanda/production-S3-vtape-RAIT/index/amdbkp-01/_test does not exist
NOTE: it will be created on the next run.
Server check took 0.220 seconds

Amanda Backup Client Hosts Check
--------------------------------
ERROR: amdbkp-01: [Can't open exclude file /test/.exclude.lst (No such file or directory)]
Client check: 1 host checked in 1.095 seconds.  1 problem found.

(brought to you by Amanda 3.3.5)

Run dump of /test. I thought it was going to take some time and I'd have to walk away and check the result later, so I used 'nohup'.

amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ nohup amdump production-vtape-RAIT &

Checking the status of the dump and it's impressively fast. Mind you, it's the percentage completed to holding disk (!) not the vTapes yet.

amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ amstatus  production-vtape-RAIT  
Using /etc/amanda/production-S3-vtape-RAIT/log/amdump
From Wed Jul 16 17:13:35 IST 2014

amdbkp-01:/test 0   4838040k dumping  1727328k ( 35.70%) (17:13:36)

SUMMARY          part      real  estimated
                           size       size
partition       :   1
estimated       :   1              4838040k
flush           :   0         0k
failed          :   0                    0k           (  0.00%)
wait for dumping:   0                    0k           (  0.00%)
dumping to tape :   0                    0k           (  0.00%)
dumping         :   1   1727328k   4838040k ( 35.70%) ( 35.70%)
dumped          :   0         0k         0k (  0.00%) (  0.00%)
wait for writing:   0         0k         0k (  0.00%) (  0.00%)
wait to flush   :   0         0k         0k (100.00%) (  0.00%)
writing to tape :   0         0k         0k (  0.00%) (  0.00%)
failed to tape  :   0         0k         0k (  0.00%) (  0.00%)
taped           :   0         0k         0k (  0.00%) (  0.00%)
3 dumpers idle  : 0
taper status: Idle
taper qlen: 0
network free kps:         0
holding space   :  92834976k ( 95.05%)
 dumper0 busy   :  0:00:00  (  0.00%)
 0 dumpers busy :  0:00:01  (100.00%)                   0:  0:00:01  (100.00%)
 1 dumper busy  :  0:00:00  (  0.00%)

And now it's writing to vTapes.

amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ amstatus  production-vtape-RAIT  
Using /etc/amanda/production-S3-vtape-RAIT/log/amdump
From Wed Jul 16 17:13:35 IST 2014

amdbkp-01:/test 0   4838041k dump done, writing to tape (1241440k done (25.66%)) (17:14:17)

SUMMARY          part      real  estimated
                           size       size
partition       :   1
estimated       :   1              4838040k
flush           :   0         0k
failed          :   0                    0k           (  0.00%)
wait for dumping:   0                    0k           (  0.00%)
dumping to tape :   0                    0k           (  0.00%)
dumping         :   0         0k         0k (  0.00%) (  0.00%)
dumped          :   1   4838041k   4838040k (100.00%) (100.00%)
wait for writing:   0         0k         0k (  0.00%) (  0.00%)
wait to flush   :   0         0k         0k (100.00%) (  0.00%)
writing to tape :   1   4838041k   4838040k (100.00%) (100.00%)
failed to tape  :   0         0k         0k (  0.00%) (  0.00%)
taped           :   0         0k         0k (  0.00%) (  0.00%)
  tape 1        :   0         0k         0k (  0.00%) PRODUCTION-002
4 dumpers idle  : 0
taper status: Writing amdbkp-01:/test
taper qlen: 0
network free kps:       600
holding space   :  92835015k ( 95.05%)
chunker0 busy   :  0:00:41  ( 97.37%)
 dumper0 busy   :  0:00:41  ( 97.35%)
 0 dumpers busy :  0:00:01  (  2.65%)                   0:  0:00:01  (100.00%)
 1 dumper busy  :  0:00:41  ( 97.35%)                   0:  0:00:41  (100.00%)

Done in 41 seconds (!). What was strange though is that it seemed to have being writing to NAS still even after amstatus reported 100% completion. Need to test it again. When I run 'df -k' right after amstatus reported backup complete, it stayed there for about a minute before I saw df output. And at the same time observing inbound traffic to the office QNAP (via web UI System status -> Bandwidth usage), it was showing ~9MB/s inboud traffic even after amstatus reported 100% completion.

amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ amstatus  production-vtape-RAIT  
Using /etc/amanda/production-S3-vtape-RAIT/log/amdump
From Wed Jul 16 17:13:35 IST 2014

amdbkp-01:/test 0   4838041k dump done, writing to tape (4838048k done (100.00%)) (17:14:17)

SUMMARY          part      real  estimated
                           size       size
partition       :   1
estimated       :   1              4838040k
flush           :   0         0k
failed          :   0                    0k           (  0.00%)
wait for dumping:   0                    0k           (  0.00%)
dumping to tape :   0                    0k           (  0.00%)
dumping         :   0         0k         0k (  0.00%) (  0.00%)
dumped          :   1   4838041k   4838040k (100.00%) (100.00%)
wait for writing:   0         0k         0k (  0.00%) (  0.00%)
wait to flush   :   0         0k         0k (100.00%) (  0.00%)
writing to tape :   1   4838041k   4838040k (100.00%) (100.00%)
failed to tape  :   0         0k         0k (  0.00%) (  0.00%)
taped           :   0         0k         0k (  0.00%) (  0.00%)
  tape 1        :   0         0k         0k (  0.00%) PRODUCTION-002
4 dumpers idle  : 0
taper status: Writing amdbkp-01:/test
taper qlen: 0
network free kps:       600
holding space   :  92835015k ( 95.05%)
chunker0 busy   :  0:00:41  ( 97.37%)
 dumper0 busy   :  0:00:41  ( 97.35%)
 0 dumpers busy :  0:00:01  (  2.65%)                   0:  0:00:01  (100.00%)
 1 dumper busy  :  0:00:41  ( 97.35%)                   0:  0:00:41  (100.00%)


amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ ls -la /vtapes-offsite/slot2/
total 4838132
drwxr-xr-x  2 amandabackup disk       4096 Jul 16 17:14 .
drwxrwxr-x 12 amandabackup disk       4096 Jul 16 17:13 ..
-rw-------  1 amandabackup disk      32768 Jul 16 17:14 00000.PRODUCTION-002
-rw-------  1 amandabackup disk 4954193920 Jul 16 17:22 00001.amdbkp-01._test.0

I wasn's sure if amanda process was gone and not writing anything at that stage, so I checked it:

amandabackup@amdbkp-01:/etc/amanda/production-vtape-RAIT$ ps -ef | grep am
root     11046   897  0 16:31 ?        00:00:00 sshd: alex [priv]
alex 11301 11046  0 16:31 ?        00:00:00 sshd: alex@pts/1
alex 11302 11301  0 16:31 pts/1    00:00:00 -bash
63998    12375 11405  0 17:12 pts/1    00:00:00 su - amandabackup
63998    12598 12376  0 17:22 pts/1    00:00:00 grep am
[1]+  Done                    nohup amdump production-vtape-RAIT

Test restore

Test encryption

Amanda clients

Firewall policies

You need the following policies in the firewall to allow Amanda server to talk to client and client to talk back to server. Below is an example for U3 pre-prod.

File:Amanda firewall rules IN.png


File:Amanda firewall rules OUT.png

If you want to setup iptables for the same, please, follow this article.

Linux

Linux (CentOS)
Client installation
  • NOTE* Linux server infra-mon-02 was used for this example deployment.

Download the package from Amanda site.

[root@infra-mon-02 ]#  cd /tmp && wget -c -t0 http://www.zmanda.com/downloads/community/Amanda/3.3.5/Redhat_Enterprise_6.0/amanda-backup_client-3.3.5-1.rhel6.x86_64.rpm

Install pre-requisite packages:

[root@infra-mon-02 ]#  yum install xinetd perl-JSON

Install Amanda backup client

[root@infra-mon-02 ]#  rpm -Uhv /tmp/amanda-backup_client-3.3.5-1.rhel6.x86_64.rpm 

Add Amanda server line to client's /var/lib/amanda/.amandahosts file

localhost amandabackup amdump
localhost.localdomain amandabackup amdump
amdbkp-01.production  amandabackup amdump

Start xinetd and confirm that it's listening on Amanda client port 10080:

[root@infra-mon-02 ]#  /etc/init.d/xinetd start
[root@infra-mon-02 ]#  netstat -anp | grep 1008
tcp        0      0 0.0.0.0:10080               0.0.0.0:*                   LISTEN      2647/xinetd  
Linux (Ubuntu)
Client installation
  • NOTE* Linux server nas2ftp-01 was used for this example deployment

Download the package from Amanda site.

# cd /tmp && wget -t0 -c http://www.zmanda.com/downloads/community/Amanda/3.3.5/Ubuntu-12.04/amanda-backup-client_3.3.5-1Ubuntu1204_amd64.deb

Install pre-requisite packages gettext and xinetd before you attempt to install Amanda client

nas2ftp-01 # apt-get update
nas2ftp-01 # apt-get install get text xinetd

Now, install Amanda client

nas2ftp-01 # dpkg -i /tmp/amanda-backup-client_3.3.5-1Ubuntu1204_amd64.deb

Add Amanda server line to client's /var/lib/amanda/.amandahosts file

localhost root amindexd amidxtaped
localhost.localdomain root amindexd amidxtaped
localhost amandabackup amdump
localhost.localdomain amandabackup amdump
amdbkp-01.production  amandabackup amdump

If you have iptables active on the client, add the below line to the rule set (usually, it's in /etc/iptables.active.rules, if the client was provisioned from Linux VM template)

#
# allow Amanda in
-A INPUT -s 192.168.0.23 -j ACCEPT

Re-load iptables rules

nas2ftp-01 # iptables-restore < /etc/iptables.active.rules
Backup configuration on backup server

Add what you want to backup to disklist file on Amanda server (below, I added /var/spool/cron/crontabs directory from nas2ftp-01 to be backup)

nas2ftp-01        /var/spool/cron/crontabs/root   tar-client-encrypt

Run check to make sure that everything is fine from Amanda server prospective (you can ignore warning about absence of .exclude.lst file, if you don't use it to exclude something from backup):

amandabackup@amdbkp-01:/etc/amanda/prod-1week-vtape-RAIT$ amcheck prod-1week-vtape-RAIT
Amanda Tape Server Host Check
-----------------------------
Holding disk /amanda: 188 GB disk space available, using 188 GB
slot {8,8}: volume 'PRODUCTION-1WEEK-008'
Will write to volume 'PRODUCTION-1WEEK-008' in slot {8,8}.
NOTE: skipping tape-writable test
NOTE: host info dir /etc/amanda/prod-1week-vtape-RAIT/curinfo/nas2ftp-01 does not exist
NOTE: it will be created on the next run.
NOTE: index dir /etc/amanda/prod-1week-vtape-RAIT/index/nas2ftp-01 does not exist
NOTE: it will be created on the next run.
Server check took 0.192 seconds

Amanda Backup Client Hosts Check
--------------------------------
ERROR: nas2ftp-01: [Can't open exclude file /var/spool/cron/crontabs/.exclude.lst (No such file or directory)]
Client check: 1 host checked in 2.099 seconds.  1 problem found.

(brought to you by Amanda 3.3.5)

Now, we can run a test backup

amandabackup@amdbkp-01:/etc/amanda/prod-1week-vtape-RAIT$ amdump prod-1week-vtape-RAIT nas2ftp-01 &
[1] 17270

amandabackup@amdbkp-01:/etc/amanda/prod-1week-vtape-RAIT$ amstatus prod-1week-vtape-RAIT
Using /etc/amanda/prod-1week-vtape-RAIT/log/amdump
From Wed Oct  8 14:57:18 IST 2014

nas2ftp-01:/var/spool/cron/crontabs 0         0g estimate done

SUMMARY          part      real  estimated
                           size       size
partition       :   1
estimated       :   1                    0g
flush           :   0         0g
failed          :   0                    0g           (  0.00%)
wait for dumping:   0                    0g           (  0.00%)
dumping to tape :   0                    0g           (  0.00%)
dumping         :   0         0g         0g (  0.00%) (  0.00%)
dumped          :   0         0g         0g (  0.00%) (  0.00%)
wait for writing:   0         0g         0g (  0.00%) (  0.00%)
wait to flush   :   0         0g         0g (100.00%) (  0.00%)
writing to tape :   0         0g         0g (  0.00%) (  0.00%)
failed to tape  :   0         0g         0g (  0.00%) (  0.00%)
taped           :   0         0g         0g (  0.00%) (  0.00%)
4 dumpers idle  : 0
taper status: Idle
taper qlen: 0
network free kps:     80000
holding space   :       188g (100.00%)
 0 dumpers busy :  0:00:00  (100.00%)

Shortly after you will have email report to say that the backup was good:

Hostname: amdbkp-01
Org     :  production. 1 week cycle.
Config  : prod-1week-vtape-RAIT
Date    : October 8, 2014

These dumps were to tape PRODUCTION-1WEEK-008.
The next 3 tapes Amanda expects to use are: 3 new tapes.
The next 3 new tapes already labelled are: PRODUCTION-1WEEK-009, PRODUCTION-1WEEK-010, PRODUCTION-1WEEK-011


STATISTICS:
                          Total       Full      Incr.   Level:#
                        --------   --------   --------  --------
Estimate Time (hrs:min)     0:00
Run Time (hrs:min)          0:00
Dump Time (hrs:min)         0:00       0:00       0:00
Output Size (meg)            0.0        0.0        0.0
Original Size (meg)          0.0        0.0        0.0
Avg Compressed Size (%)    110.0      110.0        --
DLEs Dumped                    1          1          0
Avg Dump Rate (k/s)         89.4       89.4        --

Tape Time (hrs:min)         0:00       0:00       0:00
Tape Size (meg)              0.0        0.0        0.0
Tape Used (%)                0.0        0.0        0.0
DLEs Taped                     1          1          0
Parts Taped                    1          1          0
Avg Tp Write Rate (k/s)    100.0      100.0        --

USAGE BY TAPE:
  Label                  Time         Size      %  DLEs Parts
  PRODUCTION-1WEEK-008   0:00           0G    0.0     1     1

NOTES:
  planner: Adding new disk nas2ftp-01:/var/spool/cron/crontabs.
  taper: Slot {8,8} with label PRODUCTION-1WEEK-008 is usable
  taper: tape PRODUCTION-1WEEK-008 kb 10 fm 1 [OK]


DUMP SUMMARY:
                                                                    DUMPER STATS   TAPER STATS
HOSTNAME         DISK                     L ORIG-GB  OUT-GB  COMP%  MMM:SS   KB/s MMM:SS   KB/s
------------------------------------------- ---------------------- -------------- -------------
nas2ftp-01 /var/spool/cron/crontabs 0       0       0     --    0:00   89.4   0:00  100.0

(brought to you by Amanda version 3.3.5)


Windows client

Client installation

Ref:

There is amandabackup user account in PRODUCTION Windows domain created for Amanda client.

Install Amanda client binaries:

File:Windows client installer Amanda.PNG

Supply amandabackup user password when prompted

File:Windows client Amanda password prompt.PNG

Enter Amanda server name when prompted:

File:Windows client Amanda server prompt.PNG

Check the amandabackup user is a member of local Administrators group on the Windows server:

File:Windows client Amanda server local admins.PNG

Backup configuration on backup server

On Amanda server register the new client in the configuration that it meant to be in (prod-selligent-1week-vtape-RAIT configuration in the example below):

amandabackup@amdbkp-01:/etc/amanda/prod-selligent-1week-vtape-RAIT$ amaddclient --config prod-selligent-1week-vtape-RAIT --client selrep-01.production  --diskdev "E:/SSRS/SelligentReport" --dumptype zwc-compress
Logging to /var/log/amanda/amaddclient.20141009171134.debug
/etc/amanda/prod-selligent-1week-vtape-RAIT/disklist updated
updating /var/lib/amanda/.amandahosts on amdbkp-01.production 
Attempting to update /var/lib/amanda/.amandahosts on selrep-01.production 
ssh: connect to host selrep-01.production  port 22: Connection refused
WARNING: scp from selrep-01.production  not successful.
Check selrep-01.production :/var/lib/amanda/.amandahosts file.
If entry 'amdbkp-01.production  amandabackup' is not present,
append the entry to the file manually.
Creating amanda-client.conf for selrep-01.production 
Creating  /etc/amanda/prod-selligent-1week-vtape-RAIT on selrep-01.production 
ssh: connect to host selrep-01.production  port 22: Connection refused
WARNING: Cannot create /etc/amanda/prod-selligent-1week-vtape-RAIT on selrep-01.production 
Please copy /var/lib/amanda/amanda-client.conf-selrep-01.production  to selrep-01.production  manually
File /var/lib/amanda/example/xinetd.amandaclient contains the latest Amanda client daemon configuration.
Please merge it to /etc/xinetd.d/amandaclient.

Run client check to make sure everything is working fine:

amandabackup@amdbkp-01:/etc/amanda/prod-selligent-1week-vtape-RAIT$ amcheck -c prod-selligent-1week-vtape-RAIT

Amanda Backup Client Hosts Check
--------------------------------
Client check: 1 host checked in 0.236 seconds.  0 problems found.

(brought to you by Amanda 3.3.5)

Backup profile creation

Now we want to add new backup profile for new application.

  • NOTE* Below I took BI as an example of the new profile

You will find new profile creation script in /etc/amanda directory - new_profile.sh. The script will:

  • create Amanda profile directory
  • create two configuration files - with and without RAIT defined
  • create holding disk area for the profile
  • create specified number of vTapes on both NAS units - onsite and offsite

Source of the script:

#!/bin/sh
#
# The script creates new profile for Amanda backup
# based on template file /var/lib/amanda/template.d/amanda--rait.conf
#
# NOTE: you might need to change the following by hand:
#  * dumpcycle (default: 1 weeks) 
#  * runspercycle (default: 7)
#  * tapecycle (default: 14 tapes) 
#  * tapetype (default: VTAPE-100G)
#

PROFILE_NAME=$1
NUM_OF_VTAPES=$2

PROFILE_NAME_LOW=`echo $PROFILE_NAME | awk '{print tolower($1)}'`
PROFILE_NAME_UP=`echo $PROFILE_NAME | awk '{print toupper($1)}'`

TEMPLATE=/var/lib/amanda/template.d/amanda-.conf
TEMPLATE_RAIT=/var/lib/amanda/template.d/amanda--rait.conf
AMANDA_CFG_DIR=/etc/amanda
AMANDA_HOLDING_DISK_DIR=/amanda
AMANDA_PROFILE_DIR=prod-$PROFILE_NAME_LOW-1week-vtape-RAIT
VTAPES_DIR=/vtapes-prod/RAIT/1week
VTAPES_OFFSITE_DIR=/vtapes-prod-offsite/RAIT/1week
WARNING="NOTE: you might need to change the following by hand in $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/amanda.conf:\n
   dumpcycle (default: 1 weeks)\n
   runspercycle (default: 7)\n
   tapecycle (default: 14 tapes)\n
   tapetype (default: VTAPE-100G)\n"
DISKLIST_HDR="
#\n
# The below configured based on the requirements I had at that moment\n
#\n"

if [ $# -lt 2 ]
then
   echo "\nUsage: `basename $0` <new_profile_name> <number_of_vtapes>\n"
   exit 1
fi

if [ -f $TEMPLATE ]
then
   if [ -d $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR ]
   then 
      echo "\nERROR: $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR already exists!\n"
      exit 1
   fi

   echo "Creating profile directory $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR \n"
   mkdir $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR

   # configuration without RAIT
   sed -e "s/PROFILE_NAME_UP/$PROFILE_NAME_UP/g" -e "s/PROFILE_NAME_LOW/$PROFILE_NAME_LOW/g" $TEMPLATE > $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/amanda.conf-norait

   # configuration with RAIT
   sed -e "s/PROFILE_NAME_UP/$PROFILE_NAME_UP/g" -e "s/PROFILE_NAME_LOW/$PROFILE_NAME_LOW/g" $TEMPLATE_RAIT > $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/amanda.conf-rait

   # until we finished creating vTapes, we need RAIT configuration to be active
   ln -s $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/amanda.conf-rait $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/amanda.conf
   for d in log curinfo index
   do
      mkdir $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/${d}
   done
   echo $DISKLIST_HDR > $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/disklist

   HOLDING_DISK_AREA=`amgetconf $AMANDA_PROFILE_DIR holdingdisk:hd1:directory`
   echo "Creating holding disk area for the profile - $HOLDING_DISK_AREA\n"
   if [ -d $AMANDA_HOLDING_DISK_DIR ]
   then
      if [ ! -d $HOLDING_DISK_AREA ]
      then
         mkdir $HOLDING_DISK_AREA
      else
         echo "Directory $HOLDING_DISK_AREA already exists!"
      fi
   else
      echo "ERROR: Holding disk directory $AMANDA_HOLDING_DISK_DIR does not exist!"
   fi

   echo "Creating $NUM_OF_VTAPES vTapes for the profile"
   for i in `seq -f "%03g" 1 $NUM_OF_VTAPES`
   do
      SN=`echo ${i}|sed -e 's/0*\([1-9].*\)/\1/g'`
      mkdir -p $VTAPES_DIR/$PROFILE_NAME_UP/slot$SN $VTAPES_OFFSITE_DIR/$PROFILE_NAME_UP/slot$SN && amlabel $AMANDA_PROFILE_DIR PRODUCTION-$PROFILE_NAME_UP-1WEEK-$i slot $SN
   done

   # switching to configuration without RAIT
   # should upgrade DC - HQ connection before we can use RAIT for all applications
   # for now RAIT is not in use.
   rm $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/amanda.conf
   ln -s $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/amanda.conf-norait $AMANDA_CFG_DIR/$AMANDA_PROFILE_DIR/amanda.conf

   echo "Profile $PROFILE_NAME_UP created!\n\n"
   echo $WARNING
fi

Engage the script to create BI profile with 14 vTapes:

amandabackup@amdbkp-01:/etc/amanda$ ./new_profile.sh BI 14
Creating profile directory /etc/amanda/prod-sql-1week-vtape-RAIT 

Creating holding disk area for the profile - /amanda/prod-sql-1week-vtape-RAIT 

Directory /amanda/prod-sql-1week-vtape-RAIT already exists!
Creating 14 vTapes for the profile
Reading label...
Found an empty tape.
Writing label 'PRODUCTION-SQL-1WEEK-001'...
Checking label...
Success!
Reading label...
Found an empty tape.
Writing label 'PRODUCTION-SQL-1WEEK-002'...
Checking label...
Success!
Reading label...
Found an empty tape.
Writing label 'PRODUCTION-SQL-1WEEK-003'...
Checking label...
Success!
Reading label...
Found an empty tape.
Writing label 'PRODUCTION-SQL-1WEEK-004'...
Checking label...
Success!
Reading label...
Found an empty tape.
Writing label 'PRODUCTION-SQL-1WEEK-005'...
Checking label...
Success!
Reading label...
Found an empty tape.
Writing label 'PRODUCTION-SQL-1WEEK-006'...
Checking label...
Success!
Reading label...
Found an empty tape.
Writing label 'PRODUCTION-SQL-1WEEK-007'...
Checking label...
Success!
Reading label...
Found an empty tape.
Writing label 'PRODUCTION-SQL-1WEEK-008'...
Checking label...
Success!
Reading label...
Found an empty tape.
Writing label 'PRODUCTION-SQL-1WEEK-009'...
Checking label...
Success!
Reading label...
Found an empty tape.
Writing label 'PRODUCTION-SQL-1WEEK-010'...
Checking label...
Success!
Reading label...
Found an empty tape.
Writing label 'PRODUCTION-SQL-1WEEK-011'...
Checking label...
Success!
Reading label...
Found an empty tape.
Writing label 'PRODUCTION-SQL-1WEEK-012'...
Checking label...
Success!
Reading label...
Found an empty tape.
Writing label 'PRODUCTION-SQL-1WEEK-013'...
Checking label...
Success!
Reading label...
Found an empty tape.
Writing label 'PRODUCTION-SQL-1WEEK-014'...
Checking label...
Success!
Profile BI created!


NOTE: you might need to change the following by hand in /etc/amanda/prod-sql-1week-vtape-RAIT/amanda.conf:
 dumpcycle (default: 1 weeks)
 runspercycle (default: 7)
 tapecycle (default: 14 tapes)
 tapetype (default: VTAPE-100G)

Newly created amanda.conf points to Datacenter NAS only and amanda.conf-rait has RAIT (Datacenter NAS + HQ NAS) defined.

Here is the diff between the two templates :

--- /var/lib/amanda/template.d/amanda-.conf	2014-10-20 16:43:59.794381389 +0100
+++ /var/lib/amanda/template.d/amanda--rait.conf	2014-10-20 16:45:09.847107263 +0100
@@ -16,9 +16,16 @@
 bumpmult 4              
 etimeout 1800            
 runtapes 3              
-tpchanger "chg-disk:/vtapes-prod/RAIT/1week/PROFILE_NAME_UP"
-changerfile  "vtapes-statefile-tc"                                           
+tpchanger "chg-rait:{nas-tc,nas-gqh}"
 tapetype VTAPE-100G
+define changer nas-tc {
+	tpchanger "chg-disk:/vtapes-prod/RAIT/1week/PROFILE_NAME_UP"
+	changerfile  "vtapes-statefile-tc"                                           
+}
+define changer nas-gqh {
+	tpchanger "chg-disk:/vtapes-prod-offsite/RAIT/1week/PROFILE_NAME_UP"
+	changerfile  "vtapes-statefile-gqh"                                           
+}
 maxdumpsize -1          
 labelstr "^PRODUCTION-PROFILE_NAME_UP-1WEEK-[0-9][0-9]*$"     
 holdingdisk hd1 {
  • WARNING* we should upgrade Datacenter - HQ connection before we can use RAIT for all applications. For now RAIT is not in use.

Add required backup targets to disklist file.

#
# The below configured based on the requirements 
#
SQL-01  I:/OBI/catalog          zwc-encrypt-compress-server
SQL-01  I:/OBI/GlobalCache      zwc-encrypt-compress-server
SQL-01  I:/OBI/repository       zwc-encrypt-compress-server
SQL-01  I:/Sources              zwc-encrypt-compress-server

Check that configuration is correct and working (ignore "does not exist" messages for info and index):

amandabackup@amdbkp-01:/etc/amanda$ amcheck prod-sql-1week-vtape-RAIT
Amanda Tape Server Host Check
-----------------------------
Holding disk /amanda/prod-sql-1week-vtape-RAIT: 148107264 KB disk space available, using 148004864 KB
slot 1: volume 'PRODUCTION-SQL-1WEEK-001'
Will write to volume 'PRODUCTION-SQL-1WEEK-001' in slot 1.
NOTE: skipping tape-writable test
NOTE: host info dir /etc/amanda/prod-sql-1week-vtape-RAIT/curinfo/SQL-01 does not exist
NOTE: it will be created on the next run.
NOTE: index dir /etc/amanda/prod-sql-1week-vtape-RAIT/index/SQL-01 does not exist
NOTE: it will be created on the next run.
Server check took 0.174 seconds

Amanda Backup Client Hosts Check
--------------------------------
Client check: 1 host checked in 0.301 seconds.  0 problems found.

(brought to you by Amanda 3.3.5)

Create distribution list in you directory

File:Amanda BI distribution list.png

Add members to the list.

Run backup and watch results:

amandabackup@amdbkp-01:/etc/amanda/prod-sql-1week-vtape-RAIT$ nohup /etc/amanda/amdump_and_report.sh prod-sql-1week-vtape-RAIT &
amandabackup@amdbkp-01:/etc/amanda/prod-sql-1week-vtape-RAIT$ watch -n3 amstatus prod-sql-1week-vtape-RAIT

Every 3.0s: amstatus prod-sql-1week-vtape-RAIT                                                                      Mon Oct 20 17:07:14 2014

Using /etc/amanda/prod-sql-1week-vtape-RAIT/log/amdump
From Mon Oct 20 17:06:56 IST 2014

SQL-01:"I:/OBI/GlobalCache" 0         1k finished (17:07:04)
SQL-01:"I:/OBI/catalog"   0      6001k dumping      672k ( 11.20%) (17:07:09)
SQL-01:"I:/OBI/repository" 0      2339k finished (17:07:07)
SQL-01:"I:/Sources"       0    357273k waiting for dumping

SUMMARY          part      real  estimated
                           size       size
partition       :   4
estimated       :   4               364485k
flush           :   0         0k
failed          :   0                    0k           (  0.00%)
wait for dumping:   1               357273k           ( 98.02%)
dumping to tape :   0                    0k           (  0.00%)
dumping         :   1       672k      6001k ( 11.20%) (  0.18%)
dumped          :   2      2340k      1211k (193.23%) (  0.64%)
wait for writing:   0         0k         0k (  0.00%) (  0.00%)
wait to flush   :   0         0k         0k (100.00%) (  0.00%)
writing to tape :   0         0k         0k (  0.00%) (  0.00%)
failed to tape  :   0         0k         0k (  0.00%) (  0.00%)
taped           :   2      2340k      1211k (193.23%) (  0.64%)
  tape 1        :   2      2338k      2338k (  0.00%) PRODUCTION-SQL-1WEEK-001 (2 chunks)
3 dumpers idle  : 0
taper status: Idle
taper qlen: 0
network free kps:     78976
holding space   : 147998784k (100.00%)
chunker0 busy   :  0:00:08  ( 60.97%)
 dumper0 busy   :  0:00:08  ( 60.80%)
   taper busy   :  0:00:00  (  0.35%)
 0 dumpers busy :  0:00:05  ( 39.20%)                   0:  0:00:03  ( 60.64%)
                                                        2:  0:00:02  ( 39.34%)
 1 dumper busy  :  0:00:08  ( 60.80%)                   0:  0:00:05  ( 67.10%)
                                                        5:  0:00:02  ( 32.90%)

Since everything is confirmed to be working, you can add that distribution list you created in directory to the "mailto" parameter in amanda.conf* files.

amandabackup@amdbkp-01:/etc/amanda/prod-sql-1week-vtape-RAIT$ grep mailto amanda.conf*
amanda.conf:mailto "backup-reports@corporate eng@corporate"  
amanda.conf-norait:mailto "backup-reports@corporate eng@corporate"  
amanda.conf-rait:mailto "backup-reports@corporate eng@corporate" 

Logging and monitoring

Kibana/Logstash

We had Kibana and Logstash deployed and you could see logs from every run of Amanda backup in Kibana. See example of wrapper script we had to send Amanda reports in JSON format to see it later in Kibana.

Parsing of JSON output from Amanda requires improvement. We had few issues to resolve there:

  • There is a known limitation of Logstash - it can't explode JSON arrays properly. It leads to "summary" part of the output not parsed properly.
  • Amounts sent by Amanda is in kilobytes, but Kibana works with bytes and doesn't allow to do mathematical operations with the numbers in web UI. There is an option of doing it in the input filter at logstash level, but it would be preferable not to alter the original data on the way into logstash. After some discussion we thought that the better way was to ship the stats into Graphite and use Dashboard to plot the graphs (it allows calculations in web UI)

Dashboard

We had an idea to export data into Graphite, so we could build dashboards for backups.

Amanda server statistics were available via Amanda backup server dashboard.

Personal tools