Using the HFS archive and retrieve service

TSM rebranding

With effect from version 8.1 onwards, the IBM TSM (Tivoli Storage Manager) backup / archive product has been rebranded as IBM Spectrum Protect. This is a rebranding exercise by IBM and, unless indicated specifically, there is no difference in the operation of the two brands. Version 7 of the (old) TSM client is still available for installation on older operating systems and you will see references to both IBM Spectrum Protect and TSM throughout these pages.

 

 

The configuration and use of the HFS archive / retrieve client software differs subtly from the HFS backup / restore client software.  For this reason you are strongly encouraged to read the following sections in order. 

Expand All

The HFS archive service is subject to a formal application procedure, requiring explicit permission over and above the eligibility criteria for simple backup.  If you are engaged in a project which produces data of long term value to the University, you should refer to the Policy on Computer Archiving Services.  A simple table of what constitutes suitable candidate data for archive is also listed for quick reference.

Please also bear the following considerations in mind:

  • Data retention

Data cannot be archived indefinitely. We ask project applications to state a realistic lifetime for a project. Some projects have a natural conclusion whereas others may extend into the future due to regulatory requirements or simply because the data is of great value.

Typically, projects are granted archival for 1-10 years and we require that a case for continued archival is made on every 5 year anniversary, this being not too onerous while ensuring the archive receives appropriate occasional consideration.

  • Backup accounts

Archived data must not be backed up to the HFS backup service.  Where a client machine is sending data to both the backup service and the archive service, the client software must be configured in such a way that avoids sending the same data to both backup and archive.  We can help with the setup of this configuration.  In practice this normally requires a physical separation of the archive data from backup data on the client, for example into a separate partition or root folder path.

  • Costs

Long-term projects, and those requiring more that 5TB of data, will probably be asked to contribute to the on-costs of archive storage.  For further information please see the HFS Service Level Description.

Externally funded projects should have a defined element for data storage and will be expected to contribute to these costs.

  • Encryption

It is recommended that sensitive data be sent encrypted.  See further our pages on backup and archive security and how to encrypt files for backup.

  • Size

There are limits to the amount of data the service can store.  In practice a range of between 50GB and 50TB is acceptable.  Requests at either end of those scales will, however, be subject to greater scrutiny and possibly additional restrictions.  At the lower end we might ask why local storage cannot be used.  Towards the upper end, we might ask for additional assurances on data curation, provenance and access.

Projects above 50TB may be considered but additional storage charges above 4TB may make a local storage solution more of an economic proposition.

Please use the archive request form to submit your application for your data to be archived on the HFS.  You can also apply via the HFS Portal by selecting Register new node > Project Archive account and answering some questions.  The two methods are equivalent, you will be asked the same questions and your application processed in the same way.

Archiving is in some ways a completely separate concept to that of backup.  Additionally, by its very nature, archive suggest an inherent value in the data to be secured.  For these reasons it is advisable to start using the HFS archive software by setting up a test area within the project and exploring the capabilities and features of the archive client in a test environment, before moving on to archive live project data.

Some areas to consider before archiving your project data are:

  • Archive file identification

When using the Backup Client, the location of a file, its directory/folder path, identifies that file.  This is because there can at most be only two versions of the same file held on the backup server.  In contrast the archive client allows unlimited versions of the same file to be kept and then for the source files on the local machine to be deleted.  As such, the local directory/folder structure may provide little clue as to its archive contents and will certainly provide no information as to how versions of the same file differ.

Possible solutions to the above are to add a README or INDEX file in each directory folder, listing descriptions, dates and times of each file archived in that location.

An entry in the Description field may also be used.  By default the software client populates this field with the text "Archive Date: Date" which clearly becomes useless if you archive the same file twice on one day.  We recommend that more descriptive entries in this field, up to 255 characters, be used for each file archived to group and distinguish archived files.

  • Multiple user accounts

It is also important to consider the account, or username, under which you are going to do the archiving.  On a Linux/Unix system, other than root, only the user who archived a file can see it within the archive or retrieve it.  If you are archiving repeatedly related material, when you or someone else come to retrieve it, it will probably be confusing if the archiving has been performed under different usernames.

  • Symbolic file links (Linux/Unix)

Under Unix, when you name a symbolic link in an archive operation, the object pointed to by the link is archived, not the link data.  This behaviour differs from that of backup, where the link data is backed up.

  • Local file deletion

The archive client offers an option to delete the local files immediately on successful archival to the server.  This option needs explicitly stating on the command line or setting in the Archive Options in the GUI and probably should not be used.  The archive data is secured on the archive server by making three copies to tape.  This process occurs early each morning between 00:00-01:00.  We therefore recommend that where archive clients need to delete archived material from their local machines, for example for reasons of space, they should desist from doing so until the day following the archival process for any particular file and then do so through local operating system commands.

  • Archive file deletion

Unlike the backup client, the archive client allows the deletion of files archived on the server.  Obviously care should be taken in use of this, as once deleted from the server, a file cannot later be retrieved.

The HFS Archive client uses the same software as the HFS Backup client, namely IBM TSM now rebranded as IBM Spectrum Protect.

Ensure that you have registered for an Archive account and have your HFS Archive account name (also known as the nodename), this will be of the form PROJECTNAME_ARCHIVE.DEPT.

If you are not already using the HFS backup service on this machine then:

  1. Install the HFS archive software, providing the HFS archive account name when prompted for the nodename.
  2. Read Initial considerations before using the archive software 
  3. Then continue to the section Using archive & retrieve client software on your local machine.

 

If you are already using the HFS backup service on this machine, you will need to take the following steps:

  1. Download the template archive configuration file dsm-archive.opt.
  2. Edit the downloaded dsm-archive.opt file with Notepad, changing the Nodename value from CHANGE.THIS to the archive account name.
  3. Move the dsm-archive.opt file to the folder C:\Program Files\tivoli\tsm\baclient\.
  4. Create a new shortcut by right clicking on the Windows desktop and selecting New Shortcut.
  5. Provide the following location:
    "C:\Program Files\tivoli\TSM\baclient\dsm.exe" -optfile=dsm.opt.archive:
  6. Name the shortcut something like "HFS Archive Client".
    • You will now be able to access the HFS archive server with this shortcut and the HFS backup server with the usual shortcut.
  7. Read Initial considerations before using the archive software.
  8. Then continue to the section Using archive & retrieve client software on your local machine.

If you are not already using the HFS backup service on this machine then:

  1. Install the HFS archive software, providing the HFS archive account name when prompted for the nodename.
  2. Read Initial considerations before using the archive software 
  3. Then continue to the section Using archive & retrieve client software on your local machine.

 

If you are already using the HFS backup service on this machine:

  1. Download and install the HFS archive package.
  2. Provide the HFS archive account name when prompted.
    • The application TSMARchive will be installed on your Mac.
  3. Read Initial considerations before using the archive software 
  4. Then continue to the section Using archive & retrieve client software on your local machine.

 

If you encounter errors in either install process, please send your TSM configuration and log files to hfs@ox.ac.uk for further help.

If you are not already using the HFS backup service on this machine then:

  1. Install the HFS archive software, providing the HFS archive account name as the nodename.
  2. Download the template options file dsm.sys (also reproduced below).
  3. Edit dsm.sys according to the comments at the top of the file.
  4. Save the file to the HFS software's installation folder, typically /usr/tivoli/tsm/client/ba/bin or /opt/tivoli/tsm/client/ba/bin.
    • You will now be able to access the Archive service with either dsm for the GUI or dsmc for the Command Line client.
  5. Read Initial considerations before using the archive software 
  6. Then continue to the section Using archive & retrieve client software on your local machine.

 

If you are already using the HFS backup service on this machine:

  1. Download the template options file dsm.sys (also reproduced below).
  2. Edit dsm.sys according to the comments at the top of the file.
  3. The edited options should be appended into the existing dsm.sys file.
    • The TSM Client program connects to the first server listed in the dsm.sys file, so will connect to the HFS backup server by default.  To connect to the HFS archive server the client will require a server name as an argument.
    • You will now be able to access the Archive service with either dsm -se=OX_HFS for the GUI or dsmc -se=OX_HFS for the Command Line client.
  4. Read Initial considerations before using the archive software 
  5. Then continue to the section Using archive & retrieve client software on your local machine.

 

Template dsm.sys options file

*                  TSM Archiving on a Unix System

* This is the example dsm.sys file for use with Archive to the
* HFS at Oxford from a Unix client. As supplied, it must be
* edited as described below.

* In the line
*    Nodename                CHANGE.THIS
* you must alter the CHANGE.THIS to the registered name of
* your archive client node. This will (probably) be of the
* form something.archive or something_archive.dept

* As things are set up below, you will have to type in your
* password the first time you invoke the TSM software.
* It will then be stored on your client in encrypted form
* so that the scheduler can subsequently pass it to the
* server without needing input from you. TSM will change
* your password at roughly yearly intervals, and needs to
* tell you what the new password is when it does so. The line
*    Mailprog                MAIL-PROG root
* tells TSM to use the (mail) program specified by MAIL-PROG
* to mail this new password to root. You should replace
* MAIL-PROG by a suitable mail agent executable, and perhaps
* replace root by some more suitable username. For example
*    Mailprog                /usr/bin/mailx admin

* By default all users and members of all groups are permitted
* to use TSM (permissions on the executables being ok). To
* restrict this, uncomment the *Users and *Groups lines and
* substitute as argument a space-separated list of usernames
* and groupnames. If both lines are amended in this manner
* the users specified and all members of the groups specified
* will have access to TSM.

* The other lines set up default values for various options.
* You should not change any of these unless you know what you
* are doing (i.e. have read the TSM client manual), or under
* advice from IT Services.

* If your system is already using TSM backup, you will already
* have a dsm.sys file on your system. In this case copy the
* uncommented section below (after editing it as above) 
* and append it to the end of your existing dsm.sys
* file; the file will then have two "stanzas", one for backup and
* one for archive. You will note that the names both of the TSM
* server and your node name for use with archiving are different
* from the names used for backup. You must take care to retain
* these differences

* If your system is only using TSM Archive, then save this file
* after editing it.

* This file must be placed in different places in the filestore
* depending on your platform (and in some cases operating system
* version). The following table lists where it should go, provided
* you have installed TSM software at its default location.

*    IBM AIX                  /usr/bin/dsm.sys
*    SGI IRIX                 a symbolic link from
*                             /usr/tivoli/tsm/client/ba/bin/dsm.sys
*
*    Linux                    /usr/bin/dsm.sys
*    Sun Solaris 2.6/7/8/9    a symbolic link from
*    Sun Solaris 10           /opt/tivoli/tsm/client/ba/bin/dsm.sys

Servername              OX_HFS
CommMethod              TCPIP
TCPServeraddress        dsma.ox.ac.uk
TCPPort                 1500
Nodename                CHANGE.THIS
PasswordAccess          generate
Mailprog                MAIL-PROG root
Compression             no
*Users                  space-separated-list
*Groups                 space-separated-list
*VirtualMountPoint      directory-name
TCPBuffSize             32
TCPWindowSize           63
TCPNodelay              yes
TxnByteLimit            25600

* end-of-file

 

For simplicity, usage of the archive client software is grouped according to interface type and function.  The following pages provide a tutorial on how to archive and retrieve files:

Get support


Local IT support provide your first line of on-the-spot help

FIND MY LOCAL IT TEAM

 

Common requests and fault reports can be logged using self-service

   USE IT SELF-SERVICE    

   LOG A SUPPORT CALL    

VIEW MY SUPPORT CALLS  

The central Service Desk is available 24x7 on +44 1865 6 12345

 

If you do not have an SSO account you can use this form to contact the Service Desk