Skip to content

UCS Development Group Services

The University Computing Service (UCS) was one of the parent organisations that merged to form University Information Services (UIS).

The UCS development group wrote services, some of which are still in use today. As the UCS and it's dev group no longer exist, responsibility for running these has been transferred to DevOps.

The services run by DevOps currently includes:

Deploying changes to UCS development group services

The majority of the UCS dev group services are clustered using Pacemaker.

To make a release without incurring downtime the following steps can be taken. Examples, refer to Lookup service (ucs-ibis package) and need substituting as appropriate.

Start with the standby node

Note

Which node is on standby can be determined by running crm_mon -1 or looking at the service's /adm/status page.

  1. Put the cluster into maintenance mode, this ensures pacemaker does not try to make changes to the cluster in response to a node being unavailable.

    crm configure property maintenance-mode=true
    # Check for "unmanaged" status
    crm_mon -1
    

  2. Release the lock on the package.

    # List locked packages
    zypper ll
    # Release lock on appropriate package
    zypper rl ucs-ibis
    

  3. Run the software upgrade.

    # Refresh repositories
    zypper ref
    # List updates available
    zypper lu
    # Update application specific package
    zypper up ucs-ibis
    # Restart tomcat (a single "restart" doesn't always work)
    service tomcat6 stop
    service tomcat6 start
    

  4. Check the service is running on the node with update.

    • see https://{node url}/adm/status
    • check application functionality
  5. Move the service out of maintenance mode.

    crm configure property maintenance-mode=false
    # Check for removal of "unmanaged" status
    crm_mon -1
    

  6. Reapply lock to package.

    # Release lock on appropriate package
    zypper al ucs-ibis
    # List locked packages to check
    zypper ll
    

Move to current live node

  1. Move service to already updated standby node

    crm configure edit
    # shift node weights so that the current standby node is the preferred service
    # verify that the service has moved to the previous standby:
    crm_mon -1
    

  2. Repeat steps 1 to 6 above. i.e.

    • put the service back in maintenance mode
    • unlock the package
    • update the package
    • check success
    • remove the service from maintenance mode
    • relock the package
  3. Move service to back to this node

    crm configure edit
    # shift node weights so that this nde is the preferred service
    # verify that the service has moved back:
    crm_mon -1
    

Following the above should allow a software upgrade to be deployed without any downtime. These steps will not work if the upgrade includes a breaking change to an external data source, e.g. a database migration which is not compatible with the previous version of the software. In this case downtime may need to be scheduled.

TLS certificates on UCS development group services

Installation of TLS certificates on UCS dev group services is a manual process.

Certificate locations

Some services seem to have directories (ssl.crt and ssl.key) created to hold the certificate and key files, other use the tomcat config directory, grep -i certificate /srv/www/tomcat6/base/conf/server.xml should show the path.

To install new certificates

Obtain the new certificates from the TLS certificate application.

Copy the new certificate and key files to the certificate location on the target system.

Update the tomcat configuration to use the new certificate, edit /srv/www/tomcat6/base/conf/server.xml.

Ensure that the certificate and key have the correct ownership and permissions with chown ucstomcat <file> and chmod 600 <file>.

To update the intermidiate certificate:

Create a new file, qvsslg3.crt, in the certificate location containing the new intermediate certificate, remove any blank lines from it.

Ensure that the certificate and key have the correct ownership and permissions with chown ucstomcat <file> and chmod 600 <file>.

In /srv/www/tomcat6/base/conf/server.xml, edit the line that says (path might be different on your system):

certificateChainFile="/srv/www/tomcat6/base/conf/QuoVadisGlobalSSLICAG3.crt"

to:

certificateChainFile="/srv/www/tomcat6/base/conf/qvsslg3.crt"

Restart Tomcat:

service tomcat6 restart

Database backup and restore on UCS dev group services

Examples, refer to Lookup service (ucs-ibis package) and need substituting as appropriate.

Database backups and restores are managed by a pair of scripts in /usr/share/ibis/bin, ibis-backup and ibis-restore.

Database backups

Generally ibis-backup is run from a cronjob /etc/cron.d/ibis, which, by passing in an argument hour|day|week|month|year, outputs a bzip2 compressed database dump in the /usr/share/ibis/backup/hour|day|week|month|year directory. The ibis-backup script can also take the argument now which creates a database dump in the current directory.

Database restores

The ibis-restore script takes a bzip2 compressed backup file which was created by ibis-backup and optionally the name of the database to restore to.

ibis-restore does not restore to an existing database, if we want to replace an existing database we must do a DROP DATABASE first.