Events System¶

This page gives an overview of the Events System, describing its current status, where and how it's developed and deployed, and who is responsible for maintaining it.

Service Description¶

Events scheduling and collaboration for the University's Office of External Affairs and Comms, including for the Cambridge Festivals.

Service Status¶

The Events System is currently live.

Contact¶

The main contact for the Events System is Hannah Jackson.

Issues discovered in the service or new feature requests should be opened as feature branch cloned from the repository GitLab issues in the application repository.

Environments¶

The Events System is currently deployed to the following environments:

Name	URL	Supporting VMs
Production	Admin system: https://webservices.admin.cam.ac.uk/events/	Admin: sanders.internal.admin.cam.ac.uk
	What's On (client): http://www.admin.cam.ac.uk/whatson/	(primary)timmons.internal.admin.cam.ac.uk (failover)hancock/morello (=webdb2)What's on (accessible via SSH Jumphost Service):
		web-uas-live1.srv.uis.private.cam.ac.uk, web-uas-live2.srv.uis.private.cam.ac.uk
Staging	http://staging.app4.admin.cam.ac.uk/events/ http://rich:8096/events	rich (DB & app) What's On (accessible via SSH Jumphost Service):web-uas-dev1.srv.uis.private.cam.ac.uk

Local Properties Files¶

The EMS has two properties file which are context specific:

eventLocal.propertie: path to xsl files is configured here.
jdbc.properties: context specific database connection parameters are configured here. These file live on each server (live and UAT) at /home/events/eventsDir.

The EMS has two properties file which are context:

eventLocal.properties: path to xsl files is configured here.
jdbc.properties: context specific database connection parameters are configured here.

These file live on each server (live and UAT) at /home/events/eventsDir.

Tomcat is made aware of local properties file by adding -DeventsDir=/home/events/eventsDir to its CATALINA_OPTS. Have a look at section Deployment: Tomcat specifics for more information.

Source code¶

The source code for the Events System is at the following repository

Repository	Description
https://gitlab.developers.cam.ac.uk/uis/devops/events-system-event/events	The source code for the main application server

Technologies used¶

The following gives an overview of the technologies the Events System is built on.

Category	Language	Framework(s)
Application	Java	Struts2, Spring

Operational documentation¶

The following gives an overview of how the Events System is deployed and maintained.

How and where the Events System is deployed¶

Tomcat specifics

The application runs on tomcat-8082 on byron/smart, nashe, and finch. Tomcat is proxied behind Apache.

Note: there are now two instances of tomcat called tomcat-8080 and tomcat-8082. -8082 has shared mail, database and connection pooling libraries. The application is a standard Java web application distributable as a .war file. It doesn't require any specific configuration to run in Tomcat, but 'out-of-the-box' Tomcat has some poor defaults for its startup variables; specifically, those related to memory allocation.

Include the following in /etc/init.d/tomcat-8082 to set better options ( CATALINA_OPTS):

export CATALINA_OPTS="${CATALINA_OPTS} -Xmx512m -XX:PermSize=128m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled -Duser.language=en -Duser.country=GB -DeventsDir=/home/events/eventsDir"

This also includes locale settings, and has been done on byron/smart/nashe/finch.

[More detail: The problem with Tomcat's default config is twofold: first, that the default PermSize? (64MB) is too small for our app, so it starts throwing OutOfMemoryErrors? after a while. Second, redeploying applications doesn't release the PermGen? memory used by the undeployed app. In Tomcat 6.0.32 these settings DO release memory from undeployed apps.]

Tomcat specifics¶

The application runs on tomcat-8082 on byron/smart, nashe, and finch. Tomcat is proxied behind Apache.

Note: there are now two instances of tomcat called tomcat-8080 and tomcat-8082. -8082 has shared mail, database and connection pooling libraries.

The application is a standard Java web application distributable as a .war file. It doesn't require any specific configuration to run in Tomcat, but 'out-of-the-box' Tomcat has some poor defaults for its startup variables; specifically, those related to memory allocation.

Include the following in /etc/init.d/tomcat-8082 to set better options (CATALINA_OPTS):

export CATALINA_OPTS="${CATALINA_OPTS} -Xmx512m -XX:PermSize=128m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled -Duser.language=en -Duser.country=GB -DeventsDir=/home/events/eventsDir"

This also includes locale settings, and has been done on byron/smart/nashe/finch.

[More detail: The problem with Tomcat's default config is twofold: first, that the default PermSize? (64MB) is too small for our app, so it starts throwing OutOfMemoryErrors? after a while. Second, redeploying applications doesn't release the PermGen? memory used by the undeployed app. In Tomcat 6.0.32 these settings DO release memory from undeployed apps.]

These settings have been thoroughly load tested with JMeter during w

Debugging¶

The client apps (e.g. for What's On, [donne|blunt] /httpd/web/whatson/events_client/) are written in PHP and communicate with http://webservices.internal.admin.cam.ac.uk/events/api/ to get event data.

Information is fetched as XML and cached in events_client/cache. Information is only re-fetched if it is stale, as determined by the cached files timestamps, using the If-Modified-Since: HTTP header.

In the event of the API becoming unavailable, the client apps will continue to run using cached data. New searches won't work, and occasional delays will be experienced as the client tries to fetch new data. In the event of such a failure, the cached file causing the error (if it exists) will not be checked again for some time, in the expectation that soon the situation will resolve itself.

A few name lookup timeouts occur every day; this seems to be a network issue and no cause for concern.

Other operation issues¶

Errors in the client apps (Whats On etc)¶

check the client application logs in e.g. [donne|blunt]~httpd/web/whatson/events_client/log/application.log for clues.

Errors in the admin app¶

Check the server logs in [byron|smart]/usr/local/apache-tomcat/logs for clues.

events.yyyy-mm-dd.log contains logging for the Admin application
events_api.yyyy-mm-dd.log contains logging for the API
catalina.yyyy-mm-dd.log contains logging from Tomcat itself, and additionally anything written by Java to stdout (e.g. stack traces, connection pool errors)

See https://webservices.admin.cam.ac.uk/events/admin/system/monitor (requires global admin privileges) for very detailed system config and memory usage information

Restart tomcat if there are OutOfMemory? Errors, race conditions. hangs, or other weird problems. We've never needed to do this but other applications may subsequently appear that have bugs.

To restart Tomcat (as root):

service tomcat-8082 stop
wait a few seconds for things to happen; double-check tomcat-8082 is down with ps -ef|grep tomcat
service tomcat-8082 start
check catalina.out for errors, but now all should be well.

App uses webdb and pools connections; if the DB disappears and then comes back the app should automatically come back to life after a short delay - no need to restart.

Service Management and tech lead¶

The service owner for the Event System is TBD.

The service manager for the Events System is TBD.

The tech lead for the Events System TBD .

The following engineers have operational experience with the Events System and are able to respond to support requests or incidents:

engineer name TBD.
engineer name TBD.