Holiday IT Maintenance Update
As many of you are aware, IT underwent a complete network shutdown on December 28th. This shutdown began at noon on December 28th and took three hours longer than anticipated, concluding at 3am on December 29th. This was an aggressive maintenance window that involved the upgrade of every network switch, router, and firewall (approximately 100 devices). It was also necessary to shut down approximately 160 network connected servers for the duration of the outage.
We faced some issues before and after the maintenance window that we are now satisfied are rectified. Notably;
- In advance of the maintenance window, we reconfigured our network to allow access to the web server and e-mail during the outage. This reconfiguration should not have caused any interruption to services and was part of the preparation. Unfortunately, this was not the case. This reconfiguration resulted in internet connectivity to some of our other servers being interrupted, notably Blackboard. We were notified of the Blackboard outage by an automated monitoring system at approximately 10 am on December 27th, and service was restored by about 2:30pm on that same day.
- Following the maintenance window and on December 29th, we began to receive reports of certain anomalies;
- At 9am PRD reported that our VPN service was down. This outage was discovered to be a part of a general outbound internet issue and was resolved by about 11am. Affected services were outbound internet from some buildings (not all, but notably Athletics).
- Around 2 pm we received a report of no outbound long distance (reported by Athletics and Advancement), and also that Nobelgen had no internet access. We discovered yet another routing issue that was affecting networks of tenants, long distance, and WIFI. All issues were resolved by 7pm.
While we did face some issues before and after the maintenance window, we are satisfied that services are now operating normally. I would like to send a big thank you to Andrew Bell (Manager, Digital Service Delivery and Administration) and his team for their hard work in preparing for, executing, and rectifying any issues related to our maintenance window. Their hard work ensures that we continue to enjoy a stable and secure network that we can rely on.
Associate Vice President, IT
P.S. Missed a Tech Bulletin? Please go to http://www.trentu.ca/tech-bulletins to review the Tech Bulletin archives.
Posted on December 31, 2017