Source : Perth Now news
Telstra has addressed another major blunder in its catastrophic nationwide outage in July that left tens of thousands of people without service and many unable to dial Triple Zero.
The service went black about 2am on July 7 with tens of thousands of customers affected across WA and no estimate given of when the service would be restored. The outage also affected the Aldi Mobile, Belong and Boost brands.
West Australians were continuing to report outage issues at 5.30pm, more than 12 hours after the telco’s problem was first realised.
Telstra CEO Vicki Brady in addressing investigation findings on Wednesday revealed a further eight calls to emergency services were missed that weren’t initially subject to a welfare check as per protocol.
“It shouldn’t have happened, we should have picked those eight calls up,” Ms Brady said.
“The required checks have since been completed, and Telstra is not aware of any adverse outcomes for those callers,” she said. Ms Brady accepted the network did not adequately prioritise the network timing system.
“As a result of prioritising it not at that highest level, that had some flow-on impacts, which included the TAP findings identifying in this particular area in the teams looking after this network timing system.
“We did not have sufficient resourcing and expertise. Just to be clear, we have reallocated already resourcing into that team with the expertise required, and also drawing on external experts to make sure we have that capability at the right level.”
A Technology Audit Partners investigation found the outage was “triggered by a specific technical event” involving incorrect date information propagated through parts of the network following planned maintenance on its timing system.
TAP found Telstra fell short in treating network timing as a “critical capability within the network (or a ‘sovereign function’) requiring the highest levels of oversight and protection”.
Inadequacies were also identified in technical expertise for network timing systems, with Telstra being told it needed stronger “competency and more curiosity to investigate when something looked wrong”.
It also found Telstra had shortcomings in “process discipline including change management, configuration control, incident management, documentation and follow through”.
The incident response was lacking with gaps in ownership and visibility, while operational support of the network timing system delayed the identification of the problem in the early stages of the outage.
Telstra on Wednesday declared it had made changes in the wake of the outage including reviewing critical non-timing related functions across its network, reviewing its vendor notification management processes to improve how future alerts play out.
It has also assessed its “alarm instructions and service assurance arrangements to ensure our technical teams can action all issues without delay”.
