Candidates unable to access IRPTs
- 1 Event Name
- 1.1 Issue Summary
- 1.1.1 Issue Raised by:
- 1.1.2 Issue Resolved by:
- 1.2 Resolution
- 1.3 Mitigation Actions and Next Steps
- 1.1 Issue Summary
Issue Summary
Candidates attempting to access their practice test were receiving an error message.
2:34PM - Wendy contacted me via Teams that she was receiving an influx of emails regarding users receiving an error when attempting to access their Individual practice test.
3:03PM – Dan contacted me via Teams about the issue.
3:06PM – I acknowledged both teams messages and immediately began my investigation.
3:06PM – Wendy provides 4 helpdesk ticket ids for Candidates reporting the issue that contain screenshots / copies of the error message being received
3:07PM – I accessed my test account on Production which is related to an existing 17A form and was able to successfully access it
3:08PM – I accessed my test account on Production which had never accessed any practice test and I was able to successfully access it.
Issue Raised by:
Wendy Boheim
Issue Resolved by:
Robert Adach
Resolution
After confirming that the issue was resolved, I began an investigation to determine the root cause of the issue. Due to recent changes to support the new 22A form, some logging of IPT access is no longer available. The issue impacted 11 Candidates who were accessing their IPT for the first time today. Users who had previously accessed their Individual Practice Test would not have been impacted by the issue
From the available logs and the help of the tickets provided by Wendy, I was able to identify that a script which was run to process an activate the 22A forms had not been completed successfully. This script resulted in the system not being able to identify the proper form to deliver to a candidate for their first access. A unrelated update to the database revealed that this script had not completed successfully and was rerun, unaware of the lingering access issue.
Although the script in error was run against the test environment multiple times, the particular failure was not not experiences, and the error catching implemented as part of the script did not adequately isolate the failure.
Mitigation Actions and Next Steps
Item | Description | Due Date | Owner |
---|---|---|---|
Implement additional alerting on databases | We will implement additional logging and alerting which will notify operators if a query or task takes an extended period of time to process. | 2/1/2022 | Robert Adach |
Implement additional fail safes in scripts | Investigate and implement if possible additional fail safes in database scripts to encompass all possible points of failure | 3/1/2022 | Robert Adach |
Implement additional verification steps | Ensure that deployment scripts include test steps that cover multiple scenarios to ensure that all impacted scenarios | 2/1/2022 | Robert Adach |
Tickets relating to event
Helpdesk.com N0G315, QABLCN,PARGD0, F3HE6K