AINIAA failure we could not prevent. Monitoring that could keep running.
For AINIA, a technology centre specialising in research and development, we addressed a critical continuity issue: failures when connecting to sensors could close the application and interrupt the monitoring of ongoing laboratory analyses.
Original case-study image · Click to enlarge
A memory error could interrupt a laboratory analysis in progress.
The laboratory application had to monitor several sensors to collect metrics from analyses in progress. Under certain conditions, connecting to these sensors caused a catastrophic error that closed the program and stopped monitoring.
The issue could not be resolved from within the main application: the failure corrupted memory pointers, making conventional in-process recovery impossible.
What needed protecting
Continuous monitoring
Prevent a component failure from interrupting the monitoring of samples.
Failure isolation
Protect the main application from errors in the sensor connection.
Automatic recovery
Detect an incident and restore the auxiliary service.
Three separate applications to contain failures and restore service.
We assessed the complexity of the problem and designed an architecture in which sensor communication was isolated from the main application, with a third application responsible for monitoring.
Main application
Keeps laboratory operations running independently of the failing process.
Sensor application
Handles sensor connections and can fail without closing the main application.
Monitoring application
Detects an auxiliary service failure and restores it.
Understand the failure
We analysed system behaviour and the limits of recovery after memory errors.
Separate responsibilities
We isolated the main application from the process responsible for communicating with sensors.
Recover without stopping
We created a third process to detect failures and restart the secondary component.

