Designing Services for Resilience

This technique falls under fault prevention and fault removal categories...In my opinion, scalability is a commodity now. Resiliency and Intelligence are the next frontiers.

Chaos engineering for resilient distributed systems

Errors are a fact of life in large scale distributed systems. Now while a system can have error they are only significant if they interrupt the service i.e, produce a failure. The goal of resilient systems is to mask these failures and continue functioning at an acceptable level.

One key component in the development and evaluation of such a resilient system is having ability to simulate faults in controlled manner.

Chaos engineering is basically about empirically studying the system, simulating chaos, evaluating system resilience and building confidence in system capability to withstand real-world turbulent conditions. 

Many developers don't distinguish between faults, errors and failures but otherwise this distinction is fundamental to engineering chaos in a distributed system. A fault is a defect that exists in a service but may be "active" or "dormant". An error occurs when fault becomes active. A failure occurs when the error is not suppressed and becomes visible outside the service.

Pathology of a Failure

How Complex Systems Fail

Came across this summary paper on the nature of failure, how failure is evaluated and how failure is attributed to causes. While the paper is in context of hospitals and patient safety, it is applicable for big data systems as well. Following are some highlights...

Complex systems are intrinsically hazardous systems

All of the interesting systems (e.g. transportation, healthcare, power generation) are inherently and unavoidably hazardous by the own nature.

Complex systems are heavily and successfully defended against failure

The high consequences of failure lead over time to the construction of multiple layers of defense against failure. The effect of these measures is to provide a series of shields that normally divert operations away from accidents.

Catastrophe requires multiple failures - single point failures are not enough

The array of defenses works. System operations are generally successful. Overt catastrophic failure occurs when small, apparently innocuous failures join to create opportunity for a systemic accident.

Complex systems contain changing mixtures of failures latent in them.

The complexity of these systems makes it impossible for them to run without multiple flaws being present. Because these are individually insufficient to cause failure they are regarded as minor factors during operations.

Software Resilience Patterns

Good slide deck on patterns for software resiliency...

Patterns of resilience from Uwe Friedrichsen

Scalability, Availability and Stability Patterns

Came across this sometime back. Good slide deck...

Scalability, Availability & Stability Patterns from Jonas Bonér