Designing Services for Resilience

This technique falls under fault prevention and fault removal categories...In my opinion, scalability is a commodity now. Resiliency and Intelligence are the next frontiers.

How Complex Systems Fail

Came across this summary paper on the nature of failure, how failure is evaluated and how failure is attributed to causes. While the paper is in context of hospitals and patient safety, it is applicable for big data systems as well. Following are some highlights...

Complex systems are intrinsically hazardous systems

All of the interesting systems (e.g. transportation, healthcare, power generation) are inherently and unavoidably hazardous by the own nature.

Complex systems are heavily and successfully defended against failure

The high consequences of failure lead over time to the construction of multiple layers of defense against failure. The effect of these measures is to provide a series of shields that normally divert operations away from accidents.

Catastrophe requires multiple failures - single point failures are not enough

The array of defenses works. System operations are generally successful. Overt catastrophic failure occurs when small, apparently innocuous failures join to create opportunity for a systemic accident.

Complex systems contain changing mixtures of failures latent in them.

The complexity of these systems makes it impossible for them to run without multiple flaws being present. Because these are individually insufficient to cause failure they are regarded as minor factors during operations.

Designing distributed applications with code mobility paradigms

When we are working on a distributed systems, many times during design phase we automatically assume the location of code as static. In other words, once a component is created, it cannot change either the location or the code during its life time. 

But there are also scenarios where considering concepts of code location and mobility etc during design phase makes a huge difference to the underlying distributed application in terms of fault-tolerance, concurrency, lower latencies and higher flexibility.

Rest of this post covers two notions of code mobility, various code mobility paradigms and some scenarios where a distributed application can benefit considerably by exploiting mobile code paradigms.

Techniques to ensure eventual consistency

Recently I came across a paper on eventual consistency that touches among other things techniques to ensure eventual consistency for the updates to replicas. While the term eventual consistency is fairly popular now a days, I am not sure some of its subtleties are equally well known. So a small detour before diving into the techniques to ensure eventual consistency.

Eventual consistency basically guarantees that if there were no additional updates to a given data item then all reads for that data item will eventually return the same value. 

What the above also means that in an eventually consistent system,

• The system can return any arbitrary data and still be eventually consistent. So client has no way to know if the read response is wrong behavior. 
• If there are multiple concurrent updates, under eventually consistency you do not know which update gets eventually chosen. The order is unpredictable. Only guarantee is, there will be eventually a convergence. 
• In short, what eventual consistency tells us is something good will happen eventually but no guarantees as to what happens in the interim and no behavior is ruled out in the meantime!

Consistency considerations in distributed data stores

In a large distributed system, network partitions are a given. This means we cannot achieve both consistency and availability (CAP theorem). So our choices are either to relax the consistency for system to be highly available under partitions (or) make consistency a priority and system will not be available under certain conditions. Both of these options require developers to be aware of what the system is offering.

For example, if the system is emphasizing consistency, then the developer has to deal with availability issues. So if an update fails because of system unavailability, then developer need to plan on what to do with that update.

On other hand, if the system is emphasizing availability, then the developer should assume there will be times when the reads will not return latest updates. The application needs to be tolerant i.e., work with slightly stale data.

Many faces of replication...

Recently I was looking into replication and consistency papers and thought would be good topic to summarize. Replication is one of the most studied topics and is a quite important tool for designer.

It improves system availability by removing single point of failures, improves performance by reducing communication overheads and improves scalability by enabling system to grow with acceptable response times. But the benefits of replication comes with its own challenges. Nothing comes for free in distributed systems...

For example, some of the challenges anyone dealing with replication have to address are -

• How to manage the updates i.e., replication strategy? 
• Data Consistency & Availability tradeoffs? 
• How to handle downtime during new replica creation 
• Maintenance Overhead 
• Lower write performance etc. 

Designing scalable failure detection for distributed sensor nodes

Distributed systems need to provide reliable and continuous service despite failure of some of the components or nodes. So failure detection is one of the key aspects in building these systems. Also we generally will need failure detection for lot of other stuff like consensus, group membership, high availability etc.

The challenge is designing failure detection in a way that is both accurate and efficient is not an easy task though. One reason is the delays in a network are unpredictable. Another reason is the FLP impossibility theorem i.e., in any asynchronous distributed systems, it is impossible to determine accurately whether a remote process failed or has just taking its own time to respond. 

Bright side, a node really don't need to know if a remote process has failed or just taking its own time. As long as we have a way for all nodes to come to same conclusion (i.e., failed or delay) about a remote process then we can work from there. You do this by assuming an upper bound on the delays (ex: timeouts) after which a remote process is considered as failed. In academic speak, it is called a partially synchronous model

In rest of the post we go thru 

1. Eight factors to consider in designing failure detection, 
2. Four key metrics to consider in evaluating failure detection and 
3. Static, Lazy, Gossip, Hierarchical, Adaptive and Accrual based failure detection techniques.
4. Failure detection approaches in some popular distributed systems.

Failover Strategies & Techniques in Distributed Applications

Systems fail. Networks fail. Processes fail. The secret, let it fail, but fix it fast. If services are restored quickly enough before the user notices then did the failure occur? 

Broadly there are four types of failover strategies -

1. No Failover
2. Cold Failover
3. Warm Failover
4. Hot Failover

These strategies vary in their recovery time, cost and impact. Also often a combination of these strategies are used. For example, hot failover strategy for high availability and cold failover strategy for disaster recovery to resume availability.

Next part is techniques. Typically the techniques utilized for doing hot failover are -

1. Client-based failover
2. DNS-based failover
3. Network-based failover
4. IP Address takeover
5. Gratuitous ARP based failover
6. Server-based failover

Some failover techniques like DNS-based failover are provided as a service by many cloud providers. On other extreme, techniques like Gratuitous ARP based failover are not at all cloud friendly. 

Distributed systems are all about trade-offs. Each failover strategy mentioned above has its own pro's and con's. Same with failover techniques. In rest of the post we dive deeper into these details. At the end we touch base some of the challenges in failover like tug-of-war, split-brain, quorums etc.

Designing Messaging for Scalable Distributed Systems - Part 2

Recently I had few discussions on messaging. Thought it would be an interesting follow up to explore some additional considerations that are important but typically don't surface in early phases of messaging system design.

The focus of this post is on following aspects -

1. Which scenarios in system design are better suited for async messaging?
2. What are the issues to consider during implementation of distributed messaging functionalities?
3. How can we make the distributed messaging solution easier to monitor, debug & support?

Designing Messaging for Scalable Distributed Systems

The cloud has changed considerably the scale of distributed systems. As the size of the systems grow, it becomes increasingly difficult to design them and keep them running. To avoid those difficulties, most large scale architectures use loosely coupled technologies. 

The vehicle that is often used in this journey to paradise (or inferno) of scaling the system is the message bus. If implemented properly, I think messaging is a highly valuable element in the architecture for near infinite scale. 

If implemented haphazardly...

You get the point...:-)

Scalable Distributed Systems - Introduction

In recent years couple factors have increasingly become important in design of distributed systems i.e., Scalability & Reliability of the system. Over time I picked few things related to these factors. This post series is an attempt to share my modest knowledge on scalability aspects. 

What is Scalability? 
Simply put it is ability of the system to handle increasing load whether it is addition of users or resources or both. Now typically the scale of a system has 3 dimensions -

1. The quantity dimension i.e., number of users, resources, objects etc that are part of the system
2. The distribution dimension i.e., geographical distribution of servers, services, data etc. 
3. The administrative dimension i.e., the number of organizations, multi-tenancy etc

These dimensions in turn affect a whole host of components that are needed for a distributed system. 

Building a scalable system does not happen by accident. Similarly a distributed system is not automatically a scalable system. So it is important to consider the effects of scale in these dimensions early on.