Feel free to let me know if you want me to talk about this topic publicly or privately. It will take ~20 mins to talk and you’ll be able to learn a few patterns of the large scale distributed systems and their failure modes.

Linkedin

These are the slides I used but it will be difficult to understand my messages only from the slides. I’m wondering to create a YouTube video as well if this is an interesting topic for many people.

https://speakerdeck.com/opsbr/large-scale-distributed-systems-patterns

I’ve launched the chapter 2 of “Decoding Attention”. In this chapter, you’ll learn about the first and last layers of Transformer - Embedding and Linear.

• Code: https://github.com/opsbr/decoding-attention

• Live demo: https://decoding-attention.opsbr.dev/chapter2

The progress is good as I’ve completed my own Qwen3 implementation, thanks to ’s reference implementation.

I’ll keep working on “Decoding Attention”. The next chapter is the neural network and I’ll explain from the basic perceptron to Qwen3’s MLP.

tl;dr - “Decoding Attention“

“Decoding Attention” is the course name and I’ve published the chapter 1:

“Attention” is the core of the majority of the current AI systems and the AI architecture handling Attention very well is called “Transformer”. Now, Decoding Attention is intended to describe Attention/Transformer to non-ML (Machine Learning) experts so that they can start their journey to the AI-era.

The first chapter of Decoding Attention is explaining the input and output of Transformer model so that you can at least understand Transformer as a black box function for now. As a software engineer, understanding a new thing from its in/out should be the straightforward method, I believe:

• Code: https://github.com/opsbr/decoding-attention

• Live demo: https://decoding-attention.opsbr.dev/chapter1

  • Note: It’s running on a Raspberry Pi 5, so it might be disconnected sometime!

Why do I need to understand Transformer? I’m a software engineer!

To answer this question, I’d like to introduce Andrej Karpathy’s “Software 3.0” idea.

Software 3.0

We, as a software engineer, have been working on Software 1.0 for decades and “Software was eating the world” really. However, about 10 years ago, probably because of Deep Learning, the neural network started eating the world, like visual recognition, machine translation, etc. — Software 2.0 began. Now, for the last few years, LLMs are becoming much popular and the natural language is the hottest programing language today i.e. Software 3.0.

However, this doesn’t mean all Software 1.0 territory is taken by Software 2.0 or 3.0. Today’s best applications need all phases so that you should know all of them to build a good application today. Thankfully, you already know Software 1.0. So, you can focus on learning Software 2.0 and 3.0, then you’ll be the top talent of software engineer today.

Transformer

Both Software 2.0 and 3.0 are backed by neural network (that we’ll learn later in Decoding Attention), especially Transformer architecture. People who build Software 2.0 today is basically machine learning experts and that hasn’t been changed even since before Software 2.0 began because it requires careful ML training design and execution. However, these days LLM’s training is being democratized thanks to “post” training phases. Without fully training LLM from scratch, you can fine-tune the existing models (and that was the breakthrough of ChatGPT). I don’t say it’s super easy for everyone so far, but it is getting easier e.g. Text-to-LoRA is opening this by just typing natural language. That’s why I’m very interested in post training today.

Software 3.0 is already open for everyone even who doesn’t have any coding experience e.g. I really love this video — 9-13 years old kids can vibe-code. However, it’s still rough, and just like you understand OS/CPU/etc. as a software (1.0) engineer, understanding the foundational system is quite useful to build sophisticated applications. Andrej called it “LLM OS” that I love, too:

So, going back to the original question: “Why do I need to learn Transformer?” My answer is because it’s core of Software 2.0 and 3.0.

At least to me, who didn’t know Transformer at all 2 months ago, this understanding gives the clear picture of LLM OS and it’s the basic of understanding Software 2.0 stack (I’m learning post training now). Thus, I’d love to share the same experience to you.

What’s next?

I’ll keep publishing the remaining chapters. The next chapter is one layer deeper — “Embedding”. Stay tuned!

Subscribe now

As of May 2025, most software developers see AI/LLM as:

• Coding agents

• HTTP APIs

So, if they want to use AI/LLM in their traditional software development, they typically try to find the best model and the best way to utilize a coding agent. When they need to embed AI/LLM in their products, they tend to treat it as just another third-party API dependency—i.e., a black box.

However, I see AI/LLM as an emerging computing model that is poised to dominate new ideas for at least the next few years. In this post, I’d like to explain my understanding of AI/LLM and recommend several learning resources that I have used.

Why start with the foundations?

We, as software engineers, learn about CPUs that have been dominant in computing for decades. Although most software developers today won’t build a CPU themselves, learning how a CPU works is critical because it’s the system we operate every day. That’s why many computer-science programs include a course to build a CPU from scratch!

In addition to the CPU, the OS is also important because the majority of software runs on an OS. So, software engineers learn about one or more modern OSs, including processes, memory management, system calls, etc. (One of my favorite UNIX commands is strace.)

Learning how compilers and interpreters work also has huge benefits for understanding how your programs run, because they ultimately become 0/1 binaries.

I learned these topics while working as an SRE, DBA, and SWE. I also built a CPU (8086) emulator and tried to run a MINIX binary to see how the CPU and OS actually work, and I built a Forth compiler and interpreter with LLVM to understand virtual machines. Of course, I’ve never worked on building a CPU, OS, or programming language/VM professionally, but knowing these low-level details helps me see what’s happening under the hood. Because all the software engineers I respect are experts in these areas, I tried to catch up, and these have been my foundational skills as a senior engineer for years.

Enter the Transformer

When I decided to shift my career toward AI development, the first thing I wanted to understand wasn’t how to use coding agents or build AI agents. Instead, I knew it was more important to learn the foundational technology of AI so I could anticipate what’s going on.

One friend mentioned the word “Transformer.” I’d never heard it at the time, but I now know the Transformer is the dominant technology in the AI world today (though that may change tomorrow).

Luckily, I had a little experience with machine learning, so I recalled some basics of neural networks and backpropagation from college 20 years ago. That tiny advantage became a bridge I want to share with software engineers learning AI/LLM today:

A Transformer is a special neural-network architecture that’s incredibly powerful—not only for LLMs but also for image and voice generation, among many other tasks. I originally thought an LLM was just a large stack of perceptron layers (sometimes called an MLP—Multi-Layer Perceptron). This isn’t true. The Transformer network is a purposefully designed, complex architecture. Although it contains MLP layers, its key component—the attention mechanism—is essentially a sophisticated matrix multiplication, not an MLP.

At first, I wasn’t sure what the Transformer architecture was because most online courses assumed students already knew it. Transformers are that ubiquitous today, and I felt embarrassed for not knowing about this 2017 invention sooner.

Learning resources that clicked

To understand Transformer, I watched many online video courses. For example, this is the very passionate explanation from one of the authors of the origin of today’s AI - “Attention is All You Need“ paper:

I guess this is too hard to understand without NLP background, however. Actually, before watching the video above, I went through Stanford CS224N NLP course. I recommend the entire series, but these two lectures gave me “aha” moments:

• Stanford CS224N | Spring 2024 | Lecture 7 - Attention

• Stanford CS224N | Spring 2023 | Lecture 8 - Self-Attention and Transformers

I also just found this video today that is well visualizing LLM in just 7 minutes. I highly recommend to start from here now:

After that, my instinct was: “I want to build one.” There are many prebuilt libraries and models to run Transformers in PyTorch locally, but I wanted to write the code myself. For that, Stanford CS336 was perfect. It provided the logical architecture of a Transformer plus PyTorch implementations for every component, including most popular improvements since 2017.

A great companion blog post is “Attention Wasn’t All We Needed,” which lists key techniques that evolved the Transformer—CS336 covers almost all of them.

CS336 also covers hardware—specifically GPUs. Just like CPUs, GPUs are at the core of AI/LLM, so understanding GPU architecture is essential. (Flash Attention, for example, is a hardware-aware optimization of attention calculation.)

Putting it all together

I’ll probably never build foundational LLMs myself; that work happens in huge GPU farms and belongs to LLM researchers. However, just as CPU/OS/compiler knowledge helps traditional software development, learning about GPUs, Transformers, and post-training lets me understand AI/LLM discussions.

When you build AI agents, this understanding is critical to using LLMs effectively. For instance, you won’t fully grasp why context size is limited—or how some models overcome it—until you study Transformer internals. I won’t claim to fully understand everything yet, but I can now at least imagine and research these issues.

I’ve skipped post-training here, but I believe fine-tuning is becoming a key to AI applications. It doesn’t need the massive data and hardware dictated by scaling laws, but it does require techniques beyond the Transformer itself, such as reinforcement learning (which I’m currently studying). Still, understanding GPUs and Transformers is foundational.

I hope my investment in these foundations pays off. As mentioned, I also plan to build learning content for software developers who lack machine-learning experience. Let me know if you’re interested!

Subscribe now

Share

So, in this magazine, I’ll keep posting about AI engineering, too. Let’s start from how I’m learning about the foundation of AI.

When I started to change my career to AI engineer, the biggest mysterious part for me was “What is LLM?” I knew it’s a neural network learned from the internet and it can predict the next token. But I didn’t know how it works under the hood. At that moment, I didn’t know “Transformer” nor “Attention” as my NLP knowledge wasn’t updated since Word2Vec.

Fortunately, I encountered the best online course that is actually running in this term - Stanford CS336 | Language Modeling from Scratch. I jumped on this course right away, keep watching the lecture videos and tackling the assignments at my own pace. Their YouTube list is here.

This course describes LLM in details as well as provides the very handy assignments. Let me show you what I've learned from the assignment 1.

The assignment 1 guides you to build the latest Transformer architecture from scratch, that means you don’t use PyTorch’s built-in modules but reimplement them by using only primitive functions. While it’s better to use those built-in modules for most of the cases, it’s difficult for me to understand the inside without writing the logic by myself.

I take this type of approach usually when I learn something new; I built B+Tree to understand database, Multi-Paxos to understand consensus, Forth interpreter to understand compiler and virtual machine, 8086 emulator to understand CPU. This time, I built Transformer to understand the language model.

Here is the list of building blocks I built. All have unit tests, so it’s easy to verify:

• BPE (Byte-Pair Encoding) Tokenizer

• Linear layer

• Embedding

• RMS Norm

• SwiGLU

• RoPE (Rotary Positional Embeddings)

• Softmax

• Scaled Dot-Product Attention

• Causal Multi-Head Self-Attention

• Transformer (decoder-only)

• Cross-entropy loss

• AdamW

• Learning rate scheduler with warmup and cosine

• Gradient clipping

Then, I was able to train the model on M1 MacBook Pro (lr is the learning rate and loss is the training loss) with TinyStory training data:

Here is a few token generations at the several steps during training (“Once upon a time,“ is the prompt):

Time: 2025-05-17 22:45:33.400874 Step: 1, Loss: 9.251720428466797

Once upon a time, drives aisle shop heelsluffy fact brave poking suffer achieved Rich Ally Trust jugom guide greetedip suggest terrified wishesches N doctorunnies Gus magnets tipped animalsould darling green Joey dishwasherork later roadiens guyslex stays driveway ruffaredarrasschenulptureasis sadlyael…

Step 1 is completely at random.

Time: 2025-05-17 22:52:08.402544 Step: 500, Loss: 2.468946933746338

Once upon a time, there was a big tree. The tree was very happy. The tree was very good at the tree.

One day, a little bird came to the tree. The bird wanted to find the bird, so it looked very pretty. The bird saw…

At step 500, it looks like English but the meaning is useless.

Time: 2025-05-17 22:58:22.731600 Step: 1000, Loss: 2.165599822998047

Once upon a time, there was a little boy named Tim. Tim had a toy box. He liked to play with it every day. One day, Tim found a small toy under his bed. It was a small, green toy. Tim was very happy.

Tim…

At step 1000, it’s good already. Nice progress.

Time: 2025-05-17 23:10:47.535277 Step: 2000, Loss: 1.9070618152618408

Once upon a time, there was a little girl named Lily. She had a pretty doll named Rose. Lily loved to play with Rose. They would run and jump all day long. Lily was very happy.

One day, Lily wanted to show Lily her doll to her…

At step 2000, the first paragraph looks perfect. The second paragraph looks a bit confusing.

Time: 2025-05-17 23:47:30.009432 Step: 5000, Loss: 1.763701319694519

Once upon a time, there was a little boy named Tim. Tim had a big dream. In his dream, he was a big, round ball. He loved to play with his ball every day.

One day, Tim's friend, Sue, came over to play…

Finally, it reaches 1.7 training loss at step 5000. It looks almost similar level to step 2000, though.

The learning curve wasn’t good initially so that I needed to investigate multiple aspects. This is the very first learning curve and the inferences, lol:

After stabilizing this crazy curve, I still struggled with higher loss. It ends up to my bugs e.g. clipping gradient before backward pass, and forgetting to truncate and normalize the initial parameters of attention matrixes:

These very hands-on experience gives me deep understanding of how Transformer works and how to train it e.g. I really understand how the computational stability is important through the bugs above…

Because I built and trained Transformer once, I start to understand how each component works. Since Transformer is the dominant architecture today, I can somehow understand the novel proposals that was impossible for me one month ago.

I’ll keep learning CS336 and I’m especially interested in Alignment sections upcoming because now my next mysterious point is post training alignments.

Remember, I didn’t know Transformer at all one month ago. So, you could build your own model in a month even if you don’t know Transformer today! Feel free to ask me if you have trouble to run CS336 assignment 1.

Subscribe now

The series consists of 5 short posts:

1. Introduction

2. Why is Cell-based Architecture needed?

3. What is Cell-based Architecture?

4. When to use Cell-based Architecture? (this post)

5. How to implement Cell-based Architecture?

Use cases

B2B SaaS a.k.a. Cloud services

Recent days, B2B SaaS becomes a popular business model as it’s profitable and scalable, like AWS. B2B SaaS is typically provided with some form of multi-tenant applications so that the service provider can utilize the shared resources to gain more customers than the system allows if everyone runs 100% workloads at the same time i.e. overcommit.

User-based cell partitioning

In such multi-tenant architecture, your business would be capped somewhere if you only run one application. For example, if your application has some limitations (e.g. hard limit of a cloud service) and can host up to 1K users, your business can’t scale more than 1K users.

Instead, with Cell-based Architecture, you can horizontally scale your business by simply adding more cells e.g. 1K*N users if there are N cells. Because B2B SaaS customers typically don’t require any interactions between customers, they’re easily isolated to cells. Here is the examples of user-based cell partitioning:

In addition, if you want to handle single-tenant use case, it’s pretty easily done with user-based cell partitioning because single-tenant cells are just a special variant of multi-tenant cells that have only one customer. Everything else can be identical.

Resource-based cell partitioning

Also, Resource-based cell partitioning is common since B2B SaaS providers typically charge per resources e.g. DynamoDB tables or S3 objects. This allows one customer to own more tables than one cell can serve. Let’s see the example below:

E-Commerce

The next one is a B2C example - EC site. Except product reviews, cross-user interactions are not needed in most of the cases, like B2B SaaS. So, it’s easy to partition by user i.e. one cell host whole user experiences. But, how to handle inventory? We can’t store silo of inventory to each cell because one SKU might be ordered by two users against different cells’ inventory i.e. Split brain problem.

Then, microservice comes in. Cell doesn’t have to be end-to-end full stack. For example, Frontend microservice is split by user while Inventory service is split by resource e.g. SKU. Then, Frontend service cells talk with Inventory service through its Routing layer when accessing SKU.

Physical silo

The last example is a little bit unique. Sometime your business is tightly coupled with physical locations. For example, you want to run microservices to manage operations in warehouses you physically own. Or, your service provides regional or AZ isolated features. In these cases, cells partitioned by physical locations makes more sense and whole microservices are partitioned by the same cell boundaries and only communicate within the same cell. Therefore, Routing layer isn’t needed in front of Microservice 2 in the example below:

Summary

In this post, I showed several use cases where Cell-based Architecture is used. B2B SaaS is suitable because users/resources are typically independent. EC site can be partitioned by users on frontend but backend data e.g. inventory would be served by different microservice with different cell dimension. Lastly, cells can be tied with physical locations to isolate whole system stack per location.

What’s next?

So far, we’ve learned several Cell-based Architecture patterns at very high level design. In the last post of this series, I’d like to show more detailed examples so that you can see some actual working code to wrap up this 101 series.

Thank you for reading. If you’re interested in this series, please hit the “Like“ button and share this to any social media. I would be motivated more! Also, subscribe this magazine to receive the upcoming posts for this series!

Share

Subscribe now

1. Scale is not manageable

2. Logical SPOF

Because you need to scale the only one stack of system to fit your business demand, you can’t control the size of users, resources, etc., that leads unknown scalability bottlenecks.

Also, no matter how each cloud service is designed as highly available as possible, a single resource can be a logical SPOF anytime and could down the entire system easily.

In this post, I’m going to introduce the key concepts of Cell-based Architecture and why they can address these issues.

The series consists of 5 short posts:

1. Introduction

2. Why is Cell-based Architecture needed?

3. What is Cell-based Architecture? (this post)

4. When to use Cell-based Architecture?

5. How to implement Cell-based Architecture?

Cell

As I mentioned above, one of the key issues of the single stack system is unmanageable scale. If you can control the maximum size of the system e.g. the maximum number of users or resources, it’s much easier to manage the scalability of the system. For example, during a load testing, you don’t have to worry about too large scalability way above the maximum size and it should be a good coverage if you could address all the known scalability issues found by the load testing.

This limited-scale single application stack is called “Cell“. It’s manageable in terms of scale per cell. If you want to host more users/resources than the maximum size of one cell, you simply need to add more cells. For example, UserID 1-100 are hosted in Cell-0 and UserID 101-200 are hosted in Cell-1, and so forth. See the example below:

As you can see above, you manage two completely isolated systems, each has identical full stack architecture1 but stores completely different data set.

How about availability? Now, every layer is no more logical SPOF. If you have a super problematic user i.e. Black swan/Poison pill and your database is brown away, only one of cell is affected. That means the blast radius is 50% of the total users. In other words, if you have N cells, the blast radius is 1/N. This is also helpful when deploying a bad commit. With cells, you can deploy cell by cell and monitor after each deployment and block promotions when it’s bad. So, if there is a pretty bad deployment, the first deployed cell is the only victim i.e. 50% blast radius, again.

The other notes for cells are below. Some will be covered by the remaining series:

• Running the same code

  • A cell isn’t a tailer-made solution per customer

• Silo as much as possible

  • From top to bottom; ALB to RDS, even AWS account should be isolated per cell2

  • Typically, no direct communication between cells are allowed

• Partitioning dimension differs case by case

  • Partition by user

    • Good when the number of resources per user is manageable

  • Partition by resource

    • Good when the number of resources per user is unmanageable

• Heterogeneous size is acceptable

  • One cell could host only 1 user while others host 100s per cell because of single-tenant request or isolating noisy neighbors

Summary of Cell

Cell is a very simple but powerful concept. By just splitting your users or resources, you can solve both scalability and availability problems we talked so far. It’s so simple that any type of systems can be implemented as a cell. If you have a running application, that would be called as Cell-0 today and you can prepare Cell-1 tomorrow.

Cross-cell Data plane a.k.a. Routing layer

Although the concept of cell is simple, the reality isn’t. As you might notice, I haven’t talked about how to “route“ each user to the appropriate cell. This is the uniqueness of Cell-based Architecture.

You probably want to keep your system look non-cellular from the users’ perspective because it’s internal details. Another reason for that is you might migrate several users to the other cell, then they need to change their endpoint to the new cell unless some cross-cell layer exists. In the example below, “Routing layer“ is the component to dispatch users to the appropriate cells. I would call it Cross-cell Data plane but “Routing layer“ is more famous name:

In this diagram, Routing layer looks like an inline proxy that receives all the requests and dispatch them based on their UserID. That’s one solution but sometime it can be an aside component such as CNAME e.g. user1.example.com => cell-0.example.com. With this solution, Routing layer isn’t a proxy but somehow control the routing rules. I’ll talk about more patterns in the last article of this series.

The most critical rule of Routing layer is keeping it as thin as possible. Because it’s impossible to split into cells, it can easily hit some scalability bottlenecks. Unlike the application itself, the purpose of Routing layer is just routing, so the implementation can be much simpler i.e. more scalable than each cell. It’s also a logical SPOF, so highly available design is important and deployments should be done carefully.

Cross-cell Control plane a.k.a. Cell migration

The last component of Cell-based Architecture is Cross-cell Control plane. The key difference from Cross-cell Data plane is that this component moves data across cells. So, this is also called “Cell migration“. In the example below, UserID 100 is migrated from Cell-0 to Cell-1. During the process, Cell migration touches not only both cells but also Routing layer to appropriately re-route the requests:

Cell migration is typically a workflow that carefully designed to keep data consistency during migration i.e. treating “Split brain“ problem. As you notice, it’s not a simple problem in distributed system, so Cell migration is the most advanced component of Cell-based Architecture and it’s not easy to implement correctly. Thus, it’s not a bad idea to start Cell migration with maintenance window e.g. while blocking all access by UserID 100, you copy all the data of UserID 100 to the new cell, then update Routing layer.

Why do you need to move data across cells? There are two reasons:

1. Cell balancing

2. Noisy neighbor isolation

Cell balancing is needed when some cells become too large or small. Although cell should be designed and operated under the manageable size, sometime you need to adjust such thresholds due to a new discovery. Or, you might just want to merge/split cells for better management, especially you adopt Cell-based Architecture after several years of operations.

Noisy neighbor is a user/resource that brings too high load to the cell e.g. DDoS attackers, celebrities, etc. Once you identify them, it is a good strategy to isolate them from the others for both parties. For example, DDoS attackers can be migrated to a special cell for only such users so that any normal users won’t be affected.

Anyway, the capability of Cell migration is a huge benefit and makes cell operations easier and safer. However, because of the complexity, it is recommended to implement Cell migration from day one.

Summary

In this post, I described three components of Cell-based Architecture: 1. Cell - A limited-scale and isolated full stack application, 2. Routing layer - A cross-cell data plane to route traffic to the appropriate cells, 3. Cell migration - A cross-cell control plane to move data across cells. By using Cells, you can handle scalability and availability problems we talked before pretty simply. However, Cells rises the other problems, such as separate endpoints, cell rebalancing. Both cross-cell components fill such gaps.

What’s next

You’ve learned the core concept of Cell-based Architecture here. The next topic is when it’s needed. The two problems - scalability and availability - are ubiquitous but I’m going to apply them for several use cases so that you can understand why they are the issues and Cell-based Architecture is needed there.

Thank you for reading. If you’re interested in this series, please hit the “Like“ button and share this to any social media. I would be motivated more! Also, subscribe this magazine to receive the upcoming posts for this series!

Share

Subscribe now

1. Scalability

2. Availability

However, these are very common problems in any cloud system and there are tons of solutions/design addressing them already. Why do we need to solve them again?

In this second episode of “Cell-based Architecture 101“, I’d like to talk about the limitations of the existing solutions/design in cloud so that you would be able to understand why this new architecture is needed.

The series consists of 5 short posts:

1. Introduction

2. Why is Cell-based Architecture needed? (this post)

3. What is Cell-based Architecture?

4. When to use Cell-based Architecture?

5. How to implement Cell-based Architecture?

“Highly scalable and available” in cloud

Assuming you’ve already built or designed some application in cloud. You know one of the core values of cloud is the massive scalability because the cloud services are running on the hyperscale infrastructure. For example, an Amazon S3’s bucket can contain billions of objects or more. So, when you design an application using S3 bucket, you don’t have to worry about the maximum number of objects allowed in a bucket.

Another big benefit of cloud is the capability of highly available system design. Because of elasticity and physical distribution, you can build your system using multiple instances/availability-zones/regions to make it more available. For example, Amazon RDS offers multi-AZ failover so that a DB instance isn’t Single-Point-Of-Failure a.k.a. SPOF anymore. Like Netfilx, you can shift traffic across regions to avoid even single region failures.

Therefore, with such traits of cloud, your application is typically highly scalable and available out of the box. Let me introduce a very simple example here; A request/response system, ALB for load balancing, EC2 for application servers with Auto Scaling Group to scale in/out and RDS for database:

In this system, ALB, EC2 and RDS are deployed in multi-AZ. Therefore, it’s single-AZ failure tolerant. Also, ALB, EC2 and RDS can scale elastically so that it can support from 1 req/s to millions. No need to worry about scalability nor availability.

…Well, is this true? As you expect, it’s not 100% true. Let’s find some limitations in the next section.

Scalability of cloud

In the previous example, you’ve probably noticed that it has an obvious scalability bottleneck. RDS needs a single writer instance and it must have scalability limit because it’s a single machine so that CPU, memory, network or any other resource has its physical limitation1. That’s a super easy bottleneck, but is this the only one?

ALB is more scalable than RDS but it has some default quotas. Although most of them are adjustable, there is no guarantee that your limit increase requests are approved by AWS. Some might have actual hard limits internally so that you couldn’t exceed that hard limit. For example, the default quota for the number of certificates per ALB is 25. So, if this application provides a custom domain feature, one ALB can host up to just 25 custom domains until the limit increase request is approved2. The same logic can be applied to EC2 quota for the number of instances i.e. EC2 layer might not be scalable until limit increases even though you use Auto Scaling Group properly.

Lastly, even if your system doesn’t hit any service quotas and RDS writer can handle the volume of the requests, EC2 instance itself could hit some limit. For example, if you want to cache some metadata in-memory of your application running on EC2 instance, you need to be careful about the size of the metadata. Let’s say the metadata glows proportionally with the number of users and need to be cached in-memory on every instance to achieve the performance target. As you get more users, the metadata size increases and eventually could exhaust the physical memory space. The only solution here is scaling up all the instances but it’s not infinitely scalable as you already know.

In summary, the example system could have three types of scalability bottlenecks:

• Single writer DB instance

• Service quotas

• Physical limit on every application server

You would argue that your system’s workload doesn’t require such high scalability and these are not critical. Yes, that’s true. However, once you want to accelerate the growth of your business, you could be easily blocked by these bottlenecks and take way longer time to re-architect everything to workaround them. Even worse, you might hit unknown limits whenever you scale more and they always annoy you.

Availability of cloud

How about availability? The example system uses multi-AZ architecture well, so single-AZ failure doesn’t mean 100% outage. It doesn’t look having any SPOF: ALB is distributed service and failures are handled by AWS under SLA, EC2 is a fleet of instances and instance failures are handled by Auto Scaling Group, RDS has auto failover.

Let me provide one extreme case. What would happen if you accidentally scaled Auto Scaling Group to zero instances? This could be done by just a single API call or console operation. Once it happened, this system would have no application server i.e. 100% outage. The similar event could happen on any cloud resources e.g. deleting RDS, updating Security group wrongly, etc.

The next case is already explained in the section of Scalability of cloud. Once RDS writer instance reaches the performance limit, it affects all the requests. If one EC2 instance exhausts all memory by the large metadata, the same issue happens on all EC2 instances. Any scalability related outage could be a full system outage.

The last case is more realistic and you’ve probably experienced already. That is one single bad deployment causing 100% outage. For example, if you automate DB schema migration in your deployment workflow but unfortunately you’ve deployed ALTER TABLE on a huge table requiring an exclusive lock. Until it finishes, all the DB queries using the table don’t work and it would mean 100% outage. Another example is a Black swan3; let’s say you deploy any new code to one instance first to make sure it’s safe to deploy. Unfortunately, a new code contains a bug that issues a super heavy and long running SELECT query only if it is used by just one customer who has some special condition. Once that customer hits the single instance, your DB becomes overwhelmed even though you deployed only one instance.

In summary, the example system could have 100% outage by three cases:

• Accidental modifications of cloud resources

• Scalability bottleneck

• Bad deployment

You might think some of RDS related issues can be solved by using Serverless DB like DynamoDB, but the answer is no. Once you hit WCU/RCU limits, all the workloads are affected. Wrong UpdateTable or modifying IAM policies can be 100% outage. Although DynamoDB is highly distributed, scalable and available service, a single DynamoDB table is logically SPOF.

What’s next?

In this post, I’ve described several cases that the typical “highly scalable and available cloud system“ could be easily broken. These are actually very limited examples and there are millions of cases including unknown cases.

This is the nature of cloud. You can’t avoid “Everything fails all the time“. But, you do want to build scalable and available system in cloud. Cell-based Architecture is the state-of-the-art solution here. Now, you’re interested in Cell-based Architecture, aren’t you?

In the next post, I’m going to explain the basic concept of Cell-based Architecture and how it solves these scalability and availability issues.

Thank you for reading. If you’re interested in this series, please hit the “Like“ button and share this to any social media. I would be motivated more! Also, subscribe this magazine to receive the upcoming posts for this series!

Share

Subscribe now

This series focuses on inspiring majority of the readers and making them interested in Cell-based Architecture. I won’t dive deep into details of Cell-based Architecture because it ends up to depending on each system and I don’t have enough time to provide bunch of use cases and examples of Cell-based Architecture. Instead, I’d like to stimulate your architect mindset by providing the essence of Cell-based Architecture so that you couldn’t stop considering about that when you design a software system next time :)

Outlines

The series consists of 5 short posts:

1. Introduction (this post)

2. Why is Cell-based Architecture needed?

3. What is Cell-based Architecture?

4. When to use Cell-based Architecture?

5. How to implement Cell-based Architecture?

The order matters: First, I’d like to explain the problems that Cell-based Architecture solves because some of people give up to understand it while reading like “What is cell?“ or similar. Without understanding about the reasons why Cell-based Architecture was born, it just looks like a toy world example and boring. After explaining about “Why“, I’ll introduce Cell-based Architecture’s core concepts. Then, I’ll show several use cases of Cell-based Architecture with rationales, and finally I’ll explain a few concrete examples of Cell-based Architecture implementations.1

Why Cell-based Architecture? - Introduction

Although I’ll describe this more in the next post, I’d like to briefly introduce the most important topic in this series: “Why Cell-based Architecture?“

In summary, Cell-based Architecture solves two core problems:

1. Scalability

2. Availability

Well, probably you’re already familiar with them if you’ve built any application in cloud, and notice that these look the key features that cloud providers offer already e.g. Amazon DynamoDB provides infinite scalability and higher availability SLA out of the box. Why do we need to solve them again, then?

I’ll leave this question open here and let you think about it until the next post comes. Let’s think about why you can’t naively rely on them.

What’s next?

In the next post, I’d like to answer the question above and describe about what is the fundamental problems in cloud development today:

Thank you for reading. If you’re interested in this series, please hit the “Like“ button and share this to any social media. I would be motivated more! Also, subscribe this magazine to receive the upcoming posts for this series!

Share

Subscribe now

References

• “Reducing the Scope of Impact with Cell-Based Architecture“ by AWS Well-Architected

  • In this series, I’d like to basically squeeze this document and make it easier to get started. But, if you don’t mind to read this excellent document, I highly recommend to do so because it captures more details than my posts. Probably a good way to follow up my casual posts.

… As a target, I generally recommend aiming to have something that you could deploy into production in less than one hour from the commit of any change.

By just applying this simple principal, your development process is encouraged to optimize for various aspects and gain lots of benefits, especially better quality of software release. Let’s quickly look around the benefits first. We’ll talk about how to guide in the following newsletter.

Measurements of the performance of software development

According to “Modern Software Engineering“, and we totally agree with it, there are two key measurements of the performance of software development and each is associated with two key indicators:

1. Stability

   1. Change Failure Rate

   2. Recovery Failure Time

2. Throughput

   1. Lead Time

   2. Frequency

These are amazingly simple but tangible enough to visualize such a vague measurement i.e. the performance of software development. If your development process marks high scores on just these two key measurements, your performance is high.

For example, the better stability you have, the better quality of software you deploy, meaning less negative values provided by deployments. Then, you don’t have to be scared about deployments too much and you can focus on more creative tasks.

Also, better throughput means adding more positive values provided by deployments. You release more features with faster pipeline, therefore your customers can receive more values than lower throughput processes.

However, both are interconnected, not orthogonal nor trade-off. The better stability you have, the more confident you feel to deploy faster and more frequently. Better throughput allows you to deploy improvements of stability faster e.g. quick rollback.

One-hour deployable affects every measurements

Obviously, the “Lead Time“ indicator is directly connected to the “One-hour deployable“ principal. And as we mentioned above, it affects more than “Lead Time“. To make one-hour deployable happen, you also need better monitoring system after deployments to keep better stability and give more confidence to high frequent deployments because the monitoring system can quickly detect any failure related to the new code and trigger rollback automatically, for example.

Summary

The performance of software development can be measured by two simple measurements: Stability and Throughput. Both are correlated each other and high performance teams have high score on both measurements. “One-hour deployable“ principal is stupidly simple but affects both measurements positively in various ways.

OpsBR Magazine

Ep 1: Continuous Delivery - What does deployable mean?

Welcome to OpsBR Magazine! We’ll publish short newsletters regularly to talk about various topics around operations on software systems and organizations. This is the first newsletter and we’d like to talk about the first topic about Continuous Delivery…

Read more

3 years ago · 2 likes · Ryosuke Iwanaga

However, there is another way to make code changes deployable. Let’s talk about an elegant technique called Feature flags.

Deploy a new code without activation

People typically think that code deployment equals to rolling out a new feature or changing the behavior. But, deployment and enablement don’t have to happen at once. You can deploy the new code without activation by using Feature flags like below:

function main() {
  const featureFlags = fetchFeatureFlags();
  ...
  if (featureFlags.enableCoolFeature) {
    executeCoolFeature(); // The new code
  }
  ...
}

Feature flag is a small configuration available at runtime of your application. It’s typically injected dynamically like the code above so that you can modify the value of feature flag without code deployment. Since the default value of such boolean flag is normally false, this code is deployable by nature (!) because the new code won’t be executed until you change the value of the feature flag.

Feature flags isolate deployment and release

Feature flag brings lots of benefits in terms of operations. Since you can control when you activate the new feature without deployment, your deployment itself is regression resilient. Any deployment won’t introduce new behavior so that it’s ideally identical to the code before the deployment.

This makes Continuous Deployment a.k.a. (another) CD easier and rollback is rarely needed. With a large code base application, this means that your code is most likely able to be deployed to production along with other teams’ changes without any issue. Without feature flags, your change could be rolled back by a bug introduced by the other team’ change or vice versa.

Feature flags allow both teams to release their own new features independently. If either team notice some issues with their own new feature, they can easily rollback their own feature exclusively. This isolation is huge win especially for large organizations.

You can even release the new features gradually. For example, you can enable the feature for 1% of the users first and monitor their behavior to verify whether it’s working correctly. Once confirmed, you can dial up the percentage every day up to 100%, for example.

However, feature flags are not silver bullets

Although feature flags look perfect for CD, you can’t deploy every single change under feature flags. For example, updating dependency libraries is not easy to hide behind feature flags.

The other downside of feature flags is that the number of combinations of multiple feature flags can be exploded. Theoretically, every single combination could behave differently at runtime but it’s much harder to validate all the combinations. Therefore, you might see a weird behavior uncovered by tests because of a rare combination of existing feature flags.

So, you need to carefully select which features should be under feature flags and monitor the usage of feature flags regularly. If a feature flag has been enabled for all users for a long time, it’s safe to delete the feature flag as well as the unused old code.

Lastly, even the new feature is under a feature flag, you must run the same validations steps to ensure you don’t have known regressions like integration tests. The application behavior could change by just loading a new library without calling it e.g. it might load a too large object into memory without any function calls.

Summary

Feature flags are powerful tool to make a new code deployable instantaneously. You can deploy the new code without changing the current behavior at all, then enable it by turning the feature flag on later. This can isolate multiple teams’ feature releases each other so that unnecessary rollbacks rarely happen. However, it’s not silver bullet and you need to carefully use it, monitor its usage and keep running the validations steps like integration tests.

This is the first newsletter and we’d like to talk about the first topic about Continuous Delivery.

Continuous Delivery is the center of operations

Continuous Delivery a.k.a. CD is the concept and practice to make a new code change deployable to production continuously. With CD process i.e. CD pipeline, you are confident to deploy the new changes to production because it is ensured by CD.

The more confident of deployments you are, the better your operations become because the likelihood of failure caused by the deployment should be lower ideally. Without CD pipeline, you don’t know what would happen after deployment.

What does deployable mean?

But, what does deployable mean? If the new code is built into a new Docker image automatically, is it deployable? No. The artifacts must be verified deeply to make it deployable.

Actually, preparing such deployment artifacts is one of Continuous Integration a.k.a. CI process. It is usually triggered by code merge and builds the artifacts as well as runs unit tests to ensure the new code works as expected.

However, that doesn’t automatically mean it’s deployable to production. Unit test can only verify the code without integrations. It is useful to test business logic implementations but typically omits interactions with unmanaged dependencies e.g. other microservices.

Before production deployments, such integrations must be also verified to capture any failures caused by these interactions. Therefore, CD pipeline would have automatic deployment process to pre-production environments like staging and run integrations tests there, for example. Once all the integration tests pass, you have better confidence of the new artifacts when deploying to production. This means the new code is deployable.

Summary

Continuous Delivery a.k.a. CD is the center of operations because it ensures the new code is deployable to production automatically. Continuous Integration a.k.a. CI isn’t sufficient to make the new code deployable because it usually run only unit tests. To make it deployable, CD pipeline would run integrations tests by using the new artifacts built by CI process.

We’ll keep talking about Continuous Delivery in the following newsletters. Subscribe this newsletter from the button below:

Subscribe now