A backup is not a guarantee of recovery. The crucial factor is whether a business-critical IT service can return to a secure, technically functional, and operationally consistent state within a specific timeframe and defined loss limits after a disruption. This requires business leaders, architecture, and operations to pursue the same goals – and to regularly test their achievement. Backups alone are simply not enough!

Why 3-2-1 is necessary

One day (this was in 2023), I stopped receiving emails, and my hand-built website was down. The hosting provider was equally unreachable, neither by email nor by phone. I joked that a meteorite must have struck. But in reality, it was a disaster for the company: they had fallen victim to a cyberattack that encrypted and crippled all their systems. Not only were the virtual machines affected, but also their backups! Recovery took weeks. One customer reported losses in the millions.

One affected user noted: “ Backups alone are useless; they can be infected with malware just as easily .” This insight is part of the golden rule of data backup. The 3-2-1 rule states that it3 copies of the data on 2 different mediaThere must be three copies, one of which must be located in a different location. The term "three copies" is interpreted differently, usually as the total number (i.e., one original plus two physical copies). However, storage expert Curtis Preston writes in his book *Modern Data Protection* (O'Reilly 2021, p. 43), where he essentially elevates the rule to the definition of "backup": " three additional versions of your data ." There should also be two physically different media. The condition "one in a different location" refers both geographically and logically. There should be no direct path from the protected area to the area containing the backups, otherwise the backups (as in my case) will be compromised along with the data. This is also known as the air gap , which must separate the backups in such a way that the attacker cannot bypass it.

Why 3-2-1 isn't enough

If the data is finally backed up immutably according to the golden 3-2-1 rule and additionally on WORM ( Write Once Read Many ) storage, is everything then under control? No – unfortunately, that's still not enough. The question is: Has the recovery process – i.e., bringing the infrastructure back up, restoring the backups, and restarting the services – been tested? Even if the DevOps team answers, " Sure, that's no problem, because we've been doing Infrastructure as Code for ages! ", something is still missing. Has it been tested how the software handles data loss? It may be technically functional, but there's a high risk that the distributed system will be in a functionally inconsistent state. Is it ensured that nothing unexpected will happen? Could, for example, the latest customer orders be lost, and they receive neither goods nor an apology email? Do we even know who has already paid? Or worse: Could the orders be recovered, but the delivery data be lost? Does someone have to run into the warehouse to prevent the goods from being shipped again? This restart can turn into a nightmare.

3-2-1 is not enough: Data inconsistency after recovery

Disruption - Emergency - Crisis

It doesn't even have to be a ransomware attack or a data center fire ( like the one in Korea in 2025 ) that leads to an emergency. A bug in a synchronization service or a simple user error can be the trigger. A seemingly manageable disruption can quickly spread in a complex service landscape and impair or completely halt critical business processes. The disruption has escalated into an emergency. Hopefully, you've already considered what to do in such a situation, because if you have an emergency and no plan, you're in a crisis. This is the core of the definition according to BSI Standard 200-4 Business Continuity Management (BCM) , which addresses how to respond appropriately to a disruption or failure of time-critical business processes. The tiered model of damage events (disruption, emergency, crisis) also helps to more precisely differentiate the overlapping areas of backup/restore and disaster recovery . The key difference lies not so much in the involvement of different infrastructure layers, but rather in the fact that, for example, accidentally deleted data can be recovered within "normal" operations through appropriate restore processes. Disaster recovery, in addition to various other emergency measures, involves restarting operations to a state of emergency so that core business activities can resume as quickly as possible. The return to normal operations, on the other hand, is referred to as restoration . Restart and restoration can be sequential or parallel activities.

3-2-1 is not enough: BSI Model of incident classification

Business-Impact-Analyse

A key tool in Business Continuity Management (BCM) is Business Impact Analysis (BIA), which examines which business processes are time-critical and at what point their failures have unacceptable consequences. The BIA delivers various key performance indicators (KPIs), not only for the processes themselves but also, and subsequently, for the software systems required by these processes (in BCM, these are only broadly considered "IT resources"). These KPIs have well-known names such as Maximum Tolerable Period of Disruption (MTPD), Recovery Time Objective (RTO), and Recovery Point Objective (RPO). They indicate how quickly a system must be restarted (for example, within 4 hours) and how much information or data can be lost (for example, user comments from the previous day or orders that have not yet been confirmed to the customer). Recoverability or recoverability ( depending on whether it is viewed as a quality characteristic or a capability ) can be defined as...

The ability of an IT service or software system to restore a defined operational state in a technically consistent manner after a disruption within defined time and data loss limits (RTO, RPO).

Recoverability

At this point, in contrast to the well-thought-out world of textbooks, there is a break in reality.The Business Architecture Assessment (BIA) is conceptually a top-down approach, meaning it derives requirements as key performance indicators (KPIs) from the business architecture through the enterprise architecture down to the IT architecture. In this ideal scenario, the administrator knows, based on the criticality of the business processes, which combination of full backups and incremental backups, snapshots, and continuously backed-up transaction logs is needed, and whether only discrete backup states or a comprehensive backup strategy is required.Point-in-Time Recoveryare required, and of course what storage times (retention) need to be configured.In practice, however, things often look different. DevOps colleagues will likely ask questions like: " Do I actually need to make backups for Apache Kafka, or can the topics be lost? " Or: " Doesn't the database service provider decide about backups? " The missing link in the chain is then possibly simply the missing requirement in the software architecture. Be honest: was recoverability even formulated as a requirement in your system, for example, as a classic system requirement?

“ The system must be designed and operated in such a way that, after a fault or failure, it can be restored to the defined operating state within the time specified by the RTO value. The restoration point may be no more than the period of time prior to the occurrence of the fault, as specified by the RPO value. ”

User stories are also welcome:

“ As the service owner of a business-critical IT service, I want the system to be able to be restored to the defined operational state within the specified RTO value after a disruption or failure, and for the restored data state to be no more than the specified RPO value prior to the time of the disruption, so that the supported business process can continue within the tolerable limits of interruption and data loss. ”

If no explicit requirement for recoverability is placed on the software, then a successful restart cannot be expected. Because: where there is no requirement, there is no implementation! Once the disaster has occurred, subsequent accusations that " generally accepted rules of technology " were disregarded, or discussions about better risk management, will no longer be of any help.

Only with tests

Specific requirements have another consequence: they necessitate corresponding tests. Application owners can then argue to the responsible decision-makers that sufficient time and resources must be allocated for restore and restart tests. A widespread problem is the lack of a sufficiently realistic test environment. In particular, creating and maintaining test data is usually very time-consuming. And ultimately, it's not just a matter of testing how a system can be restored; the effects of data loss and inconsistencies within the distributed system are difficult to predict. Without meaningful and robust tests, one is unprepared.

After this overview, we see that the topics of recoverability, disaster recovery, backup/restore, and data consistency cannot be addressed on just one level, but represent a comprehensive task for the organization:

The business architecture must provide requirements in the form of MTPD, RTO, and RPO, along with a BIA.
Recoverability must be anchored in the architecture as a requirement and specified with key performance indicators.
Only with the appropriate requirements can one expect the software to be implemented and operated accordingly.
The golden 3-2-1 rule for backups must be implemented.
The restart must be tested using a test system that closely resembles production conditions and is populated with realistic test data. Sufficient resources must be allocated for this purpose.

By the way: my homepage came back. Not just because a backup existed, butbecause I was able to restore them independently - simply via FTPWhat is your plan?