# How to Measure Anything in Cybersecurity Risk ![rw-book-cover](https://images-na.ssl-images-amazon.com/images/I/5191qxHDqEL._SL200_.jpg) ## Metadata - Author: [[Douglas W. Hubbard, Richard Seiersen, Daniel E. Geer, and Stuart McClure]] - Full Title: How to Measure Anything in Cybersecurity Risk - Category: #books ## Highlights ### Foreword - John Foster Dulles: “The measure of success is not whether you have a tough problem to deal with, but whether it is the same problem you had last year.” ([Location 431](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=431)) - Tags: [[favorite]] - The central truth of engineering is that design pays if and only if the problem statement is itself well understood. ([Location 434](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=434)) - The central truth of statistical inference is that all data has bias—the question being whether you can correct for it. ([Location 435](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=435)) - I say “metrics” because metrics are derivatives of measurement. ([Location 437](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=437)) - the scale of our task compared to the scale of our tools demands force multiplication. ([Location 440](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=440)) - Our opponents by and large pick the targets that maximize their return on their investment, which is a polite way of saying that you may not be able to thwart the most singularly determined opponent for whom cost is no object, but you can sure as the world make other targets more attractive than you are. ([Location 453](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=453)) ### Foreword ### Acknowledgments ### About the Authors ### Introduction #### Why This Book, Why Now? #### What Is This Book About? - We will show that we can start at a simple level and then evolve to whatever level is required while avoiding problems inherent to “risk matrices” and “risk scores.” So there is no reason not to adopt better methods immediately. ([Location 551](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=551)) #### What to Expect #### Is This Book for Me? #### We Need More Than Technology #### New Tools for Decision Makers - We need decision makers who consistently make better choices through better analysis. ([Location 577](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=577)) - We also need decision makers who know how to deftly handle uncertainty in the face of looming catastrophe. ([Location 578](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=578)) #### Our Path Forward - we need a better approach to measuring cybersecurity risk and, for that matter, measuring the performance of cybersecurity risk analysis itself. ([Location 584](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=584)) ### PART I Why Cybersecurity Needs Better Measurements for Risk ### Chapter 1 The One Patch Most Needed in Cybersecurity - resources are limited. Therefore, the cybersecurity professional must effectively determine a kind of “return on risk mitigation.” Whether or not such a return is explicitly calculated, we must evaluate whether a given defense strategy is a better use of resources than another. In short, we have to measure and monetize risk and risk reduction. ([Location 615](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=615)) #### The Global Attack Surface - Perhaps the total attack surface that concerns all citizens, consumers, and governments is a kind of “global attack surface”: ([Location 671](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=671)) - This global attack surface is a macro-level phenomenon driven by at least four macro-level causes of growth: increasing users worldwide, variety of users worldwide, growth in discovered and exploited vulnerabilities per person per use, and organizations more networked with each other resulting in “cascade failure” risks. ([Location 673](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=673)) ##### The increasing number of persons on the Internet. - The number of uses per person for online resources. ([Location 680](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=680)) ##### Vulnerabilities increase. ##### The possibility of a major breach “cascade.” - we suspect most large organizations could just be one or two degrees of separation from each other. ([Location 700](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=700)) - Our naïve, and obvious, hypothesis? Attack surface and breach are correlated. If this holds true, then we haven’t seen anything yet. We are heading into a historic growth in attack surface, and hence breach, which will eclipse what has been seen to date. ([Location 703](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=703)) #### The Cyber Threat Response - Hardening shrinks, but does not eliminate, attack surface. ([Location 713](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=713)) - Cybersecurity budgets have grown at about twice the rate of IT budgets overall. ([Location 731](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=731)) - What risks are acceptable is often not documented, and when they are, they are stated in soft, unquantified terms that cannot be used clearly in a calculation to determine if a given expenditure is justified or not. ([Location 736](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=736)) - the vast majority of organizations concerned with cybersecurity will resort to some sort of “scoring” method that ultimately plots risks on a “matrix.” This is true for both very tactical level issues and strategic, aggregated risks. ([Location 745](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=745)) - Does this last phrase, stating “low, medium, or high is sufficient,” need to be taken on faith? Considering the critical nature of the decisions such methods will guide, we argue that it should not. This is a testable hypothesis and it actually has been tested in many different ways. ([Location 778](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=778)) - There is no evidence that the types of scoring and risk matrix methods widely used in cybersecurity improve judgment. On the contrary, there is evidence these methods add noise and error to the judgment process. ([Location 785](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=785)) - There is overwhelming evidence in published research that quantitative, probabilistic methods are effective. ([Location 790](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=790)) - How cybersecurity assesses risk, and how it determines how much it reduces risk, are the basis for determining where cybersecurity needs to prioritize the use of resources. And if this method is broken—or even just leaves room for significant improvement—then that is the highest-priority problem for cybersecurity to tackle! ([Location 793](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=793)) - Tags: [[favorite]] - If risk assessment itself is a weakness, then fixing risk assessment is the most important “patch” a cybersecurity professional can implement. ([Location 796](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=796)) #### A Proposal for Cybersecurity Risk Management ##### It is possible to greatly improve on the existing methods. ##### Cybersecurity can use the same quantitative language of risk analysis used in other problems. ##### Methods exist that have already been measured to be an improvement over expert intuition. ##### These improved methods are entirely feasible. ##### You can improve further on these models with empirical data. - Even the risk analysis methods themselves can be measured and tracked to make continuous improvements. ([Location 815](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=815)) ### Chapter 2 A Measurement Primer for Cybersecurity #### The Concept of Measurement - If we incorrectly think that measurement means meeting some nearly unachievable standard of certainty, then few things will be measurable even in the physical sciences. ([Location 943](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=943)) ##### A Definition of Measurement - measurement is only a probabilistic exercise. ([Location 957](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=957)) - Definition of Measurement Measurement: A quantitatively expressed reduction of uncertainty based on one or more observations. ([Location 961](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=961)) - Tags: [[definition]] - This “uncertainty reduction” point of view is what is critical to business. Major decisions made under a state of uncertainty—such as whether to approve large information technology (IT) projects or new [[security]] controls—can be made better, even if just slightly, by reducing uncertainty. Sometimes even small uncertainty reductions can be worth millions of dollars. ([Location 980](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=980)) ##### A Taxonomy of Measurement Scales - four different scales of measurement: nominal, ordinal, interval, and ratio scales. ([Location 1002](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1002)) - Tags: [[definition]] - If the reader is thinking of Celsius or dollars as a measurement, they are thinking of an interval and ratio scale, respectively. ([Location 1002](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1002)) - A nominal scale expresses a state without saying that one state is twice as much as the other or even, for that matter, more or less than the other—each state scale is just a different state, not a higher or lower state. ([Location 1009](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1009)) - Ordinal scales, on the other hand, denote an order but not by how much. ([Location 1011](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1011)) - Tags: [[definition]] - So most mathematical operations—other than basic logic or set operations—are not applicable to nominal or ordinal scales. ([Location 1013](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1013)) - Still, it is possible for nominal and ordinal scales to be informative even though they vary from more conventional measurement scales ([Location 1014](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1014)) - So the use of ordinal scales like those often found in cybersecurity are not strictly a violation of measurement concepts, but how it is done, what it is applied to, and what is done with these values afterward actually does violate basic principles and can cause a lot of problems. ([Location 1017](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1017)) ##### Bayesian Measurement: A Pragmatic Concept for Decisions - When we talk about measurement as “uncertainty reduction,” we imply that there is some prior state of uncertainty to be reduced. And since this uncertainty can change as a result of observations, we treat uncertainty as a feature of the observer, not necessarily the thing being observed. ([Location 1037](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1037)) - When we conduct a penetration test on a system, we are not changing the state of the application with this inspection; rather, we are changing our uncertainty about the state of the application. ([Location 1040](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1040)) - you usually already have a prior uncertainty—even though you might not explicitly state probabilities. Stating priors even allows us to compute the value of additional information since, of course, the value of additional information is at least partly dependent on your current state of uncertainty before you gather the information. ([Location 1057](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1057)) - We use probability because we lack perfect information, not in spite of it. ([Location 1066](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1066)) - probability is also used to represent our current state of uncertainty about something, no matter how much that uncertainty is. ([Location 1076](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1076)) - If a decision maker or analyst engages in what they believe to be measurement activities, but their estimates and decisions actually get worse or don’t at least improve, then they are not actually reducing their error and are not conducting a measurement according to the stated definition. ([Location 1085](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1085)) #### The Object of Measurement - What we call a “clarification chain” is just a short series of connections that should bring us from thinking of something as an intangible to thinking of it as a tangible. First, we recognize that if X is something that we care about, then X, by definition, must be detectable in some way. ([Location 1114](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1114)) ##### Clarification Chain - If it is detectable, it can be detected as an amount (or range of possible amounts). ([Location 1123](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1123)) - If it matters at all, it is detectable/observable. ([Location 1123](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1123)) - If it can be detected as a range of possible amounts, it can be measured. ([Location 1124](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1124)) - It also helps to state why we want to measure something in order to understand what is really being measured. The purpose of the measurement is often the key to defining what the measurement is really supposed to be. ([Location 1134](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1134)) - Measurements should always support some kind of decision, ([Location 1136](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1136)) - Tags: [[favorite]] - Avoidably vague terms like “threat capability” or “damage to reputation” or “customer confidence” seem immeasurable at first, perhaps, only because what they mean is not well understood. ([Location 1141](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1141)) ##### Definitions for Uncertainty and Risk, and Their Measurements - Uncertainty: The lack of complete certainty, that is, the existence of more than one possibility. The “true” outcome/state/ result/value is not known. ([Location 1154](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1154)) - Tags: [[definition]] - Measurement of Uncertainty: A set of probabilities assigned to a set of possibilities. For example: “There is a 20% chance we will have a data breach sometime in the next five years.” ([Location 1156](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1156)) - Risk: A state of uncertainty where some of the possibilities involve a loss, catastrophe, or other undesirable outcome. ([Location 1158](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1158)) - Measurement of Risk: A set of possibilities, each with quantified probabilities and quantified losses. For example: “We believe there is a 10% chance that a data breach will result in a legal liability exceeding $10 million.” ([Location 1160](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1160)) - Tags: [[favorite]] #### The Methods of Measurement - Cybersecurity is not some exceptional area outside the domain of statistics but rather exactly the kind of problem statistics was made for. ([Location 1186](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1186)) - Tags: [[favorite]] ##### Statistical Significance: What’s the Significance? - Statistical significance does not mean you learned something and the lack of statistical significance does not mean you learned nothing. ([Location 1203](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1203)) - What you want to know is whether you have less uncertainty after considering some source of data and whether that reduction in uncertainty warrants some change in actions. ([Location 1207](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1207)) ##### Small Samples Tell You More Than You Think - Rule of Five There is a 93.75% chance that the median of a population is between the smallest and largest values in any random sample of five from that population. ([Location 1245](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1245)) - No matter how complex or “unique” your measurement problem seems, assume it has been measured before. ([Location 1264](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1264)) - You probably need less data than your intuition tells you—this is actually even more the case when you have a lot of uncertainty now. ([Location 1266](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1266)) - Things that are thought to be intangible tend to be so uncertain that even the most basic measurement methods are likely to reduce some uncertainty. Cybersecurity is now such a critical endeavor that even small reductions in uncertainty can be extremely valuable. ([Location 1269](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1269)) ### Chapter 3 Model Now!: An Introduction to Practical Quantitative Methods for Cybersecurity #### A Simple One-for-One Substitution - No data is required other than the information that cybersecurity analysts may already use to inform their judgments with a risk matrix. ([Location 1310](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1310)) ##### Table 3.1 Simple Substitution of Quantitative vs. the Risk Matrix #### The Expert as the Instrument - In the spirit of the one-for-one substitution we will start with, we will use the same source for an estimate as the current risk matrix—the cybersecurity expert. ([Location 1339](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1339)) - Just as experts already assess likelihood and impact on the conventional risk matrix, they can simply assess these values using meaningful quantities. ([Location 1340](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1340)) - We just need to set up a basic structure with the following steps. ... Define a list of risks. ... Define a specific period of time over which that risk event could materialize ... For each risk, subjectively assign a probability (0% to 100%) that the stated event will occur in the specified time ... For each risk, subjectively assign a range for a monetary loss if such an event occurs as a “90% confidence interval” (CI). ... Get the estimates from multiple experts if possible, but don’t have a joint meeting and attempt to reach consensus. ([Location 1342](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1342)) - Some analysts who had no problem saying likelihood was a “4” on a scale of 1 to 5 or a “medium” on a verbal scale will argue that there are requirements for quantitative probabilities that make quantification infeasible. ([Location 1359](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1359)) - there are problems in statistics that can only be solved by using a probabilistically expressed prior state of uncertainty. And these are actually the very situations most relevant to decision making in any field, including cybersecurity. ([Location 1363](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1363)) - most experts can be trained to subjectively assess probabilities and that this skill is objectively measurable (as ironic as that sounds). ([Location 1369](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1369)) - we cannot assume that whatever errors you may be introducing to the decision by using quantitative probabilities without being trained are being avoided by using qualitative methods. ([Location 1373](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1373)) #### Doing “Uncertainty Math” - So how do we add, subtract, multiply, and divide in a spreadsheet when we have no exact values, only ranges? Fortunately, there is a practical, proven solution, and it can be performed on any modern personal computer—the “Monte Carlo” simulation method. ([Location 1384](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1384)) - Monte Carlo simulations have been used to simulate risk models on power plants, supply chains, insurance, project risks, financial risks, and, yes, cybersecurity. ([Location 1395](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1395)) ##### An Introduction to Generating Random Events and Impacts in Excel - In Excel, we can write this as For example, if the event probability is .15, then this equation would produce a “1” (meaning the event occurred) 15% of the time. ([Location 1417](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1417)) - For the impact, we need to generate not just a “0” or “1,” but a continuum of values. We can do this using one of Excel’s “inverse probability functions.” ([Location 1423](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1423)) - The lognormal distribution can’t generate a zero or negative amount, but it has a tail to the right that allows for the possibility of extremely large outcomes. This is why it is often a realistic representation of the probability of various amounts of loss. ([Location 1441](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1441)) - A normal distribution wide enough to capture some extreme events could also produce illogical negative results on the other end of the scale (you can’t have a negative number of records breached or a negative downtime for a system). This is why the lognormal is also used to model a variety of quantities that can’t be negative but could possibly (but rarely) be very large. ([Location 1442](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1442)) - Here is one note of caution in using lognormal distributions. The extreme losses for a given 90% CI may be unrealistic when the upper bound is many times the lower bound. This can happen when the expert estimating the value makes the mistake of believing the upper bound represents a worst case extreme, which it is not. ([Location 1466](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1466)) - The upper bound of a 90% confidence interval allows for a 5% chance the value is higher. ([Location 1468](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1468)) ##### Adding Up the Risks - If you could somehow record every result in a few thousand trials, then you have the output of a Monte Carlo simulation. The easiest way to do this in Excel is with a “data table” in the “What-If Analysis” tools. ([Location 1487](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1487)) - This simulation-to-simulation variation would shrink if we ran 100,000 trials or a million. You might be surprised at how little time this takes in Excel on a decently fast machine. We’ve run 100,000 trials in a few seconds using Excel, which doesn’t sound like a major constraint. ([Location 1515](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1515)) #### Visualizing Risk - This amount of information cannot be plotted with a simple point on a two-dimensional chart. Instead, we can represent this with a chart called a “loss exceedance curve” or LEC. ([Location 1532](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1532)) - In these other fields, it is also variously referred to as a “probability of exceedance” or even “complementary cumulative probability function.” ([Location 1536](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1536)) - An LEC can show how a range of losses is possible (not just a point value) and that larger losses are less likely than smaller ones. ([Location 1547](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1547)) - We can also create another variation of this chart by adding a couple more curves. Figure 3.3 shows three curves: inherent risk, residual risk, and risk tolerance. ([Location 1558](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1558)) - Inherent risk might be defined instead as including only minimal required controls. Those are controls where it would be considered negligent to exclude them so there really is no dilemma about whether to include them. ([Location 1562](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1562)) - The part of the inherent risk curve that is over the risk tolerance curve is said to “violate” or “break” the risk tolerance. The residual risk curve, on the other hand, is on or underneath the risk tolerance curve at all points. If this is the case, we say that the risk tolerance curve “stochastically dominates” the residual risk curve. This simply means that the residual risks are acceptable. ([Location 1575](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1575)) ##### Generating the Inherent and Residual Loss Exceedance Curves - The residual risk curve, for example, is just the same procedure but based on your estimated probabilities and impacts (which would presumably be smaller) after your proposed additional controls are implemented. ([Location 1601](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1601)) - One disadvantage of the LEC chart is that if multiple LECs are shown, it can get very busy looking. ... However, this complexity was easily managed just by having separate charts for different categories. Also, since the LECs can always be combined in a mathematically proper way, we can have aggregate LEC charts where each curve on that chart could be decomposed into multiple curves shown on a separate, detailed chart for that curve. ([Location 1603](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1603)) - The typical low/medium/high approach lacks the specificity to say that “seven lows and two mediums are riskier than one high” or “nine lows add up to one medium,” but this can be done with LECs. ([Location 1610](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1610)) ##### Where Does the Risk Tolerance Curve Come from? - Ideally, the risk tolerance curve is gathered in a meeting with a level of management that is in a position to state, as a matter of policy, how much risk the organization is willing to accept. ([Location 1624](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1624)) - We also need to identify which risk tolerance curve we are capturing (e.g., the per-year risk for an individual system, the per-decade risk for the entire enterprise, etc.). ([Location 1628](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1628)) - Also, some may prefer to consider such a curve only for a given cybersecurity budget—as in, “That risk is acceptable depending on what it costs to avoid it.” ... But most seem willing to consider the idea that there is still a maximum acceptable risk, and this is what we are attempting to capture. ([Location 1650](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1650)) - If your concern is that upper management won’t understand this, we can say we have not observed this—even when we’ve been told that management wouldn’t understand it. In fact, upper management seems to understand having to determine which risks are acceptable at least as well as anyone in cybersecurity. ([Location 1656](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1656)) #### Supporting the Decision: A Return on Mitigation - If you have observed (as the authors have) someone asking a question like, “If we spent another million dollars, can we move this risk from a red to a yellow?” then you may have felt the dissatisfaction from this approach. ([Location 1666](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1666)) - What the CISO needs is a “Return on Control” calculation. That is the monetized value of the reduction in expected losses divided by the cost of the control. ([Location 1670](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1670)) - So “expected loss” is the average of the Monte Carlo simulation losses due to some cause. ([Location 1674](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1674)) - If the control is a one-time investment that could provide benefits over a longer period of time, then follow the financial conventions in your firm for capital investments. ([Location 1681](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1681)) #### Where to Go from Here - In the authors’ opinion, FAIR, as another Monte Carlo–based solution with its own variation on how to decompose risks into further components, could be a step in the right direction for your firm. ([Location 1705](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1705)) - These results can be properly “added up” to create aggregate risks for whole sets of systems, business units, or companies. ([Location 1721](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1721)) ### Chapter 4 The Single Most Important Measurement in Cybersecurity - We propose that the single most important measurement in cybersecurity risk assessment, or any other risk assessment, is to measure how well the risk assessment methods themselves work. ([Location 1749](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1749)) - Regulators and standards organizations must make measured performance of methods the key feature of what “compliance” means. If complying with standards and regulations does not actually improve risk management, then those standards and regulations must change. ([Location 1757](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1757)) - We assert that if firms are using cybersecurity risk-analysis methods that cannot show a measurable improvement or, even worse, if they make risk assessment worse, then that is the single biggest risk in cybersecurity, and improving risk assessment will be the single most important risk management priority. ([Location 1761](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1761)) #### The Analysis Placebo: Why We Can’t Trust Opinion Alone - these are just a few of many similar studies showing that we can engage in training, information gathering, and collaboration that improves confidence but not actual performance. ([Location 1816](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1816)) - Our arguments will be based on the published research from large experiments. Any mention of anecdotes or quotes from “thought leaders” will only be used to illustrate a point, never to prove it. ([Location 1828](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1828)) #### How You Have More Data Than You Think - A more feasible answer for an initial measurement would be to experiment with larger populations but with existing research at the component level. ([Location 1862](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1862)) - If the individual components of a method are shown to be an improvement, then a method based entirely on these elements is much more likely to be effective than a method for which the components have no such evidence or, worse yet, have been shown to be flawed. ([Location 1868](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1868)) #### When Algorithms Beat Experts - This research generated one of the most consistently replicated and impactful findings of psychology: that even relatively naïve statistical models seem to outperform human experts in a surprising variety of estimation and forecasting problems. ([Location 1880](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1880)) ##### Some Research Comparing Experts and Algorithms - It is impossible to find any domain in which humans clearly outperformed crude extrapolation algorithms, less still sophisticated statistical ones. ([Location 1928](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1928)) ##### Why Does This Happen? - Very few experts actually measure their performance over time, and they tend to summarize their memories with selected anecdotes. The expert then makes rough inferences from this selective memory, and according to the research published by Dawes, this can lead to an “illusion of learning.” That is, experts can interpret experience as evidence of performance. They assume that years of experience should result in improved performance so they assume that it does. But it turns out that we cannot take learning for granted no matter how many years of experience are gained. ([Location 1934](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1934)) - So when an expert says that, based on some experience and data, one threat is a bigger risk than another, they are doing a kind of “mental math” whether intentional or not. We aren’t saying they are literally trying to add numbers in their heads; rather, they are following an instinct for something that in many cases really could be computed. ([Location 1947](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1947)) - So even those who know the math default to their intuition and their intuition is wrong. ([Location 1957](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1957)) ##### So What? Does This Apply to Cybersecurity? - They point out three necessary conditions for experience to result in learning. First, there must be consistent feedback. ... Second, the feedback must be relatively immediate. ... Third, the feedback should be unambiguous. ([Location 1965](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1965)) - Unless we get regular, immediate, and unambiguous feedback, we are likely to have selective memory and interpret our experiences in the most flattering way. ([Location 1971](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1971)) - The performance of experts in the research mentioned so far relates only to the estimation of quantities based on subjective inferences of recalled experience. The problem is that experts often seem to conflate the knowledge of a vast set of details in their field with their skill at forecasting uncertain future events. ([Location 1983](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1983)) #### Tools for Improving the Human Component - The expert is a component of risk analysis we cannot remove but we can improve. ([Location 1992](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1992)) - The expert must help define the problem in the first place. He or she must assess situations where the data is ambiguous or where conditions do not fit neatly into existing statistical data. The expert also must propose the solutions that must be tested. ([Location 1993](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1993)) - Our goal is actually to elevate the expert. We want to treat the cybersecurity expert as part of the risk assessment system. ([Location 1996](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=1996)) ##### The Subjective Probability Component - Without training or other controls, almost all of us would assign probabilities that deviate significantly from observed outcomes (e.g., of all the times we say we are 90% confident, the predicted outcome happens much less frequently than 90%). ([Location 2026](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2026)) - There are methods, including training, that greatly improve the ability of experts to estimate subjective probabilities (e.g., when they say they are 90% confident, they turn out to be right about 90% of the time). ([Location 2028](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2028)) - Several studies over the last several decades confirm that overconfidence is a pervasive characteristic of nearly all of us. ([Location 2042](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2042)) - If we say that an event is 25% likely to occur by the end of next year, whether it happens or not is not proof the probability was unrealistic. ([Location 2058](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2058)) - In short, researchers discovered that assessing uncertainty is a general skill that can be taught with a measurable improvement. ([Location 2070](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2070)) ##### The Expert Consistency Component - We don’t have to wait for predicted events to occur in order to evaluate the consistency of that expert. ([Location 2090](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2090)) - Stability: an expert’s agreement with their own previous judgment of the identical situation (same expert, same data, different time) ([Location 2095](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2095)) - Tags: [[definition]] - Consensus: an expert’s agreement with other experts (same data, different experts) ([Location 2097](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2097)) - Why shouldn’t random, irrelevant factors like anchoring also affect the judgment of cybersecurity experts? ([Location 2114](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2114)) - Consistency is partly a measure of how diligently the expert is considering each scenario. ... Still, we see that inconsistency accounts for at least 21% of discrimination. That is a significant portion of the expert’s judgment reflecting nothing more than personal inconsistency. ([Location 2152](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2152)) - We can statistically “smooth” the inconsistencies of experts using mathematical methods that reduce estimation error of experts. ([Location 2164](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2164)) ##### The Collaboration Component - that the random stability inconsistencies of individuals can be reduced by simply averaging several individuals together. ([Location 2177](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2177)) - It is interesting to note, however, that cybersecurity experts at a particular organization provided responses that were well correlated with their peers at the same organization. ([Location 2189](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2189)) - They agree with each other to some degree and, as the previous research shows, they can predict outcomes better if we can average several experts together. ([Location 2193](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2193)) ##### The Decomposition Component - Is it possible for experts to build models, using only their current knowledge, that outperform how they would have done without the quantitative models? The research says yes. ([Location 2198](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2198)) - They found that for the most uncertain variables, a simple decomposition—none of which was more than five variables—reduced error by a factor of as much as 10 or even 100. ([Location 2210](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2210)) - Doing the math explicitly, even if the inputs themselves were subjective estimates, removes a source of error. ([Location 2213](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2213)) - If we want to estimate the monetary impact of a denial of service attack on a given system, we can estimate the duration, the number of people affected, and the cost per unit of time per person affected. Once we have these estimates, however, we shouldn’t then just estimate the product of these values—we should compute the product. ([Location 2214](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2214)) - It is possible to “overdecompose” a problem. ... If, however, we do not have less uncertainty about the variables we decompose the problem into, then we may not be gaining ground. ([Location 2224](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2224)) #### Summary and Next Steps - “In my experience . . .” is generally the start of a sentence that should be considered with caution, especially when applied to evaluating the expert themselves. ([Location 2234](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2234)) - Wherever possible, explicit, quantitative models based on objective historical data are preferred. ([Location 2239](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2239)) - Where we need to estimate probabilities and other quantities, experts can be trained to provide subjective probabilities that can be compared to observed reality. ([Location 2241](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2241)) - The inconsistency of experts can be moderated with mathematical and collaborative methods to get an improvement in estimates. ([Location 2243](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2243)) - Decomposition improves estimates, especially when faced with very high uncertainty. ([Location 2245](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2245)) ### Chapter 5 Risk Matrices, Lie Factors, Misconceptions, and Other Obstacles to Measuring Risk #### Scanning the Landscape: A Survey of Cybersecurity Professionals #### What Color Is Your Risk? The Ubiquitous—and Risky—Risk Matrix - several studies show that the types of scales used in these risk matrices can make the judgment of an expert worse by introducing sources of error that did not exist in the experts’ intuition alone. ([Location 2460](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2460)) ##### The Psychology of Scales and the Illusion of Communication - even when participants were told exactly what the terms meant, they interpreted the terms in the context of the statement they were presented in. ([Location 2510](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2510)) - Anecdotally, he has observed conversations about risks with clients where something was judged “highly likely” in part because of the impact it would have. ([Location 2541](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2541)) - “illusion of communication.” Individuals may believe they are communicating risks when they have very different understandings of what is being said. ([Location 2552](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2552)) - even when specific numbers are presented for probabilities, the listener or the presenter may conflate their own risk tolerance with the assessment of probability, or they may be assuming that the probability is for an event over a longer time period than someone else is assuming it to be. ([Location 2554](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2554)) - Professor Craig Fox of UCLA conducted studies showing that arbitrary features of how scales are partitioned have unexpected effects on responses, regardless of how precisely individual values are defined. ([Location 2561](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2561)) - We need to consider the psychology of how we assess risks and how we use these tools. ([Location 2572](https://readwise.io/to_kindle?action=open&asin=B01J4XYM16&location=2572)) ##### How the Risk Matrix Doesn’t Add Up