Dedications |
|
v | |
The Purpose of This Book |
|
xv | |
Acknowledgments |
|
xvii | |
About the Authors |
|
xix | |
|
|
1 | (16) |
|
Production Systems in the Real World |
|
|
1 | (6) |
|
Case 1---The Case of the Puzzlingly Poor Performance |
|
|
2 | (3) |
|
Case 2---The Case of the Disappearing Database |
|
|
5 | (2) |
|
Why Should I Read This Book? |
|
|
7 | (1) |
|
The Non-Functional Systems Challenge |
|
|
8 | (1) |
|
What Is Covered by Non-Functional Testing |
|
|
9 | (1) |
|
Planning for the Unexpected |
|
|
10 | (1) |
|
Patterns for Operability in Application Design |
|
|
11 | (3) |
|
Ensuring Data and Transaction Integrity |
|
|
11 | (1) |
|
Capturing and Reporting Exception Conditions in a Consistent Fashion |
|
|
11 | (3) |
|
Automated Recovery from Exception Conditions |
|
|
14 | (1) |
|
Application Availability and Health |
|
|
14 | (1) |
|
|
14 | (3) |
|
Planning and Project Initiation |
|
|
17 | (24) |
|
The Business Case for Non-Functional Testing |
|
|
17 | (5) |
|
|
17 | (2) |
|
How Far Should the System Be Tested? |
|
|
19 | (1) |
|
Justifying the Investment |
|
|
20 | (1) |
|
|
21 | (1) |
|
|
22 | (11) |
|
Determining the Scope of Non-Functional Testing |
|
|
22 | (4) |
|
Estimating Effort and Resource |
|
|
26 | (3) |
|
Estimating the Delivery Timeline |
|
|
29 | (4) |
|
Test and Resource Planning |
|
|
33 | (6) |
|
Test Types and Base Requirements |
|
|
33 | (3) |
|
|
36 | (1) |
|
|
37 | (2) |
|
|
39 | (1) |
|
|
39 | (1) |
|
|
40 | (1) |
|
Non-Functional Requirements |
|
|
41 | (28) |
|
What Are Non-Functional Requirements? |
|
|
43 | (1) |
|
Do I Need Non-Functional Requirements? |
|
|
43 | (1) |
|
Roles and Responsibilities |
|
|
44 | (1) |
|
|
45 | (1) |
|
Establishing a Business Usage Model |
|
|
46 | (10) |
|
Quantifying Human and Machine Inputs |
|
|
46 | (8) |
|
Expressing Load Scenarios |
|
|
54 | (2) |
|
Non-Functional Requirements |
|
|
56 | (11) |
|
An Important Clarification |
|
|
56 | (2) |
|
|
58 | (4) |
|
|
62 | (2) |
|
Availability Requirements |
|
|
64 | (1) |
|
|
65 | (2) |
|
|
67 | (2) |
|
Designing for Operability |
|
|
69 | (26) |
|
|
70 | (1) |
|
|
71 | (4) |
|
Retry for Fault Tolerance |
|
|
71 | (3) |
|
|
74 | (1) |
|
|
75 | (5) |
|
|
78 | (2) |
|
The Characteristics of a Robust System |
|
|
80 | (11) |
|
|
80 | (1) |
|
|
81 | (2) |
|
Transparency: Visibility into System State |
|
|
83 | (1) |
|
Traceability and Reconciliation |
|
|
84 | (2) |
|
|
86 | (1) |
|
|
87 | (4) |
|
|
91 | (1) |
|
|
91 | (1) |
|
|
91 | (1) |
|
|
92 | (2) |
|
|
94 | (1) |
|
Designing for Performance |
|
|
95 | (44) |
|
|
95 | (6) |
|
|
95 | (6) |
|
|
101 | (1) |
|
|
101 | (1) |
|
|
102 | (10) |
|
|
102 | (1) |
|
|
102 | (1) |
|
|
103 | (4) |
|
Synchronous versus Asynchronous Execution |
|
|
107 | (2) |
|
|
109 | (3) |
|
|
112 | (7) |
|
|
114 | (1) |
|
|
114 | (3) |
|
|
117 | (1) |
|
User Session Memory Consumption |
|
|
118 | (1) |
|
|
119 | (1) |
|
|
120 | (6) |
|
|
120 | (3) |
|
|
123 | (2) |
|
|
125 | (1) |
|
|
126 | (8) |
|
|
127 | (2) |
|
|
129 | (1) |
|
|
129 | (3) |
|
|
132 | (2) |
|
|
134 | (2) |
|
|
134 | (2) |
|
|
136 | (3) |
|
|
139 | (20) |
|
|
140 | (5) |
|
|
140 | (2) |
|
|
142 | (3) |
|
|
145 | (1) |
|
|
145 | (4) |
|
|
146 | (1) |
|
|
147 | (2) |
|
Additional Testing Apparatus |
|
|
149 | (1) |
|
|
150 | (1) |
|
|
150 | (1) |
|
|
151 | (3) |
|
|
151 | (2) |
|
|
153 | (1) |
|
|
154 | (1) |
|
|
154 | (3) |
|
|
157 | (2) |
|
Test Preparation and Execution |
|
|
159 | (36) |
|
|
159 | (1) |
|
|
160 | (11) |
|
Validating the Test Environment |
|
|
164 | (1) |
|
|
164 | (3) |
|
|
167 | (1) |
|
|
167 | (4) |
|
|
171 | (10) |
|
|
172 | (1) |
|
|
173 | (3) |
|
Reporting Performance Results |
|
|
176 | (1) |
|
Performance Regression: Baselining |
|
|
177 | (4) |
|
|
181 | (1) |
|
|
181 | (7) |
|
Boundary Condition Testing |
|
|
182 | (1) |
|
|
183 | (3) |
|
|
186 | (2) |
|
|
188 | (4) |
|
|
192 | (2) |
|
|
193 | (1) |
|
|
193 | (1) |
|
|
194 | (1) |
|
|
195 | (12) |
|
Procedure Characteristics |
|
|
196 | (1) |
|
|
197 | (1) |
|
|
197 | (1) |
|
|
198 | (1) |
|
|
198 | (2) |
|
|
198 | (1) |
|
The Phased Rollout Strategy |
|
|
199 | (1) |
|
|
199 | (1) |
|
|
200 | (1) |
|
Case Study: Online Banking |
|
|
200 | (2) |
|
Case Study: The Banking Front Office |
|
|
202 | (2) |
|
|
204 | (1) |
|
|
204 | (1) |
|
|
204 | (1) |
|
|
204 | (1) |
|
|
205 | (2) |
|
Resisting Pressure from the Functional Requirements Stream |
|
|
207 | (18) |
|
|
208 | (1) |
|
Pressures from the Functional Requirements Stream |
|
|
209 | (3) |
|
|
212 | (1) |
|
|
212 | (1) |
|
|
213 | (1) |
|
|
213 | (1) |
|
|
213 | (1) |
|
|
213 | (1) |
|
Setting the Stage for Success |
|
|
214 | (9) |
|
|
215 | (1) |
|
Roles and Responsibilities |
|
|
216 | (1) |
|
Raw Resources Required by the Non-Functional Requirements Stream |
|
|
216 | (5) |
|
|
221 | (1) |
|
|
222 | (1) |
|
|
222 | (1) |
|
|
223 | (1) |
|
|
223 | (2) |
|
Operations Trending and Monitoring |
|
|
225 | (32) |
|
|
225 | (16) |
|
Attributes of Effective Monitoring |
|
|
227 | (1) |
|
|
228 | (2) |
|
Infrastructure Monitoring |
|
|
230 | (3) |
|
|
233 | (5) |
|
|
238 | (1) |
|
|
239 | (2) |
|
|
241 | (2) |
|
|
241 | (1) |
|
|
241 | (2) |
|
|
243 | (2) |
|
|
244 | (1) |
|
|
245 | (1) |
|
|
245 | (11) |
|
|
245 | (3) |
|
|
248 | (1) |
|
Case Study: Online Dating |
|
|
248 | (7) |
|
|
255 | (1) |
|
Completing a Capacity Plan |
|
|
255 | (1) |
|
|
256 | (1) |
|
Troubleshooting and Crisis Management |
|
|
257 | (20) |
|
|
257 | (1) |
|
|
258 | (1) |
|
Troubleshooting Strategies |
|
|
259 | (11) |
|
Understanding Changes in the Environment |
|
|
259 | (2) |
|
Gathering All Possible Inputs |
|
|
261 | (2) |
|
Approach Based on Type of Failure |
|
|
263 | (2) |
|
Predicting Related Failures |
|
|
265 | (3) |
|
|
268 | (1) |
|
|
268 | (1) |
|
|
269 | (1) |
|
Working Around the Problem |
|
|
269 | (1) |
|
|
270 | (2) |
|
Fix versus Mitigation versus Tolerance |
|
|
270 | (1) |
|
Assessing Level of Testing |
|
|
271 | (1) |
|
|
272 | (3) |
|
|
272 | (1) |
|
|
272 | (3) |
|
|
275 | (2) |
|
Common Impediments to Good Design |
|
|
277 | (18) |
|
|
277 | (2) |
|
What Is the Definition of Good Design? |
|
|
279 | (7) |
|
What Are the Objectives of Design Activities? |
|
|
279 | (2) |
|
|
281 | (5) |
|
|
286 | (1) |
|
Contributors to Bad Design |
|
|
287 | (1) |
|
Common Impediments to Good Design |
|
|
287 | (5) |
|
Confusing Architecture with Design |
|
|
288 | (1) |
|
Insufficient Time/Tight Timeframes |
|
|
288 | (1) |
|
Missing Design Skills on the Project Team |
|
|
288 | (1) |
|
|
288 | (1) |
|
Personal Design Preferences |
|
|
289 | (1) |
|
|
289 | (1) |
|
Constantly Changing Technology |
|
|
289 | (1) |
|
|
290 | (1) |
|
|
290 | (1) |
|
|
290 | (1) |
|
|
290 | (1) |
|
|
291 | (1) |
|
Constantly Changing Requirements |
|
|
291 | (1) |
|
Bad Decisions/Incorrect Decisions |
|
|
291 | (1) |
|
|
291 | (1) |
|
|
291 | (1) |
|
|
291 | (1) |
|
|
292 | (1) |
|
|
292 | (1) |
|
Lack of Financial Resources |
|
|
292 | (1) |
|
|
292 | (1) |
|
|
293 | (2) |
References |
|
295 | (2) |
Index |
|
297 | |