Performance testing guidance for web applications : patterns & practices

Performance Testing Guidance for Web Applications provides an end-to-end approach for implementing performance testing. Whether you are new to performance testing or looking for ways to improve your current performance-testing approach, you will gain insights that you can tailor to your specific scenarios. Cover Page......Page 1 Title......Page 2 Copyrights Page......Page 3 Foreword By Alberto Savoia......Page 4 Foreword By Rico Mariani......Page 6 Scope of This Guide......Page 7 Features of This Guide......Page 8 Parts......Page 9 Approach Used in This Guide......Page 10 External Contributors and Reviewers......Page 12 Tell Us About Your Success......Page 13 Part I: Introduction to Performance Testing......Page 14 How to Use This Chapter......Page 15 Core Activities of Performance Testing......Page 16 Why Do Performance Testing?......Page 17 Project Context......Page 18 Tuning Process Overview......Page 20 Baselines......Page 21 Terminology......Page 23 Summary......Page 27 Performance Testing......Page 28 Key Types of Performance Testing......Page 29 Summary Matrix of Benefits by Key Performance Test Types......Page 30 Additional Concepts / Terms......Page 33 Summary......Page 34 Summary Matrix of Risks Addressed by Performance Testing Types......Page 35 Speed-Related Risks......Page 37 Scalability-Related Risks......Page 38 Stability-Related Risks......Page 39 Summary......Page 40 Part II: Exemplar Performance Testing Approaches......Page 41 Overview of Activities......Page 42 Summary Table of Core Performance-Testing Activities......Page 43 Core Performance-Testing Activities Walkthrough......Page 45 Activity 1. Identify the Test Environment......Page 46 Activity 2. Identify Performance Acceptance Criteria......Page 47 Activity 3. Plan and Design Tests......Page 48 Considerations......Page 49 Activity 5. Implement the Test Design......Page 50 Activity 6. Execute the Test......Page 51 Considerations......Page 52 Activity 7. Analyze Results, Report, and Retest......Page 53 Summary......Page 55 How to Use This Chapter......Page 56 Iterative Performance Testing Activities......Page 57 Activity 1. Understand the Project Vision and Context......Page 59 Activity 2. Identify Reasons for Testing Performance......Page 60 Activity 3. Identify the Value Performance Testing Adds to the Project......Page 61 Activity 4. Configure the Test Environment......Page 62 Activity 5. Identify and Coordinate Tasks......Page 63 Checklist......Page 64 Checklist......Page 65 Checklist......Page 66 Checklist......Page 67 Summary......Page 68 How to Use This Chapter......Page 69 Agile Performance-Testing Activities......Page 70 Relationship to Core Performance-Testing Activities......Page 72 Understand the System......Page 73 Understand the Performance Build Schedule......Page 74 Success Criteria......Page 75 Discussion Points......Page 76 Activity 5. Identify and Coordinate Tasks......Page 77 Activity 6. Execute Task(s)......Page 78 Activity 8. Revisit Activities 1-3 and Consider Performance Acceptance Criteria......Page 79 Additional Considerations......Page 80 Summary......Page 81 How to Use This Chapter......Page 82 CMMI Performance-Testing Activities......Page 83 Relationship to Core Performance-Testing Activities......Page 85 CMMI Performance Testing Activity Flow......Page 86 Determine the Process......Page 87 Understand the System......Page 88 Performance Goals......Page 89 Map Work Items to Project Plan......Page 90 Prioritize Work Items by Phase......Page 91 Add Detail to the Plan......Page 92 Determine Navigation Paths for Key Scenarios......Page 93 Determine the Relative Distribution of Scenarios......Page 94 Identify Metrics to Be Captured During Test Execution......Page 95 Activity 6. Configure the Test Environment......Page 96 Activity 9. Report Results and Archive Data......Page 97 Activity 12. Prepare the Final Report......Page 98 Summary......Page 99 Part III: Identify the Test Environment......Page 100 Approach for Evaluating the System......Page 101 Considerations......Page 102 Considerations......Page 103 Logical Architecture......Page 104 Physical Architecture......Page 106 Considerations......Page 107 Summary......Page 108 Part IV: Identify Performance Acceptance Criteria......Page 109 How to Use This Chapter......Page 110 Terminology......Page 111 Review the Project Plan......Page 112 Capture or Estimate Resource Usage Targets and Thresholds......Page 113 Capture or Estimate Resource Budgets......Page 114 Communicate Results......Page 115 Performance Objectives......Page 116 Performance Objectives......Page 117 Questions......Page 118 Questions......Page 119 Summary......Page 120 How to Use This Chapter......Page 121 Determine Application Functionality and Usage......Page 122 Review Project Documentation and Related Contracts......Page 123 Interview Stakeholders Who Will Influence the “Go Live” Decision......Page 124 Separate Requirements from Goals......Page 125 Quantify Captured Performance Goals......Page 126 Quantify Captured Performance Requirements......Page 127 Summary......Page 128 How to Use This Chapter......Page 129 Terminology......Page 130 Investigate End-User Requirements......Page 131 Collect Business Requirements......Page 132 Research Standards, Compliance, and Contracts......Page 133 Compare and Consolidate Performance Characteristics......Page 134 Summary......Page 135 Part V: Plan and Design Tests......Page 137 How to Use This Chapter......Page 138 Identify the Objectives......Page 139 Determine Key Usage Scenarios......Page 140 Considerations......Page 141 Considerations......Page 142 Determine Individual User Data and Variances......Page 143 Determine the Relative Distribution of Scenarios......Page 144 Quantifying the Volume of Application Usage: Theory......Page 147 Integrating Model Variance......Page 149 Considerations......Page 150 Summary......Page 151 User Delays......Page 152 Consequences of Improperly Modeling User Delays......Page 153 Step 1 – Determine User Delays......Page 154 Step 2 – Apply Delay Ranges......Page 155 Normal Distribution......Page 156 Negative Exponential Distribution......Page 157 Double Hump Normal Distribution......Page 158 Considerations......Page 159 Considerations......Page 160 Summary......Page 161 Part VI: Execute Tests......Page 162 Validate the Test Environment......Page 163 Additional Considerations......Page 164 Validate Tests......Page 165 Dynamic Data......Page 166 Additional Considerations......Page 167 Run Tests......Page 168 Additional Considerations......Page 169 Baselines......Page 170 Benchmarking......Page 171 Summary......Page 172 Part VII: Analyze Results and Report......Page 173 Exemplar Data Sets......Page 174 Data Set A......Page 175 Data Set C......Page 176 Percentiles......Page 177 Uniform Distributions......Page 178 Normal Distributions......Page 179 Statistical Significance......Page 180 Statistical Outliers......Page 181 Confidence Intervals......Page 182 Summary......Page 184 Principles of Effective Reporting......Page 185 Use the Right Statistics......Page 186 Additional Considerations......Page 187 Frequently Reported Performance Data......Page 188 Exemplar2......Page 189 Exemplar for Stakeholders......Page 190 Volumes, Capacities, and Rates......Page 191 Component Response Times......Page 192 Trends......Page 193 All Roles......Page 194 Types of Results Sharing......Page 195 Technical Reports......Page 196 Creating a Technical Report......Page 197 Exemplar Tables for Single-Instance Measurements......Page 198 Exemplar Summary Statement......Page 199 Exemplar Results Graph......Page 200 Exemplar Workload Model Graphic......Page 201 Exemplar Observations and Recommendations Statement......Page 202 Summary......Page 203 Part VIII: Performance Testing Techniques......Page 204 How to Use This Chapter......Page 205 Approach for Load Testing......Page 206 Step 1 - Identify Performance Acceptance Criteria......Page 207 Step 2 - Identify Key Scenarios......Page 208 Step 5 - Identify Metrics......Page 209 Step 6 - Design Specific Tests......Page 210 Step 7 - Run Tests......Page 211 Summary......Page 212 Examples of Stress-Related Symptoms......Page 213 Approach for Stress Testing......Page 214 Step 1 - Identify Test Objectives......Page 215 Step 3 - Identify the Workload......Page 216 Step 4 - Identify Metrics......Page 217 Test 1 – Place Order Scenario......Page 218 Usage Scenarios for Stress Testing......Page 219 Exploratory Stress Testing......Page 220 Summary......Page 221