End-to-end Test Automation Anti-Patterns

Preface

1. Purpose of the book
2. Who the book is for?
3. Send me feedback

Introduction

1. What is End-to-End Software Testing?
2. E2E Testing is mostly a form of functional testing
3. E2E Testing: Automation vs Manual
4. E2E Testing involves the entire development team
5. E2E Testing vs User Story Acceptance Testing
6. E2E Testing vs Regression Testing
7. Good E2E Regression Testing enables frequent releases

Wrong Mindset: Developer-Focused Approach

1. E2E Test Automation is Black-Box Testing!
2. Scripting Language, Frameworks and Tools
3. Most Software Developers Lack Proficiency in Test Automation
4. Hard to Steer Away even after it is Proved Wrong

Wrong Opinions

1. Anti-Pattern: E2E Test Automation is easy
2. Anti-Pattern: E2E Test Automation is not feasible, unit/API testing is enough
3. Anit-Pattern: Don’t Really Believe in Functional Test Automation
4. Anti-Pattern: Assuming E2E UI Test Automation Is Either Trivial or Impossible
5. Anti-Pattern: Assuming the Primary Effort in E2E Test Automation Is Test Creation
6. Anti-Pattern: Aiming for 100% Coverage in Test Automation

Hypes

1. Anti-Pattern: Keep Migrating to a new Framework
2. Anti-Pattern: Codeless Test Automation
3. Anti-Pattern: Cypress Component Testing
4. Anti-Pattern: Visual Regression Testing

General Practices

1. Anti-Pattern: Record and Playback
2. Anti-Pattern: Using Gherkin (Cucumber, SpecFlow) Frameworks in E2E Test Automation
3. Anti-Pattern: E2E Test Scripts are Not in an Independent Git Repository
4. Anti-Pattern: Semi-Automated Execution
5. Anti-Pattern: Defect Tracking for Failed Automated E2E Tests from Regression Testing

Scripting Practices

1. Anti-Pattern: Use of node_modules or alike
2. Anti-Pattern: Test Interdependency
3. Anti-Pattern: Test script fails after the first successful run
4. Anti-Pattern: Excessive low-value alternative and edge path tests
5. Anti-Pattern: Lack reliability
6. Anti-Pattern: Accept Extremely Inefficient Test Creation
7. Anti-Pattern: Long suite execution
8. Anti-Pattern: Use of “Await” and “Promises”

Web Test Automation

1. Anti-Pattern: Headless Browser Testing
2. Anti-Pattern: Cross Browser Testing
3. Anti-Pattern: Not Using Raw Selenium WebDriver
4. Anti-Pattern: ‘Design a new Test Automation Framework’
5. Anti-Pattern: ‘Auto-Wait’
6. Anti-Pattern: ‘Test-Replay’

API Testing

1. Anti-Pattern: Fixated with GUI tools such as SoapUI or Postman
2. Anti-Pattern: Lack of text manipulation skills
3. Anti-Pattern: Lack of Continuous Execution
4. Anti-Pattern: Lack of dynamic test data management capability
5. Anti-Pattern: Confined to a proprietary tool

Mobile/Desktop Test Automation

1. Anti-Pattern: Attempting Mobile Test Automation Before Achieving Successful Web Test Automation
2. Anti-Pattern: Not using raw Appium
3. Anti-Pattern: No Parallel Testing Lab
4. Anti-Pattern: Fixated with Real Device Testing

Test Execution

1. Anti-Pattern: Execute a Suite in the Development Tool
2. Anti-Pattern: Assuming Automated Tests Will Always Pass After Creation
3. Anti-Pattern: Run in CI Servers
4. Anti-Pattern: Full Test Suite Runs Overnight Only
5. Anti-Pattern: Prioritizing Tests in Execution
6. Anti-Pattern: Don’t run the whole suite often
7. Anti-Pattern: “Parallel Execution on one machine”

Performance Testing

1. Anti-Pattern: Conducted too late
2. Anti-Pattern: Don’t run performance testing frequently
3. Anti-Pattern: Skip AJAX Operations
4. Anti-Pattern: Optimizing Without Automated End-to-End Regression Testing
5. Anti-Pattern: Lack the comparison history

Load Testing

1. Anti-Pattern: Only Testing Simple Operations
2. Anti-Pattern: Protocol-based Load Testing for Modern Dynamic Web Apps
3. Anti-Pattern: Overestimating the Required Load
4. Anti-Pattern: Rely on Expensive Tools or Patch with Money
5. Anti-Pattern: Don’t run it frequently

Leadership

1. Anti-Pattern: Just Love The Idea of Test Automation, But No Commitment
2. Anti-Pattern: No Time or Resources Allocated for Automated Test Stabilization and Maintenance
3. Anti-Pattern: Developers delay fixing newly discovered defects by Continuous Testing
4. Anti-Pattern: Lack of Team-Wide Adoption of E2E Test Automation
5. Anti-Pattern: No Training Provided
6. Anti-Pattern: Not Seeking External Help

AI Test Automation

1. Anti-Pattern: “AI Test Automation Is Good Enough”
2. Anti-Pattern: Using AI to Generate Many Test Scenarios
3. Anti-Pattern: Very Slow AI Test Execution
4. Anti-Pattern: Lack of Flexibility
5. Anti-Pattern: Unaware of Big Cost
6. Summary

Miscellaneous

1. Anti-Pattern: Testing in Production
2. Anti-Pattern: Manual and Performance Testing Operate in Isolation, Not taking Advantage of Automated E2E Testing
3. Anti-Pattern: Production Releases Do Not Require Automated End-to-End Testing Reports

Afterword

1. My 35-Word Functional Test Automation Strategy
2. My Test Automation Formula

Appendix I: Anti-Patterns List

Appendix II: Benefits of E2E Test Automation and Continuous Testing

1. To Wise Executives
2. To Software Managers
3. To Business Analysts
4. To Developers
5. To Software Testers (Auto & Manual)
6. To Customers

Resources

1. Books
2. Tools
3. Blog

References