End-to-end Test Automation Anti-Patterns
End-to-end Test Automation Anti-Patterns
Lessons from Real-World Test Automation Failures
About the Book
Test Automation, especially end-to-end (E2E) test automation via UI, is rarely taught in university computer science programs. In the workplace, formal training in this area is equally rare. As a result, the industry is riddled with misconceptions, confusion and wrong practices in this field.
Since 2013, while visiting various software projects, I recorded the pass rates of automated E2E tests in their 'overnight builds' within so-called 'CI/CD' pipelines. The highest pass rate I encountered was just 48%, a clear sign of systemic failure.
Many of these failures can be traced to one or more of the 59 anti-patterns detailed in this book, organized into the following categories:
- Mindset and Opinions
- General and Scripting Practices
- Web Test Automation
- API Testing
- Mobile/Desktop Test Automation
- Test Execution
- Performance Testing
- Load Testing
- Leadership
This book will help you recognize and avoid these pitfalls, paving the way for successful E2E test automation.
About the Author
Table of Contents
- Preface
- Purpose of the book
- Who the book is for?
- Send me feedback
- 1:Introduction
- 1.1:What is End-to-End Software Testing?
- 1.2:E2E Testing is mostly a form of functional testing
- 1.3:E2E Testing: Automation vs Manual
- 1.4:E2E Testing involves the entire development team
- 1.5:E2E Testing vs User Story Acceptance Testing
- 1.6:E2E Testing vs Regression Testing
- 1.7:Good E2E Regression Testing enables frequent releases
- 2:Wrong Mindset: Developer-Focused Approach
- 2.1:E2E Test Automation is Black-Box Testing!
- 2.2:Scripting Language, Frameworks and Tools
- 2.3:Most Software Developers Lack Proficiency in Test Automation
- 2.4:Hard to Steer Away even after it is Proved Wrong
- 3:Wrong Opinions
- 3.1:Anti-Pattern: E2E Test Automation is easy
- 3.2:Anti-Pattern: E2E Test Automation is not feasible, unit/API testing is enough
- 3.3:Anit-Pattern: Don’t Really Believe in Functional Test Automation
- 3.4:Anti-Pattern: Assuming E2E UI Test Automation Is Either Trivial or Impossible
- 3.5:Anti-Pattern: Assuming the Primary Effort in E2E Test Automation Is Test Creation
- 3.6:Anti-Pattern: Aiming for 100% Coverage in Test Automation
- 4:Hypes
- 4.1:Anti-Pattern: Keep Migrating to a new Framework
- 4.2:Anti-Pattern: Codeless Test Automation
- 4.3:Anti-Pattern: Cypress Component Testing
- 4.4:Anti-Pattern: Visual Regresion Testing
- 4.5:Anti-Pattern: AI Test Automation
- 5:General Practices
- 5.1:Anti-Pattern: Record and Playback
- 5.2:Anti-Pattern: Using Gherkin (Cucumber, SpecFlow) Frameworks in E2E Test Automation
- 5.3:Anti-Pattern: E2E Test Scripts are Not in an Independent Git Repository
- 5.4:Anti-Pattern: Semi-Automated Execution
- 5.5:Anti-Pattern: Defect Tracking for Failed Automated E2E Tests from Regression Testing
- 6:Scripting Practices
- 6.1:Anti-Pattern: Use of node_modules or alike
- 6.2:Anti-Pattern: Test Interdependency
- 6.3:Anti-Pattern: Test script fails after the first successful run
- 6.4:Anti-Pattern: Excessive low-value alternative and edge path tests
- 6.5:Anti-Pattern: Lack reliability
- 6.6:Anti-Pattern: Accept Extremely Inefficient Test Creation
- 6.7:Anti-Pattern: Long suite execution
- 6.8:Anti-Pattern: Use of “Await” and “Promises”
- 7:Web Test Automation
- 7.1:Anti-Pattern: Headless Browser Testing
- 7.2:Anti-Pattern: Cross Browser Testing
- 7.3:Anti-Pattern: Not Using Raw Selenium WebDriver
- 7.4:Anti-Pattern: ‘Design a new Test Automation Framework’
- 7.5:Anti-Pattern: ‘Auto-Wait’
- 7.6:Anti-Pattern: ‘Test-Replay’
- 8:API Testing
- 8.1:Anti-Pattern: Fixated with GUI tools such as SoapUI or Postman
- 8.2:Anti-Pattern: Lack of text manipulation skills
- 8.3:Anti-Pattern: Lack of Continuous Execution
- 8.4:Anti-Pattern: Lack of dynamic test data management capability
- 8.5:Anti-Pattern: Confined to a proprietary tool
- 9:Mobile/Desktop Test Automation
- 9.1:Anti-Pattern: Attempting Mobile Test Automation Before Achieving Successful Web Test Automation
- 9.2:Anti-Pattern: Not using raw Appium
- 9.3:Anti-Pattern: No Parallel Testing Lab
- 9.4:Anti-Pattern: Fixated with Real Device Testing
- 10:Test Execution
- 10.1:Anti-Pattern: Execute a Suite in the Development Tool
- 10.2:Anti-Pattern: Assuming Automated Tests Will Always Pass After Creation
- 10.3:Anti-Pattern: Run in CI Servers
- 10.4:Anti-Pattern: Full Test Suite Runs Overnight Only
- 10.5:Anti-Pattern: Prioritizing Tests in Execution
- 10.6:Anti-Pattern: Don’t run the whole suite often
- 10.7:Anti-Pattern: “Parallel Execution on one machine”
- 11:Performance Testing
- 11.1:Anti-Pattern: Conducted too late
- 11.2:Anti-Pattern: Don’t run performance testing frequently
- 11.3:Anti-Pattern: Skip AJAX Operations
- 11.4:Anti-Pattern: Optimizing Without Automated End-to-End Regression Testing
- 11.5:Anti-Pattern: Lack the comparison history
- 12:Load Testing
- 12.1:Anti-Pattern: Only Testing Simple Operations
- 12.2:Anti-Pattern: Protocol-based Load Testing for Modern Dynamic Web Apps
- 12.3:Anti-Pattern: Overestimating the Required Load
- 12.4:Anti-Pattern: Rely on Expensive Tools or Patch with Money
- 12.5:Anti-Pattern: Don’t run it frequently
- 13:Leadership
- 13.1:Anti-Pattern: Just Love The Idea of Test Automation, But No Commitment
- 13.2:Anti-Pattern: No Time or Resources Allocated for Automated Test Stabilization and Maintenance
- 13.3:Anti-Pattern: Developers delay fixing newly discovered defects by Continuous Testing
- 13.4:Anti-Pattern: Lack of Team-Wide Adoption of E2E Test Automation
- 13.5:Anti-Pattern: No Training Provided
- 13.6:Anti-Pattern: Not Seeking External Help
- 14:Miscellaneous
- 14.1:Anti-Pattern: Testing in Production
- 14.2:Anti-Pattern: Manual and Performance Testing Operate in Isolation, Not taking Advantage of Automated E2E Testing
- 14.3:Anti-Pattern: Production Releases Do Not Require Automated End-to-End Testing Reports
- 15:Afterword
- 15.1:My 35-Word Functional Test Automation Strategy
- 15.2:My Test Automation Formula
- Appendix I: Anti-Patterns List
- Appendix II: Benefits of E2E Test Automation and Continuous Testing
- To Wise Executives
- To Software Managers
- To Business Analysts
- To Developers
- To Software Testers (Auto & Manual)
- To Customers
- Resources
- Books
- Tools
- Blog
- References
- Preface
The Leanpub 60 Day 100% Happiness Guarantee
Within 60 days of purchase you can get a 100% refund on any Leanpub purchase, in two clicks.
Now, this is technically risky for us, since you'll have the book or course files either way. But we're so confident in our products and services, and in our authors and readers, that we're happy to offer a full money back guarantee for everything we sell.
You can only find out how good something is by trying it, and because of our 100% money back guarantee there's literally no risk to do so!
So, there's no reason not to click the Add to Cart button, is there?
See full terms...
Earn $8 on a $10 Purchase, and $16 on a $20 Purchase
We pay 80% royalties on purchases of $7.99 or more, and 80% royalties minus a 50 cent flat fee on purchases between $0.99 and $7.98. You earn $8 on a $10 sale, and $16 on a $20 sale. So, if we sell 5000 non-refunded copies of your book for $20, you'll earn $80,000.
(Yes, some authors have already earned much more than that on Leanpub.)
In fact, authors have earnedover $14 millionwriting, publishing and selling on Leanpub.
Learn more about writing on Leanpub
Free Updates. DRM Free.
If you buy a Leanpub book, you get free updates for as long as the author updates the book! Many authors use Leanpub to publish their books in-progress, while they are writing them. All readers get free updates, regardless of when they bought the book or how much they paid (including free).
Most Leanpub books are available in PDF (for computers) and EPUB (for phones, tablets and Kindle). The formats that a book includes are shown at the top right corner of this page.
Finally, Leanpub books don't have any DRM copy-protection nonsense, so you can easily read them on any supported device.
Learn more about Leanpub's ebook formats and where to read them
