Part IV: Test Design and Maintainability
Chapter 9: Writing Maintainable Tests
As test suites grow, structural decisions made earlier in the architecture begin to show their practical impact. Patterns such as page objects, domain models, service layers, and flows are only valuable if they contribute to long-term maintainability. Without disciplined test design, even well-architected abstractions can degrade into complexity that is difficult to understand and expensive to change.
This chapter brings together the concepts introduced in earlier sections and focuses on what makes a test suite sustainable in practice. It examines readability, duplication control, and assertion design, with an emphasis on keeping tests clear, intentional, and resilient to change.
Test Readability and Intent
A maintainable test should communicate intent clearly. When a test is read in isolation, it should be immediately apparent what behaviour is being verified, without requiring the reader to trace implementation details across multiple layers.
Earlier chapters introduced flows as a way of expressing behaviour at a higher level. This principle extends more broadly: tests should be written in terms of what the system does, rather than how it is driven.
For example, a low-level test might look like this:
1 pages.auth.login.open()
2 pages.auth.login.enter_email("user@example.com")
3 pages.auth.login.enter_password("password")
4 pages.auth.login.submit()
5
6 expect(pages.dashboard.welcome_message).to_be_visible()
While explicit, this style places unnecessary cognitive load on the reader. The same behaviour can be expressed more clearly using higher-level abstractions:
1 flows.auth.login("user@example.com", "password")
2 flows.dashboard.assert_user_is_logged_in()
The second version communicates intent directly. It reduces the need to interpret sequencing details and allows the reader to focus on behaviour rather than mechanics.
Readability is not merely a stylistic preference; it directly affects maintainability. Tests that are easy to understand are easier to modify, debug, and extend.
Minimising Duplication
Duplication in test code often emerges gradually. A sequence of actions that initially appears in a single test is frequently copied into others, particularly when flows or domain abstractions are not yet established or are inconsistently applied.
Duplication typically appears in three forms:
- Repeated UI interaction sequences
- Repeated setup and teardown logic
- Repeated assertion patterns
A straightforward example is repeated login logic across multiple tests:
1 pages.auth.login.open()
2 pages.auth.login.enter_email("user@example.com")
3 pages.auth.login.enter_password("password")
4 pages.auth.login.submit()
When repeated across a suite, this becomes a maintenance burden. A change in the login flow requires updates in multiple places, increasing the risk of inconsistency.
Encapsulating this behaviour in a flow reduces duplication:
1 flows.auth.login("user@example.com", "password")
The same principle applies to setup via services. Creating test data through repeated API calls should be centralised in service methods rather than duplicated in individual tests.
However, minimising duplication does not mean eliminating repetition at all costs. Some repetition improves clarity, particularly when it makes a test self-contained. The goal is not absolute DRY adherence, but a balanced approach that prioritises maintainability without sacrificing readability.
Structuring Assertions Effectively
Assertions are a critical part of test design, yet they are often under-structured or inconsistently applied. Poorly placed or overly granular assertions can obscure intent and make tests harder to interpret.
A common issue is scattering multiple unrelated assertions throughout a test:
1 expect(page.title).to_have_text("Dashboard")
2 expect(page.get_by_text("Welcome")).to_be_visible()
3 expect(page.get_by_role("button", name="Logout")).to_be_visible()
While each assertion is valid, the grouping does not clearly communicate what behaviour is being validated.
A more structured approach is to group assertions around a single behavioural outcome. For example:
1 flows.dashboard.assert_user_is_logged_in()
Internally, this flow or helper method may contain multiple assertions, but from the test’s perspective, the behaviour is expressed as a single expectation.
This does not mean that all assertions should be hidden. In some cases, particularly when verifying specific business rules, explicit assertions improve clarity. The key is consistency: assertions should be structured in a way that reflects the level of abstraction used in the test.
It is also important to avoid embedding assertions inside unrelated abstractions. For example, page objects should not contain test-specific assertions, as this blurs the boundary between structure and validation.
Balancing Abstraction and Transparency
A recurring theme across maintainable test design is the balance between abstraction and visibility. While higher-level constructs such as flows and domain methods improve readability, excessive abstraction can make tests difficult to debug.
A test should not feel like a black box. When a failure occurs, it should be reasonably clear which part of the system is responsible. Over-compressed abstractions such as:
1 flows.user.do_all_the_things()
reduce diagnostic clarity and make troubleshooting more difficult.
A practical guideline is that each abstraction should represent a meaningful unit of behaviour. If a method name no longer reflects a clear user action or business concept, it is likely too broad.
Similarly, tests should remain anchored to observable outcomes. Even when using high-level flows, there should be a clear point at which behaviour is validated in terms of system state.
Consistency Across the Test Suite
Maintainability depends heavily on consistency. A test suite that uses multiple styles for solving the same problem becomes harder to understand and extend.
Consistency applies across several dimensions:
- Level of abstraction (pages vs flows vs domains)
- Naming conventions for actions and assertions
- Structure of setup and teardown logic
- Use of services versus UI interactions
For example, mixing direct page object usage with flows in an inconsistent manner can create confusion:
1 pages.auth.login(...)
2 flows.auth.login(...)
A more consistent approach is to define clear boundaries for when each abstraction is used. Flows might be reserved for end-to-end scenarios, while page objects are used for targeted interaction tests.
Consistency does not require rigidity, but it does require deliberate design choices that are applied uniformly across the suite.
Maintainability as an Architectural Outcome
Maintainability is not a property of individual tests alone; it is the result of the overall architecture. The patterns introduced in earlier chapters—locators, synchronisation, page objects, domain models, services, and flows—each contribute to reducing complexity at different levels.
When combined effectively, they allow tests to be:
- Easier to read
- Easier to modify
- Easier to debug
- Less sensitive to UI or backend changes
However, these benefits only materialise when abstractions are used consistently and appropriately. Overuse of abstraction can be as damaging as underuse.
Ultimately, writing maintainable tests is less about any single pattern and more about disciplined application of structure, clarity, and intent across the entire test suite.
Chapter 10: Managing Test Data
Test data is one of the most significant factors influencing the reliability of automated test suites. Even well-designed tests can become unstable if they depend on inconsistent, shared, or poorly controlled data. Failures that appear to be related to timing or UI behaviour are often, in reality, symptoms of underlying data issues.
This chapter examines strategies for creating, managing, and isolating test data in a structured way. It focuses on data generation patterns, environment considerations, and techniques for avoiding shared state. The aim is to ensure that tests remain deterministic, repeatable, and independent of external conditions.
Data Generation Patterns
A common challenge in test automation is deciding how test data should be created. Hard-coded data may be simple to implement, but it quickly becomes limiting as test suites grow. It can lead to conflicts, duplication, and a lack of flexibility when scenarios evolve.
A more scalable approach is to generate test data programmatically. This can be done through factories or dedicated helper functions that produce valid, consistent data structures.
For example:
1 import uuid
2
3 def create_user_data(email: str | None = None) -> dict:
4 unique_email = email or f"user-{uuid.uuid4()}@example.com"
5
6 return {
7 "email": unique_email,
8 "password": "SecurePassword123!",
9 "name": "Test User"
10 }
This pattern ensures that each test can operate with unique data, reducing the risk of collisions between test runs. It also makes tests more flexible, as variations can be introduced without modifying the core test logic.
In more complex systems, data builders may be extended to support different states or configurations:
1 class OrderBuilder:
2 def __init__(self):
3 self._items = []
4
5 def with_item(self, sku: str, quantity: int):
6 self._items.append({"sku": sku, "quantity": quantity})
7 return self
8
9 def build(self) -> dict:
10 return {
11 "order_id": str(uuid.uuid4()),
12 "items": self._items,
13 "status": "pending"
14 }
This approach encourages clarity and reuse while maintaining control over data structure and validity.
Environment Considerations
Test data strategies must also take into account the environment in which tests are executed. Differences between local development, staging, and CI environments can introduce inconsistencies if not properly managed.
A key principle is that tests should not depend on pre-existing data within an environment. Shared or manually maintained datasets tend to drift over time, leading to brittle tests that fail unpredictably.
Instead, environments should be treated as disposable and reproducible. Test data should be created as part of the test setup process, either through service APIs or dedicated fixtures.
For example, using the service layer introduced in Chapter 7:
1 user = services.users.create_user("user@example.com", "password")
2 order = services.orders.create_order(user["id"], [{"sku": "ABC123", "qty": 1}])
This ensures that each test operates within a known state, independent of external conditions.
It is also important to consider environment isolation. Parallel test execution, particularly in CI pipelines, increases the risk of data collisions if identifiers are not properly isolated. Techniques such as unique identifiers, namespacing, or tenant separation can help mitigate these issues.
Avoiding Shared State
Shared state is one of the most common sources of flaky tests. When multiple tests rely on or modify the same data, outcomes can become non-deterministic. A test may pass or fail depending on the order in which it is executed or the behaviour of other tests running in parallel.
The safest approach is to ensure that each test is fully self-contained. Any required data should be created at the start of the test and cleaned up afterwards, either explicitly or through automated teardown mechanisms.
For example:
1 user = services.users.create_user("user@example.com", "password")
2
3 try:
4 flows.auth.login("user@example.com", "password")
5 flows.dashboard.assert_user_is_logged_in()
6 finally:
7 services.users.delete_user(user["id"])
Where possible, teardown should be handled automatically to avoid duplication and reduce the risk of forgotten cleanup steps.
Another important consideration is avoiding dependencies between tests. Tests should not rely on data created by previous tests, as this introduces hidden coupling and makes execution order significant. Each test should be able to run independently in isolation.
Determinism Through Controlled State
A key objective of test data management is determinism. A test is deterministic when it produces the same outcome given the same inputs and environment conditions. Uncontrolled or shared data undermines this principle.
Controlled state can be achieved through a combination of strategies:
- Generating unique data per test run
- Using service layers to manage setup and teardown
- Avoiding reliance on pre-seeded environment data
- Resetting or isolating state between tests where necessary
For example:
1 def test_order_creation():
2 user = services.users.create_user("user@example.com", "password")
3
4 order = services.orders.create_order(user["id"], [{"sku": "ABC123", "qty": 1}])
5
6 flows.orders.assert_order_exists(order["order_id"])
Each test defines its own state explicitly, ensuring that external factors do not influence outcomes.
This approach improves reliability and also makes tests easier to reason about, since all dependencies are visible within the test itself.
Trade-offs in Test Data Strategy
While full isolation is ideal, it is not always practical in large systems. Creating and tearing down data for every test can introduce performance overhead, particularly in slower environments or when dealing with complex datasets.
In such cases, a balanced approach may be required. Shared reference data can be used for stable, read-only scenarios, provided it is carefully controlled and monitored. However, any shared data should be treated as part of the test infrastructure rather than assumed to be static.
Where trade-offs are made, they should be explicit and documented. Hidden dependencies on environment state are more dangerous than deliberate, well-understood compromises.
Ultimately, the goal is to prioritise reliability and repeatability, even if that requires additional effort in test setup design.
Chapter 11: Scaling the Test Suite
As a test suite grows, challenges shift from individual test design to system-wide concerns. Issues that are negligible in small projects, such as execution time, repository structure, and coordination between test modules, become increasingly significant at scale. Without deliberate design choices, even a well-structured suite can become slow to run, difficult to navigate, and expensive to maintain.
This chapter examines how to scale a UI test suite effectively. It focuses on execution strategies, parallelisation, and repository structure. The aim is to ensure that growth in test coverage does not come at the expense of performance or maintainability.
Test Execution Strategies
Test execution strategy plays a central role in how a suite behaves as it scales. Sequential execution is straightforward and easy to reason about, but it does not make efficient use of available resources. As the number of tests increases, total execution time can quickly become a bottleneck in development workflows and continuous integration pipelines.
A common approach is to categorise tests based on speed and scope. For example:
- Fast tests: unit-level or isolated UI checks
- Medium tests: domain or flow-based scenarios
- Slow tests: full end-to-end journeys
This categorisation allows selective execution depending on context. Developers may run fast tests locally during development, while slower, more comprehensive suites are reserved for CI environments.
In practice, tagging or grouping tests is often used to support this strategy:
1 import pytest
2
3 @pytest.mark.smoke
4 def test_login():
5 flows.auth.login("user@example.com", "password")
6 flows.dashboard.assert_user_is_logged_in()
This enables flexible execution:
1 pytest -m smoke
Such segmentation helps to maintain fast feedback cycles while preserving full coverage where it matters.
Parallelisation
As suites grow, parallel execution becomes pretty much essential to maintain reasonable run times. Modern test frameworks and CI systems provide native support for parallelisation, but effective use requires careful design.
The primary challenge in parallel execution is avoiding shared state. Tests must be fully independent, with no reliance on shared data or execution order. This reinforces earlier principles around test data management and isolation.
In Python-based Playwright setups, parallel execution is often achieved using tools such as pytest-xdist:
1 pytest -n auto
This distributes tests across multiple workers, reducing total execution time.
However, parallelisation introduces constraints:
- Test data must be unique per test or worker
- Environments must support concurrent access
- External dependencies must tolerate simultaneous requests
For example, user creation must avoid collisions:
1 user = services.users.create_user(f"user-{uuid.uuid4()}@example.com", "password")
Without such precautions, parallel execution can lead to intermittent and difficult-to-diagnose failures.
It is also important to consider resource contention. Running too many parallel tests can overload shared environments, leading to degraded performance and unreliable results. A balance must therefore be struck between speed and stability.
Structuring Large Repositories
As test suites expand, repository structure becomes a key factor in maintainability. Poor organisation can make it difficult to locate tests, understand coverage, or identify ownership of specific areas.
A common approach is to mirror the logical structure of the application under test, often aligned with domains introduced earlier in this document. For example:
1 tests/
2 auth/
3 test_login.py
4 test_registration.py
5 checkout/
6 test_cart.py
7 test_payment.py
8 orders/
9 test_order_creation.py
This structure improves discoverability by grouping related tests together. Developers can quickly locate tests relevant to a specific feature area without needing to search across a flat directory.
In larger teams, additional layering may be introduced:
- Feature-based grouping (auth, checkout, orders)
- Test type grouping (smoke, regression, integration)
- Component-based grouping (UI, API, end-to-end)
However, over-segmentation can become counterproductive. If tests are spread across too many directories or classification schemes, navigation becomes fragmented. A balance should be maintained between structure and simplicity.
Naming conventions also play an important role. Test names should reflect behaviour clearly, avoiding ambiguity:
1 def test_user_can_complete_checkout():
2 ...
Clear naming reduces the need to inspect implementation details when diagnosing failures.
CI/CD Integration and Feedback Loops
At scale, test suites are most valuable when integrated effectively into CI/CD pipelines. The primary objective is not only correctness, but fast and reliable feedback.
A typical strategy involves multiple stages:
- Quick smoke tests on every commit
- Broader regression suites on pull requests
- Full end-to-end runs on scheduled builds
This staged approach ensures that critical issues are detected early, while maintaining comprehensive coverage over time.
Execution time becomes a key constraint in this context. If feedback loops are too slow, developers may begin to bypass or ignore test results. Maintaining a balance between coverage and speed is therefore essential.
Managing Complexity Through Modularity
As the suite grows, modularity becomes increasingly important. The architectural patterns introduced in earlier chapters—page objects, domains, services, and flows—should be reflected in the repository structure.
A well-organised suite typically separates concerns clearly:
- Page layer: UI structure and interaction
- Domain layer: business concepts and workflows
- Service layer: backend interaction and setup
- Flow layer: end-to-end behaviour
- Test layer: assertions and validation logic
Maintaining these boundaries helps prevent entanglement between concerns. When changes occur in one area of the application, their impact on the test suite is localised rather than widespread.
Sustaining Performance and Maintainability
Scaling a test suite is not solely a technical challenge; it is also an organisational one. As teams grow, consistency in how tests are written and structured becomes as important as the underlying tooling.
Key principles include:
- Avoiding unnecessary duplication of test logic
- Ensuring consistent use of abstractions
- Maintaining clear ownership of test areas
- Regularly reviewing execution time and flaky tests
Without ongoing maintenance, even a well-designed suite can degrade over time. Continuous attention to structure and performance ensures that the suite remains a reliable signal rather than a source of noise.
Scaling is not a one-time design decision but an ongoing discipline that evolves alongside the system under test.