Home / Current Issue / Paper 1722278
Displaced and Suppressed: A Root-Cause Taxonomy of Test Failures in a Large Industrial Playwright/BDD Suite
Subject area: Science,Engineering and Technology · Area of research: Software Testing
Abstract
End-to-end (E2E) user-interface test suites at industrial scale are widely reported to make flakiness their dominant maintenance cost. Less widely examined is a second problem that compounds it: the site at which a failure is reported often diverges from the site of its true defect, and a further, under-examined subset of defects suppresses the failure signal entirely rather than merely displacing it. This paper reports an industrial case study of a 48-feature, approximately 852-scenario Playwright and Cucumber behaviour-driven test suite exercising a commercial multi-tenant software-as-a-service platform across three environments. Through practitioner-embedded root-cause analysis, latency instrumentation of the application under test, and codebase-wide pattern search, a taxonomy of seven recurring defect classes is derived and organized along a displacement/suppression axis. Five classes cause a correct expectation about test behaviour to fail at a location distant from the true cause; two classes cause a broken test to report a pass while verifying nothing. Measured latencies that motivate specific fixes are reported, the propagation of one defect class across eighteen page-object classes is quantified, and a dependency-partitioned parallel execution architecture adopted as a mitigating intervention for suite run time is described. The taxonomy is positioned against the flaky-test and test-smell literature, noting where the suppression classes are structurally analogous to previously reported “rotten green test” phenomena in unit testing, and the threats to validity that follow from a single-suite, single-team study are set out.
Keywords
Behaviour-Driven Development, End-To-End Testing, Flaky Tests, Silent Test Failure, Test Smells.
How to cite this paper
@article{1722278,
author = {Soniya V},
title = {Displaced and Suppressed: A Root-Cause Taxonomy of Test Failures in a Large Industrial Playwright/BDD Suite},
journal = {Iconic Research And Engineering Journals},
year = {2026},
volume = {10},
number = {2},
pages = {1212-1218},
issn = {2456-8880},
url = {https://www.irejournals.com/formatedpaper/1722278.pdf},
abstract = {End-to-end (E2E) user-interface test suites at industrial scale are widely reported to make flakiness their dominant maintenance cost. Less widely examined is a second problem that compounds it: the site at which a failure is reported often diverges from the site of its true defect, and a further, under-examined subset of defects suppresses the failure signal entirely rather than merely displacing it. This paper reports an industrial case study of a 48-feature, approximately 852-scenario Playwright and Cucumber behaviour-driven test suite exercising a commercial multi-tenant software-as-a-service platform across three environments. Through practitioner-embedded root-cause analysis, latency instrumentation of the application under test, and codebase-wide pattern search, a taxonomy of seven recurring defect classes is derived and organized along a displacement/suppression axis. Five classes cause a correct expectation about test behaviour to fail at a location distant from the true cause; two classes cause a broken test to report a pass while verifying nothing. Measured latencies that motivate specific fixes are reported, the propagation of one defect class across eighteen page-object classes is quantified, and a dependency-partitioned parallel execution architecture adopted as a mitigating intervention for suite run time is described. The taxonomy is positioned against the flaky-test and test-smell literature, noting where the suppression classes are structurally analogous to previously reported “rotten green test” phenomena in unit testing, and the threats to validity that follow from a single-suite, single-team study are set out.},
keywords = {Behaviour-Driven Development, End-To-End Testing, Flaky Tests, Silent Test Failure, Test Smells.},
month = {August},
}