TY  - GEN
ID  - heidok31757
KW  - testing
KW  -  very large software project
KW  -  software testing
KW  -  coverage
KW  -  object creation
KW  -  cost reduction
AV  - public
CY  - Heidelberg
TI  - Testing in Very Large Software Projects
Y1  - 2022///
N2  - Testing is an important activity for software projects. However, testing requires effort and therefore creates so-called test costs. They consist of factors such as human effort for testing, or costs for required software and hardware. Test costs can contribute considerably to the total cost of a software project. Therefore, it is desirable to reduce test costs without affecting the quality of the software. With agile development processes and short test cycles, this aim is even more important.

Consequently, prior work propose a wide range of techniques for test cost reduction. However, while studying a very large industrial software project, we found that techniques proposed in related work can often not be applied for a project of such a size. We attribute this to problems that are only significant in large projects and may not be noticed in smaller projects. 

The amount of test execution grows superlinearly over time in large projects. This superlinear increase cannot be sustainably solved by increases in hardware. Additionally, tests with long execution times have several negative consequences. A large number of changes and the long execution times of tests create a conflict that does not exist in smaller projects.

Therefore, approaches for test cost reduction must consider the specific characteristics of large projects. In our work, we tackle this challenge in multiple ways. We describe an approach to replace the superlinear increase in test executions with a linear increase by utilizing a multi-stage test strategy that provides economic incentives to decrease test costs. We also design an approach that extracts unit tests from system tests and utilizes the multi-stage test strategy to reduce test costs without reductions in quality. Our approach builds upon time-travel debugging and reverse-executes a system test to identify dependencies. We combine coverage-based differential analysis and source code analysis via a compiler plugin to enable accurate identification of the test core. Our approach extends previous work by creating maintainable source code instead of binary data and by focusing on the test core instead of extracting tests for all parts of the software.

In addition to our core contributions, we also provide two empirical studies that investigate the relationship between coverage data and faults. The first study investigates the distribution of bugs and coverage. The second study investigates Granger-causality between coverage and bugs.

Finally, as a byproduct of our core contributions, we provide approaches for automatically determining the creation of objects with complex dependencies, for automated mock recommendation, for combining combinatorial testing with coverage-based test amplification, for reductions of coverage data sizes, and we solve coverage-based algorithmic problems.
A1  - Bach, Thomas
UR  - https://archiv.ub.uni-heidelberg.de/volltextserver/31757/
ER  -