Generation of Constrained Combinatorial Tests for Multi-Label Classifiers

This study proposes a constrained combinatorial test generation method for testing multi-label classifiers. A recent study demonstrated that combinatorial testing is effective for this problem and introduced a special type of constrained covering array, called label-value covering arrays (LVCAs). An LVCA represents a test suite in which each test case specifies the presence or absence of each label. Each test case is subject to a cardinality constraint that limits the number of labels that may be present. Available approaches generate LVCAs by adaptively sampling test cases from the space of possible inputs. However, available methods do not handle constraints other than cardinality constraints, even though such constraints frequently arise in real-world scenarios. Building on previous work, our study proposes a new method for generating test suites that satisfy both cardinality constraints and additional constraints. Our method integrates sampling-based strategy with an SMT-based approach to cover combinations that are difficult to obtain trough sampling alone. We present the results of a series of experiments where the proposed approach is compared with alternative approaches using multiple benchmarks, both with and without additional logical constraints, demonstrating its effectiveness in terms of test generation time and test suite size.