Separating Integration Tests
Current problem
For a historical reason, the majority of TiDB ‘unit test’ cases are coded as integration tests. Since it is simple and quick, a lot of examples can be found everywhere. Here is 1 simple example:
func (s *testSuiteAgg) TestAggregation(c *C) {
tk := testkit.NewTestKit(c, s.store)
tk.MustExec("set @@tidb_hash_join_concurrency=1")
tk.MustExec("use test")
tk.MustExec("set sql_mode='STRICT_TRANS_TABLES'") // disable only-full-group-by
tk.MustExec("drop table if exists t")
tk.MustExec("create table t (c int, d int)")
tk.MustExec("insert t values (NULL, 1)")
tk.MustExec("insert t values (1, 1)")
tk.MustExec("insert t values (1, 2)")
tk.MustExec("insert t values (1, 3)")
tk.MustExec("insert t values (1, 1)")
tk.MustExec("insert t values (3, 2)")
tk.MustExec("insert t values (4, 3)")
tk.MustQuery("select bit_and(c) from t where NULL").Check(testkit.Rows("18446744073709551615"))
tk.MustQuery("select bit_or(c) from t where NULL").Check(testkit.Rows("0"))
...
}
It brings us a lot of troubles:
- The real unit tests are not designed and implemented.
- It brings a lot of concurrent execution issues(like ‘DATA RACE’) because modifying internal variables is easy.
Goal
We would like to move those integration tests out of the codes.
Proposal
- Provide a new directory named ‘tidb-test’ in TiDB server codebase(pingcap/tidb), just like what MySQL did: mysql-test.
- Port as many as the ‘integration cases’ in ‘xxx_test.go’ to ‘tidb-test’ directory.
Take executor/aggregate_test.go(the example given earilier in this topic) to:
# $TIDB_SRC/tidb-test/t/aggregate_suite.test
# case: TestAggregation
set @@tidb_hash_join_concurrency=1
use test
set sql_mode='STRICT_TRANS_TABLES'
drop table if exists t
create table t (c int, d int)
insert t values (NULL, 1)
insert t values (1, 1)
insert t values (1, 2)
insert t values (1, 3)
insert t values (1, 1)
insert t values (3, 2)
insert t values (4, 3)
select bit_and(c) from t where NULL
select bit_or(c) from t where NULL
...
# $TIDB_SRC/tidb-test/r/aggregate_suite.result
set @@tidb_hash_join_concurrency=1
use test
set sql_mode='STRICT_TRANS_TABLES'
drop table if exists t
create table t (c int, d int)
insert t values (NULL, 1)
insert t values (1, 1)
insert t values (1, 2)
insert t values (1, 3)
insert t values (1, 1)
insert t values (3, 2)
insert t values (4, 3)
select bit_and(c) from t where NULL
18446744073709551615
select bit_or(c) from t where NULL
0
- Update Makefile to trigger mysql-tester for
make test.
The automatic ports
Porting the cases from the code to mysql-tester manually can be a very big amount of work. We need an automatic and quick way to archive it.
There are 2 major testing frameworks: pingcap/check and stretchr/testify. Most of the test cases can be generated automatically by hacking the framework.
For pingcap/check:
// Open the test suite file in when `SetUpSuite` is called and close the file when `TearDownSuite` is called:
f = os.Open(‘$SUITE_NAME/$TESTCASE_NAME’)
caseRunner.run(case.SetUpSuite())
// ...
caseRunner.run(case.TearDownSuite())
f.Close()
// Write the `sql` statement to test file when `tk.MustExec(sql)` and `tk.MustQuery(sql)` is called:
func (tk *TestKit) MustExec(sql string) {
if tk.f != nil {
_, _ = tk.f.WriteString(sql)
_, _ = tk.f.WriteString("\n")
}
}
func (tk *TestKit) MustQuery(sql string) {
if tk.f != nil {
_, _ = tk.f.WriteString(sql)
_, _ = tk.f.WriteString("\n")
}
}
Challenges
- How to(or should we?) port the test cases with the failpoint.
- Many test cases change the configuration of TiDB dynamically (setting some internal variables at the beginning and restoring it after the case is done), which is not practical for ‘mysql-tester’.
- Performance: running integration tests on ‘mysql-tester’ will be slower compared to the current built-in code way.
Known risks
Since the ‘mysql-tester’ is our own implementation of ‘mysql-test’ (they look the same). It is likely that engineers may want to port test cases from MySQL source to TiDB server repository, which violates the license of MySQL(GPLv2).