Announcing remove required integration test check when merge pr

#1

Hi developers! On behalf of our infra team, I’m hereby announcing the adjustment on pr merge required integration tests. The following projects will remove requied integration test check label when merge pr due to idc-jenkins-ci cluster’s high pressure.

CI will trigger all integration test after your pr has been merged, integration test result will display on your merged pr releated commit id. Furthermore QA team will receive notify if your pr break integration test, so take it easy to contribute.

And one more important thing, you still could trigger all integration tests by comment /run-integration-tests. After this adjustment, comment /merge and /run-all-test will not trigger any integration tests.

PS. some trigger example if you want to run integration test on pr.

  1. run all integration tests: /run-integration-tests
  2. run one specific integration test, example for copr-test: /run-integration-copr-test
#2

Great!
It make CI a little better now

#3

Thanks for announcing this change @purelind!

Three questions here:

  1. What’s your plan on populating doc of commands to the corresponding repos? I think developers hardly remember all these command.
  2. As @skyzh mentioned here, tikv/tikv requires to turn on the integration tests “tidb_ghpr_integration_copr_test” to avoid coprocessor compatibility breaks silently. cc @breeswish
  3. Could you introduce the “QA team”? Who are they and what they will do after “receive notify if your pr break integration test”?
1 Like
#5

It’s reasonable to reduce pressure of testing infra. However,

  • Previously, we trigger integration test once before merge. Now we trigger it when being merged. The total number of runs is not changed, but now integration issues cannot be notified immediately upon merging.
  • The integration test has been broken for a week, but I don’t find any place that we could trace who breaks the test and the fixing progress.
#6

… and broken tests on master are making life harder for both of us. For QA team, you may need to notify the corresponding PR author or even revert the commit. For me, it is not impossible to tell whether a copr PR could be merged. As integration tests are broken now, I don’t know whether new PRs are working or not, as they would fail on unrelated cases.

#7
  1. A doc of trigger commands like ci.md in repo will be useful for any developers who has confused about how to trigger ci. Add a link on dev guide also is welcome.
  2. Currently all tikv/tikv integration tests are not required for pr mege. Any failed tests will alert to lark group 集成测试失败通知.
  3. @jingshanglu is now in charge of break integration tests. Once a meged pr break integration test, a lark alert will push to group 集成测试失败通知。 @jingshanglu will notify pr author to fix this issue as soon as possible, and create releated issue to trace this problom.
#8
  1. Most of time ci cluster pressure from integration test is encountered when try to merged too much pr for a new release version. For example, now release-5.1.x has 10 PRs wait to merge, ti-chi-bot firstly try to merge master then run unit-test & integration tests for every pr, so total number of runs is 𝑛×(𝑛+1)2
  2. @jingshanglu will create issue about pr break integration tests.

Any suggestions about making integration test more reasonable is welcome.

#9

Currently ti-chi-bot will not run all inregration tests when try to merge pr. We still could trigger all integration tests by comment /run-integration-tests to verify coprocessor compatibility breaks or not.

#10

As you know, community members have no access to internal Lark group. I’ll trigger integration tests when needed.