New TiChiBot service for pull requests

[wuhuizuo]

Plan

Milestone	Features	Estimated date
New-Deploy	Latest Prow features	2022-12-16
Take-Over	TiChi controll pre-merge CI builds and enable batch merging on pingcap/tidb repo	2022-12-30
Take-Over2	TiChi controll post-merge CI builds on pingcap/tidb repo	2023-01-30
Full-Control	TiChi controls all Jenkins CI builds for all public repository	2023-03-30

Before

• Old origin Prow components and plugins

• ti-community-* external plugins based on old k8s.io/test-infra codes

• Hacked tide component base on https://github.com/kubernetes/test-infra/ at 2022-01-29 old

• Core CI Pipeline triggers controlled by Jenkins Controller (ci.pingcap.net, ci2.pingcap.net, do.pingcap.net/jenkins)

  • Most trigger comments were delivered to Jenkins, faked to Prow command styles.

• To avoid feature conflicts:

  • Enabled Require branches to be up to date before merging in Github branch protection.

  • Using ti-community-tar to fresh other opened pull requests when new commits pushed to base branch (another PR merged).

• TiChiBot lacks capabilities for batch merging pull requests:

  • Prow is old

  • Core CI pipelines are not controlled by itself.
    

After

• New origin Prow components and plugins: v202212*

• ti-community-* external plugins based k8s.io/test-infra with v0.0.0-202212*

• Hacked tide component based on https://github.com/kubernetes/test-infra/ at 2022-12-06

• Core CI Pipeline triggers controlled by TiChiBot(Prow backend), other heavy CI services will act as backend workers for Prow with Jenkins-Operator (for Jenkins) and pipeline (for TektonCD) third components

• TiChiBot will own capabilities for batch merging pull requests:

  • Upgrade to the latest features of Prow(include our contribution).

  • We made the CI pipelines able to be controlled by Prow:

    • We refactored core Jenkins pipelines(current focued on pingcap/tidb repo).

    • Pre-migration jobs were done: Prow Call Jenkins

      • Run it in staging env at least a month.

        • https://prow-ee.pingcap.net/?repo=PingCAP-QE%2Fee-ops&type=presubmit call

        • https://prow-ee.pingcap.net/jenkins/job/pingcap-qe/job/ee-ops/

  • Some changes will be made in repositories one by one:

    • In github branch protection rule:

      • Disable Require branches to be up to date before merging in Github branch protection.

      • Delete the required contexts from Jenkins CI servers.

    • Disable ti-community-tar plugin in repositorie.

• To avoid feature conflicts

  • Prow Tide test multi pull requests at same time(pre-merged together)

• Trigger CI jobs smartly with the ability of prow

  • By changing scopes

  • Blocked by label

  • …

• Better experience

  • For rerun failed jobs.

  • …

References

• TiChi bot:
  • Site: https://prow.tidb.net
  • Codes: https://github.com/ti-community-infra/tichi
  • Configuration: https://github.com/ti-community-infra/configs

[purelind]

Hi everyone, We are happy to announce that Take-Over2: TiChi controll post-merge CI builds on pingcap/tidb repo has been launched.
For more details, please see TiChi controll post-merge CI builds on pingcap/tidb repo.