Tracking issue: https://github.com/pingcap/tidb/issues/28775
Introduction
Dumpling is a tool and a Go library for creating SQL dump from a MySQL-compatible database. It is intended to replace mysqldump and mydumper when targeting TiDB.
Motivation
There are three primary reasons to merge:
-
DM suffers from dependency problem about TiDB and Dumpling. Concretely, DM depends on TiDB and Dumpling, while Dumpling depends on TiDB too. Every time DM updates Dumpling’s version, TiDB’s version will be updated along with Dumpling, which is unexpected.
-
Regression during a release. Dumpling releases along with TiDB even though they are in two repos. This results in conflicts are only exposed when testing failed during a release. For example, TiDB adds a new feature, which may cause a fail in Dumpling, which Dumpling isn’t aware of. This issue cannot be found on their own tests, but the integration test of Dumpling and TiDB. This is very troublesome!
-
Reduce development costs. In TiDB, there is a function
export SQLof TiDB similar with Dumpling.
After merged, we will:
- Get rid of the DM’s dependency problem;
- Test in a monorepo and thus expose problem earlier;
- Reduce duplicated code such as implementing
export SQLfor TiDB with Dumpling.
Milestones
- [x] Milestone 1: Merge master branch of Dumpling into TiDB repo.
- [ ] Milestone 2: Keep maintaining released branches until they are end of maintenance; e.g., release-4.0.
- [ ] Milestone 3: Migrate all useful content from Dumpling to TiDB repo, e.g. issues. And then archives Dumpling repo.
Milestone 1
This section refers to Proposal: Merge BR repo into TiDB.
To achieve milestone 1, we should do as follow:
git checkout -b clone-dumpling && git subtree add --prefix=dumpling https://github.com/pingcap/dumpling.git master --squash-
git checkout -b merge-dumpling(we will update code in this branch) - Do necessary merging:
- merge go.mod, go.sum;
- update Makefile;
- update the import path for DM and Dumpling;
- merge CI scripts.
- Create a MR from
okJiang:merge-dumplingtookJiang:clone-dumplingfor reviewing locally: (link if create) - Create true PR from
okJiang/merge-dumplingtopingcap:master(link if create) - After reviewing, merge it to master
Milestone 2
In this stage, we should do lots of maintenance in both repos:
- All increment activities happened in TiDB repo and give some corresponding guidance.
- Release new version (>= 5.3) from TiDB repo.
- Maintain existing active branches released for TiDB repo and Dumpling repo.
What increment activities should happen in TiDB repo?
- All created PRs, issues about Dumpling;
- When we want to fix an existing issue, we can create a new issue in TiDB repo with a link to original issue.
What guidance should we give?
- Post migrated notice on README.md like https://github.com/pingcap/br/blob/master/README.md
- When contributors create issues or PRs, we should tell them how to re-create in TiDB repo.
How do we release new version from TiDB repo?
- Follow TiDB’s rule: https://pingcap.github.io/tidb-dev-guide/project-management/release-train-model.html
How do we maintain former active released branches?
This section refers to Proposal: Merge BR repo into TiDB
If we want to cherry-pick the specific commit <COMMIT_SHA> to Dumpling repo. DO THE FOLLOWING THINGS
- if the <COMMIT_SHA> not in <SPLIT_BRANCH>:
- In TiDB repo:
- git subtree split --prefix=dumpling -b <SPLIT_BRANCH>
- git checkout <SPLIT_BRANCH>
- git push <DUMPLING_REPO_REMOTE> <COMMIT_SHA>:refs/heads/<SPLIT_BRANCH>
- In TiDB repo:
- In Dumpling repo:
- git fetch origin <SPLIT_BRANCH>
- git checkout master
- git checkout -b <pick_from_tidb>
- git cherry-pick <COMMIT_SHA>
- Give a PR of merge <pick_from_tidb> to master.
We will maintain release 4.0, 5.0, 5.1, 5.2 in Dumpling repo.
Milestone 3
After all releases(<= 5.2) end of maintenance, it is time to archive Dumpling repo. But before the truly end, we should do some closing work:
- Migrate issues from Dumpling to TiDB
- …