TiDB Data Migration (DM) is an integrated data migration task management platform, which supports data migration from MySQL-compatible databases (such as MySQL, MariaDB, and Aurora MySQL) into TiDB.
We would like to put DM code into TiCDC repo due to these reasons:
- DM has more similarities with TiCDC in processing logic, the main function of both tools is to synchronize in real time the specified type of upstream (MySQL, MariaDB, TiDB…) data flow to downstream (MySQL, MariaDB, TiDB, Kafka…) . Currently, the two tools are independent and have independent implementations for upstream and downstream data interfaces and internal data flow logic processing. So after merging these two repos, it will help code integration and reduce maintenance costs.
- The merge of DM and TiCDC can help reduce the product concepts and provide a unified data platform for users to use and reduce the cost for users of learning to use.
Steps below migrates minimal DM codebase to TiCDC, which allow DM build and release from the TiCDC repository. (Even not the latest version.)
git checkout -b clone-dm && git subtree add -D dm https://github.com/pingcap/dm.git
git checkout -b merge-dm (to create a diff branch, so the PR would be easier to review…)
Do necessary merging.
- merge go.mod, go.sum
- update Makefile,
- update the import path for both TiCDC and DM.
- merge CI scripts.
Create a PR from merge-br targeting clone-br for reviewing.
After reviewing, merge it to master.
The change in TiCDC is add a new directory
dm, so the risk is under control.