Rewrite PD in Rust

#1

We need a PD catagory :slight_smile:

PD is an important component for TiKV (not TiDB), it should be written in rust and optionally embedded in TiKV executable.

6 Likes
#2

AFAIK, someone has been working on this.

#3

Where can I find the roadmap?

#4

I don’t think there’s any work being done on it yet, do you have a source?

#5

:shushing_face: There is no source.

#6

I have posted a similar idea in an internal discussion months ago. I suggested to rewrite PD in Rust and use Wasm to write the scheduling algorithm. So we can add/remove new scheduling policy to PD without restarting PD. And PD in Rust can reuse the raft-rs, which allows async writes and provides better performance.

4 Likes
#7

So cool, sounds exciting.

#8

Hi @BusyJay!

I can see that PD has already a plugin mechanism for scheduling strategy and also present a plugin demo.

I’d like to know that in your mind, do we implement a new plugin using cgo and load Rust Wasm manager which can add/remove new scheduling policy to PD without restarting PD?

Visually, my question is that for a plugin layout, do we have

  • Rust Wasm Manager Plugin
    • Rust Wasm Strategy 0
    • Rust Wasm Strategy 1
    • …

or

  • Rust Wasm Strategy Plugin 0
  • Rust Wasm Strategy Plugin 1
  • …

I’m not quite sure about both Golang Plugin and Rust Wasm but it seems to me the first version is reasonable.

cc @TennyZhuang @hundundm

#9

I don’t have a clear design for now. I was just exploring the possible benefits from rewriting PD in Rust. For example, if PD is written in Rust and we can embed it into TiKV, then plugins are just a type of coprocessors, they can share designs and even APIs. Being able to embed it into TiKV and get rid of one component can be a huge benefits for both deployment and maintenance.

#10

I think Jay’s idea is a distributed scheduling method, not just a centralized scheduling plug-in running on PD.

#11

But in fact, PD currently supports dynamic loading of Go plug-in shcedulers, but no customer wants to use it in a production environment.

#12

Can you elaborate the difference between

  • a distributed scheduling method
  • a centralized scheduling plug-in running on PD

I don’t get a clear image of them.

#13

I mean decentralized scheduling.

#14

sounds like a secret…

:rofl:

#15

IT’S COMING https://github.com/BusyJay/mini-pd

2 Likes
#16

by the way, is there any discuss on rewriting tidb in Rust ?

#17

You can send a post.