Proposal: TiKV Development Guide

#1

author: tison

Introduction

There are more and more contributors asking for a guide about how to participant in TiKV development activities.

In order to participant in TiKV development activities, a contributor is supposed to know the following topics:

  1. How to set up a development environment.
  2. How to follow the development process.
  3. Architecture, design, and implementation of the project.

Contributors using TiKV in production may also want to know about the project’s management especially how it releases.

Layout

TiKV is released simultaneously with TiDB, and TiDB already has its development guide containing a project management chapter. I propose setting up a reference from TiKV Development Guide to this chapter and that is enough.

Below describe the details about the rest three topics.

Get Started

This chapter answers the question “How to set up a development environment”.

  • Build TiKV from Source
  • Import TiKV into an IDE
  • Write and Run Unit Tests
  • Debug and Profile
  • Submit a Pull Request

Available resource:

Contribute to TiKV

This chapter answers the question “How to follow the development process”.

  • Community Guideline
  • Committer Guide
  • Report a Bug
  • Contribute code
  • Review a Pull Request
  • Request for Comments
  • Code Style and Quality Guide
  • Write document

Available resource:

Understanding TiKV

This chapter delivers knowledge about architecture, design, and implementation of the project.

  • RocksDB
  • Raft
  • Raft KV
  • MVCC
  • Transaction
  • Coprocessor

Available resource:

Prototype

You can see a prototype of TiKV Development Guide here.

Looking forward to your feedback.

4 Likes
#2

I feel good on most chapters and the only suggestion is on the Understanding TiKV, I propose the following:

  • Overview: The architecture.

    • RawKV: How raw get and raw put are processed.
    • TxnKV: How 2PC works, how prewrite and commit are processed. (only focus on the optimistic transaction)

  • Scalability

    • Shard: what is a region, what is split/merge, and the conditions
    • Scheduling: scoring, operations

  • High availability

    • Byzantine Failure
    • Raft: what is Raft, how it works
      • raft-rs
      • Joint consensus
      • Lease
      • Hibernate region
    • Multi-Raft: region epoch, conf version
      • Split
      • Merge: PrepareMerge, CommitMerge, RollbakMerge
    • RaftStore: batch system, actor model, tick-driven loop
      • FSM
      • write batch

  • Transaction

    • Percolator: consistency across multiple regions
      • MVCC: encode, region boundary issues, key adjustment on region split
      • TSO: how pd tso works, how global/local tso works
    • Concurrency control: consistency within single region
      • Commands: prewrite, commit, check_txn_status
      • Latch
      • Scheduler
    • Pessimistic transaction:
      • Pessimistic lock
      • deadlock detector
    • Async commit: the model, the resolve process, the fallback process
    • 1PC: memory locks
    • Stale read
    • CDC

  • Storage

    • RocksDB: write batch, compaction, snapshot
    • Import / Backup / Restore: SST file, ingest
    • IO ratelimiter

  • Coprocessor

    • vectorized execution framework
    • Coprocessor plugin

  • Client

    • Sharding: region cache, retry strategy (e.g. no_jitter)
    • Unified integration test: jepsen test
4 Likes
#3

@andylokandy thanks for your reply! I think the content is well in any form at first. Since the valuable thing is its content. Would you mind to create a PR against

… to update the layout - we don’t have to create dummy file though.

#4

I created a PR for this.

1 Like
#5

Thanks! I will review it today.

#6

Hi, tison
Is there any update recently about debugging tikv ?–>Debug and Profile
lots of people including me need it, really、really、really！！！ :laughing:
I want to use it to understand how point_get run in tikv and jump into from different code blocks.

I have tried rust-gdb or vscode + lldb on Mac M1，but not successful.

for example :



And

this is my failure steps --> How to debug tikv? experiment vscode + lldb failed