Tikv_alloc stats and snmalloc

#1

Hi, I happen to be the maintainer of snmalloc-rs and it may be possible to add support of snmalloc to tikv_alloc, whose performance is at the same level of mimalloc and seems to perform a little bit better in some async runtimes in my previous test; but it is implemented in C++, so extra c++ runtime is needed (libatomic, libstdc++).

However, I am little confused about the current allocator configuration. It seems that jemalloc has a primitive support for stats (I say primitive support because I see lots of TODOs there) but it is not the case for other allocators.

In fact, both mimalloc and snmalloc has comprehensive builtin support for stats, it should be easy to utilize the stats FFIs. So, is the stats feature used heavily or it is not that important?

#2

Hi! We use jemalloc by default and all other supported allocators are experimental. The jemalloc stats was added to monitor memory fragmentation and also useful to confirm TiKV memory usage. To support new allocator, it would be great to add stats for them, but it is optional.

1 Like
#3

Thanks for your information. I will work on it when my current assignments settled.

#4

By the way, jemalloc provides heap profiling which is also extremely helpful in my own experience to find memory leaks or investigate OOMs. It seems that other custom allocators don’t have similar features.

#5

I believe such APIs are exposed https://microsoft.github.io/mimalloc/group__analysis.html.
Yet,writing a tool like jemallocctl may still be a very tricky and tedious work.

#6

see also https://github.com/microsoft/snmalloc/blob/master/src/mem/allocstats.h

#7

It looks a little different. By traversing all allocated blocks, we still don’t have enough information about what these objects are and where they were allocated.

Jemalloc’s heap profiling gets a backtrace every time it allocates, so we can know where the used memory are from.

#8

Just get back to track. I think it great to add some tests (to CI) or benchmarks to allocators.
Any existing test? or did the CI try to run some tests (i.e. critical routines) with different allocators?

#9

It seems to be me the current allocator organization is a little bit messy: I mean too many feature flags, crossing with each other, without a specific doc.
It should be great if we can provide a document tracking the allocator API usage within the project and record why it is crucial for some crates to use specialized allocator and why it is not for others.

#10

Sorry for the mess. We really don’t pay much attention to it.

Except for the default allocator (jemalloc), other allocators are not tested by CI or in daily tests. So most developers and users should only use jemalloc. The other options exist only for people who are interested in trying out the different allocators (probably testing if a newer version has significant improvement).

Unless another allocator really show its edge over the current default one in benchmarks, we will still stick to jemalloc. There is no resource to test with all the options regularly.