From malloc to physical pages: stringing together Linux memory management links
The first seven articles have broken down Linux memory management into several parts:
- Article 1What the process sees is the virtual address, not the physical memory.
- Article 2Physical memory is organized into nodes, zones, PFNs and
struct page. - Article 3Talk about the water level inside the zone,
free_area, buddy and recycling entrance. - Article 4speak
vm_area_structIt is the legal range table of the process address space. - Article 5Let’s talk about how VMA and page tables cooperate in page missing exceptions.
- Article 6Let’s talk about how slab/SLUB allocates space for small kernel objects.
- Article 7speak
task_struct,mm_struct, user stack, kernel stack, page table and VMA are organized on the process/thread.
This article no longer introduces new core structures, but uses a question to string them together:
once malloc When is the returned address just a virtual address, and when does it actually correspond to a physical page? On this road, where do VMA, page table, buddy, slab, and task/mm appear respectively?
Let’s put the conclusion first:
malloc pointer returned
│
│ First, user mode allocator the result
▼
Pass if necessary brk / mmap Extend process address space
│
│ This step mainly creates or expands VMA
▼
Really read and write a page for the first time
│
▼
CPU page walk Failure, triggering page fault exception
│
▼
kernel check VMA, Confirm that the address and permissions are legal
│
▼
Allocate user data pages and, if necessary, page table pages
│
▼
fill PTE: virtual page -> physical page frame
│
▼
The error command was re-executed and the access was successful this time.
The most confusing thing here is the two distribution paths:
User data page: page fault -> alloc_pages -> zone / buddy -> struct page -> PTE
Kernel metadata: vm_area_struct / task_struct / inode ...
-> kmem_cache_alloc -> slab / SLUB -> when necessary buddy Page
malloc The piece of memory returned to the user program is not an object directly cut out by slab. Slab manages the kernel's own small objects; user data pages usually come from buddy, but they are not actually honored until page faults occur.
1. Put the first seven articles into one general picture
The image below is intentionally large. It is not a precise source code call graph, but puts the structural relationships of the first seven articles into a dynamic path. Pure text images are used here to avoid relying on whether the Markdown previewer supports Mermaid.
┌──────────────────────────────────────────────────────────────────────────────┐
│ once malloc Complete link strung together │
└──────────────────────────────────────────────────────────────────────────────┘
User thread is executing
No. 7 Chapter: Thread is a task, Ordinary user threads have task_struct, User stack, kernel stack
│
▼
current task_struct
│
├─ task->stack --------------------------► current task kernel stack
├─ thread / thread_info -----------------► Architecture-related thread status
└─ task->mm
│
▼
mm_struct
│
├─ VMA tree / maple tree ----------► No. 4 Chapter: Legal interval table
│ │
│ ├─ code VMA
│ ├─ heap VMA
│ ├─ mmap anonymous VMA
│ ├─ file mapping VMA
│ └─ user stack VMA
│
└─ pgd ----------------------------► No. 5 Chapter: Page Table Root
│
▼
pgd / p4d / pud / pmd / pte
┌──────────────────────────────────────────────────────────────────────────────┐
│ 1. User mode malloc path │
└──────────────────────────────────────────────────────────────────────────────┘
malloc(size)
│
├─ small pieces, and libc arena There is space in
│ │
│ ▼
│ Only in user mode allocator Cut into a piece
│ Not necessarily into the kernel, not necessarily new VMA
│
└─ arena Not enough, or the allocation is too large
│
▼
brk / mmap system call
│
▼
Enter the kernel and find current->mm
│
▼
Create, expand, merge or split VMA
│
├─ need vm_area_struct metadata
│ │
│ ▼
│ kmem_cache_alloc(vm_area_struct cache)
│ │
│ ▼
│ slab / SLUB
│ │
│ └─ slab page When it is not enough, continue to buddy Page
│
▼
Return user virtual address p
Notice:
p Is the user virtual address. So far, VMA can already exist,
but p Each virtual page covered does not necessarily already exist present PTE.
┌──────────────────────────────────────────────────────────────────────────────┐
│ 2. The first access triggers a page fault path │
└──────────────────────────────────────────────────────────────────────────────┘
User code execution: p[i] = 1
│
▼
CPU / MMU check TLB
│
├─ TLB hit
│ │
│ ▼
│ Get the physical page frame directly and access successfully
│
└─ TLB miss
│
▼
hardware page walk page lookup table
│
├─ PTE present
│ │
│ ▼
│ Establish TLB, Access successful
│
└─ PTE not present / Permission exception
│
▼
page fault Enter the kernel
│
▼
current task_struct
│
▼
task->mm
│
▼
find_vma / lock_vma_under_rcu
│
├─ Can't find coverage address VMA
│ └─ SIGSEGV
│
├─ turn up VMA, But access permissions don't match
│ └─ SIGSEGV
│
└─ VMA Legal, permission allowed
│
▼
handle_mm_fault
│
├─ Anonymous page written for the first time
│ └─ Assign anonymous user page, clear, fill PTE
│
├─ Anonymous page read for the first time
│ └─ Possibly map shared zero page
│
├─ File mapping first access
│ └─ filemap_fault, try to find page cache, Read files if necessary
│
├─ fork back COW Write
│ └─ do_wp_page, Copy or reuse page, update PTE
│
└─ swap entry
└─ do_swap_page, Fill in after replacing PTE
┌──────────────────────────────────────────────────────────────────────────────┐
│ 3. Where does the user data page come from? │
└──────────────────────────────────────────────────────────────────────────────┘
Write missing pages anonymously / COW / Page table page allocation
│
▼
alloc_pages / folio_alloc
│
▼
choose NUMA node
│
▼
choose zone: DMA / DMA32 / Normal / Movable ...
│
▼
examine zone watermark: min / low / high
│
├─ Insufficient water level
│ │
│ ├─ wake kswapd
│ ├─ current thread direct reclaim
│ ├─ Recycle page cache Or swap out the anonymous page
│ ├─ compaction Organize consecutive pages
│ └─ If you still fail, you may go OOM / Allocate failed path
│
└─ water level is sufficient
│
▼
per-cpu pageset fast path
│
├─ There is a suitable page
│ └─ return page frame
│
└─ No suitable page
│
▼
buddy allocator
│
▼
free_area[order]
│
├─ current order There are free blocks
│ └─ take out 2^order consecutive pages
│
└─ current order No
│
▼
Find higher order
│
▼
Break into big pieces
│
├─ half return
└─ The remaining part hangs back low order linked list
│
▼
struct page Describe the resulting physical page
│
├─ User data page: mapped to p[i]
├─ Page table page: save PTE/PMD Equal page table entry
└─ slab page: Continue to cut objects to the kernel object allocator
┌──────────────────────────────────────────────────────────────────────────────┐
│ 4. How page tables deliver results to users │
└──────────────────────────────────────────────────────────────────────────────┘
Get user data page / page cache Page / COW new page / swap Switch to page
│
▼
Fill in intermediate page table pages if necessary
│
▼
fill in PTE
│
├─ present
├─ writable / readonly
├─ user / supervisor
├─ dirty / accessed
└─ PFN: Point to physical page frame
│
▼
refresh or update TLB Related status
│
▼
from page fault Return to user mode
│
▼
Re-execute the failed command just now: p[i] = 1
│
▼
This time the page table can be translated successfully, and the write actually falls on the physical page
┌──────────────────────────────────────────────────────────────────────────────┐
│ 5. slab What is this side responsible for? │
└──────────────────────────────────────────────────────────────────────────────┘
The kernel requires small objects
│
├─ vm_area_struct: describe a paragraph VMA
├─ task_struct: describe a task
├─ dentry / inode: file system object
└─ kmalloc-64 / kmalloc-512 Common small blocks
│
▼
kmem_cache_alloc
│
▼
SLUB per-cpu freelist
│
├─ freelist There are free objects
│ └─ Directly pop up the object and return quickly
│
└─ freelist no free objects
│
▼
Find the current CPU of partial slab
│
├─ CPU partial Available slab
│ └─ from slab page get an object
│
└─ CPU partial None or inappropriate
│
▼
Find the current NUMA node of partial slab
│
├─ node partial Available slab
│ └─ from slab page get an object
│
└─ node partial Not available either slab
│
▼
Towards buddy Request one or more pages
│
▼
Cut into fixed size objects
│
▼
Return one of the objects
Key division of labor:
User data page:
p[i] content
page fault -> alloc_pages -> buddy -> fill PTE
Kernel metadata object:
vm_area_struct / task_struct / dentry / inode ...
kmem_cache_alloc -> slab / SLUB -> when necessary buddy want slab page
page table page:
PTE/PMD The page where the page table entry is located
Page table allocation path -> buddy
If you compress the picture into one sentence, it is:
task_struct turn up mm_struct
mm_struct Tube VMA and page table root
VMA Determine whether the virtual address is legal
The page table records whether this page has been cashed
The page fault path allows legal virtual pages to be cashed into physical pages
buddy from zone page frame
slab exist buddy Cut kernel small objects on page
2. One malloc has at least three accounts
malloc This word tends to make people lump all memories together. A more accurate way to break it down is three accounts.
| ledger | Typical objects | who is responsible | when does it happen |
|---|---|---|---|
| User virtual address range | heap, anonymous mmap interval | VMA / mm_struct | brk, mmap, mprotect, munmap time changes |
| User data physical page | real storage p[i] page frame | buddy / struct page /page table | Cashed out when the first read or write triggers a page fault |
| Kernel metadata object | vm_area_struct, task_struct, file objects, etc. | slab / SLUB | allocated when the kernel needs to maintain the structure itself |
so malloc(128 * 1024 * 1024) After success, you cannot directly say "The kernel has allocated me 128MB of physical memory." More accurately:
- libc first decides whether this allocation should be done from the existing arena or through
brk/mmapExtended address space. - If you enter the kernel, the kernel usually changes the VMA first: making a range of virtual addresses legal.
- The page table only needs present mapping when the user actually touches a page.
- The page fault path allocates physical pages for user data pages and, if necessary, page table pages.
- When creating kernel objects such as VMA, file mapping, and tasks, small kernel objects may come from slab.
3. Small malloc, big malloc and first touch
glibc malloc Just a user-space allocator. It usually maintains its own arena; the internal structures of user-mode allocators such as arena, top chunk, fastbin, small bin, large bin, and unsorted bin can be compared with the "Why does multi-threaded malloc slow down - glibc's arena to bins panorama"Read the whole article.
malloc(32KB)
│
├─ arena There are free blocks in -> Return directly
└─ arena not enough -> possible brk expand heap, or mmap new area
Large allocations are more likely to go mmap:
malloc(128MB)
│
▼
libc tune mmap
│
▼
Kernel creates anonymous VMA
│
▼
Return user virtual address
But this VMA is just "this address can be used". It is not synonymous with 128MB physical page. The real user data page has to wait here:
p[i] = 1
│
▼
CPU page lookup table: PTE not present
│
▼
page fault
│
▼
find_vma: The address is at malloc correspond VMA Within, permissions allow writing
│
▼
Anonymous page fault handling
│
▼
alloc_pages from zone/buddy get physical page
│
▼
Clear, fill in PTE, renew RSS
│
▼
Re-execute p[i] = 1
If the memory pressure is very high, this path may also lead to recycling:
alloc_pages
│
▼
zone watermark insufficient
│
├─ wake kswapd
├─ current thread direct reclaim
├─ Recycle page cache / Anonymous page swap out
├─ compaction Try to organize consecutive pages
└─ If it still fails, the allocation fails or triggers OOM path
this isArticle 3The water level and recycling entrance in it return to the path of writing memory in user mode.
4. Don’t mix VMA metadata and user data
When creating an anonymous mapping, the kernel must do at least two types of things:
mmap
│
├─ create VMA metadata
│ └─ vm_area_struct -> kmem_cache_alloc -> slab
│
└─ Waiting for user access
└─ page fault -> alloc_pages -> buddy -> User data page
That is to say,vm_area_struct This object itself is kernel metadata, usually from slab; the user data page covered by VMA is not a slab object. When users write to anonymous memory, what they ultimately need is the page frame separated by buddy, and then the page table maps the virtual page there.
Page table pages are also a separate type of overhead:
User data page: save p[i] content
Page table page: save PTE/PMD Equal page table entry
VMA Object: Save the rules of this virtual interval
All three are combined once malloc Related, but they are not the same kind of memory.
5. Experiment: Observe malloc, VMA, fault, slab and buddy in one go
The following experiment does several things:
- Print machine architecture, kernel version, page size.
- Record
/proc/self/statusinsideVmSize,VmRSS,RssAnon,VmData. - use
getrusageLog minor/major faults. - statistics
/proc/self/mapsThe number of VMA rows. - from
/proc/self/smapsFind out the covered chunksmallocVMA for pointers,observationSizeandRss. - from
/proc/slabinfoobservevm_area_structcache. - from
/proc/buddyinfoSummarize the number of free blocks for several orders. - Create 1200 anonymous mappings with page size and alternating permissions to avoid VMA being merged to observe the impact of VMA on slab.
The complete code is as follows, also placed in the same directory 08-malloc-chain-demo.c:
#define _GNU_SOURCE
#include <errno.h>
#include <inttypes.h>
#include <malloc.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/resource.h>
#include <sys/utsname.h>
#include <unistd.h>
#define BUDDY_ORDERS 11
struct slab_row {
char name[96];
unsigned long active_objs;
unsigned long num_objs;
unsigned long objsize;
unsigned long objperslab;
unsigned long pagesperslab;
};
static void die(const char *what)
{
fprintf(stderr, "%s: %s\n", what, strerror(errno));
exit(1);
}
static unsigned long count_self_maps(void)
{
FILE *fp = fopen("/proc/self/maps", "r");
char line[512];
unsigned long count = 0;
if (!fp)
die("open /proc/self/maps");
while (fgets(line, sizeof(line), fp))
count++;
fclose(fp);
return count;
}
static void print_status_memory(const char *tag)
{
FILE *fp = fopen("/proc/self/status", "r");
char line[256];
if (!fp)
die("open /proc/self/status");
printf("[%s] status\n", tag);
while (fgets(line, sizeof(line), fp)) {
if (strncmp(line, "VmSize:", 7) == 0 ||
strncmp(line, "VmRSS:", 6) == 0 ||
strncmp(line, "RssAnon:", 8) == 0 ||
strncmp(line, "VmData:", 7) == 0) {
fputs(line, stdout);
}
}
fclose(fp);
}
static void print_faults(const char *tag)
{
struct rusage ru;
if (getrusage(RUSAGE_SELF, &ru) != 0)
die("getrusage");
printf("[%s] faults minor=%ld major=%ld\n",
tag, ru.ru_minflt, ru.ru_majflt);
}
static int read_slab_row(const char *cache, struct slab_row *out)
{
FILE *fp = fopen("/proc/slabinfo", "r");
char line[512];
if (!fp)
return -1;
while (fgets(line, sizeof(line), fp)) {
struct slab_row row;
if (line[0] == '#' || strncmp(line, "slabinfo", 8) == 0)
continue;
memset(&row, 0, sizeof(row));
if (sscanf(line, "%95s %lu %lu %lu %lu %lu",
row.name, &row.active_objs, &row.num_objs,
&row.objsize, &row.objperslab,
&row.pagesperslab) != 6)
continue;
if (strcmp(row.name, cache) == 0) {
*out = row;
fclose(fp);
return 0;
}
}
fclose(fp);
return 1;
}
static void print_slab_cache(const char *tag, const char *cache)
{
struct slab_row row;
int ret = read_slab_row(cache, &row);
if (ret < 0) {
printf("[%s] slab %-18s unavailable: %s\n",
tag, cache, strerror(errno));
return;
}
if (ret > 0) {
printf("[%s] slab %-18s not found\n", tag, cache);
return;
}
printf("[%s] slab %-18s active=%lu total=%lu objsize=%lu "
"objperslab=%lu pagesperslab=%lu\n",
tag, row.name, row.active_objs, row.num_objs, row.objsize,
row.objperslab, row.pagesperslab);
}
static void print_buddy_summary(const char *tag)
{
FILE *fp = fopen("/proc/buddyinfo", "r");
unsigned long long orders[BUDDY_ORDERS] = {0};
char line[512];
if (!fp) {
printf("[%s] buddy unavailable: %s\n", tag, strerror(errno));
return;
}
while (fgets(line, sizeof(line), fp)) {
char *p = strstr(line, "zone");
int order = 0;
if (!p)
continue;
p += 4;
while (*p == ' ' || *p == '\t')
p++;
while (*p && *p != ' ' && *p != '\t')
p++;
while (order < BUDDY_ORDERS) {
char *end;
unsigned long long value;
while (*p == ' ' || *p == '\t')
p++;
if (*p == '\0' || *p == '\n')
break;
errno = 0;
value = strtoull(p, &end, 10);
if (errno != 0 || end == p)
break;
orders[order++] += value;
p = end;
}
}
fclose(fp);
printf("[%s] buddy free blocks order0=%llu order1=%llu order2=%llu "
"order9=%llu order10=%llu\n",
tag, orders[0], orders[1], orders[2], orders[9], orders[10]);
}
static int parse_maps_header(const char *line, unsigned long *start,
unsigned long *end)
{
return sscanf(line, "%lx-%lx", start, end) == 2;
}
static void print_smaps_for_addr(void *addr, const char *tag)
{
FILE *fp = fopen("/proc/self/smaps", "r");
char line[512];
unsigned long target = (unsigned long)addr;
int in_target = 0;
int printed = 0;
if (!fp)
die("open /proc/self/smaps");
printf("[%s] smaps for %p\n", tag, addr);
while (fgets(line, sizeof(line), fp)) {
unsigned long start, end;
if (parse_maps_header(line, &start, &end)) {
in_target = (start <= target && target < end);
if (in_target) {
fputs(line, stdout);
printed = 1;
}
continue;
}
if (in_target &&
(strncmp(line, "Size:", 5) == 0 ||
strncmp(line, "Rss:", 4) == 0 ||
strncmp(line, "Private_Dirty:", 14) == 0 ||
strncmp(line, "Anonymous:", 10) == 0 ||
strncmp(line, "VmFlags:", 8) == 0)) {
fputs(line, stdout);
if (strncmp(line, "VmFlags:", 8) == 0)
break;
}
}
if (!printed)
printf("address is not covered by any current VMA\n");
fclose(fp);
}
static void snapshot(const char *tag)
{
print_status_memory(tag);
print_faults(tag);
printf("[%s] self_vma_count=%lu\n", tag, count_self_maps());
print_slab_cache(tag, "vm_area_struct");
print_buddy_summary(tag);
}
static void touch_each_page(char *p, size_t len, size_t page_size)
{
for (size_t i = 0; i < len; i += page_size)
p[i] = (char)(i / page_size);
}
static void advise_no_hugepage(void *addr, size_t len, size_t page_size)
{
uintptr_t raw = (uintptr_t)addr;
uintptr_t base = raw & ~((uintptr_t)page_size - 1);
size_t offset = raw - base;
size_t rounded = (offset + len + page_size - 1) & ~(page_size - 1);
if (madvise((void *)base, rounded, MADV_NOHUGEPAGE) != 0)
printf("madvise(MADV_NOHUGEPAGE) failed: %s\n", strerror(errno));
}
static void **create_sparse_vmas(int mappings, size_t page_size)
{
void **addr = calloc((size_t)mappings, sizeof(*addr));
if (!addr)
die("calloc vma table");
for (int i = 0; i < mappings; i++) {
int prot = (i % 2 == 0) ? PROT_NONE : PROT_READ;
addr[i] = mmap(NULL, page_size, prot,
MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE, -1, 0);
if (addr[i] == MAP_FAILED) {
fprintf(stderr, "mmap %d failed: %s\n", i, strerror(errno));
exit(1);
}
}
return addr;
}
static void destroy_sparse_vmas(void **addr, int mappings, size_t page_size)
{
for (int i = 0; i < mappings; i++) {
if (munmap(addr[i], page_size) != 0) {
fprintf(stderr, "munmap %d failed: %s\n", i, strerror(errno));
exit(1);
}
}
free(addr);
}
int main(int argc, char **argv)
{
struct utsname uts;
size_t page_size = (size_t)sysconf(_SC_PAGESIZE);
size_t big_len = 128UL * 1024 * 1024;
int mappings = 1200;
char *small;
char *big;
void **sparse_vmas;
if (argc > 1)
big_len = (size_t)strtoull(argv[1], NULL, 0) * 1024 * 1024;
if (argc > 2)
mappings = atoi(argv[2]);
if (big_len == 0 || mappings <= 0) {
fprintf(stderr, "usage: %s [big_mib] [vma_count]\n", argv[0]);
return 2;
}
if (uname(&uts) != 0)
die("uname");
mallopt(M_MMAP_THRESHOLD, 128 * 1024);
printf("machine=%s sysname=%s release=%s\n",
uts.machine, uts.sysname, uts.release);
printf("page_size=%zu big_len=%zu bytes (%zu MiB) sparse_vmas=%d\n",
page_size, big_len, big_len / 1024 / 1024, mappings);
snapshot("start");
small = malloc(32 * 1024);
if (!small)
die("malloc small");
memset(small, 0x5a, 32 * 1024);
printf("small malloc ptr=%p len=32768\n", small);
snapshot("after small malloc+touch");
big = malloc(big_len);
if (!big)
die("malloc big");
printf("big malloc ptr=%p requested=%zu usable=%zu\n",
big, big_len, malloc_usable_size(big));
advise_no_hugepage(big, big_len, page_size);
snapshot("after big malloc before touch");
print_smaps_for_addr(big, "after big malloc before touch");
touch_each_page(big, big_len, page_size);
snapshot("after touching big malloc");
print_smaps_for_addr(big, "after touching big malloc");
printf("creating %d one-page VMAs with alternating permissions\n",
mappings);
sparse_vmas = create_sparse_vmas(mappings, page_size);
snapshot("after sparse mmap");
destroy_sparse_vmas(sparse_vmas, mappings, page_size);
snapshot("after sparse munmap");
free(big);
snapshot("after free big malloc");
free(small);
snapshot("after free small malloc");
return 0;
}
Compile and run command:
docker run --rm --platform linux/amd64 \
-v /Users/xyzjiao/article:/work -w /work \
gcc:13 \
bash -lc 'gcc -O2 -Wall -Wextra os/memory/08-malloc-chain-demo.c -o /tmp/malloc-chain-demo && /tmp/malloc-chain-demo 128 1200'
The actual output I ran in an x86_64 container is as follows:
machine=x86_64 sysname=Linux release=6.12.65-linuxkit
page_size=4096 big_len=134217728 bytes (128 MiB) sparse_vmas=1200
[start] status
VmSize: 287404 kB
VmRSS: 4552 kB
RssAnon: 2548 kB
VmData: 284548 kB
[start] faults minor=1967 major=0
[start] self_vma_count=33
[start] slab vm_area_struct active=3034 total=3302 objsize=152 objperslab=26 pagesperslab=1
[start] buddy free blocks order0=6093 order1=3345 order2=4226 order9=10 order10=12
small malloc ptr=0x4068a0 len=32768
[after small malloc+touch] status
VmSize: 287432 kB
VmRSS: 4676 kB
RssAnon: 2608 kB
VmData: 284576 kB
[after small malloc+touch] faults minor=1981 major=0
[after small malloc+touch] self_vma_count=33
[after small malloc+touch] slab vm_area_struct active=3034 total=3302 objsize=152 objperslab=26 pagesperslab=1
[after small malloc+touch] buddy free blocks order0=6093 order1=3345 order2=4226 order9=10 order10=12
big malloc ptr=0x7ffff75dc010 requested=134217728 usable=134221808
[after big malloc before touch] status
VmSize: 418508 kB
VmRSS: 4680 kB
RssAnon: 2612 kB
VmData: 415652 kB
[after big malloc before touch] faults minor=1982 major=0
[after big malloc before touch] self_vma_count=34
[after big malloc before touch] slab vm_area_struct active=3034 total=3302 objsize=152 objperslab=26 pagesperslab=1
[after big malloc before touch] buddy free blocks order0=6093 order1=3345 order2=4226 order9=10 order10=12
[after big malloc before touch] smaps for 0x7ffff75dc010
7ffff75dc000-7fffff5dd000 rw-p 00000000 00:00 0
Size: 131076 kB
Rss: 4 kB
Private_Dirty: 4 kB
Anonymous: 4 kB
VmFlags: rd wr mr mw me ac nh
[after touching big malloc] status
VmSize: 418512 kB
VmRSS: 135752 kB
RssAnon: 133684 kB
VmData: 415656 kB
[after touching big malloc] faults minor=34750 major=0
[after touching big malloc] self_vma_count=34
[after touching big malloc] slab vm_area_struct active=3033 total=3302 objsize=152 objperslab=26 pagesperslab=1
[after touching big malloc] buddy free blocks order0=5239 order1=2782 order2=1571 order9=10 order10=12
[after touching big malloc] smaps for 0x7ffff75dc010
7ffff75dc000-7fffff5dd000 rw-p 00000000 00:00 0
Size: 131076 kB
Rss: 131072 kB
Private_Dirty: 131072 kB
Anonymous: 131072 kB
VmFlags: rd wr mr mw me ac nh
creating 1200 one-page VMAs with alternating permissions
[after sparse mmap] status
VmSize: 423312 kB
VmRSS: 135932 kB
RssAnon: 133864 kB
VmData: 415656 kB
[after sparse mmap] faults minor=34795 major=0
[after sparse mmap] self_vma_count=1234
[after sparse mmap] slab vm_area_struct active=4238 total=4238 objsize=152 objperslab=26 pagesperslab=1
[after sparse mmap] buddy free blocks order0=5239 order1=2751 order2=1571 order9=10 order10=12
[after sparse munmap] status
VmSize: 418516 kB
VmRSS: 135936 kB
RssAnon: 133868 kB
VmData: 415660 kB
[after sparse munmap] faults minor=34796 major=0
[after sparse munmap] self_vma_count=34
[after sparse munmap] slab vm_area_struct active=4238 total=4238 objsize=152 objperslab=26 pagesperslab=1
[after sparse munmap] buddy free blocks order0=5239 order1=2745 order2=1543 order9=10 order10=12
[after free big malloc] status
VmSize: 287440 kB
VmRSS: 4864 kB
RssAnon: 2796 kB
VmData: 284584 kB
[after free big malloc] faults minor=34796 major=0
[after free big malloc] self_vma_count=33
[after free big malloc] slab vm_area_struct active=4238 total=4238 objsize=152 objperslab=26 pagesperslab=1
[after free big malloc] buddy free blocks order0=6989 order1=4298 order2=4203 order9=10 order10=12
[after free small malloc] status
VmSize: 287440 kB
VmRSS: 4864 kB
RssAnon: 2796 kB
VmData: 284584 kB
[after free small malloc] faults minor=34796 major=0
[after free small malloc] self_vma_count=33
[after free small malloc] slab vm_area_struct active=4238 total=4238 objsize=152 objperslab=26 pagesperslab=1
[after free small malloc] buddy free blocks order0=6989 order1=4298 order2=4203 order9=10 order10=12
6. How to read this set of results
First, the experiment did run in the x86_64 container:
machine=x86_64 sysname=Linux release=6.12.65-linuxkit
page_size=4096
Second, small pieces malloc(32KB) No new VMAs were made:
[start] self_vma_count=33
[after small malloc+touch] self_vma_count=33
VmSize and VmData There are only minor changes, indicating that this type of allocation is libc arena behavior in the first place. It does not necessarily correspond to a new mmap, and it should not be understood as slab cutting an object for the user.
Third, big chunks malloc(128MB) After that, the virtual address space has obviously become larger, but RSS has not followed suit:
[after big malloc before touch] VmSize: 418508 kB
[after big malloc before touch] VmRSS: 4680 kB
cover big pointer smaps Also says the same thing:
Size: 131076 kB
Rss: 4 kB
here Size is the virtual range of the VMA,Rss It is the physical page that has already resided. Just malloc Only 4KB of RSS at all, mostly allocator metadata or a handful of touched pages, not 128MB all falling into physical memory.
Fourth, after writing page by page, the RSS increased to nearly 128MB:
[after touching big malloc] VmRSS: 135752 kB
[after touching big malloc] smaps Rss: 131072 kB
minor fault also comes from 1982 increase to 34750, the increase is 32768, corresponding to the number of pages of 128MiB / 4KiB:
34750 - 1982 = 32768
In the experiment, I did this to this VMA MADV_NOHUGEPAGE, so this output is closer to the teaching model of "one 4KB page touch corresponds to one minor fault". Without this hint, modern kernels may cause the number of faults to no longer strictly equal the number of 4KB pages due to THP or large folio. The core conclusions remain unchanged:The VMA exists first and the physical page is honored via the page fault path on the first touch.
Fifth, creating a large number of VMAs can affect vm_area_struct cache:
[after touching big malloc] self_vma_count=34
[after sparse mmap] self_vma_count=1234
[after touching big malloc] slab vm_area_struct active=3033 total=3302
[after sparse mmap] slab vm_area_struct active=4238 total=4238
This shows mmap Not just "give the user an address." The kernel also maintains VMA metadata, which vm_area_struct Objects come from slab/SLUB.
sixth,munmap After that, the VMA count of the current process goes back, but /proc/slabinfo It does not necessarily return to the original value immediately:
[after sparse munmap] self_vma_count=34
[after sparse munmap] slab vm_area_struct active=4238 total=4238
This is not a contradiction./proc/slabinfo It is a global view, not a private ledger of the current process; SLUB, RCU delayed release, cache reserved free objects, and other process activities will all affect it. The conclusion that should be read here is:The number of process VMAs and the global slab cache are not a one-to-one instantaneous count, but making a large number of VMAs will clearly drive vm_area_struct cache growth.
Seventh, the buddy number can only assist observation:
[after big malloc before touch] buddy free blocks order2=4226
[after touching big malloc] buddy free blocks order2=1571
[after free big malloc] buddy free blocks order2=4203
/proc/buddyinfo It is also a global view, and other allocations, recycling, merging, and splitting in the system will change it. Do not regard the difference in a certain order as the exact number of pages consumed by the program this time. It is suitable to assist understanding: after touching the anonymous page, it will indeed put pressure on the physical page allocator; after releasing the large block mapping, the physical page can be returned to the free pool managed by buddy.
7. Review the previous seven chapters chapter by chapter and return to this path.
| Previous article | this time malloc position in path |
|---|---|
| Part 1: Virtual addresses are not memory | malloc returned 0x7ffff75dc010 Is the user virtual address;smaps Size Getting bigger does not mean physical pages have been allocated |
Article 2: node, zone, PFN,struct page | After an anonymous page fault, the physical page frame is struct page Description and belongs to a zone/node |
| Chapter 3: zone, water level, buddy | alloc_pages To check the zone water level, start from buddy's free_area[order] Take pages; may recycle when stressed |
Article 4:vm_area_struct | chunks malloc Corresponds to anonymous VMA; large number mmap Will make lots of VMAs |
| Part 5: VMA and Page Tables | smaps Size=131076KB, Rss=4KB It is the performance of "VMA already exists, but the page table has not been fully realized" |
| Article 6: slab/SLUB | vm_area_struct Such kernel metadata objects are managed by slab cache, not user data pages |
| Part 7: task/mm/stack/page table | The current thread passes current->mm find the same one mm_struct, and then enter the VMA and page table paths |
This table is actually the final map of the entire series:
user thread
│
▼
task_struct
│
▼
mm_struct
│
├─ VMA: Is the virtual interval legal?
│
└─ pgd: Page table root
│
▼
page fault: Bundle VMA Promise fulfillment into page table mapping
│
▼
buddy: Assign user page / page table page
│
▼
struct page: Describe the physical page frame
other side:
vm_area_struct / task_struct / inode / dentry
│
▼
slab / SLUB
│
▼
buddy supply slab page
8. Several points that are easy to make mistakes in the end
First,malloc Success does not mean that all physical pages have been allocated.
Large chunks of anonymous memory often first appear as VmSize Increase. Only after access, the RSS and minor faults increased significantly.
second,VMA is not a page table.
The VMA says "this virtual address can be used like this"; the page table says "which physical page frame this page is now translated to".smaps Size and Rss The difference is the observable version of this sentence.
third,slab is not the backend of user-space malloc.
User mode malloc The backend is libc arena plus brk / mmap. slab is the kernel object allocator, giving vm_area_struct, task_struct, dentry, inode, kmalloc-* Used by other objects.
fourth,buddy manages pages, slab manages objects.
slab also requires pages, it does not bypass buddy. The difference is that buddy answers "which zone to get a few pages from" and slab answers "which object slot on this page can be used by the kernel."
fifth,Threads share address space, which does not mean sharing all execution contexts.
Threads in the same process usually share mm_struct, VMA and page table, so a thread mmap The area can also be seen by other threads. But each thread still has independent task_struct, user stack and kernel stack.
sixth,/proc The observation port should distinguish between the process view and the global view..
/proc/self/maps, /proc/self/smaps Is the current process address space view;/proc/slabinfo, /proc/buddyinfo Is the global kernel allocator view. The former is suitable for observing VMA/RSS, while the latter is suitable for observing trends, but not suitable for precise accounting of a single process.
9. Ending of the whole series
If you only remember one line, just remember this one:
pointer value
│
▼
User virtual address
│
▼
VMA Determine whether it is legal
│
▼
Page table determines whether it has been honored
│
▼
Page fault path allocation or physical page found
│
▼
struct page Description page frame
│
▼
zone / buddy Manage free page sources
│
▼
slab Cut kernel small objects on the page
│
▼
task_struct / mm_struct Hook it all back to the current thread and process
Linux memory management is not a straight line of "malloc directly accessing physical memory", but a set of ledger collaboration:
- The user-mode allocator manages the small block allocations seen by user programs.
- VMA manages whether the virtual address range is legal.
- The page table manages the page-level mappings that have been fulfilled.
- buddy manages the physical page frame.
- slab tube kernel small object.
task_structandmm_structAttach these structures to the current execution context.
Look at it like this again p = malloc(128 * 1024 * 1024); p[0] = 1;, behind it is no longer a sentence of "applying for memory", but a complete link: the user mode allocator may expand the VMA, the CPU's first access triggers a page fault, the kernel checks the VMA and page table, gets the physical page from the buddy, establishes the PTE, and finally allows the same user instruction to be re-executed successfully.