Back at 42, we were speedrunning the exams. You know how it goes: you get a C problem, you solve it, you move on. Time is everything.

I was halfway through the final exam, the hardest one, carefully managing my memory like a good student. malloc the exact size, realloc when needed, free at the end. Proper stuff.

Then my mate leans over and says: “Bro, just allocate 16 gigs.”

“What?”

“16 gigs. For a string buffer. It works.”

I thought he was insane. The exam machines had maybe 8 GB of RAM. But we were speedrunning, so I tried it. And it worked. I finished that exam in 15 minutes.

That’s when I learned that Linux will say yes to almost anything.

#include <stdlib.h>
#include <stdio.h>

int main() {
    void *ptr = malloc(1024ULL * 1024 * 1024 * 1024 * 1024);
    printf("%p\n", ptr);
    return 0;
}

Run it.

If you’re on Linux, you got a valid pointer back, not NULL, a real address.

My laptop has 32 GB of RAM and it just told me “oui, 1 petabyte, pas de problème.”

Your OS Is a Filthy Liar 🔗

When you call malloc, you’re not asking for RAM, you’re asking for virtual memory, which is just a number, an address, a promise that the kernel makes to you.

The kernel says: “Sure, here’s your pointer, and if you ever touch this memory, I’ll find some RAM for you, probably, inch’Allah.”

This is called overcommit and Linux does it by default because the kernel doesn’t check if 1 PB of RAM exists, it just vibes it out.

$ cat /proc/sys/vm/overcommit_memory
0

That 0 means heuristic mode where the kernel looks at your request, thinks about it for a moment, shrugs, and says “seems fine to me.”

1 petabyte passes the vibe check.

The Three Flavors of Delusion 🔗

Value Mode What It Means
0 Heuristic “Bof, why not”
1 Always “Yes to everything”
2 Strict Checks for real

Let’s try mode 2:

$ sudo sysctl vm.overcommit_memory=2
$ ./malloc_1pb
(nil)

Now it says no, but nobody runs mode 2 because the default is mode 0.

What If You Actually Use It? 🔗

Getting a pointer is easy but let’s try to write to it:

void *ptr = malloc(1024ULL * 1024 * 1024 * 1024 * 1024);
printf("Got pointer: %p\n", ptr);
memset(ptr, 0, size);

Open another terminal and run this:

$ dmesg -w

You’ll see:

[  234.567890] Out of memory: Killed process 12345 (malloc_1pb)

The OOM killer showed up and shot your process in the head without a trial.

Here’s the sequence of events:

  1. malloc hands you a virtual address
  2. memset starts writing zeros
  3. Each page you touch makes the kernel scramble for real RAM
  4. The kernel runs out of RAM and swap space
  5. The OOM killer picks a victim
  6. SIGKILL, no appeal, no cleanup, c’est la vie

The OOM Killer Has No Loyalty 🔗

The OOM killer doesn’t always kill the process that caused the problem because it uses a badness score to pick its victim, which means it might kill your database instead of your test script.

# Make a process immune (don't do this)
echo -1000 > /proc/<pid>/oom_score_adj

# Make a process the first to die
echo 1000 > /proc/<pid>/oom_score_adj

Fun game: start two memory hogs, give one immunity, watch the other explode.

Other Systems 🔗

macOS is more conservative and returns NULL for absurd sizes, which is almost reasonable.

Windows uses a two-phase system with VirtualAlloc where you first MEM_RESERVE the address space and then MEM_COMMIT to back it with real memory.

32-bit systems can’t fit 1 PB in a 4 GB address space so math wins.

calloc Is Sneakier 🔗

void *ptr = calloc(1ULL << 50, 1);

This is not the same as malloc plus memset because the kernel has a trick called the zero page, which is one physical page full of zeros.

When you calloc, all your virtual pages point to this same zero page with copy-on-write, so calloc can “succeed” at giving you 1 PB of zeros without using any real memory.

You only pay when you write:

char *ptr = calloc(1ULL << 40, 1);
ptr[0] = 'A';     // one real page allocated
ptr[4096] = 'B';  // another one
// keep going and eventually boom

Finding the Limit 🔗

for (int exp = 30; exp < 64; exp++) {
    size_t size = 1ULL << exp;
    void *ptr = malloc(size);
    printf("2^%d: %s\n", exp, ptr ? "oui" : "non");
    free(ptr);
}

On my machine:

2^40 (1 TB): oui
2^45 (32 TB): oui
2^46 (64 TB): oui
2^47 (128 TB): non

The wall isn’t RAM, it’s the virtual address space because x86_64 uses 48-bit addresses which gives you 128 TB for userspace, and after that even the lie falls apart.

Why This Matters 🔗

Sparse structures let you allocate a huge array and only use 0.01% of it because only the pages you touch cost real RAM.

fork() needs this because the child gets a copy-on-write copy of the parent’s memory, and without overcommit every fork would need 2x the RAM upfront which would make forking a gamble.

The footgun is that your program allocates memory slowly and malloc keeps succeeding and everything looks fine until 3 AM when the OOM killer shoots your production database, and you spend the morning reading kernel mailing list archives and questioning your life choices.

Try It 🔗

// chaos.c
#include <stdlib.h>
#include <stdio.h>
#include <string.h>

int main(int argc, char *argv[]) {
    int exp = argc > 1 ? atoi(argv[1]) : 50;
    size_t size = 1ULL << exp;

    printf("malloc(2^%d)...\n", exp);
    void *ptr = malloc(size);

    if (!ptr) {
        printf("Refused.\n");
        return 1;
    }

    printf("Got %p. Press Enter to touch it.\n", ptr);
    getchar();

    memset(ptr, 0xFF, size);
    printf("Survived?\n");

    return 0;
}

Run it, watch dmesg, and when the OOM killer takes out Firefox instead of your test program, that’s on you.

Further Reading 🔗