How Syscalls Work?

I was always curious about what actually happens after stepping over the syscall instruction in GDB. Unlike a normal function call, GDB doesn’t step into it, instead, it steps over the instruction. Yet, something significant has happened by the time execution returns (the syscall we requested has already been processed). While learning kernel exploitation, I noticed that GDB running on QEMU lets me smoothly step into the syscall instruction and follow what happens inside the kernel, going through all the different stages of the syscall path.

I was curious to trace this entire process back through the kernel source code. Later, I thought it would be useful to document the call chain here for my future self. So, below is the series of calls that happened during the syscall.

Step 1 — the SYSCALL instruction

One thing that initially confused me was that SYSCALL isn’t really a function call. There’s no address sitting next to the instruction telling the CPU where to jump. Instead, the CPU already knows where the kernel’s syscall entry point is because Linux sets it up earlier during boot using an MSR (a special CPU register):

// arch/x86/kernel/cpu/common.c:2266
wrmsrq(MSR_LSTAR, (unsigned long)entry_SYSCALL_64);

MSR_LSTAR basically tells the CPU: “Whenever you execute _SYSCALL_, jump to this address.”

In this case, that address is entry_SYSCALL_64. So every SYSCALL instruction eventually lands there and switches from userspace (ring 3) to the kernel (ring 0).

The interesting part is how little the CPU actually does here. The hardware doesn’t save the whole register state or set up a stack for us. It only does a few things:

• saves the return address — the instruction immediately after SYSCALL — in rcx
• saves the current rflags in r11
• loads the new cs, ss, and rip from the corresponding MSRs

And that’s basically it.

The CPU doesn’t touch the stack and doesn’t save any of the other registers. Everything that happens after this point is handled by kernel code.

The syscall arguments are passed through registers according to the x86-64 syscall ABI:

rax = syscall number
rdi = arg0
rsi = arg1
rdx = arg2
r10 = arg3   // rcx can't be used because SYSCALL overwrites it
r8  = arg4
r9  = arg5

So, at the moment SYSCALL executes, the CPU has everything it needs to identify the requested syscall and its arguments. It then jumps to the address stored in MSR_LSTAR, which is where the kernel’s syscall entry path begins.

Step 2 — entry_SYSCALL_64, the landing pad

Source: arch/x86/entry/entry_64.S:87

SYM_CODE_START(entry_SYSCALL_64)
    UNWIND_HINT_ENTRY
    ENDBR

    swapgs
    movq    %rsp, PER_CPU_VAR(cpu_tss_rw + TSS_sp2)     /* stash user RSP */
    SWITCH_TO_KERNEL_CR3 scratch_reg=%rsp                /* KPTI page table swap */
    movq    PER_CPU_VAR(cpu_current_top_of_stack), %rsp  /* onto the kernel stack */

    pushq   $__USER_DS                 /* pt_regs->ss */
    pushq   PER_CPU_VAR(cpu_tss_rw + TSS_sp2)  /* pt_regs->sp */
    pushq   %r11                       /* pt_regs->flags */
    pushq   $__USER_CS                 /* pt_regs->cs */
    pushq   %rcx                       /* pt_regs->ip */
    pushq   %rax                       /* pt_regs->orig_ax */

    PUSH_AND_CLEAR_REGS rax=$-ENOSYS

    movq    %rsp, %rdi                 /* arg0 = &pt_regs */
    movslq  %eax, %rsi                 /* arg1 = syscall number */

    call    do_syscall_64

Now the CPU has jumped to entry_SYSCALL_64. From this point onward, everything is done by software.

Here, the kernel builds the pt_regs struct manually on stack and passes it as argument to the do_syscall_64 function. This code manually pushes every register, one pushq at a time, in a specific order, onto the kernel stack to build a struct pt_regs. . swapgs swaps in the kernel’s per-CPU data (because gs was pointing at userspace data a). The stack pointer itself gets swapped too — user rsp is stashed, and a real kernel stack (cpu_current_top_of_stack) is loaded.

Step 3 — what pt_regs actually is

The pushes above, in that exact order are this struct laid out on the stack:

// arch/x86/include/asm/ptrace.h:103
struct pt_regs {
    unsigned long r15;
    unsigned long r14;
    unsigned long r13;
    unsigned long r12;
    unsigned long bp;
    unsigned long bx;

    /* callee-clobbered, always saved on entry */
    unsigned long r11;
    unsigned long r10;
    unsigned long r9;
    unsigned long r8;
    unsigned long ax;
    unsigned long cx;
    unsigned long dx;
    unsigned long si;
    unsigned long di;

    unsigned long orig_ax;   /* syscall number lives here on entry */
    unsigned long ip;        /* where SYSCALL will return to (was in rcx) */
    unsigned long cs;
    unsigned long flags;     /* was in r11 */
    unsigned long sp;        /* user's rsp, stashed earlier */
    unsigned long ss;
};

Step 4 — do_syscall_64

Source: arch/x86/entry/syscall_64.c:87

__visible noinstr bool do_syscall_64(struct pt_regs *regs, int nr)
{
    nr = syscall_enter_from_user_mode(regs, nr);

    if (!do_syscall_x64(regs, nr) && !do_syscall_x32(regs, nr) && nr != -1)
        regs->ax = __x64_sys_ni_syscall(regs);

    syscall_exit_to_user_mode(regs);
    ...
}

Two arguments: the pt_regs we just built, and nr the syscall number, read straight out of %eax back in the asm (movslq %eax, %rsi). do_syscall_x64() bounds-checks nr and dispatches:

// arch/x86/entry/syscall_64.c:53
static __always_inline bool do_syscall_x64(struct pt_regs *regs, int nr)
{
    unsigned int unr = nr;
    if (likely(unr < NR_syscalls)) {
        unr = array_index_nospec(unr, NR_syscalls);   /* Spectre v1 guard on attacker-controlled index */
        regs->ax = x64_sys_call(regs, unr);
        return true;
    }
    return false;
}

array_index_nospec() matters: nr came straight from userspace, unvalidated, and it’s about to index a table.

Step 5 — dispatch table: x64_sys_call

Source: arch/x86/entry/syscall_64.c:35

#define __SYSCALL(nr, sym) case nr: return __x64_##sym(regs);
long x64_sys_call(const struct pt_regs *regs, unsigned int nr)
{
    switch (nr) {
    #include <asm/syscalls_64.h>
    default: return __x64_sys_ni_syscall(regs);
    }
}

At his point, we’ve reached the part where the kernel actually needs to figure out which syscall was requested.

Remember that the syscall number was originally placed in rax by userspace. We carried that value through the entry code, and eventually do_syscall_64() passes it to x64_sys_call() as nr.

The slightly confusing part is that you won’t find all those case statements written out manually in this file. They’re generated from the syscall table at build time using macros. The details of that generation aren’t really important for understanding the flow.

What matters is what we end up with conceptually:

one syscall number → one corresponding syscall handler.

For example, read has syscall number 0 on x86-64. So conceptually, the generated code contains something like:

case 0:
    return __x64_sys_read(regs);

If nr is 1, we’d get the handler for write; if it’s 2, the handler for open, and so on.

So this function is essentially the kernel’s dispatch table:

syscall number
      │
      ▼
  x64_sys_call()
      │
      ├── 0 ──► __x64_sys_read()
      ├── 1 ──► __x64_sys_write()
      ├── 2 ──► __x64_sys_open()
      ├── ...
      └── unknown ──► __x64_sys_ni_syscall()

And that’s the key thing to understand here. We’ve gone from a raw SYSCALL instruction, through the low-level assembly entry code, and now the kernel has finally used the syscall number to select the function that implements the requested operation.

For our read() example, the next stop is:

__x64_sys_read(regs);

That’s where we can finally start following the actual read syscall implementation.

Step 6 — the actual syscall body

__x64_sys_read(regs) pulls the real arguments back out of pt_regs (regs->di, regs->si, regs->dx and calls the function.

Source: fs/read_write.c:705 and fs/read_write.c:723

ssize_t ksys_read(unsigned int fd, char __user *buf, size_t count)
{
    CLASS(fd_pos, f)(fd);
    ssize_t ret = -EBADF;

    if (!fd_empty(f)) {
        loff_t pos, *ppos = file_ppos(fd_file(f));
        if (ppos) { pos = *ppos; ppos = &pos; }
        ret = vfs_read(fd_file(f), buf, count, ppos);
        if (ret >= 0 && ppos)
            fd_file(f)->f_pos = pos;
    }
    return ret;
}

SYSCALL_DEFINE3(read, unsigned int, fd, char __user *, buf, size_t, count)
{
    return ksys_read(fd, buf, count);
}

This is where the syscall actually does something: turn fd into a struct file, call vfs_read(), which goes further down into whatever filesystem/driver owns that file descriptor, update the file position, return the byte count (or a negative -errno on failure).

That return value now travels back up, unwinding through every layer we just went down: back into __x64_sys_read, back into the case 0: line, back out of x64_sys_call, and do_syscall_64 stores it into regs->ax , overwriting the syscall number that used to live there.

Step 7 — getting back to userspace

We’ve now gone all the way down the syscall path and got the return value back from read(). Now we need to get out of the kernel and return to userspace.

Back in do_syscall_64(), after the syscall handler returns, the kernel checks whether it can use the fast SYSRET path:

// arch/x86/entry/syscall_64.c:112
/* SYSRET requires RCX == RIP and R11 == EFLAGS */
if (unlikely(regs->cx != regs->ip || regs->r11 != regs->flags))
    return false;

if (unlikely(regs->cs != __USER_CS || regs->ss != __USER_DS))
    return false;

/*
 * On Intel CPUs, SYSRET with non-canonical RCX/RIP will #GP
 * in kernel space. This essentially lets the user take over
 * the kernel, since userspace controls RSP.
 */
if (unlikely(regs->ip >= TASK_SIZE_MAX))
    return false;

Remember back in Step 1: when the CPU executed SYSCALL, it automatically saved the userspace instruction pointer in rcx and rflags in r11. It also verifies that we’re actually returning to userspace (cs/ss are the expected user segments) and that the return RIP is a valid userspace address.

SYSRET expects those exact registers to contain the return state. So the kernel checks:

regs->cx  == regs->ip
regs->r11 == regs->flags

If everything checks out, control returns to the tail of entry_SYSCALL_64:

syscall_return_via_sysret:
    POP_REGS pop_rdi=0
    ...
    SWITCH_TO_USER_CR3_STACK scratch_reg=%rdi
    popq    %rdi
    popq    %rsp
    swapgs
    sysretq

Every register gets popped back out of pt_regs, except now rax holds the syscall’s return value, not the syscall number anymore. Page tables get switched back (SWITCH_TO_USER_CR3_STACK), swapgs flips gs back to userspace, and sysretq jumps back to rcx.

Your program resumes right after the SYSCALL instruction, rax holds the result, and that’s what your C library hands back to you as the return value of read().

• Wow, the mystery behind SYSCALL finally makes sense.

start to end

userspace: rax=0, rdi=fd, rsi=buf, rdx=count, then `syscall`
    │
    ▼  (hardware: rcx=return addr, r11=rflags, jump to MSR_LSTAR)
entry_SYSCALL_64            [asm]
    swapgs, switch stack, hand-build pt_regs from pushes
    │
    ▼
do_syscall_64(regs, nr=0)   [C]
    │
    ▼
x64_sys_call(regs, 0)  →  switch(0)  →  __x64_sys_read(regs)
    │
    ▼
__se_sys_read  →  __do_sys_read  →  ksys_read  →  vfs_read   (the real work)
    │
    ▼
return value stored into regs->ax
    │
    ▼
do_syscall_64 checks: cx==ip? r11==flags? cs/ss correct? ip canonical?
    │
    ├─ yes → entry_SYSCALL_64 sysretq path → back to userspace, fast
    └─ no  → iretq path → back to userspace, slow but always correct

Adding your own syscall

If everything above makes sense, adding a brand new syscall is short. Only one file is truly mandatory to touch. Everything else is either auto-generated or just good conventional practice.

1. Give it a number — the only required edit

Source: arch/x86/entry/syscalls/syscall_64.tbl

Add one line, same format as every other row in that file:

472	common	mysyscall		sys_mysyscall

That’s <number> <abi> <name> <entry point>. Pick a number bigger than whatever the current last one is.

This single line is enough to make both generated headers update themselves on the next build:

• unistd_64.h gets a new #define __NR_mysyscall 472
• syscalls_64.h gets a new __SYSCALL(472, sys_mysyscall) line

And remember from Step 5 above — __SYSCALL(472, sys_mysyscall) is exactly the macro that expands into case 472: return __x64_sys_mysyscall(regs); inside x64_sys_call()’s switch. You’re not editing that switch statement. You’re editing the one file that generates it.

2. Write the actual code

Put this anywhere that’s already being compiled, e.g. kernel/sys.c:

SYSCALL_DEFINE2(mysyscall, unsigned long, arg1, unsigned long, arg2)
{
    // whatever you want it to do
    return 0;
}

This one macro is doing all the work we already traced back in Step 6. So, adding syscall is quite easy now. :)

Wanna create rootkits?

Adding a syscall (above) needs a full rebuild. A rootkit doesn’t get that luxury. It’s a module, loaded into an already-running kernel, and it wants to change syscall behavior without a reboot. I have found two famous techniques. One of them is dead on a modern kernel. One of them still works.

Target 1: sys_call_table — dead

Old-school technique, still explained in every “write a rootkit” tutorial: find sys_call_table, disable write protection, overwrite an entry, done.

// arch/x86/entry/syscall_64.c:23-29
/*
 * The sys_call_table[] is no longer used for system calls, but
 * kernel/trace/trace_syscalls.c still wants to know the system
 * call address.
 */
#define __SYSCALL(nr, sym) __x64_##sym,
const sys_call_ptr_t sys_call_table[] = {
    #include <asm/syscalls_64.h>
};

Reason 1 — nobody reads it anymore. Real dispatch happens through x64_sys_call()’s hand-written switch(nr) (Step 5, earlier in this note) — direct case: branches compiled straight into .text. sys_call_table[] is kept around purely so the tracing subsystem can print addresses. Even a successful overwrite changes nothing, because the syscall path never indexes into this array.

Reason 2 — it’s const. That places it in .rodata, and this kernel calls mark_rodata_ro() at boot (arch/x86/mm/init_64.c:1405), which walks the page tables and clears the writable bit on every .rodata page. A plain write faults.

To change the dispatch behavior directly, you’d have to patch the kernel’s live machine code in .text. That’s a fundamentally different problem from modifying a function-pointer table. You now need a way to modify executable kernel text at runtime, deal with instruction boundaries, synchronization, CPUs executing the code concurrently, and the kernel’s text-protection mechanisms.

Target 2: MSR_LSTAR — still very much alive

Different kind of target entirely. MSR_LSTAR is a CPU register , nothing about mark_rodata_ro() touches it. Recall from Step 1: this is the one MSR that tells the CPU where to jump on SYSCALL, and the kernel sets it with a completely ordinary inline instruction wrapper:

// arch/x86/include/asm/msr.h:78, 197
static __always_inline void __wrmsrq(u32 msr, u64 val)
{
    asm volatile("1: wrmsr\n" "2:\n"
                 _ASM_EXTABLE_TYPE(1b, 2b, EX_TYPE_WRMSR)
                 : : "c" (msr), "a" ((u32)val), "d" ((u32)(val >> 32)) : "memory");
}
static inline void wrmsrq(u32 msr, u64 val) { native_write_msr(msr, val); }

It’s a static inline, so any module that includes asm/msr.h gets this inlined directly. wrmsr is a CPL0-only instruction, and loaded module code runs at the exact same ring-0 privilege as the rest of the kernel. So this genuinely works:

#include <asm/msr.h>
#include <asm/msr-index.h>

extern void my_syscall_entry(void);   // real asm, same ABI contract as entry_SYSCALL_64

static int __init hook_init(void)
{
    wrmsrq(MSR_LSTAR, (unsigned long)my_syscall_entry);
    return 0;
}

Every SYSCALL on that core now lands in my_syscall_entry instead of entry_SYSCALL_64.

The catch: it’s per-core, not global.

// arch/x86/kernel/cpu/common.c:2264-2266, 2299-2313
static inline void idt_syscall_init(void)
{
    wrmsrq(MSR_LSTAR, (unsigned long)entry_SYSCALL_64);
    ...
}

/* May not be marked __init: used by software suspend */
void syscall_init(void)
{
    wrmsr(MSR_STAR, ...);
    if (!cpu_feature_enabled(X86_FEATURE_FRED))
        idt_syscall_init();
}

• This initialization happens as each CPU is brought online. That means if your module executes:

wrmsrq(MSR_LSTAR, my_syscall_entry);

• You have only changed MSR_LSTAR on the CPU that executed that instruction. Imagine a four-core machine. A process running on CPU 0 will hit your hook. If the scheduler moves that process to CPU 2, its next syscall will go through the normal kernel entry path instead.

• So a complete system-wide hook has to update the MSR on every online CPU, typically using something such as on_each_cpu().

• And even that isn’t enough to make the hook permanent.

• A CPU can come online later through CPU hotplug, or the system can resume from suspend. During CPU initialization/resume, the kernel can run syscall_init() again and restore:

MSR_LSTAR = entry_SYSCALL_64

• So a persistent hook would also need to account for CPUs being brought online or reinitialized later.

The other catch: you’re replacing the entire syscall entry point

• Whatever my_syscall_entry is has to redo everything entry_SYSCALL_64 did by hand back in Step 2 — swapgs, get off the untrusted user stack, the KPTI CR3 switch, hand-build a valid pt_regs. Missing any of it will result in any syscall on that core corrupts kernel state.
• This is why real hooks don’t fully replace the handler , they rdmsr the original value first, install a small trampoline that does its own check, then jump to the saved original entry_SYSCALL_64.

But wait, It is not worth to do it. Too much engineering work. Let us see how modern rootkits work on eBPF section. XD