Briefly, in usual Linux distros, one needs libc in order to be able to link/use the commonly known syscalls (e.g. open,read…). When a userspace program calls, for example, ‘open’, it actually calls a wrapper implemented in libc. We need wrappers as we can’t directly call kernel code.
To make a syscall we must place the syscall number and its arguments in some arch specific registers, run an instruction that puts the current process in wait and makes the kernel start running. The kernel knows that it has to handle a syscall, takes the number from the first register and calls the associated routine. After the result (the handling was successful or not) is put yet into a particular register, it returns to user mode.
Given this, you could write your own syscall wrapper library, only consisting of wrappers that you know your application will use.
What we will do next is try to find what syscalls a given userspace uses by analyzing an object file and trying to identify the snippets that implement syscalls.
The first trial is made on my own libc, libc.so.6 (for x86_64). We decompile it (“$ objdump -lD”) and search for, let’s say, renameat:
00000000000574d0 <renameat>: renameat(): ....... 574d9: b8 08 01 00 00 mov $0x108,%eax 574de: 0f 05 syscall
If we search the syscalls table that matches the syscall name with it’s actual number, we conclude that the number of that syscall is kept in the eax register (in out case, 264). So, the first solution that comes to mind is searching for int literals that are moved into the %eax register. Although this will identify all syscalls, it will also have false positives as the register can be used for other opperations.
In case we have a real small, single purpose, embedded system, we may want to use only a few syscalls, of which we are sure our application will need. Then why not compile only these few kernel correspondent functions? For this, we need to detect those syscalls and then make it possible in the kernel to compile them out (for example, automatically generate a .config that builds a smaller kernel image supporting that userspace). In the next thread, we’ll see how it works.