Making a System Call

The key architectural feature needed to implement an operating system is a ``mode bit.'' The processor needs to support two modes: user and supervisor (aka system, kernel).

The difference between the modes is that in user mode the machine's capabilities are limited compared to system mode. There are two types of limitations needed:

  1. Some memory should be available to both the user process and the system, with other memory restricted to the system only. This defines system space and user space.
  2. Some instructions should be available to the user, and others should be available to the system only. The canonical example of an instruction that should not be accessible to a user process is halt. Machines that use in and out instructions will normally have these as supervisor-mode instructions; machines that use memory-mapped IO will normally have device registers in system space.

What I've described above is a very minimal set; it can be (and has been) expanded in nearly every direction you can think of. There have been machines that had more than two modes (VAX and Intel both have four levels, the intent being to have more than two levels of ``trustedness.''). Intel has a very elegant scheme allowing processes, on a process by process and IO-port by IO-port basis, to have access to IO on particular devices. And so forth.

Finally, the mode bit itself shouldn't be under the direct control of the user (if it were, you could just set the bit and circumvent everything). So, how do you change the mode bit? On most machines, the mechanism to do this is piggybacked onto the device interrupt mechanism. Remember, this (typically) looks something like this:

  1. Device requests interrupt while CPU has interrupts enabled.
  2. (optional) CPU switches from user stack to system stack. This keeps from corrupting the stack if user programs are ill-behaved.
  3. CPU pushes minimal state (typically PC and PSW).
  4. CPU loads new CPU according to interrupt, and sets PSW to known condition (which includes setting the mode bit).
Three key points about interrupts:
  1. The interrupt handler is executed in supervisor mode
  2. The location of the interrupt handler is determined by system tables, not the device
  3. A bit more philosophically, the interrupt can be regarded as the device requesting service from the system.
So, architectures universally supply a trap instruction (aka syscall, software interrupt) which requests an interrupt from software. Requesting a trap is the user's only way of setting the mode bit - and when you do, you don't have control of where in the OS your handler is located. So security is maintained. Incidentally, one of the better ``pragmatic'' definitions of the kernel is that it is the code you can only get to by way of interrupts or trap instructions.

That said, there are two fundamental OS organizations: a system-call approach, and a message-passing approach (which might also be called a microkernel approach).

In a system-call approach, all OS functions are implemented directly in the kernel itself. The advantage of the system-call approach is efficiency.

In a message-passing approach, the only functions implemented directly in the kernel are message sending and receiving. Everything else is performed by trusted processes that execute in supervisor mode. The advantages of the message-passing approach are modularity and safety.

As you might expect, all real OS's are somewhere in between!