Here is an interesting blog i found about system calls and switching between real and protected mode.
http://blogs.msdn.com/larryosterman/archive/2005/02/08/369243.aspx
There is also a link about the "invalid" instruction system calls ( http://blogs.msdn.com/oldnewthing/archive/2004/12/15/313250.aspx)