Only as reliable as your tools. It would be great if Nmap was always 100% accurate and it does a pretty good job especially if you are doing service detection as well with -sV. Obviously you won't have IIS running on a BSD box.
For Windows:
It's possible you could use a null session and user2sid to enumerate the SIDs and then do a compare with the entry at
http://support.microsoft.com/kb/243330 of well-known SIDs to narrow down the OS list. Supposedly
If you can get shell you can use
systeminfo | findstr /B /C:"OS Name" /C:"OS Version"
or if you just want the OS version you can use
ver
If you have an account on the box you can detect specific patches remotely with WMIC using
wmic /node:<target> qfe list full
You may also have to specify username and password if you are using a different account.
wmic /user:<username> /password:<userpassword> /node:<target> qfe list full
That doesn't exactly answer your question I know since you want to know in scanning/enumeration. I'm not sure there is a 100% way to detect but if there is I'd love to know it!
Check out
http://nmap.org/book/osdetect.html for more info