Limit access to servers as much as possible by locking down computer rooms, the network room and wiring closets. The most important servers to secure are domain controllers as they contain all the domain's username and passwords and these items are the keys to all network resources.
Only the most trusted systems/network administrators should have physical access to the servers. Once this is obtained, access to the software can be accomplished. For example, the server can be rebooted and an intruder could use the console to access hard disks from the command line.
You should log entry, exit and other actions electronically, as well as implementing video and/or security systems.