Protection in an operating system refers to a set of mechanisms and policies that control the access of subjects (e.g., processes, users) to objects (e.g., files, memory segments, CPU, devices). The primary goals of these protection mechanisms are multifaceted:
- Preventing Malicious Access: The most obvious goal is to prevent unauthorized users or processes from accessing, modifying, or deleting sensitive data or system resources. This includes preventing both intentional attacks and accidental misuse.
- Ensuring Data Integrity: To maintain the correctness and trustworthiness of data. Protection mechanisms prevent unauthorized or erroneous modifications to files, databases, or system configurations, ensuring that data remains consistent and uncorrupted.
- Maintaining System Reliability and Availability: To ensure that the operating system and its resources function correctly and are available to legitimate users when needed. Protection prevents one faulty or malicious process from adversely affecting other processes or crashing the entire system.
- Enforcing Policy: To implement and enforce specific security policies defined by administrators or users. This means ensuring that access to resources aligns with the predefined rules and restrictions (e.g., "only the owner can modify this file").
- Supporting Multiple Users/Processes: In multi-user or multi-programmed environments, protection is essential for isolating users and processes from each other, providing privacy and preventing interference. Each process should operate in its own isolated environment.
- Confidentiality: To ensure that sensitive information is only disclosed to authorized entities. This prevents unauthorized reading or viewing of data.

Several design principles guide the creation of robust protection mechanisms:
- Principle of Least Privilege: This is the most fundamental principle. It dictates that every program, user, or process should be granted only the minimum set of privileges (rights) necessary to perform its legitimate function, and no more.
- Implications:
- Reduced Attack Surface: By limiting privileges, the potential damage caused by a compromised entity (e.g., a malicious program, a hacked user account) is minimized. If a program only has read access to a specific file, it cannot write to it even if it is exploited.
- Containment of Errors: Accidental errors or bugs in a program are less likely to cause widespread system damage if the program operates with restricted privileges.
- Improved Auditability: It becomes easier to track and audit specific resource accesses when privileges are tightly controlled.
- Application: This principle applies to all levels of system design: from user accounts (e.g., regular users vs. administrators), to processes (e.g., web server running as a non-privileged user), to even specific system calls.
- Separation of Privilege: Requires that access to an object depends on more than one condition. For instance, requiring multiple keys to unlock a safe or two-factor authentication for sensitive operations. This adds an extra layer of security.
- Economy of Mechanism: Keep the protection mechanism as simple and small as possible. Simpler mechanisms are easier to analyze, test, and verify for correctness, reducing the likelihood of design flaws or bugs that could lead to security vulnerabilities.
- Open Design: The security of a mechanism should not depend on the secrecy of its design or implementation. While keys and passwords must be kept secret, the algorithms and protocols should be publicly known and reviewed. This allows for public scrutiny and identification of weaknesses.
- Complete Mediation: Every access to every object must be checked for authorization. There should be no bypasses or shortcuts around the protection mechanisms. This ensures that no unauthorized access goes unnoticed.
- Fail-Safe Defaults: The default access to an object should be denial. Only explicitly granted access should be allowed. When a new object is created, its default permissions should be restrictive.
- Least Common Mechanism: Minimize the amount of mechanism common to more than one user and depended on by all users (e.g., shared code). This reduces the potential for a vulnerability in one component to affect many users.
- Psychological Acceptability: The security mechanisms should be easy to use and intuitive, otherwise users will circumvent them, undermining their effectiveness.