HP OpenVMS Guide to System Security: OpenVMS Version 8.4 > Chapter 1 Understanding System Security


The OpenVMS operating system provides several data protection schemes. For example, by using UIC-based protection you can protect data by controlling access to files. You can use ACLs to refine access control to specific groups or individual users. For a protection scheme with yet greater security for your data, you can encrypt the files. Encrypting a file transforms it into unrecognizable, unintelligible data, even if someone manages to gain access to it.

Encryption Process

The process of encryption takes readable data, called plaintext, and uses a mathematical algorithm to transform the plaintext into an unreadable, unintelligible form, called ciphertext.

To encrypt the plaintext data, the encryption operation requires a key. The key is a variable that controls the encryption operation. The same plaintext, encrypted with different keys, results in different ciphertext. In addition, repeated encryption of the same plaintext with the same key also results in different ciphertext each time.

OpenVMS Version 8.3 integrates the former Encryption for OpenVMS software product into the operating system. This eliminates the need for a separate product installation and product license. In addition, OpenVMS Version 8.3 and later supports the Advanced Encryption Standard (AES) algorithm.

AES Encryption Algorithm

The AES algorithm allows OpenVMS users, system managers, security managers, or programmers to secure their files, save sets, or application data with AES encryption. DES and AES are similar encryption algorithms. They are both block cipher algorithms. However, encryption using AES algorithms is found to be more secure than DES encryption due to the number of rounds the plain text undergoes during its transformation to ciphered text. The number of rounds depend on the key size. For example, a key size of 128 bits invokes 10 rounds of transformation. Similarly, key sizes of 192 bits and 256 bits invoke 12 and 14 rounds, respectively. For more information on AES encryption algorithm, see “Using Encryption”

DES Encryption Algorithm

The algorithm used by OpenVMS is a software implementation of the Data Encryption Standard (DES) defined by the National Bureau of Standards (NBS). The NBS document FIPS-PUB-46 describes the operation of the DES algorithm in detail.

Because the DES algorithm is public knowledge, the security of your ciphertext files depends on the keys you define.


OpenVMS encryption uses two keys:

  • Key that you provide.

  • Key that the software randomly generates, called the data key.

The key you provide encrypts the data key, which is stored in the first block of the ciphertext file. The process uses the encrypted data key to encrypt the file. You have the option to encrypt either the data key or the file.

Table 1-2 shows the components of the encryption process.

Table 1-2 Components of the Encryption Operation

User-supplied data keyKey encryptionEncrypted key
Data (plaintext) and the encrypted data key Data encryptionEncrypted file


Figure 1-1 illustrates the data encryption operation. In this example, the input file contains the text "secret" and the key has been defined as "elmno jflghi." The output file is unreadable text.

Figure 1-1 Encrypting a File

Per-Thread Security Profile Model


To gain access to the data in an encrypted file, reverse the encryption process by performing the decryption process. Decryption uses a mathematical encryption algorithm to change ciphertext into the original plaintext.

Before decrypting a file, the software checks the validity of the key you provide. This validation is a checksum operation on the encrypted data stored in the first block of the ciphertext file.

When you specify the AES/DES algorithm to decrypt a file, use the key that is identical to the one used in the original encryption process.

NOTE: Only the correct key can decrypt your file. If you lose or forget the key, you cannot gain access to the data in any understandable, useful form.

Figure 1-2 shows the data decryption operation. In this example, the input file holds unreadable text. The key, "elmno jflghi," is the same key that was used to encrypt this file. The output file contains the readable text "secret."

Figure 1-2 Decrypting a File

Per-Thread Security Profile Model

Authentication Process

OpenVMS detects any modification made to both plaintext and ciphertext files. This process is called authentication. Authentication checks for and reports on any changes to:

  • File data

  • File location

  • Authentication key

  • Security settings

The software calculates two Message Authentication Codes (MACs): one based on file contents and one based on security settings. The software then associates them with one or more files and stores this information. When you subsequently check file integrity, the software recalculates the MACs and compares them against the stored codes.

NOTE: Currently, MAC authentication is supported only with DES algorithm.

Encryption Interfaces

To define and delete keys, and to encrypt and decrypt files, use the following Encryption interfaces:

  • DCL commands – for interactive encryption functions. These commands encrypt files and backup save sets. For more information on using DCL commands, see “Using Encryption”.

  • Callable routines – for application programming. These routines encrypt files and small blocks. For more information, see the HP OpenVMS Utility Routines Manual.


Encrypted files are fully compatible between OpenVMS systems. You can copy them from system to system and do all remote file operations that OpenVMS systems support for other kinds of files. In addition, you can encrypt files on one system and decrypt them on another system that also runs the Encryption software. Inter-system encryption operations with non-OpenVMS platforms are not supported.