IT Encyclopedia: What is MD5?

Hello, this is the Network Technology Alliance Station, and I'm Ruigo.

MD5, short for Message-Digest Algorithm 5, is a widely used cryptographic hash function that generates a 128-bit (16-byte) hash (hash) that is used to ensure that information is transmitted intact and consistently. MD5 is one of the hashing algorithms, which can take data input of any length through a series of complex operations to produce a fixed-length output - this is what we often call a hash value.

MD5 was designed and released in 1991 by United States computer scientist Ronald Rivest ·. This is an improved and enhanced version, as the previous MD2, MD3 and MD4 had certain issues in terms of security and efficiency. MD5 is designed to provide a fast and secure hashing algorithm.

MD5 was officially released in 1991, and a detailed technical description was published in a paper titled "The MD5 Message-Digest Algorithm", which described in detail the design principles and implementation details of the MD5 algorithm. Soon after its release, MD5 was widely used in cryptography, data integrity checks, and digital signatures.

The design and year of the MD5

Ronald · Levist is a professor at the Massachusetts Institute of Technology (MIT) and one of the inventors of the RSA public key cryptography algorithm. As an important figure in the field of cryptography, he designed the MD5 algorithm by combining the advantages and disadvantages of the previous version and striking a balance between security and computational efficiency.

The MD5 algorithm was released in 1991, when the Internet was just starting, and various network protocols and data transmission methods were not yet perfect. The release of MD5 at this time provides an effective solution for the integrity and security of data transmission.

How MD5 works

Hash function

A hash function is a mathematical algorithm that takes an arbitrary length of data input and produces a fixed-length output through a series of operations, known as a hash (or digest).

The core features of hash functions include:

• Fixed-length output: The length of the hash is fixed regardless of the length of the input data.
• Irreversibility: The original data cannot be deduced from the hash value.
• Uniqueness: Different input data generates different hashes (ideally).

Hash functions are widely used in computer science and cryptography, including data verification, data indexing, password encryption, digital signatures, etc. The efficiency and security of hash functions directly affect the reliability of these applications.

Steps of the MD5 algorithm

The first step of the MD5 algorithm is to fill the input data so that its length satisfies certain conditions.

The rules for populating are as follows:

• Start by adding a bit 1 (i.e., 0x80) at the end of the data.
• Then add enough bit 0s so that the padded data length is modulo equal to 448 for 512 (i.e., 64 bits less than a multiple of 512).
• Finally, 64 bits are used to represent the original length of the data and appended to the end of the padded data.

For example, for a data with a length of 448 bits, the padded length is 512 bits; For a data with a length of 500 bits, the length of the padding is 1024 bits.

The MD5 algorithm uses four 32-bit registers (A, B, C, D) as the initial state.

These four registers are initialized to the following constants:

• A = 0x67452301
• B = 0xEFCDAB89
• C = 0x98BADCFE
• D = 0x10325476

The populated data is divided into multiple 512-bit chunks, each of which is further divided into 16 32-bit sub-chunks.

The MD5 algorithm processes each 512-bit data block with the following steps:

1. Main Loop: The main loop is divided into four rounds, each of which contains 16 operations, each using a different nonlinear function (F, G, H, I) and constant values.
2. Four-round operation:
3. Round 1: The nonlinear function F is used, and the contents of the register are shifted to the left in a loop with each operation.
4. Second round: Using the nonlinear function G, similarly, the contents of the register are shifted to the left in a loop.
5. Round 3: Use the nonlinear function H to do something similar.
6. Fourth round: Use the nonlinear function I to perform the final round of operations.

After each block of data is processed, the algorithm updates the values of the registers, and the values of these registers are used to process the next block. The final hash value is concatenated from the final values of these registers.

The internal structure of the MD5

The MD5 algorithm uses four nonlinear functions (F, G, H, I) to process the data. These functions make a complex mix of the current register value with the data block to ensure the uniqueness and irreversibility of the hash value.

• 函数F：F(B, C, D) = (B & C) | (~B & D)
• 函数G：G(B, C, D) = (B & D) | (C & ~D)
• 函数H：H(B, C, D) = B ^ C ^ D
• 函数I：I(B, C, D) = C ^ (B | ~D)

The MD5 algorithm uses a set of 64 constant values (T[i]), which are pre-calculated during the design of the algorithm and defined as: [ T[i] = \lfloor 2^{32} \times |\sin(i)| \rfloor ] where i can range from 1 to 64.

The entire data processing process of the MD5 algorithm can be summarized into the following steps:

1. Initialization: Set the initial values of registers A, B, C, and D.
2. Populate Data: Populate the input data according to the rules.
3. Chunked processing: Splits the populated data into multiple 512-bit data blocks.
4. Main loop operation: Performs four rounds of nonlinear function operations and constant-value mixing operations on each data block.
5. Update Register: Updates the value of the register to process the next block of data.
6. Output Hash: Connect the value of the final register to generate a 128-bit MD5 hash.

Advantages and disadvantages of MD5:

Advantages of MD5:

The calculation is fast

A significant advantage of the MD5 algorithm is that it is very computationally fast. Due to the simple design and high data processing efficiency of the MD5 algorithm, it is very suitable for scenarios that require fast generation of hash values. For example, during file integrity checks and data transfers, MD5 is able to quickly calculate hash values, improving overall efficiency.

Simple to implement

The MD5 algorithm is relatively simple and easy to implement. This simplicity allows it to be implemented in a variety of programming languages and platforms, whether it's C, C++, Java, Python, JavaScript, or mature MD5 implementation libraries. Developers can easily integrate MD5 functionality in a variety of applications.

Wide range of applications

Since MD5 was released earlier and was considered safe for a long time, it has been widely used in numerous applications. For example, MD5 is often used to verify file integrity, generate digital signatures, and protect passwords. Even today, despite known security issues, MD5 is still used in a number of non-safety-critical applications.

Disadvantages of MD5:

Collision vulnerability

A major drawback of the MD5 is the presence of a crash vulnerability. Collision is when two different inputs generate the same hash value. Since MD5 generates a limited length of hash (128 bits), there is theoretically an infinite number of different inputs that can generate the same hash. In 2004, researchers succeeded in discovering an actual MD5 crash attack method, proving that MD5 could not defend against this attack.

Pre-image attacks

A pre-imaging attack is when a hash value is given and an input data is found to generate it. While MD5 was designed with pre-image attack protection in mind, finding pre-images becomes more feasible as computing power increases. Especially in security-critical applications such as cryptography and digital signatures, this attack poses a serious threat to MD5's security.

Rainbow Table Attack

A rainbow table is a precomputed hash table that allows an attacker to quickly find the corresponding input of a hash by pre-calculating and storing a large number of possible hash values and their corresponding input data. Due to the short hash length of MD5, it is relatively easy to generate and store rainbow tables, making MD5 more vulnerable to rainbow table attacks.

Although MD5 was widely used for cryptographic encryption and digital signatures in the early days, it was gradually replaced by more secure hashing algorithms in these areas as security issues were exposed. Especially in sensitive and security-critical applications, MD5 is no longer suitable.

While MD5 is commonly used for data integrity checks, its collision vulnerability means that it does not fully guarantee the uniqueness and security of the data. For scenarios that require high security, it is recommended to use a stronger hashing algorithm, such as SHA-256 or SHA-3.

MD5 applications

Data integrity verification

During file downloads and transfers, it is very important to ensure that the files have not been tampered with and corrupted. MD5 is often used to generate a checksum for a file, and after the user downloads the file, the user can calculate the MD5 value of the local file and compare it with the MD5 value provided by the server to verify the integrity of the file. For example, many open-source software and large file download sites provide MD5 checksums for users to verify.

In network data transmission, data packets can be faulty due to various reasons such as network interference. By calculating and comparing the MD5 hash of the packets before and after transmission, it is ensured that the packets have not been tampered with and corrupted during transmission. This is especially important in data backup and distributed systems.

Digital Signatures & Certificates

A digital signature is a hash of digital information that is encrypted to verify the authenticity and integrity of the information. MD5 was widely used to generate digital signatures because it could quickly generate fixed-length hashes for easy signing and verification. However, as MD5 security issues are exposed, more and more systems are moving to more secure hashing algorithms (such as SHA-256).

In a public key infrastructure (PKI), a digital certificate is used to verify the ownership of a public key. MD5 was used to generate a hash of the certificate to ensure the uniqueness and integrity of the certificate. However, modern PKI systems have phased out MD5 in favor of a more secure hashing algorithm due to security concerns.

Password encryption

In user password management, storing the plaintext of passwords is very insecure. By MD5 hashing the password, storing the hash value instead of the plaintext, you can increase the security of the password. Even if the database is compromised, it is difficult for an attacker to directly obtain the plaintext password. However, due to MD5's security concerns, modern systems often use more secure hashing algorithms (e.g., bcrypt, scrypt, Argon2) to store passwords.

When a user logs in, the entered password is hashed and compared to the hash value stored in the database. If the hashes match, the user entered the correct password. This way avoids exposing the user's plaintext password during transmission.

Version control in software development

In the software development process, it is very important to manage and track changes to files. By generating an MD5 hash value for a file, developers can quickly determine if a file has changed. This is useful for version control systems (e.g., Git) and configuration management tools (e.g., Ansible) to effectively manage changes to code and configuration files.

In large software projects, duplicate files waste storage space and add complexity to management. By calculating the MD5 hash value of a file, duplicate files can be quickly detected and removed, optimizing storage space and resource utilization.

MD5 security issues

Collision attack

A collision attack is when two different inputs are found so that they generate the same hash. Due to the limited length of hashes generated by MD5, there is theoretically an infinite number of different input data that can generate the same hash value. In 2004, researchers successfully discovered the actual MD5 crash attack method, which posed a serious threat to the security of MD5.

The presence of collision attacks means that attackers can forge different data with the same hash value, bypassing data integrity checks or digital signature verification. This is unacceptable in security-critical applications because it undermines the uniqueness and immutability of the hash function.

A real-world example is in 2008, when researchers successfully forged a valid SSL certificate through a crash attack to make it appear to be issued by a trusted certificate authority (CA). This attack proves that MD5 is inadequate in the security of the public key infrastructure.

Pre-image attacks

A pre-image attack is when a hash is given and an input data is found to generate it. Although MD5 was designed with pre-image attack protection in mind, finding pre-images becomes more feasible as computing power increases.

Pre-image attacks allow an attacker to forge arbitrary data so that its hash is the same as the target hash.

In cryptography applications and digital signatures, pre-imaging attacks can have a serious impact on security. For example, if a digital signature system relies on an MD5 hash to verify the integrity and authenticity of the data, an attacker could forge a legitimate signature by finding arbitrary data that matches the target hash. This attack can lead to forged transactions, contracts, or other important documents being accepted as genuine and legitimate.

Rainbow Table Attack

A rainbow table is a precomputed hash table that contains a large number of possible hashes and their corresponding input data. By pre-computing and storing these hashes, attackers can find and match the target hashes in a very short time to find the corresponding plaintext data.

Rainbow table attacks make it easier and faster to crack hashes, especially for algorithms with short hash lengths like MD5. Attackers can use rainbow tables to quickly find the raw data corresponding to the hash value and crack passwords or other sensitive information stored in the database.

To defend against rainbow table attacks, the "salting" technique can be used. Salting is when a random value (salt) is added to the raw data, such as a password, and then the hash value is calculated. In this way, even the same raw data generates different hashes in different situations, increasing the difficulty of cracking.

MD5's obsolescence issues

With the improvement of computing power and the deepening of cryptography research, the security problems of MD5 are gradually exposed. The effectiveness of crash attacks and pre-image attacks makes MD5 unsuitable for many safety-critical applications.

Due to the security concerns of MD5, many systems and applications have gradually shifted to the use of more secure hashing algorithms, such as SHA-256 and SHA-3. These algorithms are more complex in design, with longer hash lengths and greater resistance to attacks.

Comparison of MD5 with other hashing algorithms

MD5 vs. SHA-1

SHA-1 (Secure Hash Algorithm 1) is another widely used hashing algorithm designed by the National Security Agency (NSA) of United States and published by the National Institute of Standards and Technology (NIST) of the United States. SHA-1 generates a hash of 160 bits, which is longer than MD5's 128 bits and is theoretically more secure.

Although SHA-1 is designed to be more secure than MD5, with the increase in computing power, SHA-1 is gradually exposing security issues. In 2017, Google and the Netherlands Institute for Information Security (CWI) successfully carried out a SHA-1 collision attack, further proving its insecurity.

SHA-1 has been widely used in digital signatures, certificate generation, and data integrity verification. However, as SHA-1 security issues have been exposed, many systems and applications have gradually shifted to more secure hashing algorithms such as SHA-256 and SHA-3.

MD5 vs. SHA-256

SHA-256 (Secure Hash Algorithm 256) is part of the SHA-2 family and generates a 256-bit hash value, providing greater security. SHA-256 is more complex in design and has a longer hash value, making it more effective against collision attacks and pre-image attacks.

Compared to MD5 and SHA-1, SHA-256 has a significant improvement in security. SHA-256 is designed with many modern cryptographic attack methods in mind, and to date, no effective collision or pre-image attacks have been found.

SHA-256 is widely used in Bitcoin and other cryptocurrencies, TLS/SSL certificates, digital signatures, and data integrity verification. Due to its high security, SHA-256 has become the hashing algorithm of choice for many modern security protocols and systems.

MD5 vs. SHA-3

SHA-3 (Secure Hash Algorithm 3) is the latest hashing algorithm standard, released by NIST in 2015. SHA-3 is based on the Keccak algorithm and is completely different from SHA-2 in design, providing greater security and flexibility.

SHA-3 is designed with the lessons learned from SHA-1 and SHA-2 to provide greater resistance to collision attacks and pre-image attacks. SHA-3's internal structure is more complex, making it excellent in the face of modern cryptographic attacks.

SHA-3 is suitable for scenarios that require high security, such as digital signatures, cryptography protocols, and data integrity verification. Although the use of SHA-3 is not yet as widespread as that of SHA-2, more and more systems and applications will gradually adopt SHA-3 over time.

An alternative to MD5

Application of SHA-256

SHA-256 can be used for file integrity checks, similar to MD5, but provides greater security. By calculating the SHA-256 hash of the file, users can verify that the file has not been tampered with or corrupted during transmission. For example, many open source projects and software distribution sites have started providing SHA-256 checksums for users to verify downloaded files.

SHA-256 is widely used in digital signatures and certificate generation. Due to its increased security, SHA-256 is able to effectively defend against collision attacks and pre-image attacks, ensuring the authenticity and integrity of digital signatures. Modern TLS/SSL certificates mostly use SHA-256 to generate hashes to improve communication security.

Use of HMAC

Hash-based Message Authentication Code (HMAC) is a hash-based message authentication code that verifies the integrity and authenticity of data. HMAC combines a hash function with a key to generate an authentication code by hashing the message and the key multiple times.

Despite the security issues inherent in MD5, HMAC-MD5 improves its ability to resist attacks by introducing keys. However, HMAC-SHA256 is gradually becoming the more commonly used option due to safety concerns. HMAC-SHA256 combines the high security of SHA-256 with the authentication features of HMAC, and is widely used in secure communication and data protection.

HMAC is commonly used in scenarios such as security protocols such as TLS and IPsec, API authentication, and data integrity verification. By using HMAC, the system is able to effectively verify the source and integrity of the data, preventing data tampering and falsification.

A new type of hashing algorithm

• Argon2 is a modern cryptographic hash function designed for cryptographic hashing and key derivation. Argon2 is designed with high parallelism and resistance to side-channel attacks in mind, providing strong security. Argon2 is available in three versions: Argon2d, Argon2i, and Argon2id, which focus on anti-GPU attacks, anti-time attacks, and hybrid attacks, respectively.
• Bcrypt is a cryptographic hash function based on the Blowfish encryption algorithm, which is widely used for password storage. Bcrypt increases the complexity and time of hash computation by introducing salt values and configurable cost factors, making it highly effective against rainbow table attacks and brute force attacks.

Write at the end

Finally, I would like to summarize for you:

MD5 (Message-Digest Algorithm 5) is a widely used cryptographic hash function that converts input data of any length into a fixed-length 128-bit (16-byte) hash value. This hash is usually expressed in the form of 32 hexadecimal numbers.

The main use of MD5 is to verify data integrity, for example, when data is transmitted or stored over a network, the MD5 hash value of the data can be calculated to detect whether the data has been tampered with or corrupted. However, since the MD5 algorithm has been proven to have some security vulnerabilities, such as collision attacks and pre-image attacks, it is no longer considered a secure cryptographic hash function and is not recommended for use in situations where a high level of security is required. In these cases, more secure hash functions, such as SHA-256 or SHA-3, should be prioritized.