Understanding Idempotency: Managing Repetitive Requests in Software

Today, I’d like to write about what I learned about the concept of ‘Idempotency’ which I encountered in Level 2 while reviewing REST API Maturity Levels. I wanted to write this in a conversational tone because I feel more comfortable and understandable that way :)

What is Idempotency?

The word “idempotency” actually originates from mathematics. The Latin word “idem” means “same,” while “potent” means “effective.” Combining the two, we get the meaning “having the same effect.” In mathematics, if an operation or function remains constant when applied repeatedly, it is called idempotent.

While Idempotency is a noun, idempotent is an adjective, so the way it is used in writing may vary.

Example:

• The function f(x)=∣x∣ is an idempotent function because the result does not change whether it is applied once or repeatedly.
• If f(x)=x+1, it is not idempotent; the result changes with each application.

The Place of Idempotency in Software and REST API Design

REST APIs operate over the HTTP protocol. Various operations are performed using HTTP methods such as GET, POST, PUT, PATCH, DELETE, and HEAD. Each method has its own purpose and impact on the system.

Idempotency is important for understanding how these HTTP methods should behave and for constructing the API design correctly.

For this reason, let’s evaluate HTTP methods in terms of idempotency:

GET

It’s used to retrieve data from the server. Whether we execute the same GET request one or thousands of times, the result will be the same; the data will remain unchanged (assuming the data isn’t modified during the operations). Therefore, GET methods are idempotent.

POST

It’s used to create a new resource or initiate an action (like uploading a photo to Instagram or confirming a purchase). When you send the same POST request repeatedly, a new resource can be created or the action can be repeated each time. Therefore, POST is generally not idempotent. However, this can be managed with methods like “idempotency keys.”

PUT

It is used to completely update a specific resource or create it if it doesn’t exist. No matter how many times the same PUT request is made, the result is always the same; the resource always contains the same data. Therefore, PUT is idempotent.

PATCH

It is used to partially update specific parts of a resource. It is generally not idempotent because different or cumulative changes may be made with each call. However, with careful design, it can be idempotent.

For example, with one patch, an employee’s salary can be increased by 20%, and with two consecutive updates, you can increase your employee’s salary by 44%.

DELETE

It is used to delete a specific resource. Even if the same DELETE request is repeated after the resource is deleted, the resource no longer exists and its system state remains unchanged. Therefore, DELETE is idempotent.

Note: If the endpoint is:

VERB : Delete

URI : kayas.dev/posts/last

The DELETE method may behave non-idempotently.

Press enter or click to view image in full size

How Does Idempotency of HTTP Methods Affect Our Application?

Idempotency directly impacts our software’s behavior, error handling, and therefore the user experience. Let’s illustrate this with concrete examples:

1. Network Problems and Duplicate Requests

For example, a user places an order on an e-commerce site and uses the POST method. When the user clicks the payment button, the connection is lost and the server doesn’t respond. The user assumes the transaction failed and resubmits the order.

If the POST request is not idempotent, the user will order the same product more than once — frustrating enough when you empathize :)

If the idempotency key is used, the system will process only the first request when it receives the same request again, returning the previous response for the others. This prevents repeat orders.

2. User Experience

A user transfers money in a banking app, but the connection drops and there’s no response. If the user presses the “Send” button again and idempotency isn’t applied, their money could be withdrawn twice.

Thanks to the idempotency key, duplicate requests are detected and the transaction is only processed once, ensuring a secure and seamless user experience.

3. Data Consistency and Security

You’re using the PUT method to update the user profile. Sending the same update multiple times won’t create data inconsistencies because it’s idempotent.

In partial updates such as PATCH, if idempotency is not provided, repeated requests may lead to data inconsistency.

4. Performance and Scalability

For example, in a hotel reservation system, customers frequently send GET requests to check into the same room. Because GET requests are idempotent, the system can cache these requests and respond quickly.

However, if POST requests for reservations are repeated and there is no idempotency, the same room could be booked for more than one person, creating problems for both the customer and the business.

So, are there any precautions that can be taken against these non-idempotent methods?

Yes, there is. The idempotency key is used to prevent problems that might arise from repeated requests, especially in non-idempotent methods like POST. We can say that this key adds idempotency to POST methods.

Press enter or click to view image in full size

Solution with Idempotency Key

Why is it needed?

The main problem with non-idempotent methods is that the same request is sent more than once and the operation is performed more than once.

Example situations:

• User repeatedly pressing the payment button
• Repeated requests due to network problems
• Repeated requests triggered repeatedly in mobile apps

These can lead to issues like duplicate payments, duplicate orders, or recreated data.

What is the Solution?

Processing only one of the consecutive requests and returning the first response to the others.

So the system must know:
“I’ve already done this process, there’s no need to repeat it.”

We achieve this with the idempotency key.

How Does Idempotency Key Work?

Client Side — Key Generation:
1. When the page loads or before the operation starts, the client generates a unique idempotency key. This is usually in GUID format. (It can be done by generating a random GUID via a script on the page and assigning it to the idempotency key variable inside the form.)

2. Sending Along with the Request:
When the user triggers the operation, this unique key is sent to the server as an HTTP header or as part of the form data.

3. Server Side — Key Verification:
The server checks whether a request with this key has been processed before by looking it up in the cache or database.
If the operation has not been performed yet, it processes the request and stores the result together with the key. If the operation has already been done, it does not repeat it and returns the previous result.

Press enter or click to view image in full size

Real-Life Example: Payment System

The user enters their card information and clicks the “Pay” button. The connection is slow and the response is delayed. The user clicks the same button multiple times.
If there is no idempotency key, the user may be charged multiple times.
But if there is an idempotency key, the server processes the first request, ignores the duplicates, and returns the same result.

Speaking of payment systems, I read a news report about a bug in Uber Eats India that is believed to be caused by idempotency issues. Here’s a summary of the news:

1) Paytm is a payment service provider working with Uber Eats India.
2) Paytm’s payment API originally returned a uniform error message.
3) Later, with an update, they added a different error message specifically for the second failed attempt.
4) However, this change was not properly communicated to API users such as companies like Uber Eats).
5) Uber Eats’ system was only expecting the old error messages and did not handle the new error message correctly.
6) Because of this, in some cases, failed transactions were mistakenly treated as successful, and orders were processed multiple times.

This is discussed in the following video:

Conclusion

Idempotency prevents errors that may arise from repeated requests, ensures data consistency and enables performance improvements, thus directly affecting user satisfaction.

In this article, I tried to put into writing what I’ve learned about Idempotency. There may be some mistakes in my writing, please excuse me :)

I wish you a good reading.