22nd Friday Fun Session (Part 3) – 16th Jun 2017
When we have to implement an idempotent operation in web application, where the respective method execution is not idempotent when executed simultaneously, it is essential that we use concurrency control. Here we focus on .NET application deployed in IIS.
Idempotent operation
Suppose user requests to approve a certain transaction, say, transaction Id 100, hence it is data specific. We need to make sure this is an idempotent operation, meaning, no matter how many times the request comes, simultaneously or serially, the end result should be the same.
Concurrency and thread synchronization
If more than one request comes, from the same user or from multiple users simultaneously, we have seen IIS might launch multiple threads, one for each request. It is possible that they are executed simultaneously.
Let’s take a closer look. Suppose the two requests were initiated at 12:00:00:000 PM. The requests ended up in IIS at 12:00:01:000 PM, two threads starts processing them at the same time at 12:00:01:001 PM. Both the threads find that the transaction has not been approved yet, and proceed to do the same thing, like send an email, make some database entries etc. At the end both mark that the transaction is approved.
So we see that, what was supposed to be an idempotent operation has ended up sending two emails and so on, clearly failing to be one. If we don’t have any concurrency control in place, nothing is stopping from sending two emails from the two threads. After all, both the threads are executing simultaneously checking, finding and doing the same thing, at the exact same time.
Concurrency control is essential here. That means, at the beginning we need to place a gate through which only one thread can enter, at a time. Once it enters, it should mark that it is approving a certain transaction. Then all other threads who would enter the gate later serially (one after another), can detect that somebody else is processing that exact same request and quit gracefully.
Intra-process synchronization
We will walk through some options, even though they would not qualify to be the desired solution. We would do so just to explain the issues around the options, so that we can rationalize the final solution(s).
Let us start with lock, provided by C#. lock can implement critical section, meaning it can make a certain portion of the code executable only by one thread at a time. So the transaction processing method can be wrapped using a lock. That will make sure only one thread of a process is executing the method at any given point of time.
However, we have two issues here:
First, we are locking the whole method. We wanted to make sure only transaction Id 100 gets approved only by one thread at any point of time. But we end up blocking approval for all other transactions, say, transaction Id 99 or 101, when approval for transaction Id 100 is going on.
We can solve this issue by implementing a named lock, meaning the lock will have a name, say, TranApproval_100. That way, only threads executing approval requests for the same transaction will be executed serially. Instead of using lock, we can achieve the same using interned string, named mutex etc. as well.
Second, the scope of the lock is within the process. Nobody outside the process knows about it. However, we know that in web garden configuration, there might be more than one process running for the same web application and the two threads can come from two different processes. In that case, the thread in the first process would not know that the thread in the second process is having a lock. Hence both the threads would happily execute the same method simultaneously. We can solve this problem by using named mutex, an inter-process synchronization mechanism.
Inter-process synchronization
We see that using a named mutex, say, the name of the mutex is TranApproval_100, we can make sure we don’t block approval for other transactions. Meaning only thread approving transaction Id 100 will execute, without blocking approval for, say, transaction Id 99 or 101.
We also see that between the two threads running in two processes in web garden configuration, only one would do the approval and other would quit. This is because the named mutex is visible to all processes throughout the operating system.
However, we also know that in web farm configuration, the two processes, executing the two threads, both intending to serve the approval processing for transaction Id 100, might be running in two different web servers. In such a case, this inter-process synchronization mechanism that we just explored, would not work.
External and Common
So we see that we have to use something that is both external and common to web servers. If we have one common database that is used by all the processes then we can use something from the database to make sure that the threads are processing the requests one by one.
Using transaction within application
Using the concurrency control support provided by database, we can make sure only one method is doing the database operations, at any point of time. If we are using ORM, say Entity Framework, then we can use the transaction support provided from version 6 onwards (Database.BeginTransaction(), Commit, Rollback etc.). Since we want them execute serially, we know we have to use serializable isolation level System.Data.IsolationLevel.Serializable. We can begin the transaction with this isolation level parameter.
There are two problems associated with this. First, we are again serializing the whole transaction approval process, meaning blocking approval for transaction Id 101, while processing the same for transaction Id 100.
Second, we cannot stop non-database operations like sending email etc.
Named synchronization using database
The first problem can be solved by, as we have seen previously, using a named synchronization mechanism. The second problem can also be solved using the same, but we need to make sure we are using the synchronization mechanism to serialize the whole method, not just the database transaction.
Named lock table
We first create the following table that will keep track of the transactions, presently under processing.
CREATE TABLE [Web].[LockInfo]
(
[LockInfoId] [int] IDENTITY(1,1) NOT NULL,
[LockName] [nvarchar](256) NULL,
[DurationInSeconds] [int] NULL,
[CreatedOn] [datetime] NOT NULL,
CONSTRAINT [PK_LockInfoId] PRIMARY KEY CLUSTERED
(
[LockInfoId] ASC
)
WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF,
IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON)
ON [PRIMARY]
) ON [PRIMARY]
GO
ALTER TABLE [Web].[LockInfo] ADD DEFAULT ((60)) FOR [DurationInSeconds]
Lock name
LockName is the name of the lock, like TranAproval_100.
Duration
We should not hold the lock forever. If after acquiring the lock things go wrong, for example, the caller somehow misses to unlock it or there is a crash, this resource cannot be locked again by any thread in future. The created date along with the default duration of 1 minute would make sure, after a minute this lock becomes invalid. This is based on the assumption that approval processing would be done within a minute.
Using stored procedure
We create the following stored procedure.
CREATE Procedure [Web].[usp_Lock]
(@Lock BIT, @LockName NVARCHAR(256), @LockDuration INT = NULL)
AS
BEGIN SET TRANSACTION ISOLATION LEVEL SERIALIZABLE
BEGIN TRY
BEGIN TRANSACTION
DECLARE @Success AS BIT
SET @Success = 0
IF(@Lock IS NULL OR @LockName IS NULL)
BEGIN
SELECT @Success AS Success;
ROLLBACK TRANSACTION
RETURN;
END
IF(@Lock = 1) -- LOCK
BEGIN
DELETE FROM [Web].[LockInfo]
WHERE @LockName = [LockName] AND
DATEADD(SECOND, [DurationInSeconds], [CreatedOn]) < GETUTCDATE();
IF NOT EXISTS (
SELECT * FROM [Web].[LockInfo]
WHERE @LockName = [LockName])
BEGIN
IF(@LockDuration IS NULL)
INSERT INTO [Web].[LockInfo]
([LockName], [CreatedOn])
VALUES(@LockName, GETUTCDATE())
ELSE
INSERT INTO [Web].[LockInfo]
([LockName], [DurationInSeconds], [CreatedOn])
VALUES(@LockName, @LockDuration, GETUTCDATE())
SET @Success = 1
END
END
ELSE -- UNLOCK
BEGIN
DELETE FROM [Web].[LockInfo]
WHERE @LockName = [LockName]
SET @Success = 1
END
COMMIT TRANSACTION
END TRY
BEGIN CATCH
SET @Success = 0
IF(@@TRANCOUNT > 0)
ROLLBACK TRANSACTION
END CATCH;
SELECT @Success AS Success;
END
Lock and unlock
This stored procedure can be used for both lock and unlock operation of a named resource. The lock can be acquired for a given interval. In absence of any duration parameter, default duration of 1 minute will be used.
Leveraging isolation level
We make sure that this stored procedure can be executed serially, thus making sure only one transaction execute, at any point of time.
Here we focus on making the lock/unlock function faster, compromising concurrency by using the highest isolation level.
Leveraging consistency
We could have used unique constraint on lock name column in the Web.LockInfo table. In that case, we could use the default READ COMMITTED isolation of MS SQL Server, in the stored procedure, increasing concurrency. If a second thread would want to take the lock on the same resource, it would try to insert a row with the same lock name, resulting in failure, due to the unique constraint checking.
Performance
The two have different performance implications that we explain using an example.
Suppose two threads started executing simultaneously. The faster stored procedure takes one millisecond to execute. The first thread takes one millisecond to lock and then 100 milliseconds to execute the main approval processing, taking a total of 101 milliseconds.
The second stored procedure would wait one millisecond for the first stored procedure to lock. Then it would take one more millisecond to check that it is already locked, and hence it would take a total of 2 milliseconds to quit the processing.
On the other hand, suppose the stored procedure using default isolation level and unique constraint takes 3 milliseconds to lock. The first thread would take 103 milliseconds to execute.
The second thread that starts at the same time would take 3 milliseconds to fail while acquiring the lock and quit.
Based on the load and usage pattern, an appropriate mechanism can be chosen.
Usage for general purpose locking
This mechanism can be used not only for synchronizing transaction approval but all other cases that use named resources. Based on the load, more than one lock table (and/or stored procedure) can be used, each supporting certain modules.
GitHub: Named Resource Lock
Sharing Experiences 