Performance gotchas

Recently we upgraded one of our customers to .NET 5, Content Cloud 12 and Commerce Cloud 14. In the first days after launch we experienced some performance issues.

The behavior was typically that the site was super fast and performing well, but then after a while and usually with heavy traffic the site became non responsive. At this point one or several of the servers were “dead” and the only option was restarting the web app.

General impression was that code that had been running fine in .NET Framework v4, were now causing severe performance issues.

By using Diagnose and solve problems tool in Azure we quickly identified and fixed the following issues:

Do not use httpClient in using statements - may lead to SNAT port exhaustion

When an instance’s SNAT ports are exhausted, the following symptoms can be observed from the application:

• Slow and pending on connecting to the remote endpoint
• Socket exceptions when the connections timeout in the web application

using(var client = new HttpClient())
{
    //do something with http client
}

The using statement is well know in C#: once the using block is complete the disposable object, in this case HttpClient, goes out of scope and is disposed.

But for HttpClient this should be avoided. Although it implements the IDisposable interface it is actually a shared object. Instead of creating a new instance of HttpClient for each execution you should share a single instance of HttpClient for the entire lifetime of the application.

In .NET Core the recommendation is create a reuasable client through an injected IHttpClientFactory object.

// register specific client in StartUp.cs
services.AddHttpClient("MyClient", (provider, client) => {
    var settings = provider.GetRequiredService<MyClientSettings>();
    client.BaseAddress = new Uri(settings.Url);
    client.DefaultRequestHeaders.Add("x-api-key", settings.ApiKey);
 });

// Inside service that needs to connect to client
public MyService(IHttpClientFactory httpClientFactory) 
{
   _httpClient = httpClientFactory.CreateClient("MyClient");
}

We applied the same principle for DbContext connections:

// DBContext through using statements
using (var context = new DbContext()) 
{
  ...
}

// setup using preferred EF Core way with connection pooling
services.AddPooledDbContextFactory<DbContext>(o => o.UseSqlServer(efConnection.ConnectionString), 1024);


// factory for easy access
public class MyDbContextFactory : IMyDbContextDbContextFactory
{
    public MyDbContextFactory(IDbContextFactory<MyDbContext> contextFactory)
    {
        _contextFactory = contextFactory;
    }

    /// <summary>
    /// Returns a disposable DbContext
    /// </summary>
    /// <returns></returns>
    public MyDbContext GetContext()
    {
        return _contextFactory.CreateDbContext();
    }
}

Reference articles:
https://4lowtherabbit.github.io/blogs/2019/10/SNAT/#how-to-solve-a-snat-port-exhaustion-issue-for-app-service
https://www.aspnetmonsters.com/2016/08/2016-08-27-httpclientwrong/

My colleague, Valdis, has written a blog post about the same topic, related to service bus connections:
https://tech-fellow.eu/2022/01/29/how-to-easily-exhaust-snat-sockets-in-your-application/

Mixing asynchronous and synchronous calls leads to deadlocks

Avoid mixing sync code with async code as that can lead to deadlocks. This issue is probably more known, but if you (like us) inherited a code base, they might hide somewhere inside your project. So please, do a quick search and make sure these are handled.

// suspicious code - should be fixed
var result = Task.Run(async () => await someService.GetMethodAsync()).Result;

To read more about the issue, and how to solve it, please read the following blog post:
https://medium.com/rubrikkgroup/understanding-async-avoiding-deadlocks-e41f8f2c6f5d