Write to Azure Blob Storage with Serilog in .NET Core

Similar to the previous post https://jaryl-lan.blogspot.com/2022/03/write-to-elasticsearch-with-serilog-in.html. Will be using Serilog library to write to Azure Blob Storage. Before we can start, you are required to have a Azure Storage Account up and running. Do not worry if you do not have an Azure account as you can actually run the emulator on your local machine. If you have Visual Studio 2022 installed on your machine, then Azurite (To replace Azure Storage Emulator) is included. Otherwise you will have to manually install it.

For those who has Visual Studio 2022. It is located at the following directory

Enterprise:
C:\Program Files\Microsoft Visual Studio\2022\Enterprise\Common7\IDE\Extensions\Microsoft\Azure Storage Emulator

Professional:
C:\Program Files\Microsoft Visual Studio\2022\Professional\Common7\IDE\Extensions\Microsoft\Azure Storage Emulator

For those does not have Visual Studio 2022 installed. 1 alternative is to run it with docker. You may be wondering on the ports defined. Port 10000 is for blob service, 10001 is for queue service and lastly 10002 is for table service.

Pull the azurite image:
docker pull mcr.microsoft.com/azure-storage/azurite

Run the azurite:
docker run --rm -p 10000:10000 -p 10001:10001 -p 10002:10002 mcr.microsoft.com/azure-storage/azurite

Other than the Azurite, you are required to install Microsoft Azure Storage Explorer to browse the logs written into the blob storage.

Back to the code, make sure the following 2 NuGet packages is included.

Serilog.AspNetCore

Serilog.Sinks.AzureBlobStorage

You may use 1 of the following methods to be implemented into your Program.cs file.

Method 1: Configure Serilog in code.

[.NET 6] - Minimal hosting model

builder.Host.UseSerilog((hostBuilderContext, loggerConfiguration) =>
    loggerConfiguration
        .WriteTo.Console()
        .WriteTo.AzureBlobStorage(
            "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;",
            Serilog.Events.LogEventLevel.Information,
            storageContainerName: "writetoazureblobstoragewithserilog",
            storageFileName: "WriteToAzureBlobStorageWithSerilog-{yyyy}-{MM}-{dd}.txt"));
SelfLog.Enable(Console.Error);

[.NET 5 and earlier]

Host.CreateDefaultBuilder(args)
    .UseSerilog((hostBuilderContext, loggerConfiguration) => {
        loggerConfiguration
        .WriteTo.Console()
        .WriteTo.AzureBlobStorage(
            "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;",
            Serilog.Events.LogEventLevel.Information,
            storageContainerName: "writetoazureblobstoragewithserilog",
            storageFileName: "WriteToAzureBlobStorageWithSerilog-{yyyy}-{MM}-{dd}.txt");
        SelfLog.Enable(Console.Error);
    })
    .ConfigureWebHostDefaults(webBuilder =>
    {
        webBuilder.UseStartup<Startup>();
    });

Method 2: Configure Serilog from appSettings file.

[appSettings.json]

"Serilog": {
  "WriteTo": [
    {
      "Name": "Console"
    },
    {
      "Name": "AzureBlobStorage",
      "Args": {
        "ConnectionString": "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;",
        "storageContainerName": "writetoazureblobstoragewithserilog",
        "storageFileName": "WriteToAzureBlobStorageWithSerilog-{yyyy}-{MM}-{dd}.txt"
      }
    }
  ]
}

[.NET 6] - Minimal hosting model

builder.Host.UseSerilog((hostBuilderContext, loggerConfiguration) =>
    loggerConfiguration
        .ReadFrom.Configuration(hostBuilderContext.Configuration));

SelfLog.Enable(Console.Error);

[.NET 5 and earlier]

Host.CreateDefaultBuilder(args)
    .UseSerilog((hostBuilderContext, loggerConfiguration) =>
    {
        loggerConfiguration
            .ReadFrom.Configuration(hostBuilderContext.Configuration);
        SelfLog.Enable(Console.Error);
    })

For more details on the setup and the ConnectionString, do refer this link https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azurite?tabs=visual-studio

Sample Code: https://1drv.ms/u/s!Aj2AA7hoIWHm1HIzst4No3QEe1A0?e=lnWzHc

Write to ElasticSearch 8 with Serilog in .NET Core

Continuation from the previous post https://jaryl-lan.blogspot.com/2022/02/write-to-physical-file-with-serilog-in.html which talks about writing to physical file through Serilog. For this round will be to write logs to Elasticsearch with Serilog.

First and foremost, you need to have Elasticsearch up and running. So if you do not have it, you will have to install it on your local machine. Otherwise ignore the installation steps. There are few ways to go about it but I will go through using docker command with minimal steps.

There are 2 images required. Elasticsearch and Kibana. Kibana is required for you to view the logs.

1. Run these commands to grab the Elasticsearch and Kibana image

docker pull docker.elastic.co/elasticsearch/elasticsearch:8.0.1
docker pull docker.elastic.co/kibana/kibana:8.0.1

2. Run this command to create a network. To be used by Elasticsearch and Kibana late

docker network create elasticsearch

3. Run this command to start the Elasticsearch

docker run --rm --name localelasticsearch --net elasticsearch -p 9200:9200 -p 9300:9300 -t docker.elastic.co/elasticsearch/elasticsearch:8.0.1

Note: If the Elasticsearch stopped on its own, most likely it is due to error. If it shows the following error, you are then required to increase the max virtual memory.

ERROR: [1] bootstrap checks failed. You must address the points described in the following [1] lines before starting Elasticsearch.
bootstrap check failure [1] of [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

The command to set the maximum virtual memory is heavily depend on where the docker is installed. In my case the docker is on the wsl. So if is different, you may refer from this link to identify which command is suitable https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#_set_vm_max_map_count_to_at_least_262144.

So for wsl, firstly terminal into the wsl with the following command.

wsl -d docker-desktop

Optional: you can use this command to check what is the current max virtual memory value. In my case it is showing 65530.

cat /proc/sys/vm/max_map_count

Run the following command to set the max virtual memory value to 262144

sysctl -w vm.max_map_count=262144

Once done, type exit and hit enter key to exit from the wsl terminal.

exit

Run this command again to verify that the Elasticsearch did not terminate on its own.

docker run --rm --name localelasticsearch --net elasticsearch -p 9200:9200 -p 9300:9300 -t docker.elastic.co/elasticsearch/elasticsearch:8.0.1

4. Run this command to start the kibana.

docker run --rm --name localkibana --net elasticsearch -p 5601:5601 docker.elastic.co/kibana/kibana:8.0.1

5. Open up your browser and browse to http://localhost:5601. It may asked you for a token. the token can be obtained from the console output that you run on step 3. The token should display below the following text

6. After fill up the token. The portal will request for 6 digit code, which printed on the console that runs the kibana in step 4. Look for the keyword "Your verification code is:".

7. Finally, it will asked you to fill up the login credentials. The username is elastic, whereas the password is printed on the console output that runs the elasticsearch in step 3.

With this, you have completed the setup of Elasticsearch and kibana on your local machine. For more details on the setup steps, you may refer to this link https://www.elastic.co/guide/en/kibana/current/docker.html.

With the environment ready to use. Let's get back to the code. 

The NuGet packages requires are as follows

• Serilog.AspNetCore
• Serilog.Sinks.Elasticsearch

Head to the Program.cs file and configure the UseSerilog().

[Configure Serilog in code]

Note 1: The username and password is the same credentials used to login into kibana.
Note 2: As of this writing, the Serilog.Sinks.Elasticsearch is having issue writing to Elasticsearch version 8. You will get hit with the following error due to the deprecation of type parameter.

Action/metadata line [1] contains an unknown parameter [_type]

For more detail, do refer on this link https://github.com/serilog-contrib/serilog-sinks-elasticsearch/issues/375. As mention in the link, we will need to add TypeName = null. Also due to this issue, we can't use the configuration way to configure writing logs into Elasticsearch.

[.NET 6] - Minimal hosting model

builder.Host.UseSerilog((hostBuilderContext, loggerConfiguration) =>
    loggerConfiguration
        .WriteTo.Console()
        .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("https://yourusername:yourpassword@localhost:9200"))
        {
            TypeName = null,
            AutoRegisterTemplate = true,
            IndexFormat = "WriteToElasticsearchWithSerilog-{0:yyyy-MM-dd}",
        }));

SelfLog.Enable(Console.Error);

[.NET 5 and earlier]

.UseSerilog((hostBuilderContext, loggerConfiguration) => {
    loggerConfiguration
        .WriteTo.Console()
        .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("https://yourusername:yourpassword@localhost:9200"))
        {
            TypeName = null,
            AutoRegisterTemplate = true,
            IndexFormat = "WriteToElasticsearchWithSerilog-{0:yyyy-MM-dd}",
        });
    SelfLog.Enable(Console.Error);
})

The reason of adding the SelfLog.Enable(Console.Error); is to print out any errors that happens when writing logs to Elasticsearch. Which helps in troubleshooting. Also you may consider changing it to write into physical file or any other storage.

If you not able to log to Elasticsearch and received the following error:

Caught exception while preforming bulk operation to Elasticsearch: Elasticsearch.Net.ElasticsearchClientException: The SSL connection could not be established, see inner exception..
Call: Status code unknown from: POST /_bulk
 ---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
 ---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid because of errors in the certificate chain: UntrustedRoot

It means that you do not have the cert used by the Elasticsearch since Elasticsearch create its own cert (Self-Signed Certificate) when it launched the first time. There are ways to modify your code to forcefully ignore certificate error. In this case we are not going into that route. But instead add the certificate into the "Trusted Root Certification Authorities" store.

Before we can add it to the certificate store, we need to grab the certificate generated by the Elasticsearch. In the official link https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#elasticsearch-security-certificates. it specified /usr/share/elasticsearch/config/certs, which is the path of the certificate generated in the Elasticsearch container and we need the file http_ca.crt. To grab it out to your physical folder, run the following command.

Command argument to copy file from container to physical folder:

docker cp <Elasticsearch container name>:<file path in container> <Physical file path>

Sample command to copy out the http_ca.crt to c drive:

docker cp localelasticsearch:/usr/share/elasticsearch/config/certs/http_ca.crt C:/http_ca.crt

Once the file has been copied out. Locate the certificate file and run it.

1. Click Install Certificate... button.

2. Maintain as Current User and click Next button.

3. Select Place all certificates in the following store and click Browse... button.

4. Select Trusted Root Certification Authorities and click OK button

5. Click Next button. On the next screen click Finish button.

6. If it prompt a Security Warning window, click Yes button.

Once the certificate has been added to the certificate store. Rerun your application and the error will no longer appear.

If you need to rely on configuration file instead of code to configure, probably advisable to use version 7 instead due to the library yet to properly support version 8 of Elasticsearch. Do follow up from their github issue https://github.com/serilog-contrib/serilog-sinks-elasticsearch/issues.

Sample Code: https://1drv.ms/u/s!Aj2AA7hoIWHm1HCLWNJsI6ZrS1Sb?e=TnyW0T

Write to Physical File with Serilog in .NET Core

It is common for a need to log application errors or including trace log for ease of troubleshooting issues that may arise. By default, the .NET itself already provided a few classes (For example: FileStream, TextWriter, etc) that allows you to write content into file. But what if you want to include logs that are written by the .NET itself, or customize the .NET log content, or write the same content to many different destination (eg: File, ElasticSearch, BlogStorage, etc). Then Serilog can help you with it.

For the prerequisites, it is required to NuGet the following 2 packages

• Serilog.AspNetCore
• Serilog.Sinks.File

Next, head to Program.cs file. You will need to call to the extension method UseSerilog() to start using Serilog.

Method 1: Configure Serilog in code.

[.NET 6] - Minimal hosting model

builder.Host.UseSerilog((hostBuilderContext, loggerConfiguration) =>
    loggerConfiguration
        .WriteTo.Console()
        .WriteTo.File("logs/log.txt", rollingInterval: RollingInterval.Day));

[.NET 5 and earlier]

Host.CreateDefaultBuilder(args)
    .UseSerilog((hostBuilderContext, loggerConfiguration) =>
        loggerConfiguration
            .WriteTo.Console()
            .WriteTo.File("logs/log.txt", rollingInterval: RollingInterval.Day))

Method 2: Configure Serilog from appSettings file.

[appSettings.json]

"Serilog": {
  "WriteTo": [
    {
      "Name": "Console"
    },
    {
      "Name": "File",
      "Args": {
        "path": "logs/log.txt",
        "rollingInterval": "Day"
      }
    }
  ]
}

[.NET 6] - Minimal hosting model

builder.Host.UseSerilog((hostBuilderContext, loggerConfiguration) =>
    loggerConfiguration
        .ReadFrom.Configuration(hostBuilderContext.Configuration));

[.NET 5 and earlier]

Host.CreateDefaultBuilder(args)
    .UseSerilog((hostBuilderContext, loggerConfiguration) =>
        loggerConfiguration
            .ReadFrom.Configuration(hostBuilderContext.Configuration))

What the above code snippet does is to write the same log content into console and log file. 

Based on the specified path value, the file format will be log{yyyyMMdd}.txt (example: log20220227.txt). And since the configuration mentioned the rolling interval is Day, everyday a new file will be created whenever there are new content to be logged. For more details on what can be further customized for the file, you can head over to https://github.com/serilog/serilog-sinks-file.

You may be wondering what is the difference of the above code snippet compared to creating a new instance of logger with CreateLogger(). First would be to include .NET logs into the file. Second, you will be able to rely on the Microsoft logging library (Microsoft.Extensions.Logging) and inject the instance of ILogger to any of your class through the constructor. Any logs written through the ILogger will be written into your file as well as the console output.

Sample Code: https://1drv.ms/u/s!Aj2AA7hoIWHm1G4BIALQnYew9KFY?e=bIa0tq

Multi Target Framework for .NET Core Application

It is possible to target multiple framework for .NET Core application. But it have some limitations. For example, if the project target netcoreapp3.1 and net6.0, features that are available on net6.0 will not longer be supported. Due to this, it is rarely to do so unless you got hit with situation where you want to upgrade to newest .NET version but due to certain servers yet to installed the latest .NET core runtime.

First and foremost, make sure you have the targeted framework installed. Otherwise you get hit with an error while compiling your code. For example, if you are targeting netcoreapp3.1 and net6.0, make sure both of them is installed on your machine.

To get started, open up your project by double clicking on it and look for TargetFramework. Rename it to TargetFrameworks and specify net6.0;netcoreapp3.1

<TargetFrameworks>net6.0;netcoreapp3.1</TargetFrameworks>

Next item to look out for is the NuGet packages. You can specify different NuGet package version to be installed for different framework. As an example, I only have 1 NuGet package, which is Swashbuckle.AspNetCore as shown below. 

<ItemGroup>
  <PackageReference Include="Swashbuckle.AspNetCore" Version="6.2.3" />
</ItemGroup>

You are required to remove it and then create 2 separate ItemGroup as shown below. By doing so, Swashbuckle.AspNetCore version 6.2.3 will be installed for net6.0, whereas version 5.6.3 will be installed for netcoreapp3.1

<ItemGroup Condition="$(TargetFramework)=='net6.0'">
  <PackageReference Include="Swashbuckle.AspNetCore">
    <Version>6.2.3</Version>
  </PackageReference>
</ItemGroup>
<ItemGroup Condition="$(TargetFramework)=='netcoreapp3.1'">
  <PackageReference Include="Swashbuckle.AspNetCore">
    <Version>5.6.3</Version>
  </PackageReference>
</ItemGroup>

Once you saved your changes, you may notice the Background Task which locate at the bottom left of your visual studio is in progress of grabbing the NuGet packages for your project. Make sure to wait it to complete before compiling your code. Also, you will be able to see 2 different dependencies in your project under the Solution Explorer.

Sample Code: https://1drv.ms/u/s!Aj2AA7hoIWHm1G0qpM0ULUc8CBGv?e=cXWr2t