The Reformed Programmer

When you are writing Entity Framework Core (EF Core) queries (i.e. reading data from the database) you want them to be quick to run (high performance) and quick to write. It turns out that it’s possible to do that in EF, but the “quick to run” part isn’t that obvious. I make a living around writing EF Core code quickly, and that performs well (I wrote the book, “Entity Framework Core in Action”) and my approach is to use the LINQ Select method (quick to run) and the AutoMapper library (quick to write).

This article is about both “quick to run” and “quick to write”, but majors on AutoMapper, as the Select method is easy to understand, but AutoMapper does take a bit of getting used to. I also mention my EfCore.GenericServices library that uses AutoMapper, which speeds up the writing of database accesses even more.

TR;DR; – summary

LINQ Select method normally produces the fastest performing query because they only load what you need, and some calculations can be run in the database (see Posts.Count later) .
Writing Select queries contains more lines of code than other query approaches because you have to select each property you want to load.
AutoMapper’s ProjectTo method can build the Select query for you, but you need to configure the mapping.
Setting up the AutoMapper configuration isn’t total obvious (to me at least) so I give you some examples.
My GenericServices library uses AutoMapper and does some of that configuring for you.

This article is aimed at developers who use Microsoft’s Entity Framework library, so I assume you are familiar with C# code and either Entity Framework 6 (EF6.x) or Entity Framework Core library. I don’t assume you know the AutoMapper library, but I don’t detail all the features of AutoMapper and give links to its documentation. The examples in this article use EF Core, but all of the ideas are transferable to EF6.x.

Setting the scene – A quick intro to Entity Framework Core

Database accesses are normally referred to as CRUD operation (Create, Read, Update and Delete), with the read referred to in EF as a query. Queries are normally the most common operation, and often the one you want to be quick (Google and Amazon searches have conditioned us to expect very fast searches). Sometimes the query is a list of things, like products you can buy or trains you could catch, and sometimes you need a single item, like a calendar event you want to change. When the data is stored in a relational database, like SQL Server, a query might need data from multiple parts of the database, called tables, to form the exact data the user wants to see.

The EF Core library makes it easy to access a database by mapping the database tables to .NET classes (known as entity classes), and uses the database pointers (known as foreign keys) between tables to create references (known as navigational properties) to other entity classes. It also turns your LINQ commands into the correct language/format used by the database you want to access. The figure below shows four entity classes, which represent a set of Posts, each with a Blogger who wrote it, and some Tags (via a many-to-many PostTag linking entity class). The relational properties are represented in red.

In general, the LINQ Select operator in Entity Framework Core (EF Core) can produce database queries that are often perform better than any other approach. This is because the Select method allows you to pick exactly the properties you want, including looking into related classes. This performance gain is especially good when part of the query can be done inside the database, e.g. like counting how many posts a blogger has written (I show that later).

The problem is that writing a Select query, with an assignment for each property, is very repetitious and a bit boring (and boring is bad, as I can make mistakes when I am bored). In this article I show how you can use a library called AutoMapper to automatically build Select queries, which saves you time (and you don’t need to be bored anymore). I also cover one of my libraries that uses AutoMapper to make Create, Read, Update and Delete (known as CRUD) database accesses easier.

In the next section I am going to build some queries to show this database in different ways.

Part 1. Building a simple database query – a list Bloggers

The first display I want to build a list of each blogger, with the number of posts they have written. Here is the type of display I want to show.

This must pick out information from two of entity classes. Now, let’s see the different ways of obtaining that data.

1a. Using Eager loading to load the Post classes

One way to do that would be to use Eager Loading. The code would look like this.

var list = context.Bloggers
    .Include(x => x.Posts)
    .ToList();

That gets the data we need, but it pulls in ALL of the post’s content, when I only want to count them. That is very inefficient, because the Posts can be very big. If you need good performance then you don’t want to write your queries this way.

NOTE: You get EVEN WORSE performance if you load the Posts via Lazy Loading, e.g. having a virtual public ICollection<Post> of Posts. That will cause one extra trip to the database for every entry. I get paid to performance tune EF systems and the first thing I look for is Lazy Loading – it’s a major performance drag.

See the Summary of the performance table for detailed timings.

1b. Using a hand-coded Select query

Over the years I have found the LINQ Select queries to be more efficient, where I only select the properties I need into a class I call a Data Transfer Object (DTO), also known as a ViewModel in ASP.NET. Here is my ListBloggersDto class that I am going to copy the data into

 
public class ListBloggersDto 
{
    public string Name { get; set; }
    public string EmailAddress { get; set; }
    public int PostsCount { get; set; }
}

And here is a hand-coded Select query that will only read in the data I actually need.

 
var list = context.Bloggers.Select(x =>
    new ListBloggersDto
{
    Name = x.Name,
    EmailAddress = x.EmailAddress,
    PostsCount = x.Posts.Count
}).ToList();

This is more efficient as it only loads the data it needs. This is especially true for the count of the Blogger’s Posts – this is done in the database, which stops us loading all the Posts just to count them. The SQL that EF Core would create from this query looks like this (don’t worry if you don’t know SQL – take it from me that is very efficient).

 
SELECT "x"."Name", "x"."EmailAddress", (
    SELECT COUNT(*)
    FROM "Posts" AS "p"
    WHERE "x"."BloggerId" = "p"."BloggerId"
) AS "PostsCount"
FROM "Bloggers" AS "x"

Summary of the performance

In section 12.5.1 on my book “Entity Framework Core in Action” I built a query to load the data to create the book display list (see http://efcoreinaction.com/ for live example site) using three approaches. Here is the result of the three different ways of loading the data. As you can see the Select query was the fastest.

NOTE: I didn’t test Lazy loading as EF Core 2.1 wasn’t released when I was writing that chapter, but I can tell you it would be equal (or maybe worse) to the explicit loading timing.

Now, for our much simpler example the Select method would have a sub-millisecond timing because the query is so simple, and would be much quicker than any of the other loading approaches. But the table above gives you a better comparative of a complex query (see my article “Entity Framework Core performance tuning – a worked example” for more on performance)

Quick to write – AutoMapper

Having looked at the “quick to run” part lets move onto the “quick to write” part. The query examples so far are pretty easy to write, but in bigger Select queries you would have to write many assignments, and that gets tedious. This is where AutoMapper comes in, so lets now recreate the same query using AutoMapper.

1c. Using AutoMapper to build the Select query automatically

AutoMapper is what is known as an object-to-object mapper, and can selectively map from one class to another, handling any relationships. AutoMapper can also produce LINQ code via its ProjectTo<T> method (see Queryable Extensions). This allows you to let AutoMapper build your Select queries for you by matching up names. There are three stages to building a Select query using AuthoMapper:

Create your DTO
Add an AutoMapper configuration,
Then use that AutoMapper configuration in your query.

Let’s go through each part in turn

a) Create my DTO

I need to create my DTO, which is the same as the hand-coded one.

 
public class ListBloggersDto
{
    public string Name { get; set; }
    public string EmailAddress { get; set; }
    public int PostsCount { get; set; }
}

The first two properties have the same name and type as properties in the Blogger class, so they are copied. The interesting property is the PostsCount – this is translated as Posts.Count(). That’s a AutoMapper feature which is helpful.

NOTE: AutoMapper can also ‘flatten’ relationships’: what that means it can pick out a one-to-one relationship by joining together the two names without the dot in the middle. You will see this in part 2, where I list each Post, but access the Blogger’s Name by using the name of the relationship, Blogger, followed by the property’s name, Name, i.e. BloggerName.

b) Configure the AutoMapper mapping via dependency injection

There are lots of ways to configure AutoMapper mappings. For NET Core the recommends using AutoMapper’s Profile and assembly scanning with NET Core’s Dependency Injection (DI).

For this example, I use a DI approach to configuring your AutoMapper mapping in an ASP.NET Core web application. First you add a class that inherits AutoMapper’s Profile class, most likely near to where you define your DTOs. In class’s constructor you add one or more CreateMap methods to set up the mapping. Here is a simple example.

 
public class BloggerDtosProfile : AutoMapper.Profile
{
    public BloggerDtosProfile()
    {
        CreateMap<Blogger, ListBloggersDto>();
        // Add other CreateMap’s for any other configs
    }
}

Then you need to call AutoMapper’s AddAutoMapper() method. In ASP.NET Core that would be in the ConfigureServices method in the Startup class, e.g.

 
public IServiceProvider ConfigureServices(
    IServiceCollection services)
{
    // … other setups removed for clarity
    services.AddAutoMapper();
}

The AddAutoMapper() method scans all the assemblies looking for all the classes that inherit AutoMapper.Profile class. It then produces a IMapper instance containing the all the Mappings found.

NOTE: I find the configuring AutoMapper is the hard part, as it relies on me remembering to set up the AutoMapper mapping. I have tried a number of ways to make this better and I think in my EfCore.GenericServices I have got it right. In that I write my DTO and then you need to add an empty interface called ILinkToEntity<TEntity>, where TEntity is your class that EF Core maps to the database. Then the setup of EfCore.GenericServices a) finds the DTOs and then uses the TEntity part of the ILinkToEntity<TEntity> interface to, among other things, form the correct AutoMapper mappings.

c) Use AutoMapper in your query

I’m now going to show you a ASP.NET Core controller so that you can see how the DI works, but you can use AutoMapper in any application you like.

 
public class HomeController : Controller
{
    private readonly BlogContext _context;
    private readonly IMapper _mapper;

    public HomeController(BlogContext context, IMapper mapper)   
    {                                              
        _context = context;  
        _mapper = mapper;                      
    }                                              

    public IActionResult Index()            
    {
        var dtos = _context.Bloggers
            .ProjectTo<ListBloggersDto>(_mapper)     
            .ToList();   

        return View(dtos);         
    }
}

I have injected by EF Core content and the AutoMapper’s IMapper instance into the controller via the constructor. Then any of my methods in the ASP.NET Core controller. And the SQL code produced is the same as the hand-coded Select earlier in 1b.

For this small DTO then it isn’t worth using AutoMapper, but most applications have hundreds of mappings like this, and classes that contains a lot more properties. This is where AutoMapper comes into its own.

NOTE: If you are generally doing CRUD operations then you should look at my article “GenericServices: A library to provide CRUD front-end services from a EF Core database” that uses AutoMapper and EF Core to make CRUD very simple to do.

AutoMapper is clever and can work out some relationships for you, but for more complex relationships you need to do some more configuring, which I cover in the next part.

Part 2: Handling more complex database queries – list of posts

The next example is a summery list of all the articles, showing the bogger (author), the post title and the tags the post has.

This is a bit more complex, especially around getting the tag names.

2a. List posts using hand-coded select query

Let’s start with the hand-coded LINQ Select query. First, I need to show you the DTO I use to hold the resulting query.

 
public class ListPostsDto 
{
    public string BloggerName { get; set; }
    public string Title { get; set; }
    public DateTime LastUpdated { get; set; }
    public List<string> TagNames { get; set; }
}

Now the hand-coded Select query.

 
var dtos = context.Posts.Select(x => 
    new ListPostsDto
{
    BloggerName = x.Blogger.Name,
    Title = x.Title,
    LastUpdated = x.LastUpdated,
    TagNames = x.TagLinks.Select(y => y.Tag.Name).ToList()
}).ToList();

Notice the following:

Line 4: I can select the Name of the Blogger by using the navigational property. EF can handle any depth of selection, say the Bogger class had a property called Address, which has its own table holding address information, then I could access the Blogger’s Country via x.Blogger.Address.Country. EF produces efficient SQL (an INNER JOIN) to extract the data from the related tables.
Line 7: I want a list of all the Tag Name’s associated with this post. I therefore need to use the PostTag linking table to access the Tag names.

The produces fairly efficient (see note below) SQL query by only extracting the data you need.

NOTE: Handling “many” relationships, i.e. ones that return multiple results like the TagNames, is an area where EF Core can have performance issues. If I left off the .ToList() on the TagNames then it would query the database for each ListPostsDt0 (this is known as the N+1 database queries issue). But in EF Core version 2.1, adding the .ToList() turns the load of all the TagNames into one final database query. This is OK, but can still have problems – see my article “Entity Framework Core performance tuning – a worked example” to see an even better way to handle comma separated strings.

2b. List of posts using AutoMapper and special configuration

AutoMapper is clever enough to automatically map the first three properties in the ListPostsDto – in particular it maps the BloggerName property to Blogger.Name. What it can’t do is work out the mapping for the TagNames, so we need to add some configuration the CreateMap method. I use the original ListPostsDto I used in the hand-coded version, but add the AutoMapper configuration code:

 
public class ListPostsDto
{
    public string BloggerName { get; set; }
    public string Title { get; set; }
    public DateTime LastUpdated { get; set; }
    public List<string> TagNames { get; set; }
}

Now the AutoMapper configuration code.

 
public class PostsDtosProfile : AutoMapper.Profile
{
    public PostsDtosProfile()
    {
        CreateMap<Post, ListPostsDto>()
            .ForMember(
                p => p.TagNames, 
                opt => opt.MapFrom(x => 
                    x.TagLinks.Select(y => y.Tag.Name).ToList()));
    }
}

The ForMember method (lines 6 to 9) takes two parts: the first part defines the property in the DTO that you are mapping (TagNames, on line 7), and the second part (lines 8 to 9) tells AutoMapper how to generate the data to go into the TagNames property from the input class, which is the Post class.

NOTE: AutoMapper automatically turns the BloggerName into Blogger.Name – this is the “flattening” I referred to earlier. This is a very useful feature and makes handling references to a single relationship really easy.

Let’s see the list articles query using AutoMapper’s ProjectTo method. In case I’ll show you a unit test as you might need this in your system. I also show how to create the IMapper variable using a helper method I created, as that is useful too (and checks you created the Profile class correctly).

 
[Fact]
public void TestProjectionMappingPosts()
{
    //SETUP
    var config = AutoMapperHelpers
        .CreateMapperConfig<PostsDtosProfile>();

    //ATTEMPT
    var input = EfTestData.CreateBloggersWithPosts()
        .AsQueryable();
    var list = input.ProjectTo<ListPostsDto>().ToList();

    //VERIFY
    list.First().BloggerName.ShouldEqual("Ada Lovelace");
}

Where my helper method CreateMapperConfig, looks like this.

NOTE: Technical point about unit testing. ProjectTo<T> need MapperConfiguration when testing, while Map<T> needs IMapper. Have a look at my unit tests of AutoMapper here for a good example of how to write tests that include AutoMapper.

 
public static MapperConfiguration CreateMapperConfig<T>() 
    where T : Profile, new()
{
    var config = new MapperConfiguration(cfg =>
    {
        cfg.AddProfile(new T());
    });
    return config ;
}

This produces the same SQL as the hand-codes example.

NOTE: In EfCore.GenericServices you can configure the AutoMapper mappings by adding a class which inherits from PerDtoConfig<TDto, TEntity>. When EfCore.GenericServices is scanning for DTOs it also looks for matching PerDtoConfig<TDto, TEntity> classes uses that data to alter the AutoMapper’s MapperConfiguration.

Conclusion

Firstly, I showed you how to create high performance queries by using the LINQ Select method. The Select query isn’t obvious – in fact the EF Core documentation on loading relationships doesn’t even list it even though you can load relationships in a Select method. But the ASP.NET Core EF tutorial lists Select as an option, as I was asked to help develop that tutorial . If you want fast and effective queries then think about using the LINQ Select method.

The problem with the Select method is it takes some work to write it, e.g. when Selects have lots of properties in them then they get long and repetitious (i.e. boring). I showed that AutoMapper can help by automatically build the Select query for you. While AutoMapper is a great tool, but it takes a bit of getting used to, especially around the configuration. But when you have lots of queries then it’s worth the effort.

Along the way I talk about my EfCore.GenericServices library, which uses AutoMapper inside. This handles the configuration of AutoMapper for you, which makes it a bit easier. But on the other hand, you need to learn how to set up EfCore.GenericServices. I have written an introductory article, some documentation, and an example application in the repo that you can run locally (it uses an in-memory database, so it runs anywhere), so don’t be afraid to try it.

Happy coding.

I was asked by one of my clients to help build a fairly large web application, and their authentication (i.e. checking who is logging in) and authorization (i.e. what pages/feature the logged in user can access) is very complex. From my experience a knew that using ASP.NET’s Role-based approach wouldn’t cut it, and I found the new ASP.NET Core policy-based approach really clever but it needed me to write lots of (boring) policies.

In the end I created a solution for my client and this article describes the authorization part – I call it Roles-to-Permissions (the name will make more sense as you read the article). I have also build an example ASP.NET Core application, with all new code to support this article. This example application is quite different from my client’s system as I tap into ASP.NET Core built-in Identity system (the client’s system needed OAuth2). The example application contains about 60 lines that I copied (with my client’s permission) from the original implementation to create an open-source version (MIT licence) you can use.

NOTE: you can Clone the GitHub repo and run locally – it uses in-memory databases so it will run anywhere. The application was written using ASP.NET Core 2.1: parts of the ASP.NET Identity is changing, but the overall concept will work with any version of ASP.NET Core.

This is a long article, so here are links to the major parts:

TL;DR; – summary

NET Role-based authorization system works for systems with simple authorization rules, but it has limitations, like the fact that you have to republish your code if you change the authorization rules.
Roles, with names like “Manager” or “ExternalBuyer” makes sense for users (human or external services) as they define a “Use Case” of what the user should be able to do.
But Roles don’t work well when applied to ASP.NET Core actions or Razor Pages. Here you need a much more fine-grained solution, with names like “CanRequestHoliday”, “CanApproveHoliday” – I call these Permissions.
The solution is to map the user’s Roles to a group of Permissions and store these in the User’s Claims.
Then I use ASP.NET Core’s new policy-based authorization system to check that the User’s Permissions Claims contains the Permission placed on the action/page they want to access.
There is an open-source example ASP.NET Core application to go with this article.

Setting the Scene – a look at different application security needs

If you understand ASP.NET’s authorization and authentication features then you can skip this section.

There are billions of web applications and the control of what you can do ranges for “anyone can do anything”, e.g. Google search, up to some military systems where access needs keys, biometrics etc. When you need to prove you are a valid user of the system, say by logging in, that is referred to as authentication. Once you are logged in then what you can do is controlled by what is called authorization.

Authorization breaks down into two parts:

What data can I access? For instance, you can see your personal information, but not other people’s personal information.
What features you can use? For instance, are you allowed to change the title of a book that you can see?

NOTE: This article only describes a way to manage the second part, what features can you use.

ASP.NET MVC and now ASP.NET Core have various systems to help with authorization and authentication. Some systems only need a simple authorization – I could imagine a very simple e-commerce system could get away with: a) No logged in – browsing, b) Logged in – buying, and c) Admin – Add/Remove items for sale. This could be done using ASP.NET Role-based authentication.

But many business-to-business (B-to-B) systems have more complex authorization needs. For instance, think of a human resources (HR) system where people can request holiday leave and their manager has to approve those requests – there is lot going on inside to ensure only the correct users can use these features.

Systems like my example HR B-to-B system often end up with lots of complex authorization rules. My experience is that the ASP.NET Role-based authentication starts to have problems implementing this type of system, which is why I created the Roles-to-Permissions code.

Another type of application that could benefit from the Roles-to-Permissions approach are subscription systems, where the features a user can access depend on what subscription they paid for. The Roles-to-Permissions approach can control the features that as user can access based on the subscription they bought.

Role authorization: what is it and what are its limitations?

Roles authorization has been around for years in the ASP.NET MVC application, and I have used it in a number of applications. Here is an example of a ASP.NET Core controller that can only be accessed by logged in users with either the “Staff” Role or the “Manger” role.

[Authorize(Roles = "Staff,Manager")]
public ActionResult Index()
{
    return View(MyData);
}

This works for applications that have fairly simple and well-defined Roles, like User/Admin or Staff/Manager/Admin, then Roles is a good choice. But here are some of the problems I have found:

If you have lots of roles you can end up with long Authorize attributes, e.g. [Authorize(Roles = “Staff,HrManager,BizManager,DevManage,Admin,SuperAdmin”)].
Because Authorize is an attribute then the string has to be a constant, e.g. you can’t have $”{RoleConstants.StaffRole}”. This means if things change you are editing strings, and you could misspell something really easily.
The big one for me is your authorization rules are hard-coded into your code. So, if you want to change who can access a certain action you have to edit the appropriate Authorize attributes and redeploy your application.

My experience from previous applications using Roles-based authorization is me constantly having to go back and edit the authorize Roles part as I develop or refine the application. I have been looking for a better way for some time, and my client’s requirements spurred me on to find something better than Roles authorization.

The architecture of the Roles-to-Permissions system

1. Introducing Roles, Permissions and Modules

It turns out that there is nothing wrong with the idea of a user having Roles. A user (human or an external service) can typically can be described by their function or department, like “Developer” or “HR”, maybe with side Roles like “HolidayAdmin”. Think of Roles as “Use Cases for users.

NOTE: In the example application I have Roles of “Staff”, “Manager”, and “Admin.

But the problem is that Roles aren’t a good fit for the actions in the Controllers. Each Controller action has a little part to play in a Role, or to turn it around, a Role is made up of a series of Controller actions that the Role allows you to access. I decided I would call the authorization feature on each action a “Permission”, and I used an Enum to define them. A permission Enum member might be called “CanReadHoliday”, “CanRequestHoliday”, “CanApproveHoliday”, etc.

NOTE: In the example application I have Permissions on my ColorController of “ColorRead”, “ColorCreate”, “ColorUpdate”, and “ColorDelete”.

Now that we have permissions we can provide another feature that of controls access to optional features, e.g. features that a user only has based on their subscription to the service. There are many ways of handling features but by combining optional features into the permissions makes it simpler to setup and control.

NOTE: In the example application I have Permissions called “Feature1” and “Feature2” which are mapped to Modules with the same name.

2. How this is implemented

Having defined my terms, I’m going to give you an overview of the process. It consists of two parts: the login stage and the normal accesses to the web site. The login stage is the most complex with lots of magic goes on in the background. Its basic job is to convert the user’s Roles into Permissions and add it to the User’s information.

I have set up my example application to store the user’s claims in a cookie which is read in with every HTTP request and turned into a ClaimsPrincipal, which can be accessed in ASP.NET Core by the HttpContext property called “User”.

Here is a diagram of that login stage. It might not make a lot of sense yet, but I describe each part in the rest of the article. This diagram is to give you an overview.

The second part is simpler and covers what happens every time the logged-in user accesses a protected Controller action. Basically, I have a policy-based authorization with dynamic rules that checks the current User has the permission needed to execute the ASP.NET action/razor page.

NOTE: Don’t forget there is example application if you want to look at the actual code.

Now I’m going to build up the Roles-to-Permissions in stages and explain what each part does.

Why I used Enums for the Permissions

One of the down-sides of using Roles is it used strings, and I’m a little bit dyslexic. That means I can type/spell things incorrectly and not notice. Therefore, I wanted something where intellisence would prompt me and if I still typed it incorrectly it would be a compile error. But it turns out there are a couple of other reasons that make using an Enum for the permissions a good idea. Let me explain.

In a big application there could be hundreds of Permissions. This lead to two problems:

If I use Cookie Authorization there is a maximum size of 4096 bytes for the Cookie. If I had hundreds of long strings I might start to fill up the Cookie, and I want some room for other things like my data authorization. If I can store the Enums permissions as a series of integers it’s going to be much smaller than a series of strings.
Secondly, I want to help the Admin person who needs to build the mapping from Roles to permissions. If they need to scroll through hundreds of permission names it could be hard to work out which ones are needed. It turns out Enum members can have attributes, so I can add extra information to help the Admin person.

So, here is part of my Permissions Enum code

public enum Permissions
{
    //Here is an example of very detailed control over something
    [Display(GroupName = "Color", Name = "Read", Description = "Can read colors")]
    ColorRead = 0x10,
    [Display(GroupName = "Color", Name = "Create", Description = "Can create a color entry")]
    ColorCreate = 0x11,
    [Display(GroupName = "Color", Name = "Update", Description = "Can update a color entry")]
    ColorUpdate = 0x12,
    [Display(GroupName = "Color", Name = "Delete", Description = "Can delete a color entry")]
    ColorDelete = 0x13,

    [Display(GroupName = "UserAdmin", Name = "Read users", Description = "Can list User")]
    UserRead = 0x20,
    //This is an example of grouping multiple actions under one permission
    [Display(GroupName = "UserAdmin", Name = "Alter user", Description = "Can do anything to the User")]
    UserChange = 0x21,

    [Obsolete]
    [Display(GroupName = "Old", Name = "Not used", Description = "example of old permission"
    OldPermissionNotUsed = 0x40,

The things to note are:

I show two types of permissions.
- First four (lines 4 to 1) are fine-grained permissions, almost one per action.
- Next two (lines 13 to 17) are more generic, e.g. I have a specific “UserRead”, but then one permission called “UserChange” which allows create, update, delete, lock, change password etc.
Line 5, 7, etc. Notice that I give each enum a specific number. If you are operating a 24/7 application with a new version seamlessly replacing the old version, then the Permission numbed mustn’t change otherwise user’s Claims be wrong. That is why I give each enum a specific number.
Line 19. I also support the Obsolete attribute, which stops the Permission appearing in the listing. There are plenty of scary stories about reusing a number with unintended consequences. (Also, it you try to use something marked as Obsolete you get a warning).
Line 4 etc. I add a Display Attribute to each Permission Enum. This has useful information that I can show lots of useful information to help the person who is building a Role.
Line 4, 6, 8, 10. I “Group” permissions that are used in the same place. This makes it easier for the Admin person to find the things they want. I also number in Hex, which gives me 16 possible permissions in a Group (I tried 10 and you could go over that, so 16 is better).

Here is a list of some of the Permissions in my example application listed via the Users->List Permissions nav dropdown.

And the code that produced that output (link to PermissionDisplay class for the whole thing)

public static List<PermissionDisplay> GetPermissionsToDisplay(Type enumType) 
{
    var result = new List<PermissionDisplay>();
    foreach (var permissionName in Enum.GetNames(enumType))
    {
        var member = enumType.GetMember(permissionName);
        //This allows you to obsolete a permission and it won't be shown as a
        //possible option, but is still there so you won't reuse the number
        var obsoleteAttribute = member[0].GetCustomAttribute<ObsoleteAttribute>();
        if (obsoleteAttribute != null)
            continue;
        //If there is no DisplayAttribute then the Enum is not used
        var displayAttribute = member[0].GetCustomAttribute<DisplayAttribute>();
        if (displayAttribute == null)
            continue;

        //Gets the optional PaidForModule that a permission can be linked to
        var moduleAttribute = member[0].GetCustomAttribute<PermissionLinkedToModuleAttribute>();

        var permission = (Permissions)Enum.Parse(enumType, permissionName, false);

        result.Add(new PermissionDisplay(displayAttribute.GroupName, displayAttribute.Name, 
                displayAttribute.Description, permission, moduleAttribute?.PaidForModule.ToString()));
    }

    return result;
}

How to handle optional/paid-for features?

My client provides a Business-to-Business application and plans to add new features that customers can subscribe to. One way to handle this would be create different Roles, like “Manager”, “ManagerWithFeature1”, “ManagerWithFeature2” or add separate Feature Roles that you have to manually apply to a user. That works but is pretty horrible to manage, and human error could cause problems. My preferred system is mark Permissions linked to a paid-for feature filter them based on the User’s subscriptions.

Marking Permissions as linked to a module is easy to do with the Enums – I just add another attribute. Here an example of Permissions linked to a Module (see line 5).

public enum Permissions
{
    //… other Permissions removed for clarity

    [LinkedToModule(PaidForModules.Feature1)]
    [Display(GroupName = "Features", Name = "Feature1", Description = "Can access feature1")]
    Feature1Access = 0x30,
    [LinkedToModule(PaidForModules.Feature2)]
    [Display(GroupName = "Features", Name = "Feature2", Description = "Can access feature2")]
    Feature2Access = 0x31
}

The paid-for modules are again represented by an Enum, but one marked the [Flags] attribute because a user can have multiple modules that they have subscribed to. Here is my PaidForModules Enum code

[Flags]
public enum PaidForModules : long
{
    None = 0,
    Feature1 = 1,
    Feature2 = 2,
    Feature3 = 4
}

NOTE I add “: long” to the Enum which gives me up to 64 different modules in my system.

What happens is that Permissions linked to a Module that the user hasn’t subscribe to are filtered out during the login stage (I show how later). This makes the setting up the Roles much simpler, as you build each Role with all the Permissions that make sense for that role, including features mapped to a paid-for module. Then, at login time, the system will remove any Permissions the current user doesn’t have access to. That is simpler for the Admin person and more secure for the application.

How do I turn the Roles into a Permissions Claim?

I my client’s system we uses 0Auth2 authentication, but for this example I used ASP.NET Core IdentityRole to hold the Roles that a user has. That means I can use all of the ASP.NET Core built-in Identity code to set up the Users and Roles. But how do I convert the User’s Roles to a Permissions Claim?

Again there are few ways to do it, but in the end I tapped into an event in the Authorization Cookie called ‘OnValidatePrincipal’ (here is a link to the lines in the example application startup class). This calls the code below, but be warned it’s pretty complex so here is a summary of the steps it goes through:

If the Claims already have the Permissions claim type then nothing to do so return quickly.
Then we get the Roles the user has from the Role Claim
I need to access my part of the database. I can’t use dependency injection, so I use the extraAuthDbContextOptions, which is a singleton that I can provide at startup.
Then I get all the permissions for all of the roles, with a Distinct to remove unnecessary duplicates.
Then I filter out any permissions that are linked to a Module that the user doesn’t have access to.
Then I add a permissions Claim containing all the Permissions the user is allowed, but packed as hex numbers in a single string so that it doesn’t take up so much room (I used Hex format as it made debugging easier).
Finally I have to create a new ClaimsPrincipal and tell ASP.NET Core to replace the current ClaimsPrincipal, plus set the all-important ShouldRenew to true, which updates the Cookie, otherwise this complex (slow) method on every HTTP request!

public async Task ValidateAsync(CookieValidatePrincipalContext context)
{
    if (context.Principal.Claims.Any(x => 
        x.Type == PermissionConstants.PackedPermissionClaimType))
        return;

    //No permissions in the claims so we need to add it
    //This is only happens once after the user has logged in
    var claims = new List<Claim>();
    foreach (var claim in context.Principal.Claims)
    {
        claims.Add(claim);
    }

    var usersRoles = context.Principal.Claims
        .Where(x => x.Type == ClaimTypes.Role)
        .Select(x => x.Value)
        .ToList();
    //I can't inject the DbContext here because that is dynamic, 
    //but I can pass in the database options because that is a 
    //From that I can create a valid dbContext to access the database
    using (var dbContext = new ExtraAuthorizeDbContext(_extraAuthDbContextOptions))
    {
        //This gets all the permissions, with a distinct to remove duplicates
        var permissionsForUser = await dbContext.RolesToPermissions
            .Where(x => usersRoles.Contains(x.RoleName))
            .SelectMany(x => x.PermissionsInRole)
            .Distinct()
            .ToListAsync();

        //we get the modules this user is allows to see
        var userModules =
            dbContext.ModulesForUsers
                .Find(context.Principal.Claims
                     .SingleOrDefault(x => x.Type == ClaimTypes.Name).Value)
                ?.AllowedPaidForModules ?? PaidForModules.None;
        //Now we remove permissions that are linked to modules that the user has no access to
        var filteredPermissions =
            from permission in permissionsForUser
            let moduleAttr = typeof(Permissions).GetMember(permission.ToString())[0]
                .GetCustomAttribute<LinkedToModuleAttribute>()
            where moduleAttr == null || userModules.HasFlag(moduleAttr.PaidForModule)
            select permission;

          //Now add it to the claim
          claims.Add(new Claim(PermissionConstants.PackedPermissionClaimType,
              filteredPermissions.PackPermissionsIntoString()));    }

    var identity = new ClaimsIdentity(claims, "Cookie");
    var newPrincipal = new ClaimsPrincipal(identity);

    context.ReplacePrincipal(newPrincipal);
    context.ShouldRenew = true;
}

How do I convert my Permissions into Policy-based authorization?

OK, I now have access to the Permissions via the User’ Claims, but how do I get this turned into something that ASP.NET Core can use for authorization. This is where a .NET developer and friend, Jerrie Pelser helped me.

When I started this project, I emailed Jerrie Pelser, who runs the ASP.NET Weekly newsletter (great newsletter! Do sign up) as I know Jerrie is an expert in authentication & authorization. He pointed me at a few architectural things and I also found his own article “Creating Authorization Policies dynamically with ASP.NET Core” really helpful. Jerris’s article showed me how to build policies dynamically, which is exactly what I need.

I’m not going to repeat Jerrie article here (use the link above), but I will show you my PermissionHandler that is used inside the policy to check that the current User’s Permissions claim exists and contains the Permission on the action/Razor Page. It uses an extension method called ThisPermissionIsAllowed which does the check.

public class PermissionHandler : 
    AuthorizationHandler<PermissionRequirement>
{
    protected override Task HandleRequirementAsync(
        AuthorizationHandlerContext context, 
        PermissionRequirement requirement)
    {
        var permissionsClaim = context.User.Claims
            .SingleOrDefault(c => 
                 c.Type == PermissionConstants
                     .PackedPermissionClaimType);
        // If user does not have the scope claim, get out of here
        if (permissionsClaim == null)
            return Task.CompletedTask;

        if (permissionsClaim.Value
            .ThisPermissionIsAllowed(
                 requirement.PermissionName))
        {
            context.Succeed(requirement);
        }

        return Task.CompletedTask;
    }
}

There are two other classes that are involved in making dynamic policy-based authorisation work. Here are the links to them:

The AuthorizationPolicyProvider that is registered at startup.
A rather boring, but necessary, PermissionRequirement

Policies are defined by strings, but as I said I hate strings as I can make a mistake. I therefore created this very simple HasPermission attribute which allows me to apply an Authorize attribute, but using a Permissions Enum

[AttributeUsage(AttributeTargets.Method 
    | AttributeTargets.Class, Inherited = false)]
public class HasPermissionAttribute : AuthorizeAttribute
{
    public HasPermissionAttribute(Permissions permission) 
       : base(permission.ToString()) { }
}

That’s pretty simple, but it means I get intellisence when I am adding the Permission.

Putting it all together

So, we have the Permissions in the code and we can apply them using our HasPermissionAttribute to each action we want to protect via authorization. Here is one action taken from the ColorController in my example application.

[HasPermission(Permissions.ColorRead)]
public ActionResult Index()
{
    return View(MyData);
}

We also need to add two tables to whatever database your application uses. The two EF Core entity classes are:

RoleToPermissions, which maps an ASP.NET Core Role to a set of Permission Enums
ModulesForUser, which holds what modules each user is allowed to access.

Once the application is up and running an Admin-type user has to:

Create some roles, e.g. “Staff”, “Manager”, etc. using ASP.NET Core Identity code.
Create matching RoleToPermissions for each of the roles, specifying what Permissions map to each Role.

Then, for every new user an Admin person (or some automatic subscription code) has to:

Create the user (if an invite-only type application)
Add the correct Roles and ModulesForUser for that new user.

Once that is done all the code I have shown you takes over. The user logs in, gets the Permissions and what they can access is managed by ASP.NET Core’s policy-based authentication.

Things I didn’t cover elsewhere

There are a few things I didn’t cover in detail, but here are links to the items:

The important parts about registering things are shown in highlighted lines in this link to the Startup class. (NOTE: I built the application with ASP.NET Core 2.1, but I know the Identity parts are changing in 2.2, so you might have to update the code I put in the Startup class for newer versions of ASP.NET Core).
You don’t need to use ASP.NET Core Identity system at all – I said the client’s version uses an external authentication system. You just have to create a Roles-to-User table so you can assign Roles to each user.
I didn’t cover how I Packed/Unpacked the Permissions. You can find the Extension methods for doing that in PermissionPackers.
You might want to check for a permission in your razor pages to show/hide links. I created a simple method in the PermissionsExtension class, and used it in the _layout.cshtml Razor page.

Conclusion

Well that is a long article so well done by getting to the end. I have described an authentication I have built that handles complex authentication rules while being (relatively) easy to understand and manage via the Admin staff. Sure, if you have hundreds of Permissions it’s not to be hard setting up the initial RolesToPermissions, but the Admin has a lot of information to help them.

For me the Roles-to-Permissions approach solves a lot of problems I had in older systems I built using ASP.NET MVC Roles. I have to write some more code, but it makes it a) easier to change the authorization rules and b) helps the Admin person manage applications with lots of Roles/Permissions. I hope it helps you think of better ways of building better authentication systems for your projects.

TL;DR – summary

The GenericServices library bridges the gap between the database and the front-end user: the database is designed around the business domain (say, e-commerce), while the used wants pages that is appropriate to their need (say, listing all the cookery books). The library implements a read/write Adapter pattern, and a Command pattern for writes to the database.

My experience with a previous library for EF6.x was it could save 2 months on a 12 month project.

Each CRUD page takes a little bit of code, but applications have lots and lots of such pages. Thankfully these pages have a set of common patterns, and GenericServices is built to implement these patterns – you just need to provide a ViewModel/DTO (Data Transfer Object) which contains the properties for the CRUD action you want to accomplish. (I use DTO in the rest of this article)

GenericServices feature are:

Easy to use: just mark your ViewModel/DTOs to tell GenericServices how to use it.
Reduces the amount of CRUD code you need to write – just a ViewModel/DTO and two calls.
Can work with standard EF Core entity classes and Domain-Driven Design (DDD) styled entity classes.
Contains good checking and security features, especially with DDD-styled entity classes.

NOTE The GenericServices library is open-source (MIT licence) on GitHub and is available on NuGet as EfCore.GenericServices. The code, including an example ASP.NET Core Razor Pages application, is available on GitHub here. There is extensive documentation in the project’s Wiki.

Who should read this article?

If you are building applications with many front-end CRUD displays, then the GenericServices library will help you develop these pages quickly. GenericServices turns each CRUD access into a common set of calls around your display-specific ViewModel/DTO (I will use DTO from now on for this class). That minimises the code you need to write, which saves you time.

You do need to be using EF Core for your database access, as GenericServices is designed around EF Core (the NuGet package is called EfCore.GenericServices). GenericServices is designed to work with EF Core entity classes (i.e. the classes mapped to the database) that use the standard style (properties are changed by setting the property) and Domain-Driven Design (DDD) styled entity classes, where data is updated via methods in the entity class.

To keep this article short, I assume you know C# and have some idea of what the ASP.NET Core framework does. The examples I use in this article use ASP.NET Core’s Razor Pages, but in the article “How to write good, testable ASP.NET Core Web API code quickly” I show how to use GenericServices with Web APIs.

The problem that GenericServices tackles

GenericServices covers CRUD displays especially in web/mobile applications. I’m going to take an “update of an entity” to describe the typical issues that come up. My example application is an e-commerce site selling technical books and I implement a feature where an authorised user can add a sales promotion to a book, by reducing its price. I will show you two forms: a hand-coded version and a version where I use GenericServices.

To set the scene let’s look at what my “add a promotion” feature on a ASP.NET Core web site. Here is the look at the display, which the user has filled in (in red), with comments on the left about the Book properties and how they are involved in the update.

In a web/mobile application a feature like this consists of two stages:

The display of the data to the user where they can put in their changes (see figure above)
Once the user presses the button (“Add” in this case) the information is sent back and the database update is done.

In the “Add promotion” example the first stage needs to read five properties from the database to show the user. In the second stage, triggered by the user pressing the “Add” button”, the primary key (BookId) is used to select the Book to be updated, and then the NewPrice and PromotionalText values are updated in the book.

NOTE: All my examples are synchronous, but GenericServices contains async versions of every command.

1. The hand-coded version

Let’s start by looking at a hand-coded implementation of a service in an ASP.NET Core application. This sets the standard in terms of lines of code, and performance (I compare the performance of the hand-coded version with the GenericServices version near the end of this article).

I start with the DTO, and then the code to read and update the promotion

public class AddPromotionDto
{
    [HiddenInput]
    public int BookId { get; set; }

    public decimal OrgPrice { get; set; }

    public string Title { get; set; }

    public decimal ActualPrice { get; set; }

    public string PromotionalText { get; set; }
}

And now the code to read in the data and then update the Book entity with the new promotion.

public class AddPromotionService : 
    StatusGenericHandler, IAddPromotionService
{
    private readonly EfCoreContext _context;

    public AddPromotionService(EfCoreContext context)
    {
        _context = context;
    }

    public AddPromotionDto GetOriginal(int id)      
    {
        var dto = _context.Books
            .Select(p => new AddPromotionDto
            {
                BookId = p.BookId,
                Title = p.Title,
                OrgPrice = p.OrgPrice,
                ActualPrice = p.ActualPrice,
                PromotionalText = p.PromotionalText
            })
            .SingleOrDefault(k => k.BookId == id);
        if (dto == null)
            AddError("Sorry, I could not find the book you were looking for.");
        return dto;
    }

    public Book AddPromotion(AddPromotionDto dto)
    {
        var book = _context.Find<Book>(dto.BookId);
        if (book == null)
        {
            AddError("Sorry, I could not find the book you were looking for.");
            return null;
        }
        //This is a standard entity class, where you update by setting the properties
        Book.ActualPrice = dto.ActualPrice;   
	 Book.PromotionalText = dto.PromotionalText;
        _context.SaveChanges();                 
        return book;
    }
}

2. The GenericServices version

Now let’s look at how GenericServices would do this. Firstly, we change the DTO slightly, as shown below.

public class AddPromotionDto : ILinkToEntity<Book>
{
    [HiddenInput]
    [ReadOnly(true)]
    public int BookId { get; set; }

    [ReadOnly(true)]
    public decimal OrgPrice { get; set; }

    [ReadOnly(true)]
    public string Title { get; set; }

    public decimal ActualPrice { get; set; }

    public string PromotionalText { get; set; }
}

Two things have changed:

GenericServices needs to know what entity class your DTO links to, which you do by adding a ILinkToEntity<TEntity> The interface is empty, i.e. you don’t have to add anything to your DTO – it’s just there for GenericServices to find the DTO and extract the entity class type from the interface.
You also need to tell GenericServices which properties are read-only, so it won’t copy them back to the database (that’s a security feature to stop malicious updates). You do this by adding the ReadOnly(true) attributes to the DTO properties that are read-only.

NOTE: Technically you don’t need ReadOnly(true) attributes if you have DDD-styled entity classes, as your entity class method will control access to the data inside the entity class (but I do add them as it allows GenericServices to better match the correct create/update method). But you must add the ReadOnly(true) attributes is you are using standard entity classes as it tells GenericServices what properties should be updated, and which ones should not be updated.

Now the new code for reading and then adding the promotion using GenericServices’ CrudServices.

public class GenericAddPromotionService 
{
    private readonly ICrudServices _service;

    public IStatusGeneric Status => _service;

    public GenericAddPromotionService(ICrudServices service)
    {
        _service = service;
    }

    public AddPromotionDto GetOriginal(int id)
    {
        return _service.ReadSingle<AddPromotionDto>(id);
    }

    public void AddPromotion(AddPromotionDto dto)
    {
        _service.UpdateAndSave(dto);
    }
}

In fact, there is little gain in wrapping the GenericServices’ calls like that and I tend to put the GenericServices directly in the front-end. Here is an example of a Razor page using

public class AddPromotionModel : PageModel
{
    private readonly ICrudServices _service;

    public AddPromotionModel(ICrudServices service)
    {
        _service = service;
    }

    [BindProperty]
    public AddPromotionDto Data { get; set; }

    public void OnGet(int id)
    {
        Data = _service.ReadSingle<AddPromotionDto>(id);
        if (!_service.IsValid)
        {
            _service.CopyErrorsToModelState(ModelState, Data, nameof(Data));
        }
    }

    public IActionResult OnPost()
    {
        if (!ModelState.IsValid)
        {
            return Page();
        }
        _service.UpdateAndSave(Data);
        if (_service.IsValid)
            return RedirectToPage("BookUpdated", new { message = _service.Message });

        //Error state
        _service.CopyErrorsToModelState(ModelState, Data, nameof(Data));
        return Page();
    }
}

What is interesting is the C# code for the hand-coded razor page is the same l length as the GenericServices razor page. This means, if use GenericServices directly in the Razor Page I have saved me from writing 38 lines of code (The AddPromotionService is 41 lines, but I needed to add three [ReadOnly(true)] attributes to the DTO).

But more than that, the Create, Read, Update and Delete patterns are same, apart from the DTO type (line 11). This means I could copy the AddPromotion PageModel code and paste it in a PageModel to say update the publication date of the book, and all I would need to change is the type of the DTO (obviously the razor .cshtml file needs to change – I can’t do that for you (yet)).

GenericServices saves you from having to write repetitive (and boring!) CRUD front-end code. So now you can concentrate on the more complex parts of the application.

For me even a small business application will have more than 50 or more CRUD views, which mean GenericServices can save me from writing many hundreds of lines of CRUD code. I know this because I build a similar library called GenericServices for EF6.x back in 2014 and I have used it on many applications. It has saved me many months of my life from writing CRUD code (and hopefully the same for the other people who downloaded it).

NOTE For anyone who has used my original GenericServices library then the new EfCore.GenericServices provides the same feature, but is built in a very different way. Partly that is because it is designed to handle DDD-styles entity classes, and partly from feedback about making things simpler. I hope you find this new library (more) useful to you, as the original GenericServices library.

How does GenericServices work inside?

GenericServices has commands that cover the Create, Read, Update and Delete database actions. And every command can work with either an entity class or a DTO with a ILinkToEntity<TEntity> interface (apart from Delete, which only works on an entity class). But my experience is that a large proportion of CRUD accesses use DTOs, either to select that is shown to the user or to limit what gets updated – in these situations GenericServices comes into its own.

How it read data from the database

For reading, e.g. ReadSingle<AddPromotionDto>(id), GenericServices uses AutoMapper’s ProjectTo method if it’s a DTO, or simple EF Core Find<TEntity>(id) if the class you provide is an entity class. AutoMapper is a great library with impressive performance (see later) and does a great job on building LINQ Select queries.

NOTE: It’s beyond the scope of this article, but you can alter the AutoMapper read mapping that GenericServices uses by creating a PerDtoConfig<TDto,TEntity> class and overriding the AlterReadMapping property. This means you can changed the default mapping on a per-DTO basis – see the Wiki documentation for more on this.

How it creates a new entity or updates an existing entity

First, let’s look are the create or update of an entity class, which is pretty simple as its already in the correct form.

Create entity: It calls EF Core’s Add method and then SaveChanges
Update entity: If the entity has an EF Core State of Detached then it will call Update, otherwise it assumes you have updated it and will call SaveChanges.

When it comes to create and update the process used depends on whether the entity class is standard style entity or DDD-styled entity. Personally, I really like DDD-styled entity classes but I can see some simple classes, say like entity holding an Address, are quicker to write as a standard style entity.

1. Create or update a standard style entity via a DTO

In this case it uses AutoMapper’s Map method to copy the DTO properties to the entity it has loaded. The default mapping will not copy properties that have the [ReadOnly(true)] attribute on them. Also, it won’t update a property that has a private setter.

NOTE Just like in the read case you have access to the AutoMapper save mapping by creating a PerDtoConfig<TDto,TEntity> class and overriding the AlterSaveMapping property.

2. Create or update a DDD-styled entity via a DTO

GenericServices looks for a constructor/method whose parameters’ name and type match the non-read-only properties. There are lots of features here, but let me give you an example from the AddPromotion example.

The Book entity class has a method called AddPromotion, which looks like this.

public IStatusGeneric AddPromotion(decimal actualPrice, string promotionalText)                  
{
    var status = new StatusGenericHandler();
    if (string.IsNullOrWhiteSpace(promotionalText))
    {
        status.AddError(
            "You must provide some text to go with the promotion.", 
            nameof(PromotionalText));
        return status;
    }

    ActualPrice = actualPrice;  
    PromotionalText = promotionalText;

    status.Message = $"The book's new price is ${actualPrice:F}.";

    return status; 
}

NOTE: In this case the update method returns a type of IStatusGeneric, which allows it to provide a status result (methods that don’t need a status can return void).

GenericServices default process is to find an update method where the method’s parameter’s name/types match the name/type of the non-read-only DTO properties (the name match handles camel/pascal case matching). In this case GenericServices finds the method called AddPromotion and calls that method.

NOTE: There are obviously some complicated situations here, like having two methods that match, or methods that don’t need a property. This documentation for create and update goes into all the matching processes and how to override them.

As you can see from the AddPromotion update method there are several benefits over setting properties in a standard styled entity. In this case I included some checks and returned a success message that could be shown to the user. There are many advantage to using DDD-styled entity with its create and update methods – I recommend you look at my article “Creating Domain-Driven Design entity classes with Entity Framework Core” for more on this.

What about the performance of GenericServices?

I am concerned about performance and I checked this regularly during development. Here are some timings produced by BenchmarkDotNet library (which I found excellent for performance testing). I used the fastest database I could find, which turned out to be SQLite in-memory database, so that the database part was minimised. You can find the performance tests in the project called Benchmarking.

The first example is for the simplest possible update – in this case I update the publication date of a book. I opened up the Book’s PublishedOn property so I could update it via AuthoMapper as well as via a DDD-styled update method.

Method	Mean (us)	Error (us)	StdDev (us)
HandCoded, PropetyUpdate	530.9	4.003	3.343
GenericService, AutoMapper	551.4	5.624	5.261
	4%
HandCoded, MethodCall	529.1	5.583	5.223
GenericServices, MethodCall	555.7	5.828	5.451
	5%

This is about the fastest database update I could find, and the difference between hand-coded and GenericServices’ versions were only 20 to 25 us. (us = 1/1,000,000 of a second), which equates to 5% slower. Note also that calling a method is just as fast as setting the property (GenericServices builds and caches LINQ expressions to call the methods, which are very fast).

The second example is a more complex update, where it adds a new Review entity to the Book. This takes a little longer inside EF Core and the database provider.

Method	Mean (us)	Error (us)	StdDev (us)
HandCoded, MethodCall	773.8	8.882	7.874
GenericServices, MethodCall	800.7	11.262	7.874
	3%

This shows that the difference is still about 25 us, but because the database accesses take longer it now only equates to a 3% difference.

Finally, I do a complex Projection of 100 Book’s to provide the display of the user. This is using AutoMapper via GenericServices (the hand-coded version is a LINQ Select method with hand-coded mappings).

Method	Mean (ms)	Error (ms)	StdDev (ms)
HandCoded	11.35	0.2433	0.4970
GenericServices	11.23	0.0838	0.0784
	-1%

This is a bit misleading, as the GenericServices isn’t faster than the hand-coded – sometimes it comes out faster and sometimes slower. Basically, the performance cost of GenericServices is smaller than the error margin so you can’t really measure it.

How hard is it to set up GenericServices?

I have tried to make it simple to add GenericServices to an existing or new application. There is full documentation on installing GenericServices, but here is an overview, assuming an ASP.NET Core application.

You install the NuGet library EfCore.GenericServices in your ASP.NET Core project. If you hold your ViewModels/DTOs in another project, then you need to install EfCore.GenericServices in that project too.
1. You might want to include the companion NuGet library called GenericServices.AspNetCore, which includes a method to copy errors from GenericServices into ASP.NET’s ModelState.
You mark all your DTOs with the ILinkToEntity<TEntity> interface so that GenericServices can find them and work out what entity class they are linked to.
You add a call to GenericServicesSimpleSetup<TContext> in the ConfigureService method in the Startup class of a ASP.NET Core application. This registers GenericServices with the DI – here is a link to the code in my Startup class showing what I did.
You use the ICrudServices interface to inject an instance of the CrudServices into your controller or razor PageModel.

NOTE: There is other versions for registering GenericServices. Another simple one that takes a GenericServiceConfig and a more complex version to handle the case where you use multiple DbContexts to cover your database. There is also a non-DI version for registering one or two DTOs useful for unit testing or serverless applications (See this documentation).

Conclusion

I have described what the GenericServices library is designed to do – to save you time by building robust CRUD web pages. My experience with my previous library with the same name (but totally different design) for EF6.x is that it can really reduce the time it takes to build applications which have a lot of CRUD pages (see this example, which I build in 10 days using the old library).

The new EfCore.GenericServices library is designed to be easier to use and has several new features. The biggest new feature is its ability to work with standard entity classes, where you update via the properties, and DDD-styled entity classes, where you update via methods in the entity class. Personally, I have found DDD-styled entity classes are good for hiding some of the more complex rules for creating and updating entities and their aggregates (see my article that explains that).

Please do have a look at the library and see what you think. The GenericServices GitHub repo contains an example ASP.NET Core Razor Pages application – its uses an in-memory database so that it can run anywhere, and always starts at a known setting. I also have a similar example of using GenericServices with Web APIs in this GitHub repo, which also uses a in-memory database so that its easy to run.

Happy coding.

Microsoft’s documentation says “ASP.NET Core is designed from the ground up to support and leverage dependency injection”. It also says that “Dependency injection (DI) is a technique for achieving loose coupling between objects and their collaborators, or dependencies.” (read Martin Fowler’s article for the in-depth coverage of DI).

I have used DI for years and I love it – it allows me to compartmentalize my applications and have DI link everything up at run time. The only chore is making sure all my services are registered properly at startup. This article is about a library I build to automatically register classes in an assembly into Microsoft’s DI provider (known as a DI container).

TL;DR; – summary

This article introduces a small library called NetCore.AutoRegisterDi (NuGet link) that can automatically register classes with interfaces in an assembly into the built-in ASP.NET Core DI container. The reason I built this library is because ASP.NET Core’s DI container turns out to be quick – at least 5 times quicker than AutoFac. I also describe how I use this in applications that have a lot of assemblies.

Setting the scene

The typical approach shown in the Microsoft’s documentation to registering services is to add a line of code to register each service in the ConfigureServices method in ASP.NET Core’s Startup class. For me that is a horrible maintenance issue, because I might forget to register a service I need: and if I do forget then my application will fail when that service is used.

Until now my go-to solution for automatically registering my services in DI is AutoFac’s RegisterAssemblyTypes method. But when I saw David Fowler’s tweet with a link to a set of dependency Injection container benchmarks, I found out that the ASP.NET Core DI container is a lot faster than AutoFac.

I worry about performance, so I thought – how hard would it be to build a similar extension to AutoFac’s RegisterAssemblyTypes method, but for the NET Core DI container, Microsoft.Extensions.DependencyInjection? Turns out it was easy (2 hours + 2 hours for NuGet+docs). This article is about how you can use the NetCore.AutoRegisterDi in your own applications.

NOTE: Microsoft docs says, “The default services container provided by ASP.NET Core provides a minimal feature set and isn’t intended to replace other containers.” However, I find Microsoft’s DI library has all the things I need for most applications, but some features aren’t well publicised – see Steve Gordon’s articles on three Microsoft DI features that I had to find the hard way:

Introduction to DI – what is a service?

Note: If you already understand DI you can skip this section.

I refer to a service, which is a combination of a class that implements that service, known as the implementation type in the NET Core DI container, which is linked (known as registered) to a service type, which is typically an interface. Here is an example of an interface, followed by the class that implements that interface.

public interface IMyService
{
    string IntToString(int num);
}

public class MyService : IMyService
{
    public string IntToString(int num)
    {
        return num.ToString();
    }
}

There are various ways to use a service in another class. The standard way is to add a parameter to the class’s construction and the DI container will resolve that interface to the implementation class – this is known as constructor injection. Here is an example.

public class UseService
{
    private readonly IMyService _service;

    public UseService(IMyService service)
    {
        _service = service;
    }

    public string CallService()
    {
        int i = 1; //... some calculation
        return _service.IntToString(i);
    }
}

This interface/class pattern “achieving loose coupling between objects and their collaborators” (to quote Microsoft). This has lots of benefits, but here are the two main ones for me:

The interface clearly defines the properties, methods, classes etc. that user of the service can access. By using an interface I can’t access things I’m not supposed to use, so interfaces makes me focus on what I should use and ignore the rest.
The two classes are “loosely coupled”, i.e. it is easy to replace the original version of the implementing class if it conforms to the interface. This is especially useful when unit testing, as it’s easy to replace the interface with a mock that allows you to test the UseService class on its own.

How to NetCore.AutoRegisterDi works

The NetCore.AutoRegisterDi provides three extension methods – here is an example that uses all three methods.

service.RegisterAssemblyPublicNonGenericClasses(… your assembly ref goes here …)
    .Where(x => x.Name.EndsWith(“Service”))
    .AsPublicImplementedInterfaces();

The first line finds all the public, non-generic classes (see docs for detailed definition of what classes it finds) in the assembly you provide (or assemblies – it takes params Assembly[] assemblies).
The Where method in the second line is optional and it allows you to filter the classes if you need to.
The final line finds all the public, non-nested interfaces (apart from IDisposable) that each class implements and registers each interface against the class.

The AsPublicImplementedInterfaces method takes an optional parameter of the ServiceLifetime you want for the service. It defaults to Transient lifetime where a new service instance is created on every injection, which is a good default for services. The other options are Singleton (same instance used on every injection) or Scoped (a new service instance is created for each HTTP request).

TIP: The NetCore.AutoRegisterDi Where method is useful if you want different ServiceLifetime – I name my classes with an ending that tells me what sort of lifetime I need, and filter on that.

An approach to using AutoRegisterDi in real applications

Now, you could call the RegisterAssemblyPublicNonGenericClasses method directly in the ConfigureServices method in the Startup class, and it would work fine. You just need to define the assemblies you want to scan. Here is how you would do this.

public void ConfigureServices(IServiceCollection services)
{
   //... other configure code removed

   var assemblyToScan = Assembly.GetAssembly(typeof(YourClass)); //..or whatever assembly you need

   service.RegisterAssemblyPublicNonGenericClasses(assemblyToScan)
     .Where(c => c.Name.EndsWith("Service"))
     .AsPublicImplementedInterfaces();

That’s fine, but it’s not what I do. I create an extension method inside each key assembly which registers itself and any other assemblies that it ‘owns’. In a small applciation I would call my ServiceLayer assembly, and it would look after setting up all the other assemblies.

Here is an example taken from a larger application where I have multiple grouped assemblies (see Simon Brown’s package by component approach) .

public static void DevModStartup(this IServiceCollection services, 
    IConfiguration configuration)
{
    //This registers the service layer: I only register the classes who name ends with "Service" (at the moment)
    services.RegisterAssemblyPublicNonGenericClasses(
              Assembly.GetExecutingAssembly())
        .Where(c => c.Name.EndsWith("Service"))
        .AsPublicImplementedInterfaces();

    //Now I register the DevModBizLogic assembly used by DevModService
    services.RegisterAssemblyPublicNonGenericClasses(
             Assembly.GetAssembly(typeof(BuildTestTenant)))
        .AsPublicImplementedInterfaces();

    //Put any code here to initialise values from the configuration parameter
}

NOTE: This shows the two ways of getting an assembly. Assembly.GetExecutingAssembly() gets the assembly that is running and Assembly.GetAssembly(typeof(SomeType)) gets the assembly that the SomeType is defined in.

I then call this DevModStartup extension method in the ASP.NET Core ConfigureServices method like so.

public void ConfigureServices(IServiceCollection services)
{
   //... other configure code removed

   services.DevModStartup(Configuration);
   //... add other assembly startup extension methods here

I think this is cleaner, especially as it keeps the rules about what classes to scan and ServiceLifeTime in the assembly that knows what is going on.

You should also note that I provide the Configuration property to the Startup extension method. This gives the extension method access to the appsetting.json file from which it can read static values, like setting or crypto keys etc. on application startup and store them in internal static variables.

WARNING: I wouldn’t use RegisterAssemblyPublicNonGenericClasses with the ASP.NET Core assembly without a good Where clause. The ASP.NET Core assembly has a LOT of services which it sets up itself, so you don’t want to re-register those.

Conclusion

DI is a great tool and one that I use a lot. The problem is in real applications I can have many hundreds (maybe thousands) of DI services so I want the DI to a) be easy to set up and b) be fast. I fact I think not automatically registering your services is an anti-pattern because it’s so easy to forget to register a service, and your live application will fail when that service is used.

I used to use AutoFac’s RegisterAssemblyTypes method, but that was about the only feature I used. Now that I have the NetCore.AutoRegisterDi library I have an assembly scanning feature that works with ASP.NET Core’s built-in DI container, and it makes my DI faster.

PS. Andrew Lock recommends a DI library called Scrutor – see his article called “Using Scrutor to automatically register your services with the ASP.NET Core DI container“. That might fit your needs better than my library.

This article is about how to catch SQL errors, like unique index violations, and turning them into user-friendly error messages. This capability adds to the general area of validation of data being written to the database, especially around EF Core.

This article comes from needing this capability in my GenericServices and GenericBizRunner libraries while working on a project for a client. I also extended the design to handle concurrency issues in both libraries, as one of the users of my GenericServices library was asking for it.

TL;DR; – summary

When EF Core writes data out to the database it doesn’t validate that data (see Introduction to validation section for more on validation). However, a relational database will apply its own validation, such as checking that a unique index constraint hasn’t been violated, and will throw an exception if any constraint is breached.

The problem with database exception messages is that they aren’t user-friendly, and can reveal something about your database structure (which is a possible security breach), so you can’t show them directly to the user. This article describes how to capture these database exceptions in EF Core and turn them into user-friendly error messages.

Introduction to validation in EF

Note: if you know about data validation and how EF Core does (or doesn’t) validate data on save then you can skip this section.

Data validation in .NET is about checking that the data in a class fits within certain rules. Typical validation rules are things like the attribute [MaxLength(100)] on a string. Other more complex validation rules can be applied via the IValidatableObject Interface. Some of these attributes, like [MaxLength(…)], [Required], etc. are also used by EF to set the size, nullability etc. of columns in the database.

Any validation rules are typically checked at the front end, say in ASP.NET Core, but can also be checked when that data is saved to the database. The default state for EF6.x is that the data written to the database is validated, but in EF Core the data isn’t validated – the reasoning is that the data its most likely been validated earlier, so leaving out validation makes the save quicker.

Personally, when I create data in a business logic I always validate the data, as no front-end checks have been applied and the business logic might get things wrong. This is why my GenericBizRunner library defaults to validating the data on save, while my GenericServices library, which works with the front-end CRUD (Create/Read/Update/Delete) accesses, defaults to not validating the data.

But as I said at the beginning, whether you validate the data or not the relational database will have some constraints that it will always apply – you can’t bypass them for the good reason that the database wants to keep its state valid at all times. If you breach a database constraint in EF6.x or EF Core you will get an exception.

How to capture a database exception in EF Core

When you call SaveChanges/SaveChangesAsync then a range of exceptions can occur, from EF Core caught issues like concurrency issues (DbUpdateConcurrencyException, to database issues like the example of the unique index violation (DbUpdateException). Here is an example of catching a DbUpdateException taken from my book “Entity Framework Core in Action” (section 10.7.3).

try
{
   _context.SaveChanges();
}
catch (DbUpdateException e)
{
   //This either returns a error string, or null if it can’t handle that error
   var error = CheckHandleError(e);
   if (error != null)
   {
      return error; //return the error string
   }
   throw; //couldn’t handle that error, so rethrow
}

In the case of the DbUpdateException has an inner exception, which is the actual database exception containing the information about the constraint violation. The type and content of this inner exception differs for each database type.

The other thing to note is that the calling method has to return either success, or one or more error messages.

My generalized SaveChangesExceptionHandler

For my libraries I wanted to generalise the approach to cover both a) any type of exception, b) any type of databases. The signature for the SaveChangesExceptionHandler is

Func<Exception, DbContext, IStatusGeneric>

The method takes in the exception to decode and returns either a status, which can have errors or a success message, or null if it doesn’t handle the error type it was given. I also supply the current DbContext, which you would need if you were trying to handle a DbUpdateConcurrencyException.

For my example I am going to show you the code which handles DbUpdateException, where I want to capture the SQL unique index violations and turn them into user-friendly error messages. It starts with the method that detects each type of error you want to handle, and then calling the appropriate method. It’s basically a sophisticated switch pattern.

public const int SqlServerViolationOfUniqueIndex = 2601;
public const int SqlServerViolationOfUniqueConstraint = 2627;

public static IStatusGeneric SaveChangesExceptionHandler
    (Exception e, DbContext context)
{
    var dbUpdateEx = e as DbUpdateException;
    var sqlEx = dbUpdateEx?.InnerException as SqlException;
    if (sqlEx == null)
    {
        //This is a DbUpdateException on a SQL database

       if (sqlEx.Number == SqlServerViolationOfUniqueIndex ||
           sqlEx.Number == SqlServerViolationOfUniqueConstraint)
       {
          //We have an error we can process
          var valError = UniqueErrorFormatter(sqlEx, dbUpdateEx.Entries);
          if (valError != null)
          {
              var status = new StatusGenericHandler();
              status.AddValidationResult(valError);
              return status;
          }
          //else check for other SQL errors
       }   
    }

    //add code to check for other types of exception you can handle
    …

    //otherwise exception wasn't handled, so return null
    return null;
}

Having found the unique index errors I call a method to specifically handle that sort of error. I should say that I add a constraint/key name following the pattern ‘UniqueError_<EntityName>_<PropertyName> to any unique indexes or primary keys that I want to handle. This both gives me more info to show the user, and stops me trying to handle errors on properties I didn’t expect to have an error. Here is my UniqueErrorFormatter method.

private static readonly Regex UniqueConstraintRegex =
    new Regex("'UniqueError_([a-zA-Z0-9]*)_([a-zA-Z0-9]*)'", RegexOptions.Compiled);

public static ValidationResult UniqueErrorFormatter(SqlException ex, IReadOnlyList<EntityEntry> entitiesNotSaved)
{
    var message = ex.Errors[0].Message;
    var matches = UniqueConstraintRegex.Matches(message);

    if (matches.Count == 0)
        return null;

    //currently the entitiesNotSaved is empty for unique constraints - see https://github.com/aspnet/EntityFrameworkCore/issues/7829
    var entityDisplayName = entitiesNotSaved.Count == 1
        ? entitiesNotSaved.Single().Entity.GetType().GetNameForClass()
        : matches[0].Groups[1].Value;

    var returnError = "Cannot have a duplicate " +
                      matches[0].Groups[2].Value + " in " +
                      entityDisplayName + ".";

    var openingBadValue = message.IndexOf("(");
    if (openingBadValue > 0)
    {
        var dupPart = message.Substring(openingBadValue + 1,
            message.Length - openingBadValue - 3);
        returnError += $" Duplicate value was '{dupPart}'.";
    }

    return new ValidationResult(returnError, new[] {matches[0].Groups[2].Value});
}

There is some weird regex and decoding of the SQL error message, and that is because the error string isn’t that simple to decode. Here is an example error message:

Cannot insert duplicate key row in object ‘dbo.Books’ with unique index ‘UniqueError_Book_ISBN’. The duplicate key value is (9781617294563).

The UniqueErrorFormatter method reformats this into a message

Cannot have a duplicate ISBN in Book. Duplicate value was ‘ 9781617294563’.

Notice on lines 13 to 15 I try to find the entity class so that I can return a user friendly and localized name, but the name in the constraint/key name is pretty good (unless you have a TPH class).

The SaveChangesWithValidation code

The code above is called within my SaveChangedWithValidation method, which I show below. The method returns the IStatusGeneric interface I use in my libiraries. This contains either a success message if there were no errors, or a list of errors if problems were found.

public static IStatusGeneric SaveChangesWithValidation(this DbContext context, IGenericServicesConfig config)
{
    var status = context.ExecuteValidation();
    if (!status.IsValid) return status;

    context.ChangeTracker.AutoDetectChangesEnabled = false;
    try
    {
        context.SaveChanges();
    }
    catch (Exception e)
    {
        var exStatus = config?.SaveChangesExceptionHandler(e, context);
        if (exStatus == null) throw;       //error wasn't handled, so rethrow
        status.CombineStatuses(exStatus);
    }
    finally
    {
        context.ChangeTracker.AutoDetectChangesEnabled = true;
    }

    return status;
}

The ExecuteValidation code looks like this

private static IStatusGeneric ExecuteValidation(this DbContext context)
{
    var status = new StatusGenericHandler();
    foreach (var entry in
        context.ChangeTracker.Entries()
            .Where(e =>
                (e.State == EntityState.Added) ||
                (e.State == EntityState.Modified)))
    {
        var entity = entry.Entity;
        var valProvider = new ValidationDbContextServiceProvider(context);
        var valContext = new ValidationContext(entity, valProvider, null);
        var entityErrors = new List<ValidationResult>();
        if (!Validator.TryValidateObject(
            entity, valContext, entityErrors, true))
        {
            status.AddValidationResults(entityErrors);
        }
    }

    return status;
}

NOTE: You can find all of this code in the class SaveChangesExtensions in the GenericServices library.

Enabling this feature in my GenericServices and GenericBizRunner

You need to configure the libraries to take your error handler. GenericBizRunner is the simplest as it already defaults to validation on writes to the database. Here is the configuration code.

services.RegisterGenericBizRunnerBasic<MyDbContext>(
   new GenericBizRunnerConfig
{
    SaveChangesExceptionHandler = 
        GenericBizRunnerErrorHandler.SaveChangesExceptionHandler
});

The GenericServices is a bit more complex, as by default it doesn’t validate on saving to the database, so your error handler won’t be called. You can turn on validation on a case-by-case via the PerDtoConfig<TDto, TEntity> class, or turn on validation for all writes to the database via the global configuration, as shown below.

services.GenericServicesSimpleSetup<MyDbContext>(
new GenericServicesConfig
    {
        DtoAccessValidateOnSave = true,     //we use  Dto access for Create/Update
        DirectAccessValidateOnSave = true,  //And direct access for Delete
        SaveChangesExceptionHandler = GenericServiceErrorHandler.SaveChangesExceptionHandler
}, Assembly.GetAssembly(typeof(MyDto)));

Other things you could do with this

As I said earlier one of my users of the GenericServices library wanted to know if GenericServices could handle concurrency issues, and I had to say no. But, now with my generalised SaveChangesExceptionHandler both libraries can handle concurrency issues. I haven’t done this yet, but the steps would be:

In the ‘switch’ method you would detect that the exception was a DbUpdateConcurrencyException and the type of the entity it happened on. Then you would call a method designed to handle concurrency issues on that entity type.
In that method you would (try) to fix the issue. If you have then you need to call SaveChanges (within another try/catch in case the error happened again) to save the corrected update.
If successful you return a status of success, which the system would return to the user. But if there were errors it couldn’t handle it can report to the user that the process failed.

NOTE: that handling concurrency issues is quite complex and hard to test, so I’m not saying this is simple. But you can now do this in in my GenericServices and GenericBizRunner libraries.

NOTE: I cover this concurrency issues in detail in section 8.7 in my book – it takes 12 pages!

Conclusion

Catching SQL errors and turning them into user-friendly error messages is hard work, but in real applications its often the only way to ensure the system works. Otherwise the user will be presented with the rather unhelpful “There was an error” page. Certainly not acceptable in real applications.

I spent bit of my own time to generalised the error handler in my GenericServices and GenericBizRunner libraries to cover more possible exceptions and work with any database. Now you have access to this feature in both of my libraries, or you can copy my approach in your own applications.

I hope this helps you.

Happy coding!

This article is about ASP.NET Core 2.1 Web API code and ways to write code that is easy to understand, easy to test and quick to write. You might think that is quite a task, but I introduce a number of libraries that make that feat possible. Also there is an example application that you can clone from GihHub and run locally to see the libraries in action.

If you are new Web API I do start with a general introduction which covers the difference between Web APIs and using ASP.NET Core to directly deliver HTML pages. I then show you how to write simple Web API actions and set the correct HTTP response type back to the caller.

TL;DR; – summary

I describe a way to write Web API controllers that moves the database access code out to a library called GenericServices – this reduces the code needed in a Web API to a few lines. I also have another library called GenericBizRunner that does the same for business logic, which I cover in this article too. I describe a companion library called EfCore.GenericServices.AspNetCore that handles the creation of the correct response with a HTTP status code and json payload.

The overall effect of using the described pattern and libraries is that your Web API actions are small, easy to test and quick to develop.

NOTE: There is an example application on GitHub, which you can clone and run locally – it uses an in-memory database and includes Swagger, so its easy to run.

Introduction to ASP.NET Core Web API

Note: if you already know about ASP.NET Core Web API then you can skip this section.

There are now three main approaches to building a web application using ASP.NET Core version 2.1:

NET Core MVC, which uses controllers, actions and views
NET Core Razor Pages, which has pages with associated code
NET Core Web API, which can be queried by a framework, like React.js, or an external system

The first two have the ASP.NET Core web app deliver HTML pages, with the help of the built-in Razor syntax. In these cases, the ASP.NET Core web app is in charge of what is shown on the screen – see diagram below:

But in the ASP.NET Core Web API case what is shown to the user is controlled by something outside ASP.NET Core. For the human-user example its either framework (like AngularJS or React.js) or mobile application makes requests to the ASP.NET Core application to either get data or execute some business logic – see the diagram below:

There are pros and cons to both approaches, but nowadays more applications are moving over some sort of front-end code running in the browser or mobile which uses Web API to access the backend system. Also, application-to-application traffic can also use the Web API approach. Therefore, being aware on how to write good, testable Web API code quickly is a useful skill.

Going beyond Microsoft’s Web API examples

Microsoft has some good examples of writing Web API applications, including one using EF Core for the database (VS Code version, VS2017/Win version and VS2017/Mac version). These are a great place to start as they clearly show what is going. Here is an example taken from Microsoft’s example (Note: I have left out the constructor of the Web API controller, where the EF Core DbContext for the database is injected and put in the private field called _context).

[HttpPut("{id}")]
public IActionResult Update(long id, TodoItem item)
{
    var todo = _context.TodoItems.Find(id);
    if (todo == null)
    {
        return NotFound();
    }

    todo.IsComplete = item.IsComplete;
    todo.Name = item.Name;

    _context.TodoItems.Update(todo);
    _context.SaveChanges();
    return NoContent();
}

The example is very clear about what is happening, as you can see each part of the process, which is why Microsoft wrote it that way. But when the databases accesses and business logic get more complex, then putting the code in the Web API action isn’t such a good idea, for a number of reasons:

The code needed to achieve a particular action (in the example it’s a simple update) can become many lines of code, which means your controller is long and hard to understand.
Your database or business logic is likely to have extra validation/checks which it needs to return as errors, which not covered in the Microsoft example.
There is a danger of duplication of code. For instance, if another action needed to update the TodoItem you would you need to duplicate part of the code.
It wouldn’t be hard to unit test the Web API action in Microsoft’s example, but in more complex applications with say extra data injected via tokens or other parts of the HttpContext then it can get more difficult.

Over the years I have learnt to keep my controller actions as small as possible. This means moving the code out of the action into a separate method. At the same time, I have noticed patterns in database accesses and business logic calls which have culminated in my building libraries to handle the common parts. Here is my version of the TodoItem update using my GenericServices library, which is designed to handle CRUD (Create, Read, Update and Delete) functions to a EF Core database.

[HttpPut()]
public ActionResult<WebApiMessageOnly> Put(
    ChangeTodoDto dto, 
    [FromServices]ICrudServices service)
{
    service.UpdateAndSave(dto);
    return service.Response();
}

The differences are:

Its short and to the point – my Web API controller will be small and easy to read
The library uses a general ‘update TodoItem’ method which is used throughout the application – no possibility of duplication of code.
My EfCore.GenericServices (injected via the [FromServices]ICrudServices service property on line 4) provides generic CRUD functions, including the mapping from the input class to the ToDoItem class.
The Response method on line 7 returns a response that is either OK (HTTP 200) with a success message, NoContent (HTTP 204) if the item wasn’t found, or if the update logic or database validation finds errors then it returns Invalid Input (HTTP 400) with the error messages (The format of these responses are shown later).

This approach also allows me to unit test the ‘update TodoItem’ on its own, which I prefer. As I said before testing Web API actions can work, but if you have complex HTTP things going on, like cookies or tokens providing data, then testing can get very hard. I find it much simpler and easier to test the service.UpdateAndSave(dto) code on its own in a unit test.

NOTE: If you are using EF Core then see my open-source library EfCore.TestSupport for which has lots of useful methods for testing code that uses EF Core. Also, consider using in-memory databases for unit testing as they are really quick – the article “Using in-memory databases for unit testing EF Core applications” to show you how.

So, by doing all this I have:

I wrote good code: is short, obvious and does not duplicate code.
I have written testable code: it’s easy to duplicate the same code in your unit tests.
I wrote the code quickly: The EfCore.GenericServices library handles the four CRUD patterns, including copying from a ViewModel/Dto class to the EF Core entity class.

It’s my EfCore.GenericServices open-source library that makes the code shorter and quicker to write. This library is a second-generation version of my original GenericServices library built for EF6.x, and is significantly improvement over the original. My experience with the original GenericServices library was it has saved me months-per-year of development time. I expect my new library to give me a similar, if not better, productively as it now supports DDD-styled entity classes which centralises the create/update code.

I’m not going to describe the EfCore.GenericServices library in detail here, because I have already done that in the article “GenericServices: A library to provide CRUD front-end services from a EF Core database” (see also NuGet, GitHub, and documentation pages). But that article talks about MVC/Razor Pages usage.

In this article I am going to describe how to use my libraries with Web API, and to help I have updated the companion EfCore.GenericServices.AspNetCore library to convert the status output of my libraries library into the correctly formed HTTP responses.

What about business logic?

I have handled CRUD accesses to the database, but complex applications often have complex business logic. As I mentioned before I have another open-source library called EfCore.GenericBizRunner which runs business logic working with EF Core, which the companion EfCore.GenericServices.AspNetCore library also supports. You can find out more about the EfCore.GenericBizRunner library via the article called “A library to run your business logic when using Entity Framework Core”, and the GenericBizRunner’s documentation pages.

How I turn my libraries’ statuses into HTTP responses

My EfCore.GenericServices and EfCore.GenericBizRunner libraies are linked to EF Core, but could be used in any sort of application – web, desktop etc. But I mainly use them with ASP.NET Core, so I have a library called EfCore.GenericServices.AspNetCore which contains code to change the IGenericStatus that both these libraries use into either ASP.NET ModelState (for MVC and Razor Pages) into either

ActionResult<WebApiMessageOnly>, where it returns just a success message (or errors if incorrect).
ActionResult<WebApiMessageAndResult<T>>, if the Web API returns a success message and some results (or errors if incorrect).

In this article I want to look at the CreateResponse class that contain extension methods to turn my IGenericStatus into the two ActionResult forms.

NOTE: The EfCore.GenericServices.AspNetCore contains an ExampleWebApi project that contains an ASP.NET Core 2.1 Web API controller as an example of how the CreateResponse extension methods work. It uses an in-memory database and Swagger so that it’s easy to try out the Web API commands – just start the ExampleWebApi project and it should got directly to the swagger pages which lists each Web API command and allows you to test them.

Now I will describe the three main response types and what they look like. I start with the handling errors, as ASP.NET Core 2.1 has already defined that format.

InvalidInput (HTTP 400) – error return

If you create a new ASP.NET Core application the default template turns on 2.1 compatibility, which means that any Web API input is automatically validated, e.g. if a property had the attribute [Range(1,5)] then it would return an error if that property was outside that range. This is a really useful feature, as it saves you from having to add code to validate the input at the start of the Web API action.

The format that ASP.NET Core uses to return the errors is a dictionary of errors returned with Invalid Input (HTTP 400) status code. Here is an example of the format it uses:

{
    "": [
        "Global error message"
    ],    
    
    "MyPropery": [
        "The property is required",
        "Another error on the same property"
    ]
}

Therefore, my Response extension method in the EfCore.GenericServices.AspNetCore library turns any errors found by the EfCore.GenericServices or EfCore.GenericBizRunner libray into the same format. That means there is one common error format the calling code needs to process.

The other responses we need are:

OK (HTTP 200) – successful

This contains a message string, which you can show the user or log. You can set the message yourself, or my libraries will return a message relevant to the. Also, if there is any data to return, then there is a results property. Here is an example with results, which uses the WebApiMessageAndResult<T> response class.

{
  "message": "Success",
  "results": {
    "id": 1,
    "name": "Create ASP.NET Core API project",
    "difficulty": 1
  }
}

This type of response is has a result, as shown in this Get action method

[HttpGet("{id}", Name= "GetSingleTodo")]
public async Task<ActionResult<WebApiMessageAndResult<TodoItem>>> 
    GetAsync(int id, [FromServices]ICrudServicesAsync service)
{
    return service.Response(
        await service.ReadSingleAsync<TodoItem>(id));
}

The other type of action that returns a result is a create. There are two ways to handle this.

1. Create, and return url

The standard Web API way of handling a create is to return a HTTP status of Created (201), with a url to get the newly created item. This library provides that feature, by linking into the ASP.NET Core’s CreatedAtRoute method. Here is an example of this approach:

[HttpPost]
public ActionResult<CreateTodoDto> Post(CreateTodoDto item, 
    [FromServices]IActionService<ICreateTodoBizLogic> service)
{
    var result = service.RunBizAction<TodoItem>(item);
    return service.Status.Response(this, "GetSingleTodo",
         new { id = result?.Id },  item);
}

NOTE: I have to allow for the result to be null, which can happens if there are errors. Therefore I use result?.Id (see line 7) to stop me getting a null reference in the error state (thanks to Michael Wilson for spotting that).

On a successful creation this method returns a response header containing the URL you should use to obtain the newly created item. Here is an example of the header, with the url on the third line:

content-type: application/json; charset=utf-8 
date: Thu, 30 Aug 2018 11:07:48 GMT 
location: http://localhost:54074/api/ToDo/7 
... other parts removed

NOTE: There is one “gotcha” that I need to warn you about. In ASP.NET Core you need to provide the Name of action method that provides the GET method to return the crated item. The Name isn’t the name of the method, but the name provided via the HttpGet attribute, for instance [HttpGet(“{id}”, Name= “GetSingleTodo”)]. If you don’t do that you get the dreaded “No route matches the supplied values” error.

2. Create, and return the created item

The other way to handle a Create is to return the created item. This has the advantage of removing the second HTTP to get the data. The library allows you to do this, and provide the 201 status code if required. Here is an example of a create a new TodoItem, where I return the created item, with its primary key:

[HttpPost]
public ActionResult<WebApiMessageAndResult<TodoItem>> Post(
     CreateTodoDto item,
     [FromServices]IActionService<ICreateTodoBizLogic> service)
{
    var result = service.RunBizAction<TodoItem>(item);
    return service.Status.ResponseWithValidCode(result, 201);
}

NOTE: You will see I use the [FromServices] attribute to allow the dependency injection (DI) provider to inject a service as a parameter in the action. I find this more efficient than the normal constructor injection, as it means each action creates just the service it needs.

NoContent (HTTP 204) – when the requested data lookup returned null

In Web APIs if something like a Get(id) is used, and no data is found (i.e. its null) then then it returns a status code of NoContent (HTTP 204). Therefore, if that happens then you get a NoContent status with a message, but no results. Here is an example of the json that is returned.

{
  "message": "The Todo Item was not found."
}

UPDATE: In version 2.0.0 of the EfCore.GenericServices.AspNetCore library I returned the NotFound status (HTTP 404) because Microsoft used NoFound status for nulls. But following feedback from others, and looking at other documentation, I released version 3.0.0 which returns a NoContent (HTTP 204) return on a null result. Version 3.0.0 also has a more fully defined return types, which helps Swagger to properly describe the API.

If you want to set your own Success and SuccessWithNullResult status you can the ResponseWithValidCode method, e.g. ResponseWithValidCode (result, 201, 404). The second parameter is the status code to return on a successful, non-null result, while the third parameter (which is optional – it defaults to 204) is the status code to return on a successful, but null result.

Telling GenericServices that null is not an error

To make the null handler to work you need to set a configuration parameter when you set up the GenericServices library. This is because in MVC or Razor Pages applications not finding the data asked for is an error, but in a Web API it isn’t. Here is an example of how you would configure GenericServices for WebAPI in the Configuration method in the Startup class – see line 4 for the important configuration option

services.GenericServicesSimpleSetup<ExampleDbContext>(
    new GenericServicesConfig
    {
        NoErrorOnReadSingleNull = true 
    }
    ,Assembly.GetAssembly(typeof(ChangeNameDto)));

UPDATE: Now with unit test support

I’m working on a client’s project which has lots of Web APIs, and we want to check them, because we found errors could creep up if the dtos weren’t properly setup. I therefore added a new set of extension methods to the EfCore.GenericService.AspNetCore project (look for version 2.0.0 or higher on NuGet) to help test these.

The new extension methods decode the HTTP IActionResult and ActionResult<T> into a form that allows you to access all the parts of the information you sent from your GenericService/GenericBizRunner methods. This makes unit testing or integration testing much easier to do.

Here is an example integration test taken from the IntegrationTestToDoController, which you can find in the the EfCore.GenericService.AspNetCore GitHub project.

[Fact]
public void TestPutNameOk()
{
    //SETUP
    var options = SqliteInMemory.CreateOptions<ExampleDbContext>();
    using (var context = new ExampleDbContext(options))
    {
        context.Database.EnsureCreated();
        context.SeedDatabase();

        var controller = new ToDoController();
        var utData = context.SetupSingleDtoAndEntities
             <ChangeNameDto>(_genericServiceConfig);
        var service = new CrudServices(context, utData.ConfigAndMapper);

        //ATTEMPT
        var dto = new ChangeNameDto()
        {
            Id = 2,
            Name = "Test",
        };
        var response = controller.PutName(dto, service);

        //VERIFY
        response.GetStatusCode().ShouldEqual(200);
        var rStatus = response.CopyToStatus();
        rStatus.IsValid.ShouldBeTrue(rStatus.GetAllErrors());
        rStatus.Message.ShouldEqual("Successfully updated the Todo Item");
    }
}

Looking through the lines of code:

Lines 5 to 9: I create a in-memory Sqlite database a seed it with some known data for the test.
Line 11: I create an instance of the Web API controller I want to test. Because I inject the services via a parameter on the action method call, then there aren’t any parameters on the Web API controller.
Lines 12 to 14: Here I create the service that the Web API action method needs. The code you see is for GenericServices, but you can find an example of setting up GenericBizRunner here.
Lines 17 to 22: I call my Web API method, with the required dto class.
Line 25: The GetStatusCode extension method gets the HTTP status code, which I then check.
Line 26: The CopyToStatus method turns the Web API IActionResult/ActionResult<T> response back into a IStatusGeneric/IStatusGeneric<T> status (it started in from, so its easy to convert it back). This gives you access to any Errors, the Message, and the Result (if it has one).
Line 27 and 28 I check the parts of the IStatusGeneric to see what was send

NOTE: I could have added a test of the database to check that the name of the TodoItem with primary key 2 was changed too. It depends on whether you have unit tested the underlying GenericService UpdateAndSave of this entity elsewhere.

The extensions method can be found in UnitTesting.ResponseDecoders class.

NOTE: If you want to do a full integration test of your Web APIs then have a look at the articles “How to Test ASP.NET Core Web API” which uses the Microsoft.AspNetCore.TestHost package, or “Painless Integration Testing with ASP.NET Core Web API” which uses Microsoft’s Microsoft.AspNetCore.Mvc.Testing package.

Conclusion

I have described the pattern and libraries that allow me to build good, testable Web API code quickly. Previous experience says that this approach makes me a much faster developer – I developed a MVC5 web application around the AdventureWorks 2012 Lite database in 10 days, which is quick (see http://complex.samplemvcwebapp.net/). Real applications that use Web APIs can have hundreds, if not thousands of Web API actions – saving even 25% development time on each Web API action could equate to a saving of many months of your time and the project’s time.

This article and the example Web API in the EfCore.GenericServices.AspNetCore GitHub repo give you a quick way to see the use of EfCore.GenericServices and I threw in an example of calling business logic via my EfCore.GenericBizRunner library (see the Post action which creates a new TodoItem) to build Web APIs.

The examples uses DDD-styled entity classes because I like the way it centralises the create/update code inside the entity class. Using DDD-styled entity classes is optional, as EfCore.GenericServices can work with normal EF Core entity classes as well, but I find it adds an extra level of the robustness of the final solution.

Do clone the the example code and look at the code and run the ExampleWebApi application – maybe using this approach will save you time too.

Thanks to the readers who left comments on the NotFound/NoContent argument! I have updated the EfCore.GenericServices.AspNetCore library to use NoContent, and better response definitions, which helps Swagger to define the API.

Happy coding!

I’m a big fan of Domain-Driven Design (DDD) and build all my Entity Framework Core (EF Core) entity classes using a DDD approach, i.e. all properties have private setters and all changes are made via method in the entity class. This makes it very clear what, and how, properties and relationships can be updated. (See the article “Creating Domain-Driven Design entity classes with Entity Framework Core” for more details on implementing DDD in EF Core).

But there are times where a fully DDD-styled entity can create more work for yourself. I’m working with a client on a large application that will have hundreds of Web APIs feeding an Angular front-end. In this case using the JSON Patch feature will save the frontend developers a lot of time, because there are libraries that will process a form and create a JSON Patch to update the properties that have changes. (see the article “JSON Patch With ASP.NET Core“ for a fuller description of JSON Patch).

In this article I talk about how I merged the use of JSON Patch approach into my DDD entity class approach. I also describe how I added support for JSON Patch in my EfCore.GenericServices library, while keeping all the good attributes of a DDD-styled entity class.

Introducing the hybrid DDD entity class

JSON Patch, with its update via the property’s setters, seems at odds with the DDD approach to entity classes where properties have private setters. But you can’t ignore the fact that JSON Patch, and associated libraries, can significantly speed up the building of single-page applications, using Angular, React, Vue etc.

DDD is all about making the entity class business-focused, where it deals in terms of the business needs rather than simple properties. So, having methods in the entity class with names like MarkOrderAsDispatched is better than setting the OrderStatus to Dispatched. Also, DDD wants to provide the developer with a clear idea of if/how a property can be updated, especially around relationships.

When building my EfCore.GenericServices library I made it capable of updating “standard” entity classes, i.e. with public setters, and also work with DDD-styled entities, i.e. where updates are done via calling public methods. In all of this I did allow what I call a hybrid DDD entity class, where some properties were updated by method calls, and some were updated via their public setters.

The typical use case for a hybrid DDD entity class is updating an address, with its various street, city, state, zip/postcode properties, inside entity class. I wanted to keep DDD’s tight control over the relationships, but in many cases updating the address properties has no business effects so it simplified things to make the properties’ setters public (the library would notice that and call AutoMapper to update the properties from a DTO).

While some might say that a hybrid DDD entity class breaks the DDD approach, but I believe it follows the spirit of what Eric Evans was getting at in his book, Domain-Driven Design. The ‘intent’ of what can be done with property is clearly communicated.

If a property has a public setter then there are no business logic implications (other than validation).
If the property has a private setter then it either has business logic implications or you wanted to provide a business-named access method, like my earlier example of MarkOrderAsDispatched. Either way you must update it via a method call because the setter is private.

Altering a DDD-styled entity to allow JSON Patch to work

By default JSON Patch will fail on a property with non-public setter. That is actually useful, as we can differentiate between the properties that we allow JSON patch to update, and properties that should only be updated by ctors or methods. This is where the hybrid DDD entity class comes in.

Here is an example from my EfCore.GenericServices.AspNetCore library, which contains some simple entity classes and Web API to show how things work (you can clone and run this locally to try this out – it uses an in-memory which is seeded at startup to make it easier to play with).

I created a simple hybrid entity class called TodoItemHybrid, as shown below

public class TodoItemHybrid
{
    public int Id { get; private set; }

    [Required(AllowEmptyStrings = false)]
    public string Name { get; private set; }

    [Range(1, 5)]
    public int Difficulty { get; set; }

    //ctor and access methods
    public TodoItemHybrid(string name, int difficulty)
    {
        Name = name;
        Difficulty = difficulty;
    }

    public IStatusGeneric ChangeName(string name)
    {
        var status = new StatusGenericHandler();
        if (name.EndsWith("!"))
            status.AddError("Business logic says the name cannot end with !", nameof(Name));

        Name = name;
        return status;
    }
}

You can see that the Name property (line 6) has a private setter and can only be updated via the ChangeName method (lined 18 to 26). That method has some (fake) business logic connected to it (see lines 21 to 22), so that must be updated via the ChangeName method.

On the other hand, the Difficulty property (line 9) is a simple property with no business logic attached (apart from validation), so I have given it a public setter. This means I can update it via the Microsoft.AspNetCore.JsonPatch library.

The way that JSON Patch works is you can provide a list of changes. In this simple case there is only one property, Difficulty, but you can provide an array of updates. Here is simple example containing two updates to the same class:

[
	{ "op": "replace", "path": "/FirstName", "value": "Jon" }
	{ "op": "replace", "path": "/LastName", "value": "Smith" }
]

Using JSON Patch can cut down the amount of work required in the front-end code. A JavaScript library such as https://www.npmjs.com/package/rfc6902 can automate the creating of the JSON patches to send back to the server. This package has a createPatch feature that compares original data with new data (for instance, from a form) and generate a patch with all the changes to send to the server.

A look at the ASP.NET Core Web APIs

To apply such an update from a Web API controller, then you would need something like this

[HttpPatch("{id}")]
public IActionResult HandCodedUpdate(int id, 
     JsonPatchDocument<TodoItemHybrid> patch, 
     [FromServices]ExampleDbContext context)
{
    var entity = context.Find<TodoItemHybrid>(id);
    if (entity == null)
    {
        return NoContent();
    }
    patch.ApplyTo(entity);
    context.SaveChanges();
    return Ok();
}

The call sends the primary key of the entity you want to update (end of line 2), plus a JsonPatchDocument<T> patch (line 3) created by the caller. Then you have to load the entity you want to update (line 6) and if found you apply the patch and save the changes to the database (lines 11 to 12).

To make this more streamlined I have added a JSON Patch feature to my EfCore.GenericServices library. This wraps the ApplyTo with extra checks, optional validation and success/error messages. Here is the same example but using the GenericServices library:

[HttpPatch("{id}")]
public ActionResult<WebApiMessageOnly> Update(int id, 
    JsonPatchDocument<TodoItemHybrid> patch, 
    [FromServices]ICrudServices service)
{
    service.UpdateAndSave(patch, id);
    return service.Response();
}

Conclusion

DDD can provide many benefits, such as access methods which are named in a business-focused way, and these access methods are the definitive way to do something, as there is no other way in. I really like that level of isolation.

But most database tables have plenty of simple read/write columns that don’t need the level of access control. Writing access methods still works, but sometimes a simpler copy of data is sufficient. And when approaches like JSON Patch make the developers life simpler its worth adopting it in the right places.

In the end DDD is more a philosophy than a standard pattern – Eric Evans was saying “make the domain (business) problem the focus, not the tools and technology you use”. My hybrid DDD entity class might not be liked by some, but I think it still conveys the intent of the data, and it makes you a quicker developer.

Happy coding!

On my article “Creating Domain-Driven Design entity classes with Entity Framework Core” @ardalis commented that “your entities all are tightly coupled to EF Core. That’s not good…”. Then I did a podcast with Bryan Hogan where we discussed Domain-Driven Design (DDD) and he goes further than my CRUD-only (Create, Read, Update, and Delete) approach – he says that the entity classes is the perfect place for business logic too.

NOTE: My discussion with Bryan Hogan is now out. You can find the PodCast here.

With such diverging views on the best way to implement DDD in Entity Framework Core (EF Core) I decided to write an article that a) compares normal approach with a DDD approach, and b) compare three different ways to implement DDD in EF Core. This is a detailed look at the issues, hence it is very long! But hopefully useful to those looking to use DDD, or developers that want to consider all the options available to them.

TL;DR; – summary

NOTE: DDD is a massive topic, with many facets. In this article I only look at the entity class issues, which is a tiny part of what DDD is about. I really recommend Eric Evan’s book Domain-Driven Design for a full coverage.

The DDD approach to writing entity classes in EF Core makes every property read-only. The only way to create or update entity data is constructors (ctors), factories or methods in the entity class. This article introduces the DDD entity style by comparing the standard, non-DDD approach, against a basic DDD-styled entity class. This gives us a clear starting point from which I can go on to compare and contrast three difference DDD approaches:

A DDD-styled entity, but not including any references to EF Core commands.
A DDD-styled entity which has access to EF Core’s DbContext and implements CRUD methods.
A DDD-styled entity which has access to EF Core’s DbContext and contains all code that interacts with the entity, i.e. CRUD and more complex business logic.

Comparing a standard entity class with a DDD approach

Let’s start by comparing an implementing something using a standard entity class and a DDD-styled entity class. The aim of this section is to describe the major differences and sets the scene for a look at the subtler differences between the three DDD approaches.

The diagram below shows a standard entity class (i.e. a class that EF Core maps to the database) with read-write access.

And this diagram shows the same entity class but using a DDD approach.

The big difference is that the standard entity class can be created/changed by any external code, but in the DDD-styled entity class you can only create/change data via specific constructors/methods. But why is this a good idea? – here is a list of benefits:

External code now has clearly named methods/static factories to call. This makes it much clearer to developers what the entity class supports in terms of creating/changing that class.
The code to create/change the class is contained in the class itself, which keeps the code co-located with the data. This makes writing/refactoring of the class much simpler.
It stops duplication of code, and in multi-person projects it stops different developers (or even the same developer!) applying different business rules to the same feature.

DDD is about making the domain (i.e. business) rules the focus of the code, so having methods like ChangeDeliveryDate or MarkOrderAsDispatched in your entity class encapsulates business rules with a meaningful name. There is much less possibility of getting things wrong this way.

Pros and Cons of Standard & DDD approaches

Here is summary of the Pros/Cons (advantages/disadvantages) of the two approaches

Approach

Pros

Cons

Standard

Simple.

Minimum code.

Big systems can become quite hard to understand.

Possibly of duplicate code happening.

DDD-styled

More obvious, i.e. meaningful named methods to call.

Better control of access to data.

Puts the code next to the data.

Slightly more code to write.

Different views on building DDD-styled entity classes

Having introduced the DDD approach I now want to look at three different approaches to implementing DDD entity classes. Here is a diagram to show you the three DDD approaches, showing what code the entity classes contain. I use two terms that I need to define:

Repository: A repository pattern provides a set of methods to access the database. These method ‘hide’ the code needed to implement the various database features you need.
C(r)UD: This is Create, (Read), Update, and Delete – I use the term C(r)UD to separate the Read from the functions that change the database (Create, Update are Delete).
Query Objects: This is a design pattern for building efficient database queries for EF. See this article from Jimmy Bogard on this topic.
Business logic (shortened to Biz Logic): This is more complex processes that go beyond simple validation. They may have complex calculations (e.g. pricing engine) or require access to external systems (e.g. sending an email to a customer).

As you will see in the diagram below, the further to the right you go the more external code is moved inside the entity classes.

NOTE: I will use the terms POCO-only class, C(r)UD only and C(r)UD & Business Logic when referring to these three approaches.

1. The different ways of reading data

The first big difference is how the data is read from the database. When using a repository pattern, you tend to read and as well as write via the repository pattern. I personally have found (some) repository patterns can lead to poor performing code, mainly because it’s hard to build a fully featured repository with the correct adapters for the front-end. I therefore use Query Objects, normally defined in the services layer so that they can adapt the data to the needs of the front end.

I’m not going to cover this here as I have an article called “Is the repository pattern useful with Entity Framework Core?” which goes into it in detail, with this section on query objects.

2. The different ways of updating relationships

The big difference between the POCO-only class approach and the other DDD versions is that the POCO-only version doesn’t have any access to the DbContext, so it can’t load relationships. To show why this matter, let’s introduce a Book entity class a collection of Reviews, e.g. (5 stars – “It’s a great book” says Jill). Now we want to implement a method to allow a user to add a new Review entity class to an existing Book entity instance. The steps are:

Get the new review information, with the Book’s primary key.
Load the Book entity instance using the primary key.
Call a method in the loaded Book instance to create a new Review and add it to the Book’s Reviews collection.

The problem comes if the Reviews collection hasn’t been loaded when the Book entity instance was loaded. If we assume the Reviews collection was loaded and its wasn’t, then adding a new review will either a) fail with a null collection or b) silently delete all the other reviews! Therefore, something has to make sure the collection is loaded. Here are some possible implementations for the POCO-only version, and the other two DDD versions:

1.a Handling adding a Review to POCO-only version

In a POCO-only entity class there is no references to EF Core. This means you need something else to handle the correct loading of the Book with its Review collection. The standard way to do this is via a repository pattern. The repository would contain an AddReview method that would load the Book entity instance, along with loading the Reviews collection before it calls the Book’s AddReview method to update the Books _reviews backing field. Here is some example code:

1.b Handling adding a Review within the DDD-styled entity class

The limitation of the POCO-only entity class is you need to rely on the caller to pre-load the Reviews collection. Because the POCO-only entity class can’t access any of EF Core’s classes or methods. In the other two types you can access EF Core, so you can move loading of the Reviews collection inside the entity class. Here is the code:

NOTE: When using the C(r)UD only approach the code in the ASP.NET controller has a repetitive pattern: load the entity and call a named method, which makes it a candidate for building library to handle this. This is exactly what I have done with the EfCore.GenericServices library, which can work with normal and DDD-styled entity classes. This the article “GenericServices: A library to provide CRUD front-end services from a EF Core database” for more on this.

Comparing the two approaches to adding a review

The differences are subtle, but important. In the POCO-only class the update relies on an external piece of code, in this case a repository pattern, to correctly update the reviews. In the C(r)UD only case the DDD access method, AddReview, handles the whole process.

Now, if you are using a repository pattern everywhere then this difference is small. However, it does allow a small loophole in which someone could bypass the repository and calls AddReview directly, which with the code I have written would cause a null reference exception (other designs I have seen silently delete existing reviews if you don’t preload the collection).

In this case the C(r)UD only approach is better because it encapsulates the whole of the process so that calling methods don’t have to know anything about the process. Therefore, the C(r)UD only (and the C(r)UD & Business Logic) approach follows the Single Responsibility Principle, i.e. the AddReview is responsible for adding a new review to a Book, and entirely encapsulates the code to add that review.

However, the C(r)UD only approach has one big down side – it refers to EF Core. Why is this a problem? The entity classes, like Book and Review, are core parts of your domain design and Eric Evans says in chapter 6 of his book that repositories should be used to “decouple application and domain design from persistence technology”. There are many people who like having entity classes that contain no database access logic so that they the class is focused on domain (i.e. business) logic.

You have to remember that the repository pattern was the recommended approach when Eric Evan’s book was published in 2004. Personally I, and others, don’t think the repository pattern appropriate for EF Core (see my article “Is the repository pattern useful with Entity Framework Core?”).

In the next section we look more carefully at the business logic and how that is handled, but before I leave this comparison, I should talk about performance. In the two examples the POCO-only class approach is quicker, as it loads the reviews at the same time as the book, while the C(r)UD only approach needs two trips to the database. I used similar code two examples to make it easier for you to compare the two approaches, but in my actual code (shown below) I use a much more efficient approach if the _reviews collection is not loaded/initialised: It just creates a new review with the correct foreign key, which is much more efficient.

public void AddReview(int numStars, string comment, 
    string voterName, DbContext context) 
{
    if (_reviews != null)    
    {
        //Create a new Book: the _reviews HashTable is set to empty
        _reviews.Add(new Review(numStars, comment, voterName));   
    }
    else if (context.Entry(this).IsKeySet)  
    {
        //Existing Book: create a new review directly
        context.Add(new Review(numStars, comment, voterName, BookId));
    }
    else                                     
    {                                        
        throw new InvalidOperationException(
             "Could not add a new review.");  
    }
}

3. Different ways of handling business logic

The first example was a Create, i.e. adding a new Review (known as an aggregate entity in DDD) to the Book entity (known as the root entity in DDD). These CRUD operations might include some form of validation, such as making sure the numStars value in the Review is between 1 and five. But when the rules get more complicated than validating properties then we tend to call the code business logic.

When someone wants an application written there are rules on how things work. For instance, placing an order might require certain checks on stock, payment etc. and kick off other tasks such as processing the order and delivery. The steps in the example order are called business rules, or in DDD-speak, domain problems.

In Eric Evans book he says “When the domain is complex, this is a difficult task, calling for the concentrated effort of talented and skilled people”. In fact, most of Eric Evan’s book is about domain problems: how to describe them (ubiquitous language), how to group domain problem (bounded context), etc. But in this section, I’m just going to look at the three DDD approaches and how they handle business logic.

For this I am going to use the example of processing a customer’s order for books. To make it easier to understand I am only showing 5 stages (e.g. I left out payment processing, shipment, etc.):

Load the Books ordered using the primary keys provided by the higher layers
Get the address that the order should be sent to
Create the Order
Save the Order to the database
Send an email to the user saying the Order was successfully

I am also going to show the layering of the assemblies, i.e. in what order the assemblies reference each other. This matters because in .NET you can’t have circular references to assemblies (that causes problems at compile time), which constraints how you put the assemblies together. In most cases the entity classes are the lowest assembly, because everything else needs to access these key classes. (NOTE: I also add a red oval on the assembly that references EF Core).

2.a. POCO-only class approach

The POCO-only class approach uses a repository for database accesses so when the business logic needs data it relies on the repository. Because you can’t have circular references to assemblies this means part of the business logic moves into the repository. Typically, this results in repository taking on a coordination of the building of the order (Command Pattern). The figure to the right shows the assembly references.

And below is a diagram of one possible implementation of handling a customer order:

2.b The C(r)UD only approach

Over the years I have developed a series of rules to help me in implement business logic. And one of them is “Business logic should think it’s working on in-memory data”. Having decided that generic repository pattern doesn’t work well with EF Core (see my article “Is the repository pattern useful with Entity Framework Core?” on why that is) I had to come up with another way to do this. My solution is to have two new assemblies, which are:

BizLayer: This contains classes that are pure business logic, i.e. they don’t access EF Core directly, but rely on their own mini-repository for all database actions
BizDBAccess: This is handles all database accesses for a single business logic class, i.e. it’s a dedicated repository for the business logic.

It may seem odd to say I don’t like the repository pattern, and then build a dedicated repository for each business logic. But it’s because writing generic repository is actually very difficult – for instance EF Core is a generic repository/unit of work pattern and think how much effort it’s taken to write that! But writing a dedicated repository for a specific piece of business logic is easy. (I agree with Neil Ford’s comment from his book Building evolutionary architectures – “The more reusable the code is, the less usable it is.”).

Now, I’m not going to show the code here, as I have a much longer article called “Architecture of Business Layer working with Entity Framework (Core and v6) – revisited” where I covers the same example of placing an Order (NOTE: this article doesn’t include a “SendMail” part, but I think you can see that goes in the business logic).

2.c. C(r)UD and business logic

The final version has methods/factories for ALL manipulation of the data in the entity class. This means any there is no repository and no business layer because it’s all handled by the entity class itself.

In this case the business logic code that was in the repository (see 2.a) is moved completely into the entity class. The technical issue with this is the SendMail method is in an assembly that is linked to the assembly containing the entity classes, which stops you referencing the SendMail directly. It’s pretty simple to fix this by defining an interface (e.g. ISendMail) in the entity class and using dependency injection to provide the SendMail instance at run time.

At this point every operation other that read (and maybe some deletes) will all be entity class, and those method will use EF Core to access the database directly from the entity itself.

Comparing approaches to business logic

In the end the differences are around the ‘scope’ of the code inside the entity class. The POCO-only has the simplest code, with no references to EF Core. The C(r)UD only approach uses EF Core to be a one-stop-shop for doing database changes but has the business logic in another assembly. While the C(r)UD + Biz Logic approach has everything that changes the database inside the entity class.

Here is a table where I try to list the advantages/disadvantages of each approach.

Approach	Pros	Cons
1. POCO-only	Follows the original DDD description of handling entities.	Hard to write generic repositories. Repository pattern in data layer can cause performance problems. The repository can be bypassed.
2. C(r)UD only	Access methods follow the single responsibility principle. Business logic is separated from database accesses.	More work writing the business logic and its dedicated repository.
3. C(r)UD + Biz Logic	Everything in one place.	Can suffer with the “God Object” anti-pattern. Problems calling methods in “higher” assemblies.

While I think a DDD approach is useful, it’s a trade-off of features and limitations whichever way you go. In the end it depends on a) your application and b) what you feel comfortable with. For instance, if your application has some really complex business logic, with lots of calls to other parts of the system then the C(r)UD + Biz Logic approach might not work that well or you.

Conclusion

I do believe using a DDD approach with EF Core has many benefits – it can make the code clearer and more robust (e.g. no way to get it wrong). But coming up with a good DDD approach can take a while – I built my first DDD business logic approach in 2015, and it’s been through two iterations in 2016 and 2017 as I have improved it. This article is just another part of my continuing journey to learn and improve my skills around using DDD.

I am also want to be an efficient developer, so whatever approach must allow me to build quickly. But with a bit of help from some libraries, like the EfCore.GenericServices, I can build both robust, well-performing systems quickly. And when new features, like JSON Patch, become useful I try to adapt my approach to see if it can be added to my DDD style (see my article “Pragmatic Domain-Driven Design: supporting JSON Patch in Entity Framework Core” to see the outcome).

I hope this article helps you decide for yourself whether it’s worth using the DDD approach with EF Core and which approach suits your needs.

Happy coding!

TR;DR; – summary

LINQ Select method normally produces the fastest performing query because they only load what you need, and some calculations can be run in the database (see Posts.Count later) .
Writing Select queries contains more lines of code than other query approaches because you have to select each property you want to load.
AutoMapper’s ProjectTo method can build the Select query for you, but you need to configure the mapping.
Setting up the AutoMapper configuration isn’t total obvious (to me at least) so I give you some examples.
My GenericServices library uses AutoMapper and does some of that configuring for you.

Setting the scene – A quick intro to Entity Framework Core

In the next section I am going to build some queries to show this database in different ways.

Part 1. Building a simple database query – a list Bloggers

The first display I want to build a list of each blogger, with the number of posts they have written. Here is the type of display I want to show.

This must pick out information from two of entity classes. Now, let’s see the different ways of obtaining that data.

1a. Using Eager loading to load the Post classes

One way to do that would be to use Eager Loading. The code would look like this.

var list = context.Bloggers
    .Include(x => x.Posts)
    .ToList();

NOTE: You get EVEN WORSE performance if you load the Posts via Lazy Loading, e.g. having a virtual public ICollection<Post> of Posts. That will cause one extra trip to the database for every entry. I get paid to performance tune EF systems and the first thing I look for is Lazy Loading – it’s a major performance drag.

See the Summary of the performance table for detailed timings.

1b. Using a hand-coded Select query

 
public class ListBloggersDto 
{
    public string Name { get; set; }
    public string EmailAddress { get; set; }
    public int PostsCount { get; set; }
}

And here is a hand-coded Select query that will only read in the data I actually need.

 
var list = context.Bloggers.Select(x =>
    new ListBloggersDto
{
    Name = x.Name,
    EmailAddress = x.EmailAddress,
    PostsCount = x.Posts.Count
}).ToList();

 
SELECT "x"."Name", "x"."EmailAddress", (
    SELECT COUNT(*)
    FROM "Posts" AS "p"
    WHERE "x"."BloggerId" = "p"."BloggerId"
) AS "PostsCount"
FROM "Bloggers" AS "x"

Summary of the performance

NOTE: I didn’t test Lazy loading as EF Core 2.1 wasn’t released when I was writing that chapter, but I can tell you it would be equal (or maybe worse) to the explicit loading timing.

Quick to write – AutoMapper

1c. Using AutoMapper to build the Select query automatically

Create your DTO
Add an AutoMapper configuration,
Then use that AutoMapper configuration in your query.

Let’s go through each part in turn

a) Create my DTO

I need to create my DTO, which is the same as the hand-coded one.

 
public class ListBloggersDto
{
    public string Name { get; set; }
    public string EmailAddress { get; set; }
    public int PostsCount { get; set; }
}

NOTE: AutoMapper can also ‘flatten’ relationships’: what that means it can pick out a one-to-one relationship by joining together the two names without the dot in the middle. You will see this in part 2, where I list each Post, but access the Blogger’s Name by using the name of the relationship, Blogger, followed by the property’s name, Name, i.e. BloggerName.

b) Configure the AutoMapper mapping via dependency injection

There are lots of ways to configure AutoMapper mappings. For NET Core the recommends using AutoMapper’s Profile and assembly scanning with NET Core’s Dependency Injection (DI).

 
public class BloggerDtosProfile : AutoMapper.Profile
{
    public BloggerDtosProfile()
    {
        CreateMap<Blogger, ListBloggersDto>();
        // Add other CreateMap’s for any other configs
    }
}

Then you need to call AutoMapper’s AddAutoMapper() method. In ASP.NET Core that would be in the ConfigureServices method in the Startup class, e.g.

 
public IServiceProvider ConfigureServices(
    IServiceCollection services)
{
    // … other setups removed for clarity
    services.AddAutoMapper();
}

The AddAutoMapper() method scans all the assemblies looking for all the classes that inherit AutoMapper.Profile class. It then produces a IMapper instance containing the all the Mappings found.

NOTE: I find the configuring AutoMapper is the hard part, as it relies on me remembering to set up the AutoMapper mapping. I have tried a number of ways to make this better and I think in my EfCore.GenericServices I have got it right. In that I write my DTO and then you need to add an empty interface called ILinkToEntity<TEntity>, where TEntity is your class that EF Core maps to the database. Then the setup of EfCore.GenericServices a) finds the DTOs and then uses the TEntity part of the ILinkToEntity<TEntity> interface to, among other things, form the correct AutoMapper mappings.

c) Use AutoMapper in your query

I’m now going to show you a ASP.NET Core controller so that you can see how the DI works, but you can use AutoMapper in any application you like.

 
public class HomeController : Controller
{
    private readonly BlogContext _context;
    private readonly IMapper _mapper;

    public HomeController(BlogContext context, IMapper mapper)   
    {                                              
        _context = context;  
        _mapper = mapper;                      
    }                                              

    public IActionResult Index()            
    {
        var dtos = _context.Bloggers
            .ProjectTo<ListBloggersDto>(_mapper)     
            .ToList();   

        return View(dtos);         
    }
}

NOTE: If you are generally doing CRUD operations then you should look at my article “GenericServices: A library to provide CRUD front-end services from a EF Core database” that uses AutoMapper and EF Core to make CRUD very simple to do.

AutoMapper is clever and can work out some relationships for you, but for more complex relationships you need to do some more configuring, which I cover in the next part.

Part 2: Handling more complex database queries – list of posts

The next example is a summery list of all the articles, showing the bogger (author), the post title and the tags the post has.

This is a bit more complex, especially around getting the tag names.

2a. List posts using hand-coded select query

Let’s start with the hand-coded LINQ Select query. First, I need to show you the DTO I use to hold the resulting query.

 
public class ListPostsDto 
{
    public string BloggerName { get; set; }
    public string Title { get; set; }
    public DateTime LastUpdated { get; set; }
    public List<string> TagNames { get; set; }
}

Now the hand-coded Select query.

 
var dtos = context.Posts.Select(x => 
    new ListPostsDto
{
    BloggerName = x.Blogger.Name,
    Title = x.Title,
    LastUpdated = x.LastUpdated,
    TagNames = x.TagLinks.Select(y => y.Tag.Name).ToList()
}).ToList();

Notice the following:

Line 4: I can select the Name of the Blogger by using the navigational property. EF can handle any depth of selection, say the Bogger class had a property called Address, which has its own table holding address information, then I could access the Blogger’s Country via x.Blogger.Address.Country. EF produces efficient SQL (an INNER JOIN) to extract the data from the related tables.
Line 7: I want a list of all the Tag Name’s associated with this post. I therefore need to use the PostTag linking table to access the Tag names.

The produces fairly efficient (see note below) SQL query by only extracting the data you need.

NOTE: Handling “many” relationships, i.e. ones that return multiple results like the TagNames, is an area where EF Core can have performance issues. If I left off the .ToList() on the TagNames then it would query the database for each ListPostsDt0 (this is known as the N+1 database queries issue). But in EF Core version 2.1, adding the .ToList() turns the load of all the TagNames into one final database query. This is OK, but can still have problems – see my article “Entity Framework Core performance tuning – a worked example” to see an even better way to handle comma separated strings.

2b. List of posts using AutoMapper and special configuration

 
public class ListPostsDto
{
    public string BloggerName { get; set; }
    public string Title { get; set; }
    public DateTime LastUpdated { get; set; }
    public List<string> TagNames { get; set; }
}

Now the AutoMapper configuration code.

 
public class PostsDtosProfile : AutoMapper.Profile
{
    public PostsDtosProfile()
    {
        CreateMap<Post, ListPostsDto>()
            .ForMember(
                p => p.TagNames, 
                opt => opt.MapFrom(x => 
                    x.TagLinks.Select(y => y.Tag.Name).ToList()));
    }
}

NOTE: AutoMapper automatically turns the BloggerName into Blogger.Name – this is the “flattening” I referred to earlier. This is a very useful feature and makes handling references to a single relationship really easy.

 
[Fact]
public void TestProjectionMappingPosts()
{
    //SETUP
    var config = AutoMapperHelpers
        .CreateMapperConfig<PostsDtosProfile>();

    //ATTEMPT
    var input = EfTestData.CreateBloggersWithPosts()
        .AsQueryable();
    var list = input.ProjectTo<ListPostsDto>().ToList();

    //VERIFY
    list.First().BloggerName.ShouldEqual("Ada Lovelace");
}

Where my helper method CreateMapperConfig, looks like this.

NOTE: Technical point about unit testing. ProjectTo<T> need MapperConfiguration when testing, while Map<T> needs IMapper. Have a look at my unit tests of AutoMapper here for a good example of how to write tests that include AutoMapper.

 
public static MapperConfiguration CreateMapperConfig<T>() 
    where T : Profile, new()
{
    var config = new MapperConfiguration(cfg =>
    {
        cfg.AddProfile(new T());
    });
    return config ;
}

This produces the same SQL as the hand-codes example.

NOTE: In EfCore.GenericServices you can configure the AutoMapper mappings by adding a class which inherits from PerDtoConfig<TDto, TEntity>. When EfCore.GenericServices is scanning for DTOs it also looks for matching PerDtoConfig<TDto, TEntity> classes uses that data to alter the AutoMapper’s MapperConfiguration.

Conclusion

Happy coding.

This article is part of a series on authorization in ASP.NET Core

Part 1: A better way to handle authorization in ASP.NET Core (this article).
Part 2: Handling data authorization in ASP.NET Core and Entity Framework Core.

NOTE: you can Clone the GitHub repo and run locally – it uses in-memory databases so it will run anywhere. The application was written using ASP.NET Core 2.1: parts of the ASP.NET Identity is changing, but the overall concept will work with any version of ASP.NET Core.

This is a long article, so here are links to the major parts:

TL;DR; – summary

NET Role-based authorization system works for systems with simple authorization rules, but it has limitations, like the fact that you have to republish your code if you change the authorization rules.
Roles, with names like “Manager” or “ExternalBuyer” makes sense for users (human or external services) as they define a “Use Case” of what the user should be able to do.
But Roles don’t work well when applied to ASP.NET Core actions or Razor Pages. Here you need a much more fine-grained solution, with names like “CanRequestHoliday”, “CanApproveHoliday” – I call these Permissions.
The solution is to map the user’s Roles to a group of Permissions and store these in the User’s Claims.
Then I use ASP.NET Core’s new policy-based authorization system to check that the User’s Permissions Claims contains the Permission placed on the action/page they want to access.
There is an open-source example ASP.NET Core application to go with this article.

Setting the Scene – a look at different application security needs

If you understand ASP.NET’s authorization and authentication features then you can skip this section.

Authorization breaks down into two parts:

What data can I access? For instance, you can see your personal information, but not other people’s personal information.
What features you can use? For instance, are you allowed to change the title of a book that you can see?

NOTE: This article only describes a way to manage the second part, what features can you use.

Role authorization: what is it and what are its limitations?

[Authorize(Roles = "Staff,Manager")]
public ActionResult Index()
{
    return View(MyData);
}

This works for applications that have fairly simple and well-defined Roles, like User/Admin or Staff/Manager/Admin, then Roles is a good choice. But here are some of the problems I have found:

If you have lots of roles you can end up with long Authorize attributes, e.g. [Authorize(Roles = “Staff,HrManager,BizManager,DevManage,Admin,SuperAdmin”)].
Because Authorize is an attribute then the string has to be a constant, e.g. you can’t have $”{RoleConstants.StaffRole}”. This means if things change you are editing strings, and you could misspell something really easily.
The big one for me is your authorization rules are hard-coded into your code. So, if you want to change who can access a certain action you have to edit the appropriate Authorize attributes and redeploy your application.

The architecture of the Roles-to-Permissions system

1. Introducing Roles, Permissions and Modules

NOTE: In the example application I have Roles of “Staff”, “Manager”, and “Admin.

NOTE: In the example application I have Permissions on my ColorController of “ColorRead”, “ColorCreate”, “ColorUpdate”, and “ColorDelete”.

NOTE: In the example application I have Permissions called “Feature1” and “Feature2” which are mapped to Modules with the same name.

2. How this is implemented

Here is a diagram of that login stage. It might not make a lot of sense yet, but I describe each part in the rest of the article. This diagram is to give you an overview.

NOTE: Don’t forget there is example application if you want to look at the actual code.

Now I’m going to build up the Roles-to-Permissions in stages and explain what each part does.

Why I used Enums for the Permissions

In a big application there could be hundreds of Permissions. This lead to two problems:

If I use Cookie Authorization there is a maximum size of 4096 bytes for the Cookie. If I had hundreds of long strings I might start to fill up the Cookie, and I want some room for other things like my data authorization. If I can store the Enums permissions as a series of integers it’s going to be much smaller than a series of strings.
Secondly, I want to help the Admin person who needs to build the mapping from Roles to permissions. If they need to scroll through hundreds of permission names it could be hard to work out which ones are needed. It turns out Enum members can have attributes, so I can add extra information to help the Admin person.

So, here is part of my Permissions Enum code

public enum Permissions
{
    //Here is an example of very detailed control over something
    [Display(GroupName = "Color", Name = "Read", Description = "Can read colors")]
    ColorRead = 0x10,
    [Display(GroupName = "Color", Name = "Create", Description = "Can create a color entry")]
    ColorCreate = 0x11,
    [Display(GroupName = "Color", Name = "Update", Description = "Can update a color entry")]
    ColorUpdate = 0x12,
    [Display(GroupName = "Color", Name = "Delete", Description = "Can delete a color entry")]
    ColorDelete = 0x13,

    [Display(GroupName = "UserAdmin", Name = "Read users", Description = "Can list User")]
    UserRead = 0x20,
    //This is an example of grouping multiple actions under one permission
    [Display(GroupName = "UserAdmin", Name = "Alter user", Description = "Can do anything to the User")]
    UserChange = 0x21,

    [Obsolete]
    [Display(GroupName = "Old", Name = "Not used", Description = "example of old permission"
    OldPermissionNotUsed = 0x40,

The things to note are:

I show two types of permissions.
- First four (lines 4 to 1) are fine-grained permissions, almost one per action.
- Next two (lines 13 to 17) are more generic, e.g. I have a specific “UserRead”, but then one permission called “UserChange” which allows create, update, delete, lock, change password etc.
Line 5, 7, etc. Notice that I give each enum a specific number. If you are operating a 24/7 application with a new version seamlessly replacing the old version, then the Permission numbed mustn’t change otherwise user’s Claims be wrong. That is why I give each enum a specific number.
Line 19. I also support the Obsolete attribute, which stops the Permission appearing in the listing. There are plenty of scary stories about reusing a number with unintended consequences. (Also, it you try to use something marked as Obsolete you get a warning).
Line 4 etc. I add a Display Attribute to each Permission Enum. This has useful information that I can show lots of useful information to help the person who is building a Role.
Line 4, 6, 8, 10. I “Group” permissions that are used in the same place. This makes it easier for the Admin person to find the things they want. I also number in Hex, which gives me 16 possible permissions in a Group (I tried 10 and you could go over that, so 16 is better).

Here is a list of some of the Permissions in my example application listed via the Users->List Permissions nav dropdown.

And the code that produced that output (link to PermissionDisplay class for the whole thing)

public static List<PermissionDisplay> GetPermissionsToDisplay(Type enumType) 
{
    var result = new List<PermissionDisplay>();
    foreach (var permissionName in Enum.GetNames(enumType))
    {
        var member = enumType.GetMember(permissionName);
        //This allows you to obsolete a permission and it won't be shown as a
        //possible option, but is still there so you won't reuse the number
        var obsoleteAttribute = member[0].GetCustomAttribute<ObsoleteAttribute>();
        if (obsoleteAttribute != null)
            continue;
        //If there is no DisplayAttribute then the Enum is not used
        var displayAttribute = member[0].GetCustomAttribute<DisplayAttribute>();
        if (displayAttribute == null)
            continue;

        //Gets the optional PaidForModule that a permission can be linked to
        var moduleAttribute = member[0].GetCustomAttribute<PermissionLinkedToModuleAttribute>();

        var permission = (Permissions)Enum.Parse(enumType, permissionName, false);

        result.Add(new PermissionDisplay(displayAttribute.GroupName, displayAttribute.Name, 
                displayAttribute.Description, permission, moduleAttribute?.PaidForModule.ToString()));
    }

    return result;
}

How to handle optional/paid-for features?

Marking Permissions as linked to a module is easy to do with the Enums – I just add another attribute. Here an example of Permissions linked to a Module (see line 5).

public enum Permissions
{
    //… other Permissions removed for clarity

    [LinkedToModule(PaidForModules.Feature1)]
    [Display(GroupName = "Features", Name = "Feature1", Description = "Can access feature1")]
    Feature1Access = 0x30,
    [LinkedToModule(PaidForModules.Feature2)]
    [Display(GroupName = "Features", Name = "Feature2", Description = "Can access feature2")]
    Feature2Access = 0x31
}

The paid-for modules are again represented by an Enum, but one marked the [Flags] attribute because a user can have multiple modules that they have subscribed to. Here is my PaidForModules Enum code

[Flags]
public enum PaidForModules : long
{
    None = 0,
    Feature1 = 1,
    Feature2 = 2,
    Feature3 = 4
}

NOTE I add “: long” to the Enum which gives me up to 64 different modules in my system.

How do I turn the Roles into a Permissions Claim?

If the Claims already have the Permissions claim type then nothing to do so return quickly.
Then we get the Roles the user has from the Role Claim
I need to access my part of the database. I can’t use dependency injection, so I use the extraAuthDbContextOptions, which is a singleton that I can provide at startup.
Then I get all the permissions for all of the roles, with a Distinct to remove unnecessary duplicates.
Then I filter out any permissions that are linked to a Module that the user doesn’t have access to.
Then I add a permissions Claim containing all the Permissions the user is allowed, but packed as hex numbers in a single string so that it doesn’t take up so much room (I used Hex format as it made debugging easier).
Finally I have to create a new ClaimsPrincipal and tell ASP.NET Core to replace the current ClaimsPrincipal, plus set the all-important ShouldRenew to true, which updates the Cookie, otherwise this complex (slow) method on every HTTP request!

public async Task ValidateAsync(CookieValidatePrincipalContext context)
{
    if (context.Principal.Claims.Any(x => 
        x.Type == PermissionConstants.PackedPermissionClaimType))
        return;

    //No permissions in the claims so we need to add it
    //This is only happens once after the user has logged in
    var claims = new List<Claim>();
    foreach (var claim in context.Principal.Claims)
    {
        claims.Add(claim);
    }

    var usersRoles = context.Principal.Claims
        .Where(x => x.Type == ClaimTypes.Role)
        .Select(x => x.Value)
        .ToList();
    //I can't inject the DbContext here because that is dynamic, 
    //but I can pass in the database options because that is a 
    //From that I can create a valid dbContext to access the database
    using (var dbContext = new ExtraAuthorizeDbContext(_extraAuthDbContextOptions))
    {
        //This gets all the permissions, with a distinct to remove duplicates
        var permissionsForUser = await dbContext.RolesToPermissions
            .Where(x => usersRoles.Contains(x.RoleName))
            .SelectMany(x => x.PermissionsInRole)
            .Distinct()
            .ToListAsync();

        //we get the modules this user is allows to see
        var userModules =
            dbContext.ModulesForUsers
                .Find(context.Principal.Claims
                     .SingleOrDefault(x => x.Type == ClaimTypes.Name).Value)
                ?.AllowedPaidForModules ?? PaidForModules.None;
        //Now we remove permissions that are linked to modules that the user has no access to
        var filteredPermissions =
            from permission in permissionsForUser
            let moduleAttr = typeof(Permissions).GetMember(permission.ToString())[0]
                .GetCustomAttribute<LinkedToModuleAttribute>()
            where moduleAttr == null || userModules.HasFlag(moduleAttr.PaidForModule)
            select permission;

          //Now add it to the claim
          claims.Add(new Claim(PermissionConstants.PackedPermissionClaimType,
              filteredPermissions.PackPermissionsIntoString()));    }

    var identity = new ClaimsIdentity(claims, "Cookie");
    var newPrincipal = new ClaimsPrincipal(identity);

    context.ReplacePrincipal(newPrincipal);
    context.ShouldRenew = true;
}

How do I convert my Permissions into Policy-based authorization?

public class PermissionHandler : 
    AuthorizationHandler<PermissionRequirement>
{
    protected override Task HandleRequirementAsync(
        AuthorizationHandlerContext context, 
        PermissionRequirement requirement)
    {
        var permissionsClaim = context.User.Claims
            .SingleOrDefault(c => 
                 c.Type == PermissionConstants
                     .PackedPermissionClaimType);
        // If user does not have the scope claim, get out of here
        if (permissionsClaim == null)
            return Task.CompletedTask;

        if (permissionsClaim.Value
            .ThisPermissionIsAllowed(
                 requirement.PermissionName))
        {
            context.Succeed(requirement);
        }

        return Task.CompletedTask;
    }
}

There are two other classes that are involved in making dynamic policy-based authorisation work. Here are the links to them:

The AuthorizationPolicyProvider that is registered at startup.
A rather boring, but necessary, PermissionRequirement

[AttributeUsage(AttributeTargets.Method 
    | AttributeTargets.Class, Inherited = false)]
public class HasPermissionAttribute : AuthorizeAttribute
{
    public HasPermissionAttribute(Permissions permission) 
       : base(permission.ToString()) { }
}

That’s pretty simple, but it means I get intellisence when I am adding the Permission.

Putting it all together

[HasPermission(Permissions.ColorRead)]
public ActionResult Index()
{
    return View(MyData);
}

We also need to add two tables to whatever database your application uses. The two EF Core entity classes are:

RoleToPermissions, which maps an ASP.NET Core Role to a set of Permission Enums
ModulesForUser, which holds what modules each user is allowed to access.

Once the application is up and running an Admin-type user has to:

Create some roles, e.g. “Staff”, “Manager”, etc. using ASP.NET Core Identity code.
Create matching RoleToPermissions for each of the roles, specifying what Permissions map to each Role.

Then, for every new user an Admin person (or some automatic subscription code) has to:

Create the user (if an invite-only type application)
Add the correct Roles and ModulesForUser for that new user.

Once that is done all the code I have shown you takes over. The user logs in, gets the Permissions and what they can access is managed by ASP.NET Core’s policy-based authentication.

Things I didn’t cover elsewhere

There are a few things I didn’t cover in detail, but here are links to the items:

The important parts about registering things are shown in highlighted lines in this link to the Startup class. (NOTE: I built the application with ASP.NET Core 2.1, but I know the Identity parts are changing in 2.2, so you might have to update the code I put in the Startup class for newer versions of ASP.NET Core).
You don’t need to use ASP.NET Core Identity system at all – I said the client’s version uses an external authentication system. You just have to create a Roles-to-User table so you can assign Roles to each user.
I didn’t cover how I Packed/Unpacked the Permissions. You can find the Extension methods for doing that in PermissionPackers.
You might want to check for a permission in your razor pages to show/hide links. I created a simple method in the PermissionsExtension class, and used it in the _layout.cshtml Razor page.

Conclusion

TL;DR; – summary

You can control what data a user sees by adding [Authorize(…] attributes to your controller actions or configuring razor pages, but it’s an all-or-nothing control – see the Part1 article for how to do this.
If you want to filter the rows in a table, say only returning data associated to the current logged-in user, the you need a per-row data protection system – see this article on how to do that.
The best way to implement per-row data protection is to put all the code inside EF Core’s DbContext. That way the protection is on be default to every class/table that you want protected.
The process to add per-row protection to an entity class (table) is
- Mark every new row added to a table with a protect key, and…
- All queries for that entity class/table are filtered using a key provided from the user’s information.
This article starts with the protection of personal data, then moves onto the handling grouped user access (known as multi-tenant systems) and finishes with the more complex hierarchical (e.g. manager->staff) system example.
All the code in this article is available as an open-source repo on GitHub – see PermissionAccessControl

Setting the Scene – the different ways for protecting data

An ASP.NET Core web application typically has two parts:

Static parts that is fixed by the code, e.g. how a page looks or the type of Web API it provides. Typically, you need to update your code and publish it to change the static parts.
Dynamic data that can change, e.g. data held in a database or data obtained from external services.

In this article I’m going to look how you can protect data in a database. When we say we want to protect data we are talking about controlling who can see, or change, data stored in the database. If going to discuss two forms of data protection: all-or-nothing and per-row protection – see diagram below

The all-or-nothing data protection is where the user can either access all of the data or can’t access it at all. In ASP.NET applications this is done by placing authorization controls on the controller actions or Razor Pages. This does allow separate control over different types of access to the data, for instance, we can allow a user to read all of the data, but not allow the same user to create, update or delete any of that data.

Many applications only need all-or-nothing data protection, but some application types need per-row protection. Here are some examples:

Protection of personal data, e.g. e-commerce systems which hold personal data for its users.
Applications with groups of users that need to be isolated from each other. As a developer I’m sure you know GitHub, which is one (extreme) example of such a system. Generally, these types of systems are referred to as multi-tenant
Applications where have hierarchical data needs, e.g. where a Manager-to-staff relationship exists and say staff can access one set of data, but the Manager can see all the data their staff can access. I add this as it covers the implementation of filtering data in different ways depending on the type of user, e.g. a manager or a staff user.

The rest of this article is about applications that need per-row protection and how you might implement that with EF Core.

The two elements of per-row protection

In its core per-row protection needs two things:

A protect key in each row used to define who can access that data.
The user needs to provide a corresponding key than can be compared with protect key in the row.

The protect key I refer to in the list above about could be a unique value related to an individual user, a group of users, or some form of access key. This protect key is sometimes referred to as the user Id or the tenant Id, and in this article I also use the name “OwnedBy”.

Typically, the “OwnedBy” protect key is held in the information about the currently logged-in user. In ASP.NET Core this is called the ClaimsPrincipal, and it contains a series of Claims (see this article for an introduction on Claims and ClaimsPrincipal). Below is a screenshot of the basic claims of a logged-in user called Staff@g1.com.

As you can see the User has a nameidentifier claim, which a unique string for that user, often referred to as the UserId. In the diagram below I use the UserId as the protect key to select only the addresses in the UsersAddresses table that are link this this user. At the same time the UsersAddresses table has a corresponding protect key held in the “OwnedBy” property/column.

This diagram shows the filter part of the per-row protection system: we get a protect key from the ClaimsPrincipal and use it to filter any read of the protected data. The key will vary, and the filter expression will vary, but the approach will be the same in all cases.

The other part is how we add the protect key to an entity class/database table (what I call “marking an entity”), but that I will explain in the implementation section.

Example 1: Implementation of per-row protection on personal data

I am now going to implement the two elements of the protect key, which are:

Marking data with the per-row protect key.
Filtering data using the user-provided protect key.

There are a number of ways to do this, but I will be placing as much of the per-row protection code inside EF Core’s DbContext. This simplifies your own CRUD/business code, and more importantly it makes sure the per-row protection doesn’t get forgotten in the rush to get your code finished.

What I describe below is the same for all per-row protection, but I use the example of protecting data linked to a user. I go into detail of each step in this process, but in later examples I assume you have read this section for the general approach.

1. Marking data with the per-row protect key

The first step is to “mark” each row with the correct protect key, which I refer to at the “OwnedBy” property, i.e. that row is “owned” by the given key. I want to automatically mark any new protected table rows with the correct protect key deep inside the application’s DbContext. I do this by creating an interface, IOwnedBy, (see below) which I apply to every entity class that needs row-level protection. That interface allows me to detect inside SaveChanges any newly created data that needs the OwnedBy” property filled in before saving to the database.

Let’s look at parts of the code that do this, starting with the IOwnedBy interface.

1a. IOwned interface

public interface IOwnedBy
{
    //This holds the UserId of the person who created it
    string OwnedBy { get; } 
    //This the method to set it
    void SetOwnedBy(string protectKey);
}

We also create a class called OwnedByBase that implements that interface and inherits that interface, which makes it easier to add the protection code to any entity classes (i.e. classes that are mapped to the database by EF Core). Here is a link to the PersonalData class in the example code that shows this in action.

1b. Override the SaveChanges and calling MarkCreatedItemAsOwnedBy

The next part is override the SaveChanges/SaveChangesAsync in the application’s DbContext. Here is SaveChanges version (Async version the same, but with async), so I have a common method called MarkCreatedItemAsOwnedBy which sets the “OwnedBy” property.

public override int SaveChanges
    (bool acceptAllChangesOnSuccess)
{
    this.MarkCreatedItemAsOwnedBy(_userId);
    return base.SaveChanges(acceptAllChangesOnSuccess);
}

Note the _userId on line 4. That is the UserId taken from the nameidentifier Claim – I show how you get that in the next section.

The MarkCreatedItemAsOwnedBy extension method uses EF Core’s ChangeTracker to find the newly created entities and then checks if the entity has the IOwnedBy interface. The code looks like this:

public static void MarkCreatedItemAsOwnedBy(
    this DbContext context, string userId)
{
    foreach (var entityEntry in context.ChangeTracker.Entries()
        .Where(e => e.State == EntityState.Added))
    {
        if (entityEntry.Entity is IOwnedBy entityToMark)
        {
            entityToMark.SetOwnedBy(userId);
        }
    }
}

The effect of all that is that the OwnedBy property is filled in with the current user’s UserId on any class that has the IOwnedBy interface.

1c. Getting the UserId into the application’s DbContext

Ok, the first two parts are fairly basic, but getting the UserId into the DbContext is a bit more complex. In ASP.NET Core its done by a series of dependency injection (DI) steps. This can be a bit hard to understand so here is a diagram of how it works.

Here is the method extract the UserId from the current user’s Claims.

public class GetClaimsFromUser : IGetClaimsProvider
{
    public GetClaimsFromUser(IHttpContextAccessor accessor)
    {
        UserId = accessor.HttpContext?
            .User.Claims.SingleOrDefault(x => 
                x.Type == ClaimTypes.NameIdentifier)?.Value;
    }

    public string UserId { get; private set; }
}

There are lots of null handling because a) there will be no HttpContext on start, and b) the NameIdentifier Claim won’t be there if no one is logged in. This class is registered in DI in the ConfigureServices method inside the Startup class. That means when the application’s DbContext is created the UserIdFromClaims class is injected.

NOTE: In this case I am injecting the UserId, which is found in the nameidentifier Claim. But you will see later that the same process will allow us to access any Claim in the User’s data.

Here is constructor of the application’s DbContext showing the normal options parameter and the added IGetClaimsProvider data.

public class PersonalDbContext : DbContext
{
    private readonly string _userId;

    public PersonalDbContext(
        DbContextOptions< PersonalDbContext> options, 
        IGetClaimsProvider userData)
        : base(options)
    {
        _userId = userData.UserId;
    }
    //... other code left out
}

The end result of all that is the application’s DbContext has access to the user’s id.

2. Filtering per-row protected data

Now that I have the per-row protected data marked with a protect key (in this case the UserId) I need to apply a filter to only allow users with the correct protect key to see that data. You could write your own code to restrict access to certain data, but with EF Core there is a much better approach – Global Query Filters (shortened to Query Filters in this article).

You can apply Query Filters to any entity class used by EF Core and they pre-filter every read of that entity class. Here is the code that sets this up for the entity class PersonalData in the application’s DbContext.

protected override void OnModelCreating
    (ModelBuilder modelBuilder)
{
    modelBuilder.Entity<PersonalData>()
        .HasQueryFilter(x => x.OwnedBy == _userId);
}

In the UserId example that means a user can only retrieve the PersonalData entity classes that have their UserId, and that filtering is done inside the application’s DbContext so you can’t forget.

NOTE: There is a way to remove any Query Filter from an entity class by adding the IgnoreQueryFilters method to any query. Obviously, you need to be careful where to do this as it removed your security, but it’s pretty clear what you are doing.

Example 2: Implementing a multi-tenant system

The first example was dealing with personal data, which is useful but it’s a simple problem which could have written into your CRUD or business logic code. But when it comes more complex per-row protection problems, like multi-tenant systems, then all that code I just showed you is a much better solution to hand-coding data protection.

As an example of a multi-tenant system I’m going to produce an application to manage the stock in a shop. To cover the cost of the development and hosting I want to sell the application to lots of shops. To do this I need to make sure that each shop’s inventory data is kept separate from other shops. To do this I need to:

When a new shop is signed up I need to create a unique protect key for that shop.
An admin person needs to assign a user to a specific shop.
I need to add the shop’s protect key (called ShopKey) as a claim in the logged-in user’s ClaimPrincipal.
I need to add the per-row protection to all the entity classes/tables that are used by a shop.

Let’s look at each of these stages in turn.

1. Creating a unique protect key for each shop

When a new shop wants to join our inventory control application I need to add a new row to the Shops table. This might be done automatically via some signup system or via an admin person. In my example I use the primary key of the new Shop entity class as the protect key.

2. Assigning a user to a specific shop

Someone (or some service) needs to manage the users that can access a shop. One good approach is to make the person who signed up for the service an admin person, and they can register new users to their shop.

There isn’t anything built in to ASP.NET Core’s authorization system to do this, so we need to add an extra class/table to match UserId’s to the shop key. Here is my MultiTenantUser class that holds the data protection information for each shop user.

public class MultiTenantUser : IShopKey
{
    [Required]
    [MaxLength(40)]
    [Key]
    public string UserId { get; set; }
    public int ShopKey { get; private set; }

    //other code left out for clarity
}

This is very simple: I use the UserId provided by ASP.NET Core as the primary key and then set the ShopKey to the admin person’s ShopKey.

3. Lookup the user’s ShopKey and add it as a claim in the logged-in user’s ClaimPrincipal.

In the personal data example I used an existing claim, nameidentifier, in the ClaimsPrincipal as the protect key. In the multi-tenant example the big difference is I need to add a externally held protect key to the ClaimsPrincipal.

NOTE: In my implementation I use cookie-base authentication, but I could have used JWT tokens, social logins etc. The approach I share will work with any of these, but the implementation of how you add a new claim to the ClaimsPrincipal will be different for each authentication type.

In part 1, a better way to handle authorization in ASP.NET Core, I already showed how to add extra Claims to the User – in that article it was a Permissions Claim to help with feature authorization. In this article use the same approach as Part 1 to add the ShopKey to the user’s Claims. That is, I tapped into an event in the Authorization Cookie called ‘OnValidatePrincipal’ (here is a link to the lines in the example application startup class that sets that up). This calls the code below:

public class DataCookieValidate
{
    private readonly DbContextOptions<MultiTenantDbContext> 
         _multiTenantOptions;

    public DataCookieValidate(DbContextOptions<MultiTenantDbContext>
        multiTenantOptions)
    {
        _multiTenantOptions = multiTenantOptions;
    }

    public async Task ValidateAsync(CookieValidatePrincipalContext context)
    {
        if (context.Principal.Claims.Any(x => 
            x.Type == GetClaimsFromUser.ShopKeyClaimName))
            return;

        //No ShopKey in the claims, so we need to add it. 
        //This is only happens once after the user has logged in
        var claims = new List<Claim>();
        claims.AddRange(context.Principal.Claims); 

        //now we lookup the user to find what shop they are linked to
        using (var multiContext = new MultiTenantDbContext(
              _multiTenantOptions, new DummyClaimsFromUser()))
        {
            var userId = context.Principal.Claims.Single(x => 
                 x.Type == ClaimTypes.NameIdentifier).Value;
            var mTUser = multiContext.MultiTenantUsers
                 .IgnoreQueryFilters()
                 .SingleOrDefault(x => x.UserId == userId);
            if (mTUser == null)
                throw new InvalidOperationException($"error…”);
            claims.Add(new 
                Claim(GetClaimsFromUser.ShopKeyClaimName, 
                      mTUser.ShopKey.ToString()));
        }

        var identity = new ClaimsIdentity(claims, "Cookie");
        var newPrincipal = new ClaimsPrincipal(identity);
        context.ReplacePrincipal(newPrincipal);
        context.ShouldRenew = true;  
    }
}

NOTE: In Part1 I give you a detailed description of the stages in the ValidateAsync method, but for authorization. However the steps are very similar so see “How do I turn the Roles into a Permissions Claim?” for a step-by-step breakdown of how the code works.

This code is called for every HTTP request, but the first line quickly returns on all but the first request after a login. On first login the ShopKey claim isn’t in the ClaimsPrincipal so it has to look up the user in the MultiTenantUsers table to add that key to the ClaimsPrincipal. The last line of the code updates the ASP.NET Core authentication cookie content so we don’t have to do this again, which means it’s very quick once it has run once.

4. Add the per-row protection to all the entity classes linked to a shop

This is the same process as I described in the first example of personal data, so I’m going to whiz through the code.

First, I need an interface that every protected shop entity class has to have. In this example it’s called IShopKey, and here is it applied to the StockInfo entity class, with lines 7 to 11 implementing the IShopKey requirements.

public class StockInfo : IShopKey
{
    public int StockInfoId { get; set; }
    public string Name { get; set; }
    public int NumInStock { get; set; }

    public int ShopKey { get; private set; }
    public void SetShopKey(int shopKey)
    {
        ShopKey = shopKey;
    }

    //---------------------------------------
    //relationships

    [ForeignKey(nameof(ShopKey))]
    public Shop AtShop { get; set; }
}

Then I need to extract the ShopKey claim so I can access it in the Shop application’s DbContext. Here is the variant of the GetClaimsProvider that does that.

public class GetClaimsFromUser : IGetClaimsProvider
{
    public const string ShopKeyClaimName = "ShopId";
    public int ShopKey { get; private set; }

    public GetClaimsFromUser(IHttpContextAccessor accessor)
    {
        var shopKeyString = accessor.HttpContext?
            .User.Claims.SingleOrDefault(x => 
                  x.Type == ShopKeyClaimName)?.Value;
        if (shopKeyString != null)
        {
            int.TryParse(shopKeyString, out var shopKey);
            ShopKey = shopKey;
        }
    }
}

In this case I convert the claim value, which is a string, back into an integer as that is more efficient in the database. The query filters in the DbContext’s OnModelCreating method look like this.

protected override void OnModelCreating(ModelBuilder modelBuilder)
{
    modelBuilder.Entity<MultiTenantUser>()
        .HasQueryFilter(x => x.ShopKey == ShopKey);
    modelBuilder.Entity<Shop>()
        .HasQueryFilter(x => x.ShopKey == ShopKey);
    modelBuilder.Entity<StockInfo>()
        .HasQueryFilter(x => x.ShopKey == ShopKey);
}

NOTE: You need to think hard about the query inside the Query Filters, as they are called on every database read of that emtity. Make sure they are as optimised as possible and add indexes to the protect key column in the database for in each protected entity class.

The DbContext parts can be found at:

MultiTenantDbContext class with the injection of the ShopKey and the overridden SaveChanges
MarkCreatedItemWithShopKey extension method which adds the ShopKey to any newly created Shop-related entity class (row).

Example 3 (advanced): Handling hierarchical accesses

One of my clients needed multi-tenant system that handled the idea of a group manager, district managers etc. that can get data for reports from a series of shops. This means that the filter rules change depending on the how high the user is in the hierarchy.

I have extended my multi-tenant example such that shops can have district managers. This means:

A user linked to a shop, say Dress4U, can only access the information about that shop.
But district managers can access the group of shops, say Dress4U, Shirt4U and Tie4U, can see information for all the shops they manage.

This requires us to add four new parts to the existing multi-tenant shop inventory application. They are:

The MultiTenantUser class must change to tell us if the user is a shop worker or a district manager.
When a user logs in I need to see if they are a district manager, and if they are I need to add a DistrictManagerId claim to the ClaimsPrincipal.
The Shop and StockInfo need a reference to an (optional) district manager.
The query filters now need to change depending on whether the current user is a shop-only user or a district manager.

I’m not going to show you the code for all the steps in detail as this article is already very long. What I want to focus on is the Query Filters.

In ASP.NET Core if you ask for an instance of the application’s DbContext, then a new one is created per HTTP request (a scoped lifetime). But, for performance reasons EF Core builds the configuration on first use and caches it. This means you can’t change Query Filters using an If-then-else command – instead you can use the ?: operators in the Query Filter’s LINQ expression.

So, for the multi-tenant application with district manager support the OnModelCreating method looks like this.

 protected override void OnModelCreating(
    ModelBuilder modelBuilder)
{
    //Altered query filter to handle hierarchical access
    modelBuilder.Entity<Shop>().HasQueryFilter(
        x => DistrictManagerId == null
            ? x.ShopKey == ShopKey
            : x.DistrictManagerId == DistrictManagerId);
    modelBuilder.Entity<StockInfo>().HasQueryFilter(
        x => DistrictManagerId == null 
            ? x.ShopKey == ShopKey
            : x.DistrictManagerId == DistrictManagerId);

    //… other query filters removed for clarity
}

Note that the DistrictManagerId is injected into the DbContext using the same system and for the ShopKey. Have a look in the DataLayer for the multi-tenant entity classes and the multi-tenant DbContext.

Conclusion

Another long article, but I have given you a lot of details on how to implement a per-row data protection system. As I said at the beginning feature authorisation (covered in Part1) is much more common than per-row data protection. But if you need per-row data protection then this article shows how you can implement it.

I started with an example of using the user’s unique id to control access to personal data. That also introduced all the parts needed for per-row data protection. I then went onto two more complex per-row protection usages: a multi-tenant system and a hierarchical system. The hierarchical system is complex, but one of my clients needed that (and more!), so it is used.

Security is a very important topic, and one that can be tough to implement with no security loopholes. That is why my implementation places all of the per-row data protection is inside the application’s DbContext. That reduces duplication of code and makes sure per-row data protection is on by default. Personally, I also add unit tests that check I haven’t forgotten to add the query filters and any protection interfaces to the entity classes – you can never be too careful on security.

Happy coding.

Don’t forget to sign up to Jerrie Pelser’s, ASP.NET Weekly newsletter if you are interested in ASP.NET or Entity Framework. It is the most important newsletter I get.

There is a concept in Domain-Drive Design (DDD) called the anti-corruption layer which, according to Microsoft explanation of an anti-corruption layer “Implements a façade or adapter layer between different subsystems that don’t share the same semantics”. DDD uses anti-corruption layers between subsystems (known in DDD as a bounded contexts), say inventory, orders, fulfilment/delivery in an e-commerce system.

But I think the anti-corruption layer concept can also help us inside even small applications. In fact, I expect many of you are already the anti-corruption layer concept in your application, but you call them “adapters”, or “interfaces”, or “ViewModels”.

In this article I’m going to explore the application of the anti-corruption layer not to big subsystems, but to individual NET assemblies in even a small web application, e.g. between the database, the business logic, and the front-end. I hope to show how to implement anti-corruption layers and why they will make your applications easier to write and maintain.

This article was inspired by a problem I encountered in the anti-corruption feature in my EfCore.GenericBizRunner library. I had configured the DTOs (data transfer objects) with the AutoMapper library (which I use for class-to-class mapping) in such a way that some AutoMapper config properties were turning up in my output classes/DTOs. That can cause a bit of a problem when using DTOs in Web APIs, so I fixed it in version 3 (out now).

Other articles about business logic and GenericBizRunner:

Architecture of Business Layer working with Entity Framework (Core and v6)
A library to run your business logic when using Entity Framework Core.
How to write good, testable ASP.NET Core Web API code quickly.
Wrapping your business logic with anti-corruption layers – NET Core (this article)

The examples I am using are based around ASP.NET Core web applications and Entity Framework Core (EF Core) for database accesses.

TL;DR; – summary

DDD defines an anti-corruption layer is an adapter pattern that isolates one part of a system, known in DDD as a bounded context, from another bounded context. Its job is to ensure that the semantics in one bounded concept do not “corrupt” the other bounded concept’s semantics.
I think that the anti-corruption layer concept can also be used between assemblies even in small application. Some examples of using an anti-corruption layer in a web application are:
- Business logic: Your business logic shouldn’t have to concern itself about how the database or the front-end works.
- Front-end: Human user’s need the data that suits them, not how the database was built.
- Web API: Your Web API must provide a service that fits the external needs.
Anti-corruption layers can help with security too, by only passing the exact data that the other area needs.

Business logic – the classic area for anti-corruption layers

In the book Domain-Driven Design, Eric Evan that “Domain Model” (what I call “business logic”) is “the heart of the Software” (see Eric Evans book, page 4). Eric goes on to say “When the domain is complex, this is a difficult task, calling for the concentrated effort of talented ad skilled people”.

I wrote an article called “Architecture of Business Layer working with Entity Framework” where I go through a design for handling business logic. That article is worth a read, but in this article I focus on the anti-corruption layers as applied to my business logic.

From Eric Evan’s book, and my own experience, I want to have no distractions when I’m writing my business logic. That translates into the following “don’t wants”:

I don’t want to worry about how the database works.
I don’t want to worry about how my business logic communicates to a user.
I don’t want to know I am using an external service – I just want methods to call.

How I achieve this isolation differs for each part: in this case I use a repository pattern for the database side and my GenericBizRunner library to talk to the front-end. I will describe these two anti-corruption layer implementations later, but first here is a diagram give you an overview of my approach.

I’m now going to describe each of these two anti-corruption layers, i.e.

The database to business logic anti-corruption layer.
The business logic to front-end anti-corruption layer.

1. Implementing the database to business logic anti-corruption layer

Eric Evans talks in his book about separating the business logic from the database and EF Core’s navigational properties makes this possible. The navigational properties allow the business logic to work with normal (POCO) classes that have properties that references other classes. That means the business logic doesn’t have to know about primary keys, foreign keys, etc.

Let’s look at a specific piece of business logic, placing an order for some books, to show this in action. The user selects a series of books (front-end lineItems) and the business logic creates an Order containing database LineItems. To do this my business logic, called PalceOrder, has to take in the front-end lineitems and create a new Order class and adds a series of database LineItem classes to it.

That’s fine, but it does need to access the database for two things:

The front-end lineItems contain the primary key of each book, but the database LineItem class needs the Book entity class to create it. This needs a database lookup.
The PlaceOrder code creates a new Order class, but somehow that class needs to be added to the database.

My solution is to use a repository pattern to provide the business logic with methods to handle these two database requirements. Here is the code, which the business logic accesses via constructor injection via the IPlaceOrderDbAccess interface.

public class PlaceOrderDbAccess : IPlaceOrderDbAccess
{
    private readonly EfCoreContext _context;

    public PlaceOrderDbAccess(EfCoreContext context)
    {
        _context = context;
    }

    public LineItem BuildLineItem(int bookId, byte numBooks)
    {
        return new LineItem(bookId, _context.Find<Book>(bookId), numBooks);
    }

    public void Add(Order newOrder)                 
    {                                               
        _context.Orders.Add(newOrder);              
    }                                               
}

This repository is small because it only implements database access methods for just the PlaceOrder business logic. A typical repository has tens or hundreds of methods to be used all over the application, but I don’t find that that approach very useful. I have applied the single responsibility principle to the repository pattern and come up with a per-business logic repository, which I call a “mini-repository”.

The benefit of the mini-repository pattern is that the code is much simpler to write and maintain. It also means later changes in architecture, such as splitting a monolith architecture up into microservices or serverless, is easier to do as the business logic and the associated mini-repository can move together.

NOTE: I haven’t talked about saving the data to the database – in EF Core that means calling the method called SaveChanges. I could have included that in my mini-repository (most likely in the Add method), but my GenericBizRunner library handles calling the SaveChanges method. That allows the library to add validation of the entities to be saved, which I think is important for business logic.

2. Implementing the business logic to front-end anti-corruption layer

The business logic and the front-end have a common goal: to place an order for books. But both parts have their own concepts – the business logic is about applying a business rule while the front-end wants to provide an easy-to-use user interface. So, in fact the anti-corruption cuts both ways – stopping concepts/data that are useful in one assembly from leaking into the other assembly.

Here is a diagram that shows the various NET assemblies and what happens in each stage of the process of running some business logic that accesses the database. It’s quite detailed, but it conveys in one picture all the main things that are going on. I include a commentary on what my GenericBizRunner library is doing – you could of course write your own code to handle these jobs.

NOTE: The Service Layer is an assembly that I use for adapting data from lower layers to the front-end. I got this concept from Dino Esposito (who got it from Martin Fowler). You can get more information in the section on the Service Layer in my article “Six ways to build better Entity Framework (Core and EF6) applications”.

2a. front-end to business logic anti-corruption layer

The front-end might need all sorts of data to show a page to the user, but the business logic only wants the list of books with their unique IDs (primary key in this case). In ASP.NET the HTML page often sends back extra data in case there is an error and the needs to be redisplay. But the business logic doesn’t need that data, and in some case that data is in NET Type that the business logic doesn’t even know about, i.e. a SelectList type. How do we handle that?

The way I handle that is to use AutoMapper in my library to copy the front-end ViewModel/DTO into the input class that my business logic needs. Here is an example from another piece of business logic to change the delivery date of an order. The diagram below displays the front-end focused DTO and the business focused input DTO.

You can see that the front-end needs extra information to show the user what order they are changing (left-hand side of diagram) while on the right-hand side the business logic only needs the first three properties. I (or to be more correct my GenericBizRunner library) uses AutoMapper to copy the data across before the business logic is called.

I think this is a very good example of an anti-corruption process going on. The front-end has extra properties to display a meaningful page to the user, but the business logic doesn’t even know about these front-end only properties. Also, the anti-corruption layer means that the business layer doesn’t have to know about or support front-end types such as SelectList etc.

My GenericBizRunner has a few features to help in a case like this. It provides the front-end DTO with a method called SetupSecondaryData, which can be run to fill in extra data needed for the display to the user – in this case filling the extra properties to show or re-show the display. Here is the code in the ASP.NET Core MVC controller to display and ChangeDelivery page and called the business logic process the user’s request.

public IActionResult ChangeDelivery(int id, 
    [FromServices]IActionService<IChangeDeliverAction> service)
{
    //This runs SetupSecondaryData to fill in dropdownlist etc.
    var dto = service.GetDto<WebChangeDeliveryDto>(x =>
    {
        x.OrderId = id;
        x.UserId = GetUserId(HttpContext);
    });
    return View(dto);
}

[HttpPost]
[ValidateAntiForgeryToken]
public IActionResult ChangeDelivery(WebChangeDeliveryDto dto,
    [FromServices]IActionService<IChangeDeliverAction> service)
{
    if (!ModelState.IsValid)
    {
        //Rerun SetupSecondaryData if need to redisplay
        service.ResetDto(dto); 
        return View(dto);
    }

    service.RunBizAction(dto);

    if (!service.Status.HasErrors)
    {
        return RedirectToAction("ConfirmOrder", "Orders", 
            new { dto.OrderId, message = service.Status.Message });
    }

    //Otherwise copy errors in and redisplay the page 
    service.Status.CopyErrorsToModelState(ModelState, dto);
    service.ResetDto(dto);
    return View(dto); //redisplay the page, with the errors
}

You can see on lines 2 and 16 that I inject an instance of the GenericBizRunner, with the specific business logic, referred to via the interface IChangeDeliverAction, as a parameter to the action. This is more efficient than using the normal constructor DI injection as we only inject the exact service we need for each action method.

2b. business logic to front-end anti-corruption layer

Sometimes its also useful to not send everything that the business logic sends back. In the PlaceOrder example the business logic works with classes and (in theory) shouldn’t need to think what the front-end needs. Therefore, the PlaceOrder business logic will return the Order class.

The problem is that Order class might have all sorts of data that is private, like information on the payment, but the front-end only wants to have the primary key of the Order so that it can display an “order success” page. The solution is the same as the input: only copy over the properties that the front-end needs – in this case only copy the OrderId from the Order class that the business logic returns, but only after SaveChanges has been called and the primary key is available.

To make this work I need two things:

An OrderIdDto class with just one property in it called OrderId, which should hold the primary key of the created Order. This is the only bit of data that my front-end needs.
Once SaveChanges has been called I copy over the OrderId property in the Order into the OrderIdDto class’s OrderId property.

Here is the controller action that calls PlaceOrder. Note that the user’s basket of items is held in a cookie, so you will see some code to get and clear the checkout cookie.

[HttpPost]
[ValidateAntiForgeryToken]
public IActionResult PlaceOrder(PlaceOrderInDto dto, 
    [FromServices]IActionService<IPlaceOrderAction> service)
{    
    if (!ModelState.IsValid)
    {
        //model errors so return to checkout page, showing the basket
        return View("Index", FormCheckoutDtoFromCookie(HttpContext));
    }

    //This runs the PlaceOrder business logic
    var orderDto = service.RunBizAction<OrderIdDto>(dto);

    if (!service.Status.HasErrors)
    {
        //If successful I need to clear the checkout cookie
        ClearCheckoutCookie(HttpContext);
        return RedirectToAction("ConfirmOrder", "Orders", 
            new { orderDto.OrderId, Message = "Your order is confirmed" });
    }

    //There were errors so redisplay the basket from the cookie
    var checkoutDto = FormCheckoutDtoFromCookie(HttpContext);      
    //This copies the errors to the ModelState
    service.Status.CopyErrorsToModelState(ModelState, checkoutDto);
    return View("Index", checkoutDto);
}

You can see the GenericBizRunner’s call the PlaceOrder’s method on line 13, via the service injected as a parameter. In that call I tell it that the return type should be OrderIdDto, which I have created as a DTO type that GenericBizRunner knows needs to be mapped from the Order class. This means only primary key of the order ends up in the front-end and all the other properties in the Order class is discarded. The front-end can then pass on this primary key to the ConfirmOrder page which knows how to display a successful order.

GenericBizRunner and Web APIs

As I said at the start I found a problem that placed unwanted properties in the DTOs. To me it matters a lot that I get the data returned from the business logic in the exact format needed by the Web API. This really mattered on one of my client’s project which used Swagger and NSwagStudio to create Angular 6 code to match the Web API. NSwagStudio was amazingly useful, as it automates the creation of the Angular code to interface to the Web APIs, BUT you must get exactly the return type to make it work property.

Now, is this an anti-corruption feature, or just the normal adaption that we are already used to? I think adaption of data is what is mainly going on, but thinking “what anti-corruption issues are there?” is useful. Here are my thoughts:

Anti-corruption says we need to only pass the data that is required by the Web API definition.
Anti-corruption makes us thing about the data we do pass: is it in the correct form, are the names of the objects right for the Web API, etc.

A look at GenericBizRunner in a controller

With the help of my companion EfCore.GenericServices.AspNetCore library using GenericBizRunner in Web API is pretty simple. Here is a simple example taken from the EfCore.GenericServices.AspNetCore’s ExampleWebAPI. It uses some business logic to create a TodoItem.

[ProducesResponseType(typeof (TodoItem), 201)]
[HttpPost]
public ActionResult<TodoItem> Post(CreateTodoDto item, 
    [FromServices]IActionService<ICreateTodoBizLogic> service)
{
    var result = service.RunBizAction<TodoItem>(item);
    return service.Status.Response(this, 
        "GetSingleTodo", new { id = result?.Id }, result);
}

Let’s look at each line of this code

Line 1: I am using Swagger, and I want to tell Swagger a) the type of result it provides, and b) the status code on success.
Line 3: It pass in a CreateTodoDto, which contains the properties that the business logic needs to create the TodoItem.
Line 4: This is where I inject the BizRunner instance, linked via the interface to the business logic to run.
Line 6: I run the business logic, which returns the created TodoItem instance.
Lines 7 and 8: The Response method is from the EfCore.GenericServices.AspNetCore library. This form returns a “CreatedAsRoute” result, which fits the definition of the HTTP 201 create response.

In this case I have returned the created class, but I have found in real applications you really don’t want to do that. It turns out that EF Core’ entity classes very often have navigational properties. These are properties that link to other entity classes in the database, e.g. a Book has a collection of Reviews.

The problem is that many SPA’s (single page applications, like Angular and React.js) only want the non-navigational data. In these cases, we need to use DTOs that remove those navigational properties and/or applying some form of transformation to the data when sending database classes to the Web API. Thankfully the GenericBizRunner can do that.

Conclusion

Well, I hope I have made you think about the anti-corruption concept and whether it would improve your business logic. You are most likely doing something similar with adapter patterns and ViewModels/DTO, but you might not call it an anti-corruption layer. That’s fine, but thinking “anti-corruption” can help you implement a good pattern for your business logic.

The down-side of the approach I have described is it does create more code. For small bit of business logic then it might feel like overkill, but when the business logic gets more complicated then I think some separation of concerns pattern is essential. The pattern I have described is one way, but there are plenty of other approaches that achieve the same result.

I created the EfCore.GenericBizRunner library to reduce the amount of extra code I must write, especially in the front-end where it makes it very easy to call. Also, having a standard pattern for my business logic helps me write my business code quickly. I do have to write the mini-repository as well, but that database access code had to go somewhere and my mini-repository per business logic makes it easy to change or performance tune. So far this approach and the GenericBizRunner library has served me well.

Happy coding.

Andrew Lock wrote an excellent series called “Running async tasks on app startup in ASP.NET Core” in which he used “migrating a database” as an example of something you could do on startup. In part 3 of the series he covered why migrating a database on startup isn’t always the best choice. I decided to write a series about the different ways you can safely migrate a database, i.e. change the database’s schema, using Entity Framework Core (EF Core).

This is the first part of the series, which looks how to create a migration, while part 2 deals how to apply a migration to a database (coming soon), specifically a production database. I have used a similar approach to Andrew’s when writing this article, i.e. I try to cover all the way to create a migration script when working with EF Core with their pros, cons and limitations.

NOTE: Andrew and I know each other as we were both writing books for Manning Publications at the same time: Andrew’s book is “ASP.NET Core in Action” and mine is “Entity Framework Core in Action”. We shared the toils and joys of being an author, but Andrew has the harder job with ASP.NET Core – his book is 700 pages long and mine is “only” 500 pages long.

TL;DR – summary of creating a migration

NOTE: Click the links to go directly to the section covering that point.

There are two types of migrations that can be applied to a database:
- Adds new tables, columns etc, known as a non-breaking change (easy).
- Changes columns/tables and needs to copy data, known as a breaking change (hard).
There are five ways to create a migration in EF Core
- Use EF Core to create a migration – easy, but doesn’t handle all possible migrations.
- Use EF Core to create a migration and then hand-modify the migration – medium to hard, but handles all possible migrations.
- Use a third-partly migration builder to write a C# migration – hard, as you need to write the migration yourself, but you don’t need to know SQL.
- Use a SQL database comparison tool to compare databases and output a SQL changes script – easy, but you do need some understanding of SQL.
- Write your own SQL migration scripts by copying EF Core’s SQL – hard-ish, gives great control but you do need to understand SQL.
How to be sure your migration is valid – use CompareEfSql tool.

Setting the scene – what questions should we ask about creating a migration?

There are lots of ways to migrate a database’s schema, and in development you can use almost any approach. But when it comes to migrating a production database (i.e. the one that is being used by real users) then it becomes very serious. Getting it wrong will, at the minimum, inconvenience your users and at worse mangle or lose you (precious) data in your database!

But before we get the scary part of updating a database schema we need to build the migration script that will contain the schema, and possibly data, changes. To build the appropriate migration script we need to ask ourselves some important questions about the type of changes we need to apply to the database. Is the required migrations going to be:

A non-breaking change, i.e. it only adds new columns, tables, etc., which could be applied while the old software was still running, i.e. the old software would work (not break) with the database after the migration.
A breaking change, i.e. some data must be copies or transformed during the migration, could NOT be applied while the old software, i.e. the old software would encounter errors (break) with the database after the migration.

The other part in this article is that we are working with EF Core, which brings both some benefits and constraints. The benefits are that EF Core can, in most cases, create the required migrations automatically. The constraints are that the database after a migration has been applied must match the software Model of the database that EF Core builds by looking at your DbContext and mapped classes – I refer to EF Core’s Model with a capital M, because there is a property called Model in the DbContext that contains the complete mappings between classes and the database.

NOTE: I’m going to cover migrations where you are in control of the control of the classes mapped to the database and the EF Core configuration – sometimes known as the Code-First Approach. The other alternative, which I’m NOT going to cover, is where you directly control the database and you use an EF Core command called scaffolding to create the entity classes and EF Core configuration for you. That’s because migrations are trivial – just re-scaffold your database.

PART1. The five ways to create a migration script

As I said in the last section, any migration script we create must migrate the database to a state that matches EF Core Model. For instance, if a migration added a new column to a table then the entity class mapped to that table must have a property to match that new column. If EF Core’s Model of the database schema does match the database, then you can get errors occurring in queries or writes. I refer to the migration script as creating a “valid” database if it matches EF Core’s Model of that database.

The validity of a migrations created by EF Core isn’t in doubt – EF Core created it so it will be valid. But if we need to edit a migration, or we take on the job of building the migration ourselves, then we need to be very careful that the migration creates a “valid” database as far as EF Core is concerned. This is something I have thought about a lot.

This is the list of ways you can create a migration script.

Creating C# migration script
1. Standard EF Core migration script: use EF Core’s Add-Migration command to create a C# migration script.
2. Hand-modified EF Core migration script: Use EF Core’s Add-Migration command to create a C# migration script and then hand-edit it to add the bits that EF Core missed.
3. Use a third-partly migration builder, e.g. FluentMigrator. Tools like this allow you to write your own migration script in C#.
Creating SQL migration script.
1. Use a SQL database comparison tool. This compares the last database schema with the schema of a new database created by EF Core and produces a SQL script that will migrate the old database to the new database schema.
2. Write your own SQL migration scripts. A competent writer of SQL could produce a SQL migration scripts, with some help by capturing the SQL EF Core would use to create a database.

Here is a summary diagram to give you an overall review of the five approaches, with my personal view on the ease of use and level of limitations.

Now let’s look at each one of these in turn.

1a. Standard EF Core C# migration script

This is the standard migration technique provided by EF Core. It’s well documented by Microsoft, but in summary you run a command called Add-Migration which adds three C# files to your application that contain the changes needed to migrate an existing database using migraions to match the current EF Core setup/configuration.

Good parts	· Builds a migration automatically · You don’t need to learn SQL · Includes a revert migration feature
Bad parts
Limitations	· Standard migration cannot handle breaking changes (but see 1b). · No handling of SQL features, e.g. SQL user defined functions (but see 1b).
Tips	· Watch out for error messages when you run the Add-Migration method. If EF Core detects a change that could lose data it will output an error message, but still creates the migration files. YOU MUST alter the migration script otherwise you will lose data – see section 1b. · If your DbContext is in another assembly from where the DbContext is registered you need the MigrationsAssembly method on your build and you most likely you need to implement a IDesignTimeDbContextFactory in the DbContext assembly.
My verdict	This is a really easy way to handle migrations and it works well in many cases. The problem is, what happens if the migration doesn’t cover what you want. Thankfully there are many ways to handle that.

Reference: Microsoft’s documentation on creating a migration.

1b. Hand-modified EF Core C# migration script

The nice thing about EF Core’s Add-Migration command is it creates the C# migration files as the starting point, but you can then edit these files yourself to add code to handle breaking changes or add/update SQL parts of the database. Microsoft give an example of handling a breaking change with copying data.

Good parts	Same as standard migration + · Ability to customise the migration. · Ability to include SQL features, e.g. SQL user defined functions.
Bad parts	· You need to understand what is happing in the database. · Can be difficult to decide on how to edit the file, e.g. do you keep everything EF Core added and then alter it, or remove EF Core parts and do it yourself?
Limitations	No simple way to check the migration is correct (but see CompareEfSql later).
Tips	Same as standard migration.
My verdict	Great for small alterations, but big changes can be hard work as you are often mixing C# commands with SQL. That is one reason why I don’t use EF Core migrations.

Reference: Microsoft’s example of hand-modifying a migration.

1c. Use a third-partly C# migration builder

It was Andrew Lock who pointed out to me an approach using the FluentMigrator (see this documentation) to write your migrations. This works similarly to EF migrations, but you have to do all the hard work of detailing the changes. The good thing is FluentMigrator’s commands are very obvious.

Good parts	· Don’t need to learn SQL. · Very obvious what the changes are, i.e. “code as documentation”.
Bad parts	· You must work out what the changes are yourself. · Not guaranteed to produce a “correct” migration (but see CompareEfSql later).
Limitations	– none –
Tips	Note that FluentMigrator has a “Migration Runners” which can apply the update to the database, but it can also output SQL scripts instead.
My verdict	No real experience myself. It feels like it is a clearer syntax that EF Core’s migration, but you have to do all the work yourself.

Reference: FluentMigrator of GitHub.

2a. Use SQL database comparison tool

There are free and commercial tools that will compare two databases and create a SQL change script that will migrate the old database schema to the new database schema.

A “free” comparison tool is built into Visual Studio 2017 (all versions) called SQL Server Object Explorer, under the View tab. If you right-click on a database, then you can access the “Compare Schema” tool (see figure above right) which can produce a SQL change script.

The SQL Server Object Explorer tool is very good, but there isn’t much documentation on this (pity). Other commercial systems include Redgate’s SQL Compare.

Good parts	Builds the correct SQL migration script for you.
Bad parts	· You need a little bit of understanding of databases. · No all SQL compare tools produce a revert script.
Limitations	Does not handle breaking changes – needs human input.
Tips	Watch out for SQL compare tools that outputs every setting under the sun to make sure it gets it right. EF Core’s migrations are straightforward, like “CREATE TABLE…”, so should do that same. If you have any specific settings, then build them into the database create.
My verdict	I have used SQL Server Object Explorer on a big migration that was hard to hand-code. Very useful and especially good for people who aren’t comfortable with the SQL language.

2b. Hand-coding SQL migration scripts

This sounds really hard – writing your own SQL migrations, but there is plenty of help on hand, either from SQL compare tools (see above) or by looking at the SQL EF Core would use to create the database. This means I have SQL I can look at and copy to build my SQL migration script.

Good parts	Total control over the database structure, including parts that EF Core won’t add, e.g. user defined functions, column constraints, etc.
Bad parts	· You must understand basic SQL like CREATE TABLE etc. · You must work out what the changes are yourself (but there is help) · No automatic revert migration. · Not guaranteed to produce a “correct” migration (but see CompareEfSql later).
Limitations	– none –
Tips	· I use a unit test that captures the logging output of EF Core’s EnsureCreated method. That gets me the actual SQL EF Core would output. I then look for the differences for the last database. That makes writing the SQL migrations much easier. · I unit test a migration by creating a database by applying all the migrations, including the new migration, and then running CompareEfSql to check the database matches EF Core’s current Model of the database.
My verdict	This is what I use, with a little help from a CompareEfSql tool. If EF Core’s migration feature is so good, why do I go to all this trouble? Here are my reasons: · Total control over the database structure, including parts that EF Core won’t add, e.g. user defined functions, column constraints, etc. · Because I am writing the SQL it makes me think about the database aspects of the change – should that property be nullable? do I need an index? etc. · Handing breaking changes by hand-modifying EF Core’s migration system isn’t that easy. I might as well stick with SQL migrations. This is for developers that wants total control and visibility of the migrations.

You can capture the SQL output by EF Core’s for creating a database but capturing the EF Core’s logging while calling the method EnsureCreated (the EnsureCreated method is meant for creating unit test database). Because setting up logging for EF Core is a little bit complex I added helper methods to my EfCore.TestSupport library to handle that. Here is an example unit test that creates a new SQL database and catches the SQL commands that EF Core produces.

[RunnableInDebugOnly]
public void CaptureSqlEfCoreCreatesDatabaseToConsole()
{
    //SETUP
    var options = this.CreateUniqueClassOptionsWithLogging<BookContext>(
        log => _output.WriteLine(log.Message));
    using (var context = new BookContext(options))
    {

        //ATTEMPT
        context.Database.EnsureDeleted();
        context.Database.EnsureCreated();
    }
}

Let’s look at each line of this code

Line 5. This is an EfCore.TestSupport method that creates the options for your DbContext. This version uses a database name that includes the class name. I do this because xUnit test classes are run in parallel, so I want a unique database for this unit test class.
Line 6. I use the version of the option builder ending with …WithLogging, which allows me to capture the log outputs. In this case I output the Message part of the log directly to the unit test output window.
Lines 11 & 12. First, I ensure the database is deleted so that when I call EnsureCreated a fresh database will be created with a schema defined by the current DbContext’s configuration and mapped classes.

Below is part of the output captured in the unit test output. This provides you with the exact SQL that EF Core would use to create the whole schema. You do need to extract just the parts that relate to your migration, but at least you can cut-and-paste the parts you need into your SQL migration script.

CREATE DATABASE [EfCore.TestSupport-Test_TestEfLogging];
Executed DbCommand (52ms) [Parameters=[], CommandType='Text', CommandTimeout='60']
IF SERVERPROPERTY('EngineEdition') <> 5
BEGIN
    ALTER DATABASE [EfCore.TestSupport-Test_TestEfLogging] SET READ_COMMITTED_SNAPSHOT ON;
END;
Executed DbCommand (5ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
CREATE TABLE [Authors] (
    [AuthorId] int NOT NULL IDENTITY,
    [Name] nvarchar(100) NOT NULL,
    CONSTRAINT [PK_Authors] PRIMARY KEY ([AuthorId])
);
Executed DbCommand (1ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
CREATE TABLE [Books] (
    [BookId] int NOT NULL IDENTITY,
    [Title] nvarchar(256) NOT NULL,
-- rest of SQL left out

How to be sure your migration is valid – use CompareEfSql tool

I have mentioned CompareEfSql a few times in the descriptions of creating migrations. This tool compares a database with the model of the database that EF Core creates on first use for a DbContext. This model, accessed via the Model property in the DbContext instance, is built up my EF Core by looking at the DbContext configurations and DbSet and DbQuery properties.

This allows a developer to test an existing database against EF Core Model and gives you error messages if they are different. I find this a very powerful tool which allows me to hand-code SQL migrations and be sure that they are correct (there are some small limitations). Here is an example unit test that will fail if the database schema doesn’t match EF Core’s Model.

[Fact]
public void CompareViaContext()
{
    //SETUP
    var options = … options that point to the database to check;
    using (var context = new BookContext(options))
    {
        var comparer = new CompareEfSql();

        //ATTEMPT
        //This will compare EF Core model of the database 
        //with the database that the context's connection points to
        var hasErrors = comparer.CompareEfWithDb(context);

        //VERIFY
        //The CompareEfWithDb method returns true if there were errors.
        //The comparer.GetAllErrors property returns a string
        //where each error is on a separate line
        hasErrors.ShouldBeFalse(comparer.GetAllErrors);
    }
}

I love this tool, which is in my EFCore.TestSupport open-source library. It allows me to build migrations and be sure they are going to work. I also run it as a normal unit test and it tells me immediately if I, or another team mate, has changed EF Core’s setup.

You can get a much longer description of this tool in the article called EF Core: taking full control of the database schema and its many features and configurations can be found in the CompareEfSql documentation pages.

NOTE: I build a version of this originally for EF6.x (see this old article), but it was limited because EF6.x didn’t fully expose its internal model. With EF Core I could do so much more (EF Core rocks!) and now I can check almost everything, and because I tap into EF Core’s scaffolder service it works for any database that EF Core supports (see this doc).

Conclusion – Part 1

This part of the series covers creating a valid migration, while part 2 (coming soon) deals with applying a migration to a database. This article lists all the applicable approaches to creating a database migration when working with EF Core – with their pros, cons and limitations. And as you can see EF Core’s Add-Migration command is really good, but it doesn’t cover every situation.

Its up to you to decide what types of migrations you might encounter, and what level of control do you want over the database schema. If you can get away with just using EF Core’s standard migrations (1a) then that makes life easier for you. But if you expect breaking-changes, or you need to set up extra SQL features then you now know the options available to you.

The scary part comes in the part2 – applying a migration to a production database. Changing a database which contains business-critical data needs (demands!) careful planning and testing. You need to think about what you are going to do if (when!) a migration fails with an error.

The original reason I moved away from EF migrations in EF6 was its automatic migration on startup was working fine, until it threw an error on deployment! It was really hard to track down an error in the migration – that alone got me moving away from using EF migrations (read this old article on my thoughts back then).

EF Core’s migration handling is better than EF6: automatic migrations have (thankfully!) gone and EF Core migration are more git-merge friendly, to mention just two changes. But the way I build SQL migration scripts makes me think much more carefully about what I am doing than just running Add-Migration. EF Core is a really great O/RM but it does hide the database too well sometime. Creating SQL migration scripts makes me think through a migration from the database point of view, and I often spot little tweaks to the database, and the C# code, to make the database better or more robust.

See you in part2 (coming soon).

This is the second in the series on migrating a database using Entity Framework Core (EF Core). This article looks at applying a migration to a database and follows on from part 1 which covered how to create a migration script. If you haven’t read the part 1 some parts of this article won’t make sense, so here is a very quick review of part 1.

There are two types of migrations that can be applied to a database:
- Adds new tables, columns etc, known as a non-breaking change (easy).
- Changes columns/tables and needs to copy data, known as a breaking change (hard).
There are two main ways to apply a migration to a database
- Use EF Core migration feature
- Use EF Core to create a migration and then hand-modify the migration.
- Use a third-partly migration builder write in C# the migration.
- Use a SQL database comparison tool to compare databases and output a SQL changes script.
- Write your own SQL migration scripts by copying EF Core’s SQL.

So, now that you know how to create migration scripts now I’m going to look at the different ways you can apply a migration to a production database, with all the pros, cons and limitations.

TL;DR – summary of the content

NOTE: Click the links to go directly to the section covering that point.

The type of application you have affects the migration approach can use.
You must think about what errors could occur and have a plan.
There are four ways to apply a migration to a database
- Calling context.Database.Migrate() on startup – easy, but has some significant problems that limits its usefulness.
- Calling context.Database.Migrate() via a console app – easy, and works well, especially in a deployment pipeline
- Outputting EF Cores migration as a SQL script and execute that script on the target database – hard, but gives you great control.
- Use a database migration application tool to apply your own SQL scripts – hard, but gives you great control.
Three different levels of applying a migration.

Setting the scene – What sort of application have you got?

In part 1 we were focused on creating migrations that were “valid” and whether the migration is a non-breaking change or breaking change (see quick definition at start of this article, or this link for section in part 1).

Now we are looking at applying a migration to a database, but the options we have depends on the application (or applications) that are accessing the database. Here are questions you need to think about.

Is there only one application accessing that database, or is your application a web app which is scaled-out, i.e. there are multiple versions of your application running at the same time. If your application is scaled-out, then this removes one of the options.
Can you stop your application while you apply a migration to the database, or is your application providing a continuous (24/7) service? Updating continuous service applications bring some challenges when it comes to applying a breaking change.

When it comes to migrating a production database being a bit paranoid is OK.

As I said at the end of part 1 – the scary part comes when you apply a migration to a production database. Changing a database which contains business-critical data needs (demands!) careful planning and testing. You need to think about what you are going to do if (when!) a migration fails with an error.

When considering the different ways to apply a migration you should have in the back of your mind “what happens if there is an error?”. This might push you to a more complex migration approach because its easier to test or revert. I can’t give you rules or suggestions as each system is different but being a bit paranoid about failures isn’t a bad thing to have. I should make you build a system for migrating your application and its database that is more robust.

PART2: How to apply a migration to a database.

The list below gives the different ways you can apply a migration to a database. I list three options for the EF Core case: the first being the simplest, but it has limitations which the other two options don’t have. The SQL migration has no real limitations, but it does need a database migration application tool to apply the SQL scripts only once and in the right order.

Here is the list of ways you can apply a migration.

EF Core migration
1. Calling context.Database.Migrate() on startup
2. Calling context.Database.Migrate() via a console app or admin command
3. Outputting the migration as a SQL script and execute that script on the target database.
SQL migrations
1. Use a database migration application tool.

In the end, how you apply your migration depends on the type of migration (breaking or non-breaking) and the type of application you are updating (single app, multiple apps running in parallel or an app that mustn’t stop). Here is a diagram to try and convey all these permutations.

The outer dark blue shows that SQL migrations can be applied in all cases, then the lighter, inner boxes show where different types of EF Core migrations can be added. Here are some clarifying notes about the diagram:

The diagram shows standard EF migration and hand-modified EF migration, but when I am talking about applying the migration then there is no distinction between the two – we are simple applying an EF Core migration.
The “five-stage app update” red box in the diagram represents the complex set of stages you need to apply a breaking change to a application that cannot be stopped. I cover that at the end of the article.

Now I will go through each of the ways of applying a migration in detail.

1a. Calling context.Database.Migrate() on startup

This is by far the easiest way to apply a migration, but it has a big limitation – you should not run multiple instances of the Migrate method at the same time. That can happen if you scale-out a web application. To quote Andrew Lock, “It’s not guaranteed that this will cause you problems, but unless you’re extremely careful about ensuring idempotent updates and error-handling, you’re likely to get into a pickle” – see this section of his post “Running async tasks on app startup in ASP.NET Core”.

Good parts	· It is relatively easy to implement (see tips) · It ensures the database is up to date before your application runs.
Bad parts	· You must NOT run two or more Migrate methods in parallel. · If the migration has an error, then your application won’t be available. · It’s hard to diagnose startup errors
Limitations	Does not work with continuous service systems
Tips	I quite like this option in Andrew Lock’s article for running a migration on startup. I use a similar approach in some of my demo systems that use in-memory databases that need initializing (see this example)
My verdict	If you are running a single web app or similar and you can update the system when no one is using it then this might work for you. I don’t use this as many of my systems I work on use scale-out.

1b. Calling context.Database.Migrate() via a console app or admin command

If you can’t run multiple Migrate methods in parallel, then one way to ensure this is to call the Migrate method inside a standalone application designed to just execute the Migrate method. You might add a console application project to your main web app solution which has access to the DbContext and can call Migrate. You can either run it yourself or let your deployment system run it (Note to EF6.x users – this the equivalent of running Migrate.exe, but with the application dll compiled in).

Good parts	· It works in all situations. · Works well with deployment systems.
Bad parts	A bit more work.
Limitations	– none – , but watch out for continuous, five-stage app update
Tips	If your console application takes in a connection string to define which database to apply the migration to, then it will be easier to use in your deployment pipeline.
My verdict	A good option if you have a deployment pipeline, as you can execute the console application as part of the deployment. If you are manually applying the migration, then there is the command Update-Database.

1c. Turning EF Core migration into a script and applying it to the database

By using the Script-Migration command EF Core will convert a specific migration, or by default all your migrations, into a SQL script. You can then apply this using something that can execute SQL on the specific database you want updated. You can manually execute the SQL in SQL Server Management Studio, but typically you have something in your release pipeline to do that at the right time.

Good parts	· It works in all situations. · Works well with deployment systems which can use SQL scripts. · You can look at the SQL before its run to see if it looks OK.
Bad parts	· More work than the console app (2b) · You need some application to apply the script to the correct database.
Limitations	– none – , but watch out for continuous, five-stage app update
Tips	The SQL contains code to update the migration history, but you must include the impotent option in the Script-Migration command to get the checks that stops a migration from being applied twice.
My verdict	If you want to use EF Core’s Migrate method, then I would suggest using 2b, the console app. It’s as safe as using the scripts and does the same job. But if you pipeline already works with SQL change scripts then this is a good fit for you.

2a. Using a migration tool to apply a SQL script

If you create a series of SQL migrations scripts, then you need something to a) apply them in the right order and b) apply them only once. EF Core’s migrations contain code that implments the “right order” and “only once” rules, but when we write our own migration scripts we need a tool that will provides those features.

I, and many others, use an open-source library called DbUp that provides these features (and more) and also supports a range of database types. I order my migration scripts alphabetically, e.g. “Script0001 – initial migration”, “Script0002 – add seed data” for DbUp to apply. Just like EF Core migrations, DbUp uses a table to list what migrations have been applied to the database and will only apply a migration if it isn’t in that table.

Other migration tools are available, for instance Octopus Deploy, and various RedGate tools (but I haven’t used them so check they have the correct features).

Good parts	· It works in all situations. · Works well with deployment systems.
Bad parts	· You have to manage the scripts.
Limitations	– none – , but watch out for continuous, five-stage app update
Tips (for DbUp)	· I make a console application that takes in the connection string and then runs DbUp, so I can use it in my deployment pipeline. · For testing I make the method that runs DbUp available to my unit test assembly in a “only run in debug mode” unit test that migrates my local database correctly using my CompareEfSql tool (see the section about testing migrations in part 1 of this series).
My verdict	I use this approach on projects that use EF Core.

The application and applying migrations

When you apply a migration to the database you can stop the application or in some circumstances you can apply the migration while it is running. In this section I look at the different options available to you.

1. Stopping the application while you migrate the database

This is the safest option and works with breaking and non-breaking changes, but your users and your business might not be so happy. I call this the “site down for maintenance” approach. In the “site down” approach is you don’t want to stop an application while users are inputting data or maybe finished an order. That’s how you or your company gets a bad reputation.

I had this problem myself back in 2015 and I created a way to warn people that the site was going to close and then stopped all but the admin person from accessing the application. I chose this approach because for the web application I was working on it was a less costly approach than supporting breaking changes while keeping the web app running (I cover applying breaking to a continuous service application later). You may have come across “this site is down for maintenance” on services you use, normally at weekends and overnight.

NOTE: I wrote an article called How to take an ASP.NET MVC web site “Down for maintenance” which you might like to look at – the code was for ASP.NET MVC5 so it will need some work, but the idea is still valid.

Applying non-breaking migrations while the application is running

With non-breaking changes you can, in theory, apply them to the database while the old application is running, but there are some issues that can catch you out. For instance, if you added a new, non-null column with no SQL default and old software, which doesn’t know about that new column, tries to INSERT a new row you will get a SQL error because the old software hasn’t provided a value for a non-null column.

But if you know your non-breaking migration doesn’t have a problem then applying the migration while the old application is running provides continuous service to your users. There are various ways to do this, depending on which of the migration application approach you have chosen, one that come to mind are Azure’s staging slots, which have been around for ages, and the newer Azure Pipelines.

Applying breaking changes to a continuous running application: The five-stage app update.

The hardest job is applying a breaking change to a continuously running application. In the diagram showing the different approaches to will see a red box called “five-stage app update” in the top-right. The name comes from the fact that you need to migrate in stages, typically five but can be more.

Here is a diagram taken from in section 11.5.3 of my book “Entity Framework Core in Action” which shows the five stages needed to add a breaking change that split an existing CustomerAndAddress table into two tables, Customers and Addresses.

As you can see an update like this is complex to create and complex to apply, but that’s the cost of running a continuous system. There aren’t any real alternatives to the five stages, other than you never apply a breaking change to a continuous running system (I have heard one person who said that was their approach).

NOTE: I cover the continuous, five-stage app update in section 11.5.3 on my book “Entity Framework Core in Action” and you can also find a coverage of this in chapter 5 of the book “Building Evolutionary Architectures” by Neil Ford et al.

Conclusion

If the data in your database and the availability of your service is important to your organisation, then you must take a database migration seriously. In part 1 I covered the different ways create a migration script and this article covers how you might apply those migrations to a production database. The aim of this series is to provide you with the options open to you, with their pros, cons and limitations, so that you can take an informed decision about how to handle migrations.

As I said in the first article my first run-ins with EF migrations was with EF6. I know EF6 very well and having written the book “Entity Framework Core in Action” I know EF Core even better. The change from EF6 to EF Core around migrations typifies the change in the whole approach in EF Core.

EF6 had lots of “magic” going on to make it easier to use – automatic migration on startup was one of them. The problem was, when EF6’s “magic” didn’t quite work, then it was hard to sort it out. EF Core’s approach to migrations is that its up to you where and how you use it – no automatic “magic”. And lots of other small changes to migrations in EF Core come from listening to users of EF4 to 6.

So, migrations on production database is scary, but its always been scary. I have given you some insights into the options but that’s only really the minimum for production database changing. Backups, policies, pre-prod testing and deployment pipelines need to be added as required to make a reliable system.

Happy coding.

Back in 2017 I wrote about a system I built using EF Core 2.0 which used Command and Query Responsibility Segregation (CQRS) pattern that combines a SQL Server write database a RavenDB NoSQL read database. The title of the article is “EF Core – Combining SQL and NoSQL databases for better performance” and it showed this combination gave excellent read-side performance.

Ever since then I have been eagerly waiting for Entity Framework Core’s (EF Core) support of Cosmos DB NoSQL database, which is in preview in EF Core 2.2. RavenDB worked well but having a NoSQL database that EF Core can use natively makes it a whole lot easier to implement, as you will see in this article.

I’m writing this using the early EF Core 2.2.0-preview3 release of the Cosmos database provider. This preview works but is slow, so I won’t be focusing on performance (I’ll write a new article on that when the proper Cosmos DB provider is out in EF Core 3). What I will focus on is providing a robust implementation which ensures that the two databases are kept in step.

The original CQRS design had a problem if the NoSQL RavenDB update failed: at that point the two databases were out of sync. That was always nagging me, and Roque L Lucero P called me out on this issue on the original article (see this set of comments on that topic). I decided to wait until EF Core support of Cosmos DB was out (that has taken longer than originally thought) and fix this problem when I did the Cosmos DB rewrite, which I have now done.

NOTE: This article comes with a repo containing all the code and a fully functional example – see https://github.com/JonPSmith/EfCoreSqlAndCosmos. You can run the code need SQL Server (localdb is fine) and the Cosmos DB emulator. It will auto-seed the database on startup.

TR; DR; – summary

For an introduction to CQRS pattern read this excellent article on the Microsoft’s site.
I implement a two-database CQRS pattern, with the write database being a relational (SQL Server) database and the read database being a NoSQL database (Cosmos DB).
This type of CQRS scheme should improve the performance of read-heavy applications, but it is not useful to write-heavy applications (because the writes take longer).
My implementation uses EF Core 2.2 with the preview Cosmos database provider.
I implement the update of the NoSQL inside EF Core’s SaveChanges methods. This means a developer cannot forget to update the read-side database because its done for them by the code inside SaveChanges.
I use a SQL transaction to make sure that both the SQL and the NoSQL database updates are both done together. This means the two databases with always be in step.
This design of CQRS pattern is suitable for adding to a system later in its development to fix specific performance issues.
There is an example application on GitHub available to go with this article.

Setting the scene – using caching to improve read performance

You can skip this section if you already understand caching and the CQRS pattern.

In many applications the database accesses can be a bottleneck on performance, i.e. the speed and scalability of the application. When using EF Core there are lots of things you can do to improve the performance of database accesses, and its also really easy to do things that give you terrible performance.

There are only two hard things in Computer Science: cache invalidation and naming things. – Phil Karlton.But what do you do when even the best SQL database queries are deemed “too slow”? One typical approach is to add caching to your application, which holds a copy of some data in a form that can be accessed quicker than the original source. This is very useful, but making sure your cached version is always up to date is very hard (the Phil Karton quote comes from an article on CQRS written by Mateusz Stasch)

Caching can occur in lots of places, but in this article I cover caching at the database level. At the database level the caching is normally done by building “ready to display” versions for data. This works well where the read requires data from multiple tables and/or time-consuming calculations. In my book “Entity Framework Core in Action” I use a book selling application (think super-simple Amazon) as an example because it contains some complex calculations (form the list of authors, calculate the average review stars etc.). You can see a live version of the book app at http://efcoreinaction.com/.

You can do this yourself by building a pre-calculated version of the book list display (I did it in section 13.4 of my book), but its hard work and requires lots of concurrency handling to ensure that the pre-calculated version is always updated property. But the CQRS pattern makes this easier to do because it splits the write and read operations. That makes it simpler to catch the writes and deliver the reads. See the figure taken from Microsoft’s CQRS article (See this link for authors and Creative Commons Attribution Licence for this figure).

A further step I have taken is to have two databases – one for write and one for reads. In my case the read data store is a Azure Cosmos DB – according to Microsoft an “highly responsive, low latency, high availability, scalable” NoSQL database. The figure below gives you a high-level view of what I am going to describe.

The rest of the article describes how to build a two-database CQRS database pattern using EF Core with its new support for the Cosmos DB NoSQL database. The design also includes one way to handle the “cache invalidation” problem inherent in having the same data in two forms, hence the “robust” word in the title.

Describing the software structure

With any performance tuning you need to be clear what you are trying to achieve. In this case my tests show it gets slow as I add lots of books but buying a book (i.e. creating a customer order) is quick enough. I therefore decide to minimise the amount of development work and only apply the CQRS approach for the book list, leaving the book buying process as it was. This gives me a software structure where I have one Data Layer, but it has two part: SQL Server (orange) and Cosmos DB (purple).

I am really pleased to see that I can add a CQRS implementation only where I really need it. This has two big benefits:

I only need to add the complexity of CQRS where it’s needed.
I can add a CQRS system to an existing application to just performance tune specific areas.

Most applications I see have lots of database accesses and many of them are admin-type accesses which are needed, but their performance isn’t that important. This means I only really want to add CQRS where it’s needed, because adding CQRS is more work and more testing. I want to be smart as to where I spend my time writing code and many database accesses don’t need the performance improvements (and complexities!) that CQRS provides.

But the best part is by implementing my CQRS system inside EF Core’s SaveChanges methods I know that any existing database changes HAS to go though me code. That means if I’m adding my CQRS system to an existing project I know I can catch all the updates so my NoSQL (cache) values will be up to date.

Making my update robust – use a transaction

As well as using the new Cosmos DB database provider in EF Core I also want to fix the problem I had in my first version of this CQRS, two-database pattern. In the previous design the two database could get out of step if the write to the NoSQL database failed. If that happens then you have a real problem: the book information you are showing to your users is incorrect. That could cause problems with your customers, especially if the price on the book list is lower that the price at checkout time!

There are lots of ways to handle this problem, but I used a feature available to me because I am using a SQL database as my primary database – a SQL transaction (see previous diagram). This is fairly easy to do, but it does have some down (and up) sides. The main one is that the write of data is slower because SaveChanges only returns when both writes have finished. But there is an up side to this: it solves what is known as the “eventually consistent” problem where you do an update but when the app returns the data on your screen hasn’t updated yet.

Jimmy Bogard has an excellent series called “Life Beyond Distributed Transactions: An Apostate’s Implementation” in the 8^th article in the series he talks about using a transaction in a SQL database to ensure the second update is done before exiting. Jimmy is very clear that too many people ignore these errors – as he says in his tweet “hope is not a strategy”!

Jimmy’s approach is easy to understand, but if I used his approach I would have to find and replace every update path with some special code. In EF Core I can fix that by moving the code inside the SaveChanges methods, which means all the checks and updates are done whenever I create, update or delete anything that would change the book list display. That way I, or any of my colleagues, can’t forget to do the NoSQL update.

Let’s get into the code!

The whole process is contained in the SaveChanges (sync and async) methods. Below is the sync SaveChanges code.

public override int SaveChanges(bool acceptAllChangesOnSuccess)
{
    if (_bookUpdater == null)
        //if no bookUpdater then run as normal
        return base.SaveChanges(acceptAllChangesOnSuccess);

    try
    {
        var thereAreChanges = _bookUpdater
            .FindBookChangesToProjectToNoSql(this);
        //This stops ChangeTracker being called twice
        ChangeTracker.AutoDetectChangesEnabled = false; 
        if (!thereAreChanges)
            return base.SaveChanges(acceptAllChangesOnSuccess);
        return _bookUpdater
            .CallBaseSaveChangesAndNoSqlWriteInTransaction(
               this,
               () => base.SaveChanges(acceptAllChangesOnSuccess));
    }
    finally
    {
        ChangeTracker.AutoDetectChangesEnabled = true;
    }
}

There are lots of lines, many of them are to make the code run efficiently, but in the code there are two methods that manage the book list update.

FindBookChangesToProjectToNoSql (Lines 8 and 9). This uses EF Core’ ChangeTracker to find changes to entity classes that will affect the book list display.
CallBaseSaveChangesAndNoSqlWriteInTransaction (lines ?? to ??). This is only called if NoSQL writes are needed and it handles the secure update of both the SQL database and the Cosmos database.

Now we will look at the two parts – finding the changes and then saving the changes securely.

Finding the book changes

Finding the book changes is complex, but this article is really about making a robust CQRS system with Cosmos DB, so I’m only going to skip over this code and just give you a diagram of how the entity’s State is used to decide what changes should be applied to the NoSQL database.

The diagram starts with the book list that the NoSQL database is going to provide with the list of entity’s that effect that book list. The table at the bottom shows how I use the entity’s State to decide what changes need to be applied to the NoSQL database.

The basic idea is the Book’s State takes precedent, with changes in the associated relationships only causing an update to the book list. There are some subtle items, especially around soft delete, which you can see in the BookChangeInfo class.

NOTE: The actual code to work out the updates needed is quite complex, but you can see it in the accompanying example repo by starting in the SQL DbContext’s SaveChanges and follow the code. I cover how I decoded the entity State in section 14.2.4 of my book “Entity Framework Core in Action”.

The end of all this is there a series of book list changes that must be applied to the NoSQL database to make it match what the data in the SQL database. The trick is to make sure that anything will make the two databases out of step from each other, which I cover next.

Updating the databases in a secure manner

To make sure my SQL and NoSQL databases always in step I apply the both database updates inside a SQL transaction. That means if either of the updates, SQL or NOSQL, fail then neither are applied. The code below shows how I do that.

private int RunSqlTransactionWithNoSqlWrite(
    DbContext sqlContext, Func<int> callBaseSaveChanges)
{
    if (sqlContext.Database.CurrentTransaction != null)
        throw new InvalidOperationException(
            "You can't use the NoSqlBookUpdater if you are using transactions.");

    var applier = new ApplyChangeToNoSql(sqlContext, _noSqlContext);
    using (var transaction = sqlContext.Database.BeginTransaction())
    {
        var result = callBaseSaveChanges(); //Save the SQL changes
        applier.UpdateNoSql(_bookChanges);  //apply changes to NoSql database
        _noSqlContext.SaveChanges();        //And Save to NoSql database
        transaction.Commit();
        return result;
    }
}

Using a SQL transaction is a nice way to implement this, but you must apply the NoSQL database update at the end of the transaction. That’s because Cosmos DB database provider does not support transactions (most NoSQL don’t support transactions) which means the NoSQL write cannot be rolled back (i.e. undone). That means you can’t use this approach inside another transaction, as you could do something after the NoSQL update that errored (hence the check on line 4).

Here is the sync UdateNoSql method (there is a similar Async version). I use AutoMapper’s ProjectTo method to create the book list version needed for the display.

public bool UpdateNoSql(IImmutableList<BookChangeInfo> booksToUpdate)
{
    if (_noSqlContext == null || !booksToUpdate.Any()) return false;

    foreach (var bookToUpdate in booksToUpdate)
    {
        switch (bookToUpdate.State)
        {
            case EntityState.Deleted:
            {
                var noSqlBook = _noSqlContext.Find<BookListNoSql>
                    (bookToUpdate.BookId);
                _noSqlContext.Remove(noSqlBook);
                break;
            }
            case EntityState.Modified:
            {
                //Note: You need to read the actual Cosmos entity because of the extra columns like id, _rid, etc.
                //Version 3 of EF Core might make Attach work.
                //See https://github.com/aspnet/EntityFrameworkCore/issues/13633
                var noSqlBook = _noSqlContext.Find<BookListNoSql>
                     (bookToUpdate.BookId);
                var update = _sqlContext.Set<Book>()
                    .ProjectTo<BookListNoSql>(SqlToNoSqlMapper)
                    .Single(x => x.BookId == bookToUpdate.BookId);
                SqlToNoSqlMapper.CreateMapper().Map(update, noSqlBook);
                break;
            }
            case EntityState.Added:
                var newBook = _sqlContext.Set<Book>()
                    .ProjectTo<BookListNoSql>(SqlToNoSqlMapper)
                    .Single(x => x.BookId == bookToUpdate.BookId);
                _noSqlContext.Add(newBook);
                break;
            default:
                throw new ArgumentOutOfRangeException();
        }
    }

    return true;
}

Alternative ways of make the update robust

I think using a transaction is a simple way to ensure both databases are in step, but as Rafael Santos say in his comment it does make the SaveChanges take a lot longer, because it only returns when both writes are finished. In my first approach using RavenDB the write went from 13ms to 35ms.

That means this approach is OK for say an e-commerce product list (i.e. products that customer can buy) where there are lots of reads but only a few write, but it wouldn’t work so well for say stock control (i.e. what you have in warehouse), where there are lots of writes happening.

I have been thinking about this and here are some ideas for you think about. Note: all of these approaches suffer with the the “eventually consistent” problem I mentioned before, i.e. the system will return to the user before the data they were looking at has been updated.

1. Send a message inside a transaction.

In this case you would fire-and-forget a message to another system via a reliable queue, like RabbitMQ or Azure Service Bus. Then it is the job of the system that gets that message to make sure the NoSQL update is repeated until it works. The SaveChanges should return more quickly because queuing the message will be quick.

This is what Jimmy Bogard does in his relational article in his series called Life Beyond Distributed Transactions: An Apostate’s Implementation. Do have a read.

2. Run a background task to fix any failures.

If you added a LastUpdated DateTime to all the SQL entities, and a similar LastUpdated in the NoSQL cached version, then you have a method to find mismatches. This means you could looks for changes since it last run and check the SQL and NoSQL version have matching LastUpdated values (Cosmos DB has a “_ts” Unix-styled timestamp that may be useful).

Either you run the method every x mins (simple, but not that good) or you catch a NoSQL error and run the method looking for updates equal to LastUpdated time of the SQL update.

3. (Advanced) Use the ChangeTracker.StateChanged Event.

There is a really nice, but not much talked about, feature called ChangeTracker.StateChanged event that happens after the SaveChange has completed. This gives you a nice solution that only kicks in for a specific update error.

Basically you could kick off an timer for every NoSQL write, which is cancelled by the NoSQL ChangeTracker.StateChanged event that occurs on a successful write (status changes to Unchanged if successful). If the timer timed out, then you know the NoSQL update failed and you could take remedial action to fix it.

This if advanced stuff needing a ConcurrentDictionary to track each write. I have thought about it, but not implemented it yet. However if a client wanted me to add a CQRS pattern with a quick writes to their application, then this is most likely what I would build.

Limitations in Cosmos support in 2.2 preview

This application was built with EF Core 2.2 and the Microsoft.EntityFrameworkCore.Cosmos package 2.2.0-preview3-35497 . This is a very early version of Cosmos DB support in EF Core with some limitations on this application. They are

This Cosmos DB preview is very slow! (like hundreds of milliseconds). Version 3 will use a new Cosmos DB SDK, which will be faster.
I would have liked the Cosmos id value to be the same as the BookId GUID string.
I couldn’t use Attach for the update of an existing NoSQL database, which would have been quicker.

You can track what is happening to EF Core support for Cosmos here. I will most likely update the application when Version 3 is out and write an article on its performance.

Conclusion

I am very happy to see EF Core supporting NoSQL databases alongside relational databases. It gives me more flexibility in using the right database based on business needs. Also EF Core has the depth and flexibility for me to implement quite complex state management of my database writes, which I needed to implement my CQRS two-database design.

Personally, I like the CQRS two-database design because it too allows me flexibility – I can add it only to the queries that need performance tuning and its also fairly easy to add retrospectively to an application that uses EF Core and relational (SQL) databases. Most performance tuning is done late, and my design fits in with that.

The next stage is to see what performance gains I can get with EF Core version 3. In my original version of CQRS with RavenDB I got very good performance indeed. I’ll let you know how that goes when EF Core 3 is out!

Happy coding.

This isn’t one of my long articles, but just a bit of fun I had trying to convert Entity Framework Core’s (EF Core) CommandExecuted logs back into real SQL. EF Core’s logging is much improved over EF6.x and it returns very readable SQL (see this example below)

var id = 1;
var book = context.Books.Single(x => x.BookId == id);

Produces this log output

Executed DbCommand (1ms) [Parameters=[@__id_0='1'],
    CommandType='Text', CommandTimeout='30']
SELECT TOP(2) [p].[BookId], [p].[Description], [p].[ImageUrl]
    , [p].[Price], [p].[PublishedOn], [p].[Publisher]
    , [p].[SoftDeleted], [p].[Title]
FROM [Books] AS [p]
WHERE ([p].[SoftDeleted] = 0) AND ([p].[BookId] = @__id_0)

Now I spend quite a bit of time understanding and performance tuning EF Core code, so it’s very useful if I can copy & paste the SQL into something like Microsoft’s SQL Server Management Studio (SSMS) to see how they perform. The problem is I have to hand-edit the SQL to add the correct values to replace any parameters (see @__id_0 in last code).

So, in my spare time (??), I decided to try to create some code that would automatically replace the property reference with the actual value. It turns out it’s quite difficult and you can’t quite get everything right, but its good enough to help in lots of places. Here is the story of how I added this feature to my EfCore.TestSupport library.

The steps to building a my DecodeMessage method

The steps I needed to do were:

Capture EF Core’s logging output
Turn on EnableSensitiveDataLogging
Catch any EF Core CommandExecuted logs
Decode the Parameters
Replace any property references in the SQL with the ‘correct’ parameter value

Now I did say I was going to keep this article short so I’m going to give you some code that handles the first three parts. You can see the EnableSensitiveDataLogging method near the end of building the options.

var logs = new List<LogOutput>();
var options = new DbContextOptionsBuilder<BookContext>()
    .UseLoggerFactory(new LoggerFactory(new[] 
          { new MyLoggerProviderActionOut(l => logs.Add(l))}))
    .UseSqlite(connection)
    .EnableSensitiveDataLogging()
    .Options;
using (var context = new BookContext(options)) 
{
    //… now start using context

NOTE: Sensitive data logging is fine in your unit tests, but you should NOT have sensitive data logging turned on in production. Logging the actual data used is a security risk and could break some user privacy rules like GPRS.

In fact I have methods in my EFCore.TestSupport library that handle building the options and turning on sensitive logging, plus a load of other things. Here is an example of one helper that creates an in-memory database options, with logging.

var logs = new List<LogOutput>();
var options = SqliteInMemory.CreateOptionsWithLogging
     <BookContext>(log => logs.Add(log));
using (var context = new BookContext(options)) 
{
    //… now start using context

The EfCore.TestSupport library has another version of this that works for SQL Server. It creates a unique database name per class, or per method, because xUnit (the favourite unit test framework for NET Core) runs each test class in parallel.

NOTE: The EfCore.TestSupport uses a logging provider that calls an action method for every log. This makes it easy to write logs to the console, or capture them into list.

Decoding the Parameters

Having captured the EF Core’s logs now I need to decode the first line that has the parameters. There are a few permutations, but it’s clear that Regex is the way to go. This problem is I’m not an expert on Regex, but LinqPad came to my rescue!

LinqPad 5.36 has a very nice Regex tool – the best I have found so far. Here is a screenshot of its regex feature, which is called up via Ctrl+Shift+F1.

WARNING: It’s a great tool, but I thought if I saved the code it would keep the pattern I had created, but it doesn’t. I spent hours getting the regex right and then lost it when I entered something else. Now I know its all OK, but be warned.

All my trials came up with the following Regex code

new Regex(@"(@p\d+|@__\w*?_\d+)='(.*?)'(\s\(\w*?\s=\s\w*\))*(?:,\s|\]).*?");

If you don’t know regex then it won’t mean anything to you, but it does the job of finding the a) param name, b) param value, and c) extra information on the parameter (like its size). You can see the whole decode code here.

Limitations of the decoding

It turns out that EF Core’s logged data doesn’t quite give you all you need to perfectly decode the log back to correct SQL. Here are the limitations I found:

You can’t distinguish the different between an empty string and a null string, both are represented by ”. I decided to make ” return NULL.
You can’t work out if it’s a byte[] or not, so byte[] is treated as a SQL string. This will FAIL in SQL Server.
You can’t tell if something is a Guid, DateTime etc., which in SQL Server need ” around them. In the end I wrapped most things in ”, including numbers. SQL Server accepts numbers as strings (but other databases won’t).

Example of a different decoded SQL

If we go back to the book lookup at the start of this article then the decoded result is shown below

SELECT TOP(2) [p].[BookId], [p].[Description]
   , [p].[ImageUrl], [p].[Price], [p].[PublishedOn]
   , [p].[Publisher], [p].[SoftDeleted], [p].[Title]
FROM [Books] AS [p]
WHERE ([p].[SoftDeleted] = 0) AND ([p].[BookId] = '1')

As you can see on the last line the integer is represented as a string. This isn’t the normal way to do this but works in SQL Server. I took the decision to wrap things that I didn’t think were strings because this what is needed to make other types, such as GUIDs, Datetime etc. to work.

My really complex test contained lots of different NET Types, and here is the output.

SET NOCOUNT ON;
INSERT INTO [AllTypesEntities] ([MyAnsiNonNullString]
    , [MyBool], [MyBoolNullable], [MyByteArray], [MyDateTime]
    , [MyDateTimeNullable], [MyDateTimeOffset], [MyDecimal]
    , [MyDouble], [MyGuid], [MyGuidNullable], [MyInt]
    , [MyIntNullable], [MyString], [MyStringEmptyString]
    , [MyStringNull], [MyTimeSpan])
VALUES ('ascii only', 1, NULL, '0x010203', '2000-01-02T00:00:00',
NULL, '2004-05-06T00:00:00.0000000+01:00', '3456.789', '5678.9012', 
'ba65d636-65d4-4c07-8ddc-50c615cef539', NULL, '1234', NULL, 
'string with '' in it', NULL, NULL, '04:05:06');
SELECT [Id]
FROM [AllTypesEntities]
WHERE @@ROWCOUNT = 1 AND [Id] = scope_identity();

In this complex version the parts that fail are:

The MyByteArray has ” around it and FAILS – taking off the string delimiters fixes that.
The MyStringEmptyString is set to NULL instead of an empty string.

Not perfect, but quite usable.

How can you access this code?

If you just want to use this feature its build into the latest EfCore.TestSupport NuGet package (1.7.0 to be precise). Its build into the LogOutput class which is used by the loggers in this library. There are methods that create options for SQLite (in-memory) and SQL Server database that allow logging. There are plenty of examples of these in the library – have a look at the unit tests for this in the TestEfLoggingDecodeBookContext class.

If you want to play with the code yourself then take a copy of the EfCoreLogDecoder class which contains the decode parts.

Conclusion

Well it was a bit of fun, maybe not something I would do on a job but still a useful tool. I was a bit disappointed I couldn’t decode the log completely but what it does is still useful to me. Maybe you will find it useful to you too.

Now I need to get back to my real work for my clients. See you on the other side!

Happy coding.

I read Jimmy Bogard’s article called “AutoMapper’s Design Philosophy”, which he wrote to help people understand what job AutoMapper was designed to do. This sort of article helps people who get frustrated with AutoMapper because they are trying to use it to do something it wasn’t really designing to do. I use AutoMapper in my libraries and I was glad to see my usage is right in line with what AutoMapper is designed to do.

I thought I would write a similar article for my GenericServices library (see summary for what GenericServices does) to help anyone who uses, or wants to use my GenericServices library (and associated libraries). While the use of my GenericServices library is tiny compared to AutoMapper (about 0.05% of AutoMapper’s downloads!) I too have issues or requests for features that don’t fit into what GenericServices’s is designed to do. Hopefully this article will help people understand my GenericServices library better, and I also add a few tips and techniques that I have found useful too.

TL; DR; – Summary

The GenericServices library is designed to speed up the development of building front-end Entity Framework 6 and Entity Framework Core (EF Core) databases accesses.
GenericServices does this by automating the mapping of database classes to DTOs (Data Transfer Object, also known as a ViewModel in ASP.NET) in a way that builds efficient database accesses.
My personal experience I would say that my GenericServices library saved me 2 months of development time over a 12-month period.
GenericServices also has a feature where it can work with Domain-Driven Design (DDD) styled EF Core database classes. It can find and call methods or constructors inside a DDD-styled EF Core database class. That gives very good control over creates and updates.
This article tells you what GenericServices can, and cannot, do.
I then list five GenericServices tips and techniques that I use when using this library.

Setting the scene – what does GenericServices do?

TIP: This is a shortened version of a section from the introduction article on EFcore.GenericServices. The original has more code in it.

GenericServices is designed to make writing front-end CRUD (Create, Read, Update and Delete) EF Core database accesses much easier. It handles both the database access code and the “adaption” of the database data to what the front-end of your application needs. It does this by providing a library with methods for Create, Read, Update and Delete that uses either a EF Core database class or a DTO (Data Transfer Object, also known as a ViewModel in ASP.NET) to define what EF Core class is involved and what data needs to be read or written.

I’m going to take page used to update a database class to describe the typical issues that come up. My example application is an e-commerce site selling technical books and I implement a feature where an authorised user can add a sales promotion to a book, by reducing its price. The ASP.NET Core web page is shown below with the user’s input in red and comments on the left about the Book properties and how they are involved in the update.

In a web/mobile application a feature like this consists of two stages:

1. Read data to display

The display to the user needs five properties taken from the Book and I use a DTO (Data Transfer Object, also known as a ViewModel) than contains the five properties I want out of the Book entity class. GenericServices uses AutoMapper to build a query using LINQ which EF Core turns into an efficient SQL SELECT command that just reads the five columns. Below is the DTO, with the empty interface ILinkToEntity<TEntity> (see line 1) that GenericServices uses to find and map the DTO to the EF Core classes.

public class AddPromotionDto : ILinkToEntity<Book>
{
    [HiddenInput]
    public int BookId { get; set; }

    [ReadOnly(true)] // Tells GenericServices not copy this back to the database
    public decimal OrgPrice { get; set; }

    [ReadOnly(true)] //Tells GenericServices not copy this back to the database
    public string Title { get; set; }

    public decimal ActualPrice { get; set; }

    [Required(AllowEmptyStrings = false)]
    public string PromotionalText { get; set; }
}

Below is the GenericService code that reads in data into the DTO, with the id holding the Book’s primary key (see this link for full list of all the code)

 
var dto = _service.ReadSingle<AddPromotionDto>(id);

NOTE: AutoMapper is great for “Flattening” relationships, that is it can pick properties/columns in related classes – see the article “Building high performance database queries using Entity Framework Core and AutoMapper” for more on this.

2. Update the data

The second part is the update of the Book class with the new ActualPrice and the PromotionalText. This requires a) the Book entity to be read in, b) the Book entity to be updated with the two new values, and c) the updated Book entity to be written back to the database. Below is the GenericService code that does this (see this link for full list of all the code)

_service.UpdateAndSave(dto);

Overall the two GenericService calls replaces about 15 lines of hand-written code that does the same thing.

The problem that GenericServices is aimed at solving

I built GenericServices to make me faster at building .NET applications and to remove some of the tedious coding (e.g. LINQ Selects with lots of properties) around building front-end CRUD EF Core database accesses. Because I care about performance I designed the library to build efficient database accesses using the LINQ Select command, i.e. only loading the properties/columns that are needed.

With the release of EF Core, I rewrote the library (GenericServices (EF6) -> EfCore.GenericServices) and added new features to work with a Domain-Driven Design (DDD) styled database classes. DDD-styled database classes give much better control over how creates and updates are done.

GenericServices is meant to make the simple-to-moderate complexity database reads easy to build. It can also handle all deletes and some single-class creates and updates with normal database classes, but because EfCore.GenericServices supports DDD-styled database classes it can call constructors/methods which can handle every type of create or update.

Overall, I find GenericServices will handle more than 60% of all front-end CRUD accesses, but with DDD-styled database classes this goes up to nearly 90%. It’s only the really complex reads that can be easier to write by hand. The trick is to know when to use GenericServices, and when to hand-code the database access – I cover that next.

What EfCore.GenericServices can/cannot handle

OK, let’s get down to the details of what GenericServices can and cannot do, with a list for good/bad usages.

GenericServices is GOOD at:

All reads that use flattening (see Note1)
All deletes
Create/Update
- Normal (i.e. non-DDD) database classes: of a single class (see Note2)
- DDD-styled database classes: any create/update.

GenericServices is BAD at:

Any read that needs extra EF Core commands like .Include(), .Load(), etc. (see Note1)
Create/Update
- Normal (i.e. non-DDD) database classes: with relationships (see Note2)

Note1: Read – flatten, not Include.

The GenericServices reads are designed for sending to a display page or a Web API, and I can normally implement any such read by using AutoMapper’s “Flattening” feature. However, sometimes the effort to set up special AutoMapper’s configurations (see docs) can take more effort than just hand-coding the read. Don’t be afraid to build your own read queries if this simpler for you.

You cannot use GenericServices for reads that needs .Include(), .Load(), etc. Typically that sort of read is used in business logic and I have a separate library called EfCore.GenericBizRunner for handling that (see articles “A library to run your business logic when using Entity Framework Core” and “Architecture of Business Layer working with Entity Framework (Core and v6)” for more about handling business logic).

NOTE: Using .Include(), Load() or Lazy Loading is inefficient for a simple read as it means you are either loading data you don’t need, and/or making multiple trips to the database, which is slow.

Note2: Create/Update – single or relationships

When using normal (non-DDD) database classes GenericServices will only create/update a single class mapped to the database via EF Core. However, you can get around this because GenericServices is designed to work with a certain style of DDD entity classes, i.e. GenericServices can find and call constructors or methods inside your EF Core class to do a create or update, which allows your code to handle any level of complexity of a create or update.

GenericServices also gives you the option to validate the data that is written to the database (off by default – turn on via the GenericServiceConfig class). This, coupled with DDD constructor/methods, allows you to write complex validation and checks. However, if I think the code is getting too much like business logic then I use EfCore.GenericBizRunner.

Tips and Techniques

Clearly really know my library very well, and I can do things other’s might not think of. This is a list of things I have do that you might find useful. Here is a list to save you scrolling down to see what’s there.

Try using DDD-styled entity classes with EfCore.GenericServices
Don’t try to use GenericServices for business logic database accesses
How to filter, order, page a GenericService read query
Helper library for using GenericServices with ASP.NET Core Web API
How to unit test your GenericServices code

a. Try using DDD-styled entity classes with EfCore.GenericServices

Personally, I have moved over to using DDD-styled database classes with EF Core, so let me explain the differences/advantages of DDD.

Non-DDD classes have properties with public setters, i.e. anyone can alter a property, while DDD-styled classes have private setters which means you must use a constructor or a method to create/update a property/ies. So DDD-styled classes “locks down” any changes so that no one can bypass the create/update code in that class (see my article “Creating Domain-Driven Design entity classes with Entity Framework Core” for more on this).

Yes, DDD-styled database classes do take some getting used to, but it gives you an unparallel level of control over create/update, including altering not only properties but relationships as well. EfCore.GenericServices works with DDD-styled EF Core classes and finds constructors/methods by matching the parameter name/types (see GenericServices DDD docs here).

b. Don’t try to use GenericServices for business logic database accesses

When I think about database accesses in an application I separate them into two types:

CRUD database accesses done by the front-end, e.g. read this, update that, delete the other.
Business logic database accesses, e.g. create an order, calculate the price, update the stock status.

The two types of accesses are often different – CRUD front-end is about simple and efficient database accesses, while business logic database accesses are about rules and processes. GenericServices is designed for CRUD database for the front-end and won’t do a good job for business logic database accesses – I use my GenericBizRunner library for that.

Sure, it can get hazy as to whether a database access is a simple CRUD access or business logic – for instance is changing the price of an item a simple CRUD update or a piece of business logic? But there are two things that GenericService on its own can’t do when working with normal (non-DDD) : a) create/update a class and its relationships, and b) it can’t run any extra code. But with DDD-styled database classes then both limitations are removed, as EfCore.GenericServices can call the constructors/methods in the database class.

However, there are some actions, like update the stock status which can trigger a restocking order, that are clearly business logic and should be handled separately (see my article “Architecture of Business Layer working with Entity Framework (Core and v6)” on how I handle business logic).

c. How to filter, order, page a GenericService read query

The EfCore.GenericServices’s ReadManyNoTracked<T>() method returns an IQueryable<T> result, which allows you to filter, order, page the data after it has been projected into the DTO. By adding LINQ commands after the ReadManyNoTracked method EF Core will turn them into efficient SQL commands. You then end the query with something like .ToList() or .ToListAsync() to trigger the database access.

Filtering etc. after the mapping to the DTO normally covers 90% of your query manipulation but what happens if you need to filter or change a read prior to the projection to the DTO? Then you need ProjectFromEntityToDto<TEntity,TDto>(preDtoLinqQuery).

The ProjectFromEntityToDto method is useful if you:

Want to filter on properties that isn’t in the DTO version.
Want to apply the method .IgnoreQueryFilters() to the entity to turn off any Query Filter on the entity, say if you were using a Query Filters for soft delete.

NOTE: If you are using Query Filters then all the EfCore.GenericServices’s methods obey the query filter, apart from the method DeleteWithActionAndSave. This turns OFF any query filters so that you can delete anything – you should provide an action that checks the user is allowed to delete the specific entry.

d. Helper library for using GenericServices with ASP.NET Core Web Core

I use ASP.NET at lot over the years and I have generated several patterns for handling GenericServices (and GenericBizRunner), especially around Web APIs. I have now packaged these patterns into a companion library called EfCore.GenericServices.AspNetCore.

For ASP.NET MVC and Razor Pages the EfCore.GenericServices.AspNetCore has a CopyErrorsToModelState extension method that copies GenericServices’s status into the ASP.NET Core Model so they become validation errors.

The features for Web API are quite comprehensive.

GenericServices supports JSON Patch for updates – see my article “Pragmatic Domain-Driven Design: supporting JSON Patch in Entity Framework Core” for full details of this feature.
For Web API it can turn GenericServices’s status into the correct response type, with HTTP code, success/errors parts and any result to send. This makes for very short Web API method with a clearly defined output type for Swagger – see example below

[HttpGet("{id}")]
public async Task<ActionResult<WebApiMessageAndResult<TodoItem>>> 
    GetAsync(int id, [FromServices]ICrudServicesAsync service)
{
    return service.Response(
        await service.ReadSingleAsync<TodoItem>(id));
}

For more on EfCore.GenericServices and ASP.NET Core Web APIs have a look at my article “How to write good, testable ASP.NET Core Web API code quickly”

e. How to unit test your GenericServices code

I’m a big fan of unit testing, but I also what to write my tests quickly. I therefore have built-in methods to help to unit test code that uses EfCore.GenericServices. I also have a whole library called EfCore.TestSupport to help with unit testing any code that uses EF Core.

EfCore.GenericServices has a number of methods that will set up the data that GenericServices would normally get via dependency injection (DI). See line 11 for one such method in the code below. The other methods, like SqliteInMemory.CreateOptions on line 5, come from my EfCore.TestSupport library.

[Fact]
public void TestProjectBookTitleSingleOk()
{
    //SETUP
    var options = SqliteInMemory.CreateOptions<EfCoreContext>();
    using (var context = new EfCoreContext(options))
    {
        context.Database.EnsureCreated();
        context.SeedDatabaseFourBooks();

        var utData = context.SetupSingleDtoAndEntities<BookTitle>();
        var service = new CrudServices(context, utData.Wrapped);

        //ATTEMPT
        var dto = service.ReadSingle<BookTitle>(1);

        //VERIFY
        service.IsValid.ShouldBeTrue(service.GetAllErrors());
        dto.BookId.ShouldEqual(1);
        dto.Title.ShouldEqual("Refactoring");
    }
}

I also added a ResponseDecoders class containing a number of extension method to my EfCore.GenericServices.AspNetCore that will turn a Web API response created by that library back into its component parts. This makes testing Web API methods simpler. This link to a set of unit tests gives you an idea of how you could use the extension methods in integration testing.

Also see the unit testing section of my article “How to write good, testable ASP.NET Core Web API code quickly”.

Conclusion

I hope this article helps people to get the best out of my EfCore.GenericServices library and associated libraries like EfCore.GenericServices.AspNetCore and EfCore.GenericBizRunner. All these libraries were built to make me faster at developing applications, and also to remove some of the tedious coding so I can get on with coding the parts that need real thought.

The important section is “What EfCore.GenericServices can/cannot handle” which tells you what the library can and cannot do. Also note my comments on the difference between front-end CRUD (GenericServices) and business logic (GenericBizRunner). If you stay in the “sweet spot” of each of these libraries, then they will work well for you. But don’t be afraid to abandon either library and write your own code if it’s easier or clearer – pick the approach that is clear, but fast to develop.

I also hope the tips and techniques will alert you to extra parts of the EfCore.GenericServices library that you might not know about. I used my libraries on many projects and learnt a lot. The list are some things I learnt to look out for and links to other libraries/techniques that help me be a fast developer.

Happy coding.

I was building an application using Entity Framework Core (EF Core) for a client and I needed to set up test data to mimic their production system. Their production system was quite complex with some data that was hard to generate, so writing code to set up the database was out of the question. I tried using JSON files to save data and restore data to a database and after a few tries I saw a pattern and decided to build a library to handle this problem.

My solution was to serialize the specific data from an existing database and store it as a JSON file. Then in each unit test that needed that data I deserialized the JSON file into classes and use EF Core to save it to the unit test’s database. And because I was copying from a production database which could have personal data in it I added a feature that can anonymize personal data so that no privacy laws, such as GDPR, were breached.

Rather than writing something specific for my client I decided to generalise this feature and add it to my open-source library EfCore.TestSupport. That way I, and you, can use this library to help you with create better test data for unit/performance tests, and my client gets a more comprehensive library at no extra cost.

TL; DR; – summary

The feature “Seed from Production” allows you to capture a “snapshot” of an existing (production) database into a JSON file which you can use to recreate the same data in a new database for testing your application.
This feature relies on an existing database containing the data you want to use. This means it only works for updates or rewrites of an existing application.
The “Seed from Production” is useful in the following cases.
- For tests that need a lot of complex data that is hard to hand-code.
- It provides much more representative data for system tests, performance tests and client demos.
The “Seed from Production” feature also includes an anonymization stage so that no personal data is stored/used in your application/test code.
The “Seed from Production” feature relies on EF Core’s excellent handing of saving classes with relationships to the database.
The “Seed from Production” feature is part of my open-source library TestSupport (Version 2.0.0 and above).

Setting the scene – why do I need better unit test data?

I spend a lot of time working on database access code (I wrote the book “Entity Framework Core in Action”) and I often have to work with large sets of data (one client’s example database was 1Tbyte in size!). If you use a database in a test, then the database must be in a known state before you start you test. In my experience what data the unit test’s needs break down into three cases:

A small number of tests you can start with an empty database.
Most tests only need few tables/rows of data in the database.
A small set of tests need more complex set of data in the database.

Item 1, empty database, is easy to arrange: either delete/create a new database (slow) or build a wipe method to “clean” an existing database (faster). For item 2, few tables/rows, I usually write code to set up the database with the data I need – these are often extension methods with names like SeedDatabaseFourBooks or CreateDummyOrder. But when it comes to item 3, which needs complex data, it can be hard work to write and a pain to keep up to date when the database schema or data changes.

I have tried several approaches in the past (real database and run test in a transaction, database snapshots, seed from Excel file, or just write tests to handle unknown data setup). But EF Core’s excellent approach to saving classes with relationships to a database allows me to produce a better system.

Personally, I have found having data that looks real makes a difference when I am testing at every level. Yes, I can test with the book entitled “Book1” with author “Author1”, but having a book named “Domain-Driven Design” with author “Eric Evans” is easier to spot errors in. Therefore, I work to produce “real looking” data when I am testing or demonstrating an application to a client.

One obvious limitation of the “Seed from Production” approach is you do need an existing database that contains data you can copy from! Therefore, this works well when you are updating or extending an existing application. However, I have also found this useful when building a new application as the development will (should!) soon produce pre-production data that you can use.

NOTE: Some people would say you shouldn’t be accessing a database as part of a unit test, as that is an integration test. I understand their view and in some business logic I do replace the database access layer with an interface (see this section in my article about business logic). However, I am focused on building things quickly and I find using a real database makes it easier/quicker to write tests (especially if you can use an in-memory database) which means my unit test also checks that my database relationships/constraints work too.

How my “seed from production” system works

When I generalised the “seed from production” system I listed what I needed.

A way to read data from another database and store in a file. That way the “snapshot” data becomes part of your unit test code and is cover by source control.
The data may come from a production database that contains personal data. I need a way to anonymise that data before its saved to a file.
A way to take the stored “snapshot” and write it back out to a new database for use in tests.
I also needed the option to alter the “snapshot” data before it was written to the database for cases where a particular unit test needed a property(s) set to a specific value.
Finally, I need a system that made updating the “snapshot” data easy, as the database schema/data is likely to change often.

My “Seed from Production” feature handles all these requirements by splitting the process into two parts: an extract part, which is done once, and the seed part, which runs before each test to setup the database. The steps are:

Extract part – only run if database changes.
1. You write the code to read the data you need from the production database data.
2. The DataResetter then:
  1. Resets the primary keys and foreign keys to default values so that EF Core will create new versions in the test database.
  2. Optionally you can anonymise specific properties that need privacy, e.g. you may want to anonymise all the names, emails, addresses etc.
3. Converts the classes to a JSON string.
4. Saves this JSON string to a file, typically in you unit test project.
Seed database part – run at start of every unit test
1. You need to provide an empty database for the unit test.
2. The DataResetter reads the JSON file back into classes mapped to the database.
3. Optionally you can “tweak” any specific data in the classes that your unit test need
4. Then you add the data to the database and call SaveChanges.

That might sound quite complicated but most of that is done for you by the library methods. The diagram below shows the parts of the two stages to make it clearer – the parts shown as orange are the parts you need to write while the parts in blue are provided by the library.

Show me the code!

This will all make more sense when you see the code, so in the next subsections I show you the various usage of the “Seed from Production” library. They are:

Extract without no anonymisation.
Extract showing an example of anonymisation.
Seed a unit test database, with optional update of the data.
Seed database when using DDD-styled entity classes.

In all the examples I use my book app database which I used the book I wrote for Manning, “Entity Framework Core in Action”. The book app “sells” books and therefore the database contains books, with authors, reviews and possible price promotions – you can see this in the DataLayer/BookApp folder of my EfCore.TestSupport GitHub project.

NOTE: You can see a live version of the book app at http://efcoreinaction.com/

1. Extract without no anonymisation

I start with extracting data from a database stage, which only needs to be run when the database schema or data has changed. To make it simple to run I make it a unit test, but I define that test in such a way that it only runs in debug mode (that stops it being run when you run all the tests).

[RunnableInDebugOnly(DisplayName = "Needs database XYZ")]
public void ExampleExtract()
{
    var sqlSetup = new SqlServerProductionSetup<BookContext>
        ("ExampleDatabaseConnectionName");
    using (var context = new BookContext(sqlSetup.Options))
    {
        //1a. Read in the data to want to seed the database with
        var entities = context.Books
            .Include(x => x.Reviews)
            .Include(x => x.Promotion)
            .Include(x => x.AuthorsLink)
                .ThenInclude(x => x.Author)
            .ToList();

        //1b. Reset primary and foreign keys
        var resetter = new DataResetter(context);
        resetter.ResetKeysEntityAndRelationships(entities);

        //1c. Convert to JSON string
        var jsonString = entities.DefaultSerializeToJson();
        //1d. Save to JSON local file in TestData directory
        sqlSetup.DatabaseName.WriteJsonToJsonFile(jsonString);
    }
}

The things to note are:

Line 1: I use the RunnableInDebugOnly attribute (available in my EfCore.TestSupport library) to stop the unit test being run in a normal run of the unit tests. This method only needs to be run if the database scheme or data changes.
Line 4: the SqlServerProductionSetup class takes the name of a connection in the appsetting.json file and sets up the options for the given DbContext so that you can open it.
Line 9 to 14: Here I read in all the books with all their relationships that I want to save.
Lines 17 and 18: In this case the Resetter resets the primary keys and foreign keys to their default value. You need to do this to ensure EF Core works out the relationships via the navigational properties and creates new rows for all the data.
Line 21: This uses a default setting for Newtonsoft.Json’s SerializeObject method. This works in most cases, but you can write your own if you need different settings.
Line 23: Finally, it writes the data in to a file in the TestData folder of your unit tests. You can supply any unique string which is used as part of the file name – typically I use the name of the database it came from, which the SqlServerProductionSetup class provides.

2. Extract showing an example of using anonymisation

As I said before you might need to anonymise names, emails, addresses etc. that were in your production database. The DataResetter has a simple, but powerful system that allows you to define a series of properties/classes that need anonymising.

You define a class and a property in that class to anonymise and the DataResetter will traverse the whole sequence of relationships and will reset every instance of the class+property. As you will see you can define lots of classes/properties to be anonymised.

The default anonymisation method uses GUIDs as strings, so the name “Eric Evans” would be replaced with something like “2c7211292f2341068305309ff6783764”. That’s fine but it’s not that friendly if you want to do a demo or testing in general. That is why I provide a way to replace the default anonymisation method, which I show in the example (but you don’t have to if GUID strings are OK for you).

The code below is an example of what you can do by using an external library to provide random names and places. In this implementation I use the DotNetRandomNameGenerator NuGet package and create a few different formats you can call for, such as FirstName, LastName, FullName etc.

public class MyAnonymiser
{
    readonly PersonNameGenerator _pGenerator;

    public MyAnonymiser(int? seed = null)
    {
        var random = seed == null ? new Random() : new Random((int)seed);
        _pGenerator = new PersonNameGenerator(random);
    }

    public string AnonymiseThis(AnonymiserData data, object objectInstance)
    {
        switch (data.ReplacementType)
        {
            case "FullName": return _pGenerator.GenerateRandomFirstAndLastName();
            case "FirstName": return _pGenerator.GenerateRandomFirstName();
            case "LastName": return _pGenerator.GenerateRandomLastName();
            case "Email": //… etc. Add more versions as needed
            default: return _pGenerator.GenerateRandomFirstAndLastName();
        }
    }
}

The things to note are:

Lines 5 to 9: I designed my anonymiser to take an optional number to control the output. If no number is given, then the sequence of names has a random start (i.e. it produces different names each time it is run). If a number is given, then you get the same random sequence of names every time. Useful if you want to check properties in your unit test.

NOTE: See the Seed from Production Anonymization documentation link on the AnonymiserFunc and its features. There are several pieces of information I have not described here.

The code below shows an extract method, which is very similar to the first version, but with some extra code to a) link in the MyAnonymiser, and b) defines the class+property that needs to be anonymised.

[RunnableInDebugOnly(DisplayName = "Needs database XYZ")]
public void ExampleExtractWithAnonymiseLinkToLibrary()
{
    var sqlSetup = new SqlServerProductionSetup<BookContext>
        ("ExampleDatabaseConnectionName");
    using (var context = new BookContext(sqlSetup.Options))
    {
        //1a. Read in the data to want to seed the database with
        var entities = context.Books
            .Include(x => x.Reviews)
            .Include(x => x.Promotion)
            .Include(x => x.AuthorsLink)
                .ThenInclude(x => x.Author)
            .ToList();

        //1b-ii. Set up resetter config to use own method
        var myAnonymiser = new MyAnonymiser(42);
        var config = new DataResetterConfig
        {
            AnonymiserFunc = myAnonymiser.AnonymiseThis
        };
        //1b-ii. Add all class/properties that you want to anonymise
        config.AddToAnonymiseList<Author>(x => x.Name, "FullName");
        config.AddToAnonymiseList<Review>(x => x.VoterName, "FirstName");
        //1b. Reset primary and foreign keys and anonymise given class+property
        var resetter = new DataResetter(context, config);
        resetter.ResetKeysEntityAndRelationships(entities);

        //1c. Convert to JSON string
        var jsonString = entities.DefaultSerializeToJson();
        //1d. Save to JSON local file in TestData directory
        "ExampleDatabaseAnonymised".WriteJsonToJsonFile(jsonString);
    }
}

The things to note are:

Line 17: I create MyAnonymiser, and in this case I provide a see number. This means the same sequence of random names will be created whenever the extract is run. This can be useful if you access the anonymised properties in your unit test.
Lines 18 to 21: I override the default AnonymiserFunc by creating a DataResetterConfig class and setting the AnonymiserFunc properly to my replacement AnonymiserFunc from my MyAnonymiser class.
Lines 23 and 24: I add two class+property items that should be anonymised via the AddToAnonymiseList<T> method in the DataResetterConfig instance. As you can see you can provide a string that defines what type of replacement you want. In this case the Author’s Name needs a full name, e.g. “Jon Smith” and the Review’s VoterName just needs a first name, e.g. Jane.
Line 26: The creation of the DataResetter now has a second parameter with the DataResetterConfig instance with the new AnonymiserFunc.

All the rest of the method is the same.

NOTE: The ResetKeysEntityAndRelationships method follows all the navigational links so every instance of the given class+property that is linked to the root class will be reset. It also uses Reflection, so it can anonymise properties which have private setters.

3. Seed a unit test database from the JSON file

Now I show you a typical unit test where I seed the database from the data stored in the JSON file. In this case using a Sqlite in-memory, which is very fast to setup and run (see my article “Using in-memory databases for unit testing EF Core applications” for when and how you can use this type of database for unit testing).

[Fact]
public void ExampleSeedDatabase()
{
    //SETUP
    var options = SqliteInMemory.CreateOptions<BookContext>();
    using (var context = new BookContext(options))
    {
        //2a. make sure you have an empty database
        context.Database.EnsureCreated();
        //2b. read the entities back from the JSON file
        var entities = "ExampleDatabase".ReadSeedDataFromJsonFile<List<Book>>();
        //2c. Optionally “tweak” any specific data in the classes that your unit test needs
        entities.First().Title = "new title";
        //2d. Add the data to the database and save
        context.AddRange(entities);
        context.SaveChanges();

        //ATTEMPT
        //... run your tests here

        //VERIFY 
        //... verify your tests worked here
    }
}

The things to note are:

Line 9. In this case I am using an in-memory database, so it will be empty. If you are using a real database, then normally clear the database before you start so that the unit tests as a “known” starting point. (Note that you don’t have to clear the database for the seed stage to work – it will just keep adding a new copy of the snapshot every time, but your unit test database will grow over time.)
Line 11: my ReadSeedDataFromJsonFile extension method reads the json file with the reference “ExampleDatabase” (which was the name of the database that was imported – see extract code) and uses Newtonsoft.Json’s DeserializeObject method to turn the JSON back into entity classes with relationships.
Line 13: Optionally you might need to tweak the data specifically for the test you are going to run. That’s easy as you have access to the classes at this stage.
Line 15. You use Add, or AddRange if it’s a collection, to Add the new classes to the database.
Line 16. The last step is to call SaveChanges to get all the entities and relationships created in the database. EF Core will follow all the navigational links, like Reviews, to work out what is linked to what and set up the primary keys/foreign keys as required.
Lines 19 onward. This is where your tests and asserts go in your unit test.

4. Seed database when using DDD-styled entity classes

If you have read any of my other articles you will know I am a great fan of DDD-styled entity classes (see my article “Creating Domain-Driven Design entity classes with Entity Framework Core” for more about this). So, of course, I wanted the Seed from Production feature to work with DDD-styled classes, which it does now, but you do need to be careful so here are some notes about things.

Problem occur if Newtonsoft.Json can’t find a way to set a property at serialization time. This fooled me for quite a while (see the issue I raised on Newtonsoft.Json GitHub). The solution I came up with was adding a setting to the serialization (and deserialization) that tells Newtonsoft.Json that it can set via private setters. This works for me (including private fields mapped to IEnumerable), but in case you have a more complex state there are other ways to set this up. The most useful is to create a private constructor with parameters that match the properties by type and name, and then place a [JsonConstructor] attribute on that constructor (there are other ways too – look at Newtonsoft.Json docs).

NOTE: The symptoms of Newtonsoft.Json failing to serialize because it can’t access a property aren’t that obvious. In one case Newtonsoft.Json threw an unexplained “Self referencing loop detected” exception. And when I changed the JsonSerializerSettings to ignore self-referencing loops it incorrectly serialized the data by adding a duplicate (!). You can see the gory details in the issue I raised on Newtonsoft.Json.

Aside – how does this seeding work?

NOTE: This is an optional section – I thought you might like to learn something about EF Core and how it handles classes with relationships (known as “navigational links” in EF Core).

You may be wondering how this seeding feature works – basically it relies on some magic that EF Core performs.

Firstly, EF Core must work out what State a class is in, e.g. is it an Added (new) entry or an existing (tracked) entry, which tells it whether it needs to create a new entry or just refer to an existing entry in the database. This means EF Core sets the state of the class instance coming from the json as Added and will write them out tot the database.
The second part is how EF Core works out the relationships between each class instance. Because the ResetKeysEntityAndRelationships method reset the foreign keys (and the primary keys) EF Core relies on the navigational properties to work out the relationships between each class instance.

These two things mean that the database is updated with the correct data and foreign keys, even if the relationships are complex. This is feature makes EF Core so nice to work with not just in this feature but for any adding or updating of linked classes.

Here is a simple example taking from one of my talks with the code and two diagrams showing you the before and after. In this case I create a new book, with a new many-to-many BookAuthor to an existing Author.

var existingAuthor = context.Authors.First();
var newBook = new Book { Title = “New Book”};
newBook.AuthorLinks = new List<BookAuthor>
{ 
     new BookAuthor
     { 
          Book = newBook, 
          Author = existingAuthor, 
          Order = 1
     };
}

After that the classes look like this: note red classed are new, while blue have been read from the database (i.e. are tracked by EF Core).

Then we save this to the database with the following code.

context.Add(newBook);
context.SaveChanges();

After that the classes would look like this, with all the primary and foreign keys filled in and all navigational links set up.

EF Core has done the following to make this work:

During the Add stage is sets up all the navigation links and copies the primary keys of existing instances into the correct foreign keys (in this case the Author’s existing primary key into the AuthorId foreign key in the BookAuthor class).
In the SaveChanges part it does the following within a transaction
1. It inserts a new row for this Book, which sets its primary key
2. It then copies the Book’s primary key into the BookId foreign key in the BookAuthor class.
3. It then inserts a new row for the BookAuthor class.

This makes handling linked classes so simple in EF Core.

Conclusion

As I said earlier I had tried over the years different ways to set up complex databases, both with ADO.NET and EF6.x. EF Core has a number of features (good access to the database schema and better handling of adding linked classes to a database) which now make this much easier to implement a generic solution to this problem.

For my client the “Seed from Production” feature works really well. Their database contains data that is hard to create manually, and a pain to keep up to date as the application grows. By copying a database set up by the existing application we captured the data to use in our unit tests and some performance tests too. Also, the test data becomes part of the test code and therefore covered by the standard source control system. Another bonus is it makes it simple to run any tests in a DevOps pipeline as the test databases as it can be created and seeded automatically, which saves use from having to have specific database available in the DevOps path.

You won’t need this feature much, as most unit tests should use very basic data, but for those complex systems where setting up the database is complicated then this library can save you (and me) lots of time. And with anonymisation stage happening before the json file is created you don’t have to worry about having personal data in your unit tests.

Happy coding!

About six months ago I wrote the article “A better way to handle authorization in ASP.NET Core” which quickly became the top article on my web site, with lots of comments and questions. I also gave a talk at the NDC Oslo 2019 conference on the same topic which provided even more questions.

In response to all the questions I have developed a new, improved example application, called PermissionAccessControl2 (referred to as “version 2”) with series of new articles that cover the changes. The version 2 articles which cover improvements/changes based on questions and feedback.

Part 3: A better way to handle authorization – six months on (this article).
Part 4: Building robust and secure data authorization with EF Core (coming soon!).
Part 5: A better way to handle authorization – refreshing users claims (coming soon!).

Original articles:

Part 1: A better way to handle authorization in ASP.NET Core – original article.
Part 2: Handling data authorization in ASP.NET Core and Entity Framework Core.

NOTE: Some people had problems using the code in the original web application (referred to a version 1) in their applications, mainly because of the use of in-memory databases. The version 2 example code is more robust and supports real databases (e.g. SQL Server).

TL;DR; – summary (and links to sections)

This article answers comments/questions raised by the first version and covers the following changes from the original feature authorization article. If you are new to this topic the original article is a better place to start as it explains the whole system.
The changed parts covered in this article are:
- Simpler claims setup: Since the first article I have found a much simpler way to setup the users claims on login if you don’t need a refresh of the users claims.
- Roles: In the original article I used ASP.NET Core’s identity Roles feature, but that adds some limitations. The version 2 example app has its own UserToRole class.
- Using Permissions in the front-end: I cover how to use Permissions in Razor pages and how send the permissions to a JavaScript type front-end such as AngularJS, ReactJS etc.
- Provides a SuperUser: Typically, a web app needs a user with super-high privileges to allow them set up a new system. I have added that to the new example app.
The version 2 example app is a super-simple SaaS (Software as a Service) application which provides stock and sales management to companies with multiple retail outlets – see this section.

Setting the scene – my feature/data authorization approach

As I explained in the first article in this series, I was asked to build a web application that provided a service to various companies that worked in the retail sector. The rules about what the users could access were quite complex and required me to go beyond the out-of-box features in ASP.NET Core. Also, the data was multi-tenant and hierarchical, i.e. each company’s data must be secured such that a user the data they are allowed to access.

Many (but not all) of the issues I solved for my client are generally useful so, with the permission of my client, I worked on an open-source example application to capture the parts of the authentication and authorization features that I feel are generally useful to other developers.

If you not familiar with ASP.NET’s authorization and authentication features, then I suggest you read the ”Setting the Scene” section in the first article.

Below is a figure showing the various parts/stages in my approach to feature/data authorization. I use the ASP.NET Core authentication part, but I replace the authorization stage with my own code.

My replacement authorization stage provides a few extra features over ASP.NET Core Role-based authorization.

I use Roles to represent the type of user that is accessing the system. Typical Roles in a retail system might be SalesAssistant, Manager, Director, with other specific-job Roles like FirstAider, KeyHolder.
In the ASP.NET Core code I use something I call Permissions, which represent a specific “use case”, such as CanProcessSale, CanAuthorizeRefund. I place a single Permission on each ASP.NET Core pages/web APIs I want to protect. For instance, I would add CanProcessSale Permission to all the ASP.NET Core pages/web APIs than are used in the process sale use case.
The DataKey is more complex because of the hierarchical data and I cover all the bases on how to be sure that this multi-tenant system is secure – see part 4 (coming soon!)

There are other parts that I am not going to cover in this article as they are covered in other articles in this series. They are

Why using Enums to implement Permissions was a good idea (see this section in first article).
How I managed paid-for features (see this section in the first article).
How I setup and use a DataKey for segregate the data – see the part 4 article (coming soon!).

What the rest of this article does is deal with improvements I have made in the version 2 example application.

A simpler way to add to the User’s Claims

My approach relies on me adding some claims to the User, which is of type ClaimsPrincipal. In the original article I did this by using an Event in ApplicationCookie, mainly because that is what I had to use from my client. While that works I have found a much simpler way that adds the claims on login. This approach is much easier to write, works for Cookies and Tokens, and is more efficient. Thanks to https://korzh.com/blogs/net-tricks/aspnet-identity-store-user-data-in-claims for writing about this feature.

We do this by implementing a UserClaimsPrincipalFactory and registering it as a service. Here is my implementation of the UserClaimsPrincipalFactory.

public class AddPermissionsToUserClaims : 
UserClaimsPrincipalFactory<IdentityUser>
{
    private readonly ExtraAuthorizeDbContext _extraAuthDbContext;

    public AddPermissionsToUserClaims(UserManager<IdentityUser> userManager, 
        IOptions<IdentityOptions> optionsAccessor,
        ExtraAuthorizeDbContext extraAuthDbContext)
        : base(userManager, optionsAccessor)
    {
        _extraAuthDbContext = extraAuthDbContext;
    }

    protected override async Task<ClaimsIdentity> 
        GenerateClaimsAsync(IdentityUser user)
    {
        var identity = await base.GenerateClaimsAsync(user);
        var userId = identity.Claims
           .SingleOrDefault(x => x.Type == ClaimTypes.NameIdentifier).Value;
        var rtoPCalcer = new CalcAllowedPermissions(_extraAuthDbContext);
        identity.AddClaim(new Claim(
            PermissionConstants.PackedPermissionClaimType,
            await rtoPCalcer.CalcPermissionsForUser(userId)));
        var dataKeyCalcer = new CalcDataKey(_extraAuthDbContext);
        identity.AddClaim(new Claim(
            DataAuthConstants.HierarchicalKeyClaimName, 
            dataKeyCalcer.CalcDataKeyForUser(userId)));
        return identity;
    }
}

You can see on line 17 I get the original claims by calling the base GenerateClaimsAsync. I then use the UserId to calculate the Permissions and the DataKey, which I add to the original claims. After this method has finished the rest of the login code will build the Cookie or Token for the user.

To make this work you need to register in the Configure method inside the Startup code using the following code:

services.AddScoped<IUserClaimsPrincipalFactory<IdentityUser>, AddPermissionsToUserClaims>();

NOTE: The original code and this approach means that the claims are fixed until the user logs out and logs back in again. In the Part 5 article (coming soon!) I show ways to refresh the user’s claims when the roles/permissions have been changed by an admin person.

Adding our own UserToRole class

In the version 1 example app I used ASP.NET Core’s Role/RoleManger for a quick setup. But in a real application I wouldn’t do that as, I only want ASP.NET Core’s Identity system to deal with the authentication part.

The main reason for providing my own user to role class is because you can then do away with ASP.NET Core’s built-in identity database if you are using something like the OAuth 2 API (which my client system used). Also, if you are splitting the authorization part from the authentication part, then it makes sense to have the User-to-Roles links with all the other classes in the authentication part.

Note: Thanks to Olivier Oswald for his comments where he asked why I used the ASP.NET Core’s Role system in the first version. I was just being a bit lazy, so in version 2 I have done it properly.

In the version 2 example app I have my own UserToRole class, as shown below.

public class UserToRole
 {
     private UserToRole() { } //needed by EF Core

     public UserToRole(string userId, RoleToPermissions role) 
     {
         UserId = userId;
         Role = role;
     }
 
     public string UserId { get; private set; }
     public string RoleName { get; private set; }

     [ForeignKey(nameof(RoleName))]
     public RoleToPermissions Role { get; private set; }
    
     //… other code left out
}

The UserToRole class has a primary key which is formed from both the UserId and the RoleName (known as a “composite key”). I do this to stop duplicate entries linking a User to a Role, which would make managing the User’s Role more difficult.

I also have a method inside the UserToRole class called AddRoleToUser. This adds a Role to a User with check as adding a duplicate Role to a User will cause a database exception, so I catch that early and sent a user-friendly error message to the user.

Using Permissions in the front-end

The authorization side of ASP.NET returns HTTP 403 (forbidden) if a user isn’t allowed access to that method. But to make a better experience for a user we typically want to remove any links, buttons etc. that the user isn’t allowed to access. So how do I do that with my Permissions approach?

Here are the two ways you might be implementing your front-end, and how to handle each.

1. When using Razor syntax

If you are using ASP.NET Core in MVC mode or Razor Page mode, then you can use my extension method called UserHasThisPermission. The code below comes from the version 2 _layout.cshtml file and controls whether the Shop menu appears, and what sub-menu items appear.

@if (User.UserHasThisPermission(Permissions.SalesRead))
{
    <li class="nav-item">
        <div class="dropdown">
            <a class="nav-link text-dark dropdown-toggle" role="button" 
                id="dropdown1MenuButton" data-toggle="dropdown" 
                aria-haspopup="true" aria-expanded="false">
                Shop
            </a>       
            
            <div class="dropdown-menu" aria-labelledby="dropdown1MenuButton">
                @if (User.UserHasThisPermission(Permissions.SalesSell))
                {
                    <a class="nav-link text-dark" asp-area="" asp-controller="Shop" asp-action="Till">Till</a>
                }
                <a class="nav-link text-dark" asp-area="" asp-controller="Shop" asp-action="Stock">Stock</a>
                @if (User.UserHasThisPermission(Permissions.SalesRead))
                {
                    <a class="nav-link text-dark" asp-area="" asp-controller="Shop" asp-action="Sales">Sales</a>
                }
            </div>
        </div>
    </li>
}

You can see me using the UserHasThisPermission method on lines 1, 12 and 17.

2. Working with a JavaScript front-end framework

In some web applications you might use a JavaScript front-end framework such as AngularJS, ReactJS etc. to manage the front-end. In that case you need to pass the Permissions to your front-end system so that you can add code to control what link/buttons are shown to the user.

It is very simple to get access to the current user’s Permissions via the HttpContext.User variable which is available in any controller. I do this via a Web API and here is the code from my FrontEndController in my version 2 application.

[HttpGet]
public IEnumerable<string> Get()
{
    var packedPermissions = HttpContext.User?.Claims.SingleOrDefault(
        x => x.Type == PermissionConstants.PackedPermissionClaimType);
    return packedPermissions?.Value
        .UnpackPermissionsFromString()
        .Select(x => x.ToString());
}

This action returns an array of Permission names. Typically you would call this after a login and store the data in a SessionStorage for use while the user is logged in. You can try this in the version 2 application – run the application locally and go to http://localhost:65480/swagger/index.html. Swagger will then display the FrontEnd API which has one command to get the user’s permissions.

NOTE: In part 5, where the permissions can change dynamically I show a way that the front-end can detect that the permissions have changed so they can update their local version of the Permissions.

Enabling a SuperAdmin User

In most web application I have built you need one user that has access to every part of the system – I call this user SuperAdmin. And typically, I have some code that will make sure there is a SuperAdmin user in any system that the application runs. That way you can run the app with a new database and then use the SuperAdmin user to set up all the other users you need.

A logged in SuperAdmin User needs all the Permissions so that they can do anything, but that would be hard to keep updated as new permissions are added. Therefore, I added a special Permission called AccessAll and altered the UserHasThisPermission extension method to return true if the current user has the AccessAll Permission.

public static bool UserHasThisPermission(
    this Permissions[] usersPermissions, 
    Permissions permissionToCheck)
{
    return usersPermissions.Contains(permissionToCheck) 
        || usersPermissions.Contains(Permissions.AccessAll);
}

Now we have the concept of a SuperAdmin we need a way to create the SuperAdmin user. I do this via setup code in the Program class. This method makes sure that the SuperAdmin user is in the current user database, i.e. it adds a SuperUser if . Note: I am using the C# 7.1’s Main Async feature to run my startup code.

public static async Task Main(string[] args)
{
    (await BuildWebHostAsync(args)).Run();
}

public static async Task<IWebHost> BuildWebHostAsync(string[] args)
{
    var webHost = WebHost.CreateDefaultBuilder(args)
        .UseStartup<Startup>()
        .Build();

    //Because I might be using in-memory databases I need to make 
    //sure they are created before my startup code tries to use them
    SetupDatabases(webHost);

    await webHost.Services.CheckAddSuperAdminAsync();
    await webHost.Services.CheckSeedDataAndUserAsync();
    return webHost;
}

The CheckAddSuperAdminAsync obtains the SuperAdmin Email and password from the appsetting.json file (see Readme file for more information).

NOTE: The SuperAdmin user is very powerful and needs to be protected. Use a long, complicated password and make sure you provide the SuperAdmin email and password by overriding the appsettings.json values on deployment. My startup code also doesn’t allow a new SuperAdmin user to be added if there is a user already in the database that has the SuperAdmin role. That stops someone adding a new SuperAdmin with their own email address once a system is live.

Other things new in version 2 of the example application

The PermissionAccessControl2 application is designed to work like a real application, using both feature and data authorization. The application pretends to be a super-simple retail sales application service for multiple companies, i.e. a multi-tenant application. I cover the multi-tenant data authorization in part 4 (coming soon!).

The differences in version 2 of version 1 that I have not already mentioned are:

The code will work with either in-memory databases or SQL Server databases, i.e. it will check if users and data is already present and is so won’t try to add the user/data again
You can choose various application setups, such as database type and simple/complex claims setup. This is controlled by data in the appsettings.json file – see Readme file for more information on this.
The multi-tenant data access is hierarchical, with much more complex and robust than in version 1 – see Part 4: Handling hierarchical data authorization – six months on (coming soon!) for more on this.
There are some unit tests. Not a lot, but enough to give you an idea of what is happening.

Conclusion

The first article on my approach authorization in ASP.NET Core has been very popular, and I had great questions via my blog and at my NDC Oslo talk. This caused me to build a new version of the example app, available via a GitHub repo, with many improvements and some new articles that explains the changes/improvements.

In this article (known as Part 3) I focused on the ASP.NET Core authorization part and provided a few improvements over the version 1 application and added explanations of how to use these features. In article Part 4 I cover data authorization with EF Core, and the Part 5 article covers the complex area of updating a user’s permissions/data key dynamically.

Hopefully the whole series, with the two example applications, will help you design and build your own authorization systems to suit your needs.

Happy coding!

If you have a ASP.NET Core or Entity Framework Core problem that you want help on then I am available as a freelance contractor. Please send me a contact request via my Contact page and we can talk some more on Skype.

This article covers how to implement data authorization using Entity Framework Core (EF Core), that is only returning data from a database that the current user is allowed to access. The article focuses on how to implement data authorization in such a way that the filtering is very secure, i.e. the filter always works, and the architecture is robust, i.e. the design of the filtering doesn’t allow developer to accidently bypasses that filtering.

This article is part of a series and follows a more general article on data authorization I wrote six months ago. That first article introduced the idea of data authorization, but this article goes deeper and looks at one way to design a data authorization system that is secure and robust. It uses a new, improved example application, called PermissionAccessControl2 (referred to as “version 2”) and a series of new articles which cover other areas of improvement/change.

Part 3: A better way to handle authorization – six months on.
Part 4: Building robust and secure data authorization with EF Core (this article).
Part 5: A better way to handle authorization – refreshing users claims (coming soon!).

Original articles:

Part 1: A better way to handle authorization in ASP.NET Core – original article.
Part 2: Handling data authorization in ASP.NET Core and Entity Framework Core.

TL;DR; – summary

This article provides a very detailed description of how I build a hierarchical, multi-tenant application where the data a user could access depends on which company and what role they have in that company.
The example application is built using ASP.NET Core and EF Core. You can look at the code and run the application, which has demo data/users, by cloning this GitHub repo.
This article and its new (version 2) example application is a lot more complicated than the data authorization described in the original (Part 2) data authorisation article. If you want to start with something, then read the original (Part 2) first.
The key feature that makes it work are EF Core’s Query Filters, which provides a way to filter data in ALL EF Core database queries.
I break the article into two parts:
- Making it Secure, which covers how I implemented a hierarchical, multi-tenant application that filters data based on the user’s given data access rules.
- Making it robust, which is about designing an architecture that guides developers so that they can’t accidently bypass the security code that has been written.

Setting the scene – examples of data authorization

Pretty much every web application with a database will filter data – Amazon doesn’t show you every product it has, but tried to show you things you might be interested in. But this type of filtering for the convenience of the user and is normally part of the application code.

Another type of database filtering is driven from security concerns – I refer to this this as data authorization. This isn’t about filtering data for user convenience, but about applying strict business rules that that dictate what data a user can see. Typical scenarios where data authorization is needed are:

Personal data, where only the user can read/alter their personal data.
Multi-tenant systems, where one database is used to support multiple, separate user’s data.
Hierarchical systems, for instance a company with divisions where the CEO can see all the sales data, but each division can only see their own sales data.
Collaboration systems like GitHub/BitBucket where you can invite people to work with you on a project etc.

NOTE: If you are new to this area then please see this section of the original article where I have a longer introduction to data protection, both what’s it about and what the different part are.

My example is a both a multi-tenant and a hierarchical system, which is what one of my client’s needed. The diagram below shows two companies 4U Inc. and Pets2 Ltd. with Joe, our user in charge of the LA division of 4U’s outlets.

The rest of this article will deal with how to build an application which gives Joe access to the sales and stock situation in both LA outlets, but no access to any of the other divisions’ data or other tenants like Pets2 Ltd.

In the next sections I look at the two aspects: a) making my data authorization design secure, i.e. the filter always works, and, b) its architecture is robust, i.e. the design of the filtering doesn’t allow developer to accidently bypasses that filtering.

A. Building a secure data authorization system

I start with the most important part of the data authorization – making sure my approach is secure, that is, it will correctly filter out the data the user isn’t allowed to see. The cornerstone of this design is EF Core’s Query Filters, but to use them we need to set up a number of other things too. Here is a list of the key parts, which I then describe in detail:

Adding a DataKey to each filtered class: Every class/table that needs data authorization must have a property that holds a security key, which I call the DataKey.
Setting the DataKey property/column: The DataKey needs to be set to the right value whenever a new filtered class is added to the database.
Add the user’s DataKey to their claims: The user claims need to contain a security key that is matched in some way to the DataKey in the database.
Filter via the DataKey in EF Core: The EF Core DbContext has the user’s DataKey injected into it and it uses that key in EF Core Query Filters to only return the data that the user is allowed to see.

A1. Adding a DataKey to each filtered class

Every class/table that needs data authorization must have a property that holds a security key, which I call the DataKey.

In my example application there are multiple different classes that need to be filtered. I have a base IDataKey interface which defines the DataKey string property. All the classes to be filter inherit this interface. The DataKey definition looks like this

[Required]
[MaxLength(DataAuthConstants.HierarchicalKeySize)]
public string DataKey { get; private set; }

As you will see the DataKey is used a lot in this application.

A2. Setting the DataKey property/column

The DataKey needs to be set to the right value whenever a new filtered class is added to the database.

NOTE: This example is complex because of the hierarchical data design. If you want a simple example of setting a DataKey see the “personal data filtering” example in the original, Part 2 article.

Because of the hierarchical nature of my example the “right value” is bit complex. I have chosen to create a DataKey than is a combination of the primary keys of all the layers in the hierarchy. As you will see in the next subsection this allows the query filter to target different levels in the multi-tenant hierarchy.

The diagram below shows you the DataKeys (bold, containing numbers and |) for all the levels in the 4U Company hierarchy.

The hierarchy consists of three classes, Company, SubGroup and RetailOutlet, which all inherit from an abstract class called TenantBase. This allows me to create relationships between any of the class types that inherit from TenantBase (EF Core to treat these classes as a Table-Per-Hierarchy, TPH, and stores all the different types in one table).

But setting the DataKey on creation is difficult because the DataKey needs the primary key, which isn’t set until its created in the database. My way around this is to use a transaction. Here is the method in the TennantBase class that is called by the Company, SubGroup or RetailOutlet static creation factory.

protected static void AddTenantToDatabaseWithSaveChanges
    (TenantBase newTenant, CompanyDbContext context)
{
    // … Various business rule checks let out

    using (var transaction = context.Database.BeginTransaction())
    {
        //set up the backward link (if Parent isn't null)
        newTenant.Parent?._children.Add(newTenant);
        context.Add(newTenant);  //also need to add it in case its the company
        // Add this to get primary key set
        context.SaveChanges();

        //Now we can set the DataKey
        newTenant.SetDataKeyFromHierarchy();
        context.SaveChanges();

        transaction.Commit();
    }
}

The Stock and Sales classes are easier to handle, as they use the user’s DataKey. I override EF Core’s SaveChanges/SaveChangesAsync to do this, using the code shown below.

public override int SaveChanges(bool acceptAllChangesOnSuccess)
{
    foreach (var entityEntry in ChangeTracker.Entries()
        .Where(e => e.State == EntityState.Added))
    {
        if (entityEntry.Entity is IShopLevelDataKey hasDataKey)
            hasDataKey.SetShopLevelDataKey(accessKey);
    }
    return base.SaveChanges(acceptAllChangesOnSuccess);
}

A3. Add the user’s DataKey to their claims:

The user claims need to contain a security key that is matched in some way to the DataKey in the database.

My general approach as detailed in the original data authorization article is to have claim in the user’s identity that is used to filter data on. For personal data this can be the user’s Id (typically a string containing a GUID), or for a straight-forward multi-tenant it would by some form of tenant key stored in the user’s information. For this you might have the following class in your extra authorization data:

public class UserDataAccessKey
{
    public UserDataAccessKey(string userId, string accessKey)
    {
        UserId = userId ?? throw new ArgumentNullException(nameof(userId));
        AccessKey = accessKey;
    }

    [Key]
    [Required(AllowEmptyStrings = false)]
    [MaxLength(ExtraAuthConstants.UserIdSize)]
    public string UserId { get; private set; }

    [MaxLength(DataAuthConstants.AccessKeySize)]
    public string AccessKey { get; private set; }
}

In this hierarchical and multi-tenant example it gets a bit more complex, mainly because the hierarchy could change, e.g. a company might start with simple hierarchy of company to shops, but as it grows it might move a shop to sub-divisions like the west-coast. This means the DataKey can change dynamically.

For that reason, I link to the actual Tenant class that holds the DataKey that the user should use. This means that we can look up the current DataKey of the Tenant when the user logs in.

NOTE: In Part 5 I talk about how to dynamically update the user’s DataKey claim of any logged in user if the hierarchy they are in changes.

public class UserDataHierarchical
{
    public UserDataHierarchical(string userId, TenantBase linkedTenant)
    {
        if (linkedTenant.TenantItemId == 0)
            throw new ApplicationException(
                "The linkedTenant must be already in the database.");

        UserId = userId ?? throw new ArgumentNullException(nameof(userId));
        LinkedTenant = linkedTenant;
    }

    public int LinkedTenantId { get; private set; }

    [ForeignKey(nameof(LinkedTenantId))]
    public TenantBase LinkedTenant { get; private set; }

    [Key]
    [Required(AllowEmptyStrings = false)]
    [MaxLength(ExtraAuthConstants.UserIdSize)]
    public string UserId { get; private set; }

    [MaxLength(DataAuthConstants.AccessKeySize)]
    public string AccessKey { get; private set; }
}

This dynamic DataKey example might be an extreme case but seeing how I handled this might help you when you come across something that is more complex than a simple value.

At login you can add the feature and data claims to the user’s claims using the code I showed in Part 3, where I add to the user’s claims via a UserClaimsPrincipalFactory, as described in this section of the Part 3 article.

The diagram below shows how my factory method would lookup the UserDataHierarchical entry using the User’s Id and then adds the current DataKey of the linked Tenant.

A4. Filter via the DataKey in EF Core

The EF Core DbContext has the user’s DataKey injected into it and it uses that key in EF Core Query Filters to only return the data that the user is allowed to see.

EF Core’s Query Filters are a new feature added in EF Core 2.0 and they are fantastic for this job. You define a query filter in the OnModelCreating configuration method inside your DbContext and it will filter ALL queries, that comprises of LINQ queries, using Find method, included navigation properties and it even adds extra filter SQL to EF Core’s FromSql method (FromSqlRaw or FromSqlInterpolated in EF Core 3+). This makes Query Filters a very secure way to filter data.

For the version 2 example here is a look the CompanyDbContext class with the query filters set up by the OnModelCreating method towards the end of the code.

public class CompanyDbContext : DbContext
{
    internal readonly string DataKey;

    public DbSet<TenantBase> Tenants { get; set; }
    public DbSet<ShopStock> ShopStocks { get; set; }
    public DbSet<ShopSale> ShopSales { get; set; }

    public CompanyDbContext(DbContextOptions<CompanyDbContext> options,
        IGetClaimsProvider claimsProvider)
        : base(options)
    {
        DataKey = claimsProvider.DataKey;
    }

    //… override of SaveChanges/SaveChangesAsync left out

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        //… other configurations left out

        AddHierarchicalQueryFilter(modelBuilder.Entity<TenantBase>());
        AddHierarchicalQueryFilter(modelBuilder.Entity<ShopStock>());
        AddHierarchicalQueryFilter(modelBuilder.Entity<ShopSale>());
    }

    private static void AddHierarchicalQueryFilter<T>(EntityTypeBuilder<T> builder) 
         where T : class, IDataKey
    {
        builder.HasQueryFilter(x => x.DataKey.StartsWith(Datakey));
        builder.HasIndex(x => x.DataKey);
    }
}

As you can see the Query Filter uses the StartsWith method (line 30) to compare the user’s DataKey and the DataLeys of the tenants and their Sales/Stock data. This means if Joe has the DataKey of 1|2|5| then he can see the Stock/Sales data for the shops “LA Dress4U” and “LA Shirt4U” – see diagram below.

NOTE: You can try this by cloning the PermissionAccessControl2 repo and running it locally (by default it uses an in-memory to make it easy to run). Pick different users to see the different data/features you can access.

B. Building a robust data authorization architecture

We could stop here because we have covered all the code needed to secure the data. But I consider data authorization as a high-risk part of my system, so I want to make it “secure by design”, i.e. it shouldn’t be possible for a developer to accidently write something that bypasses the filtering. Here are the things I have done to make my code robust and guide another developer on how to do things.

Use Domain-Driven design database classes. Some of the code, especially creating the Company, SubGroup and RetailOutlet DataKeys, are complicated. I use DDD-styled classes which provides only one way to create or update the various DataKeys and relationships.
Build a filter-Only DbContext: I build a specific DbContext to contain all the classes/tables that need to be filtered.
Unit test to check you haven’t missed anything: I build unit tests that ensure the classes in the filter-Only DbContext have a query filter.
Adding fail-safe checks: With an evolving application some features are left for later. I often add small fail-safe checks in the original design to make sure any new features follow the original design approach.

B1. Use Domain-Driven design (DDD) database classes

DDD teaches us that each entity (a class in .NET world) should contain the code to create and update itself and its aggregates. Furthermore, it should stop any other external code from being able to bypass theses methods so that you are forced to use the methods inside the class. I really like this because it means there is one, and only one, method you can call to get certain jobs done.

The effect is I can “lock down” how something is done and make sure everyone uses the correct methods. Below the TenantBase abstract class which all the tenant classes inherit from showing the MoveToNewParent method that moves a tenant to another parent, for instance moving a RetailOutlet to a different SubGroup.

public abstract class TenantBase : IDataKey
{
    private HashSet<TenantBase> _children;

    [Key]
    public int TenantItemId { get; private set; }
    [MaxLength(DataAuthConstants.HierarchicalKeySize)]
    public string DataKey { get; private set; }
    public string Name { get; private set; }

    // -----------------------
    // Relationships 

    public int? ParentItemId { get; private set; }
    [ForeignKey(nameof(ParentItemId))]
    public TenantBase Parent { get; private set; }
    public IEnumerable<TenantBase> Children => _children?.ToList();

    //----------------------------------------------------
    // public methods

    public void MoveToNewParent(TenantBase newParent, DbContext context)
    {
        void SetKeyExistingHierarchy(TenantBase existingTenant)
        {
            existingTenant.SetDataKeyFromHierarchy();
            if (existingTenant.Children == null)
                context.Entry(existingTenant).Collection(x => x.Children).Load();

            if (!existingTenant._children.Any())
                return;
            foreach (var tenant in existingTenant._children)
            {                   
                SetKeyExistingHierarchy(tenant);
            }
        }

        //… various validation checks removed

        Parent._children?.Remove(this);
        Parent = newParent;
        //Now change the data key for all the hierarchy from this entry down
        SetKeyExistingHierarchy(this);
    }
    //… other methods/constructors left out
}

The things to note are:

Line 3: The Children relationship is held a private field which cannot be altered by outside code. It can be read via the IEnumerable<TenantBase> Children property (line 17) but you can’t add or remove children that way.
Lines 6 to 16: Similarly, all the properties have private setters, so it can only be changed by methods inside the TenantBase class.
Lines 22 to the end: This has the code to change the relationship and then immediately runs a recursive method to change all the DataKeys of the other tenants underneath this one.

B2. Build a filter-Only DbContext

One possible problem could occur if a non-filtered relationship was present, say a link back to some authorization code (there is such a relationship linking a tenant to a UserDataHierarchical class). If that happened a developer could write code that could expose data that the user shouldn’t see – for instance accessing the UserDataHierarchical class could expose the user’s Id which I wish to keep secret.

My solution was to create a separate DbContext for the multi-tenant classes, with a different (but overlapping) DbContext for the extra authorization classes (see the diagram below as to what this looks like). The effect is to make a multi-tenant DbContext which any contains the filtered multi-tenant data. For a developer makes it clear what classes you can access when using multi-tenant DbContext.

NOTE: having multiple DbContexts with a shared table can make database migrations a bit more complicated. Have a look at my article “Handling Entity Framework Core database migrations in production” for different ways to handle migrations.

B3. Using unit tests to check you haven’t missed anything

With feature and data authorization I add unit tests that check I haven’t left a “hole” in my security. Because I have a DbContext specifically for the multi-tenant data I can write a test to check that every class mapped to the database has a Query Filter applied to it. Here is the code I use for that.

[Fact]
public void CheckQueryFiltersAreAppliedToEntityClassesOk()
{
    //SETUP
    var options = SqliteInMemory.CreateOptions<CompanyDbContext>();
    using (var context = new CompanyDbContext(options, 
        new FakeGetClaimsProvider("accessKey")))
    {
        var entities = context.Model.GetEntityTypes().ToList();

        //ATTEMPT
        var queryFilterErrs = entities.CheckEntitiesHasAQueryFilter().ToList();

        //VERIFY
        queryFilterErrs.Any().ShouldBeFalse(string.Join('\n', queryFilterErrs));
    }
}

//Extension method to test all the classes mapped to the database
public static IEnumerable<string> CheckEntitiesHasAQueryFilter(
    this IEnumerable<IEntityType> entityTypes)
{
    foreach (var entityType in entityTypes)
    {
        if (entityType.QueryFilter == null
            && entityType.BaseType == null //not a TPH subclass
            && entityType.ClrType
                 .GetCustomAttribute<OwnedAttribute>() == null //not an owned type
            && entityType.ClrType
                 .GetCustomAttribute<NoQueryFilterNeeded>() == null) //Not marked as global
            yield return 
                 $"The entity class {entityType.Name} does not have a query filter";
    }
}

B4. Adding fail-safe checks

You need to be careful of breaking the Yagni (You Aren’t Gonna Need It) rule, but a few fail-safe checks on security stuff makes me sleep better at night. Here are the two small things I did in this example which will cause an exception if the DataKey isn’t set properly.

Firstly, I added a [Required] attribute to the DataKey property (see below) which tells the database that the DataKey cannot be null. This means if my code fails to set a DataKey then the database will return a constraint error.

[Required] //This means SQL will throw an error if null
[MaxLength(DataAuthConstants.HierarchicalKeySize)]
public string DataKey { get; private set; }

My second fail-safe is also to do with the DataKey, but in this case I’m anticipating a future change to the business rules that could cause problems. The current business rules say that only the users that are directly linked to a RetailOutlet can create new Stock or Sales entries, but what happens if (when!) that business rule changes and divisional managers can create items in a RetailOutlet. The divisional managers don’t have the correct DataKey, but a new developer might miss that and you could “lose” data.

My answer is to add a safely-check to the retail outlet’s DataKey. A retail outlet has a slightly different DataKey format – it ends with a * instead of a |. That means I can check a retail outlet format DataKey is used in the SetShopLevelDataKey and throw an exception if it’s not in the right format. Here is my code that catches this possible problem.

public void SetShopLevelDataKey(string key)
{
    if (key != null && !key.EndsWith("*"))
        //The shop key must end in "*" (or be null and set elsewhere)
        throw new ApplicationException(
             "You tried to set a shop-level DataKey but your key didn't end with *");

    DataKey = key;
}

This is a very small thing, but because I know that change is likely to come and I might not be around it could save someone a lot of head scratching working out why data doesn’t end up in the right place.

Conclusion

Well done for getting to the end of this long article. I could have made the article much shorter if I only dealt with the parts on how to implement data authorization, but I wanted to talk about how handling security issues should affect the way you build your application (what I refer to as have a “robust architecture”).

I have to say that the star feature in my data authorization approach is EF Core’s Query Filters. These Query Filters cover ALL possible EF Core based queries with no exceptions. The Query Filters are the cornerstone of data authorization approach which I then add a few more features to manage user’s DataKeys and do clever things to handle the hierarchical features my client needed.

While you most likely don’t need all the features I included in this example it does give you a look at how far you can push EF Core to get what you want. If you need a nice, simple data authorization example, then please look at the Part 2 article “Handling data authorization in ASP.NET Core and Entity Framework Core” which has a personal data example which uses the User’s Id as the DataKey.

Happy coding!

TR;DR; – summary

Setting the scene – A quick intro to Entity Framework Core

Part 1. Building a simple database query – a list Bloggers

1a. Using Eager loading to load the Post classes

1b. Using a hand-coded Select query

Summary of the performance

Quick to write – AutoMapper

1c. Using AutoMapper to build the Select query automatically

a) Create my DTO

b) Configure the AutoMapper mapping via dependency injection

c) Use AutoMapper in your query

Part 2: Handling more complex database queries – list of posts

2a. List posts using hand-coded select query

2b. List of posts using AutoMapper and special configuration

Conclusion

TL;DR; – summary

Setting the Scene – a look at different application security needs

Role authorization: what is it and what are its limitations?

The architecture of the Roles-to-Permissions system

1. Introducing Roles, Permissions and Modules

2. How this is implemented

Why I used Enums for the Permissions

How to handle optional/paid-for features?

How do I turn the Roles into a Permissions Claim?

How do I convert my Permissions into Policy-based authorization?

Putting it all together

Things I didn’t cover elsewhere

Conclusion

Further reading

TL;DR – summary

Who should read this article?

The problem that GenericServices tackles

1. The hand-coded version

2. The GenericServices version

How does GenericServices work inside?

How it read data from the database

How it creates a new entity or updates an existing entity

1. Create or update a standard style entity via a DTO

2. Create or update a DDD-styled entity via a DTO

What about the performance of GenericServices?

How hard is it to set up GenericServices?

Conclusion

TL;DR; – summary

Setting the scene

Introduction to DI – what is a service?

How to NetCore.AutoRegisterDi works

An approach to using AutoRegisterDi in real applications

Conclusion

TL;DR; – summary

Introduction to validation in EF

How to capture a database exception in EF Core

My generalized SaveChangesExceptionHandler

The SaveChangesWithValidation code

Enabling this feature in my GenericServices and GenericBizRunner

Other things you could do with this

Conclusion

TL;DR; – summary

Introduction to ASP.NET Core Web API

Going beyond Microsoft’s Web API examples

What about business logic?

How I turn my libraries’ statuses into HTTP responses

InvalidInput (HTTP 400) – error return

OK (HTTP 200) – successful

1. Create, and return url

2. Create, and return the created item

NoContent (HTTP 204) – when the requested data lookup returned null

Telling GenericServices that null is not an error

UPDATE: Now with unit test support

Conclusion

Introducing the hybrid DDD entity class

Altering a DDD-styled entity to allow JSON Patch to work

A look at the ASP.NET Core Web APIs

Conclusion

TL;DR; – summary

Comparing a standard entity class with a DDD approach

Pros and Cons of Standard & DDD approaches

Different views on building DDD-styled entity classes

1. The different ways of reading data

2. The different ways of updating relationships

1.a Handling adding a Review to POCO-only version