Azure Functions offers another hosting model called Azure Functions Premium Plan. With premium plans, instead of paying per function execution, you pay for the underlying compute instances that are hosting your functions. This is often more expensive, but it also ensures there are always a pre-set number of warmed instances ready to execute your function.
That's great, but what if I only really need those pre-warmed instances for a short period of time when I'm expecting a lot of incoming traffic. The rest of the time, I would rather use a Consumption Plan to save on hosting costs.
I thought the choice of hosting plan was something you needed to make up front but it turns out that you can actually move an Azure Function App from a consumption plan to a premium plan (and back again).
Thanks to Simon Timms for starting this discussion on Twitter. We got very helpful responses from folks on the Azure Functions team:
Jeff Hollan has a great sample using an Azure Durable Function to scale an Azure Function App to a premium plan for a specified amount of time, then automatically scale back down to a consumption plan.
— Jeff Hollan (@jeffhollan) April 2, 2020
This is a super cool sample. It uses the Azure Resource Manager REST API to make changes to the target function app resources. For my project however, I didn't really want to spin up another Azure Function to manage my Azure Functions. I just wanted an easy way to scale my 12 function apps up to premium plans for a couple hours, then scale them back down to a consumption plan.
I decided to try using the AZ CLI for this and it turned out really well. I was able to write a simple script to scale up and down.
First up, install the az cli.
Once installed, you'll need to login to your Azure Subscription.
1 | az login |
A browser window will popup, prompting you to log in to your Azure account. Once you've logged in, the browser window will close and the az cli will display a list of subscriptions available in your account. If you have more than one subscription, make sure you select the one you want to use.
1 | az account set --subscription YourSubscriptionId |
You will need a resource group for your Storage and CDN resources. If you don't already have one, create it here.
1 | az group create --name DavesFunctionApps --location WestUS2 |
Most commands will require you to pass in a --resource-group
and --location
parameters. These parameters are -g
and -l
for short, but you can save yourself even more keystrokes by setting defaults for az
.
1 | az configure -d group=DavesFunctionApps |
There is a strange requirement with Azure Functions / App Service. As per Jeff Hollan's sample:
The Azure Functions Premium plan is only available in a sub-set of infrastructure in each region. Internally we call these "webspaces" or "stamps." You will only be able to move your function between plans if the webspace supports both consumption and premium. To make sure your consumption and premium functions land in an enabled webspace you should create a premium plan in a new resource group. Then create a consumption plan in the same resource group. You can then remove the premium plan. This will ensure the consumption function is in a premium-enabled webspace.
First, add an Azure Functions Premium plan to the resource group.
1 | az functionapp plan create -n dave_temp_premium_plan --sku EP1 --min-instances 1 |
You can delete this premium plan using the command below after you've deployed a function app to this resource group . Don't forget to delete the premium plan. These cost $$$
1 | az functionapp plan delete -n dave_temp_premium_plan |
There are many options for creating a new function app. I really like the func
command line tool which I installed using npm. Check out the Azure Functions Core Tools GitHub Repo for details on other options for installing the func
tooling.
1 | npm i -g azure-functions-core-tools@3 --unsafe-perm true |
The focus of this blog post is around scaling a function app. If you don't already have an app built, you can follow along with this walkthrough to create a function app.
A function app requires a Storage Account resource. An Application Insights resource is also highly recommended as this really simplifies monitoring your function app after it has been deployed. Let's go ahead and create those 2 resources.
1 | az storage account create -n davefuncappstorage |
Now we can create our Azure Function App resource with a consumption plan, passing in the name of the storage account and app insights resources that we just created. In my case, I'm specifying the dotnet runtime on a Windows host.
1 | az functionapp create --consumption-plan-location WestUS2 --name davefuncapp123 --os-type Windows --runtime dotnet --storage-account davefuncappstorage --app-insights davefuncappinsights --functions-version 3 |
Remember to delete that temporary Premium Hosting Plan now!
1 | az functionapp plan delete -n dave_temp_premium_plan |
This is a bit outside the scope of this blog post but I like using the az
cli to deploy my function apps because it's easy to incorporate that into my CI/CD pipelines. Since my app is using the dotnet runtime, I use the dotnet publish
command to build the app.
1 | dotnet publish -c release |
Then, zip the contents of the publish folder (bin\release\netcoreapp3.1\publish\
).
In PowerShell:
1 | Compress-Archive -Path .\bin\release\netcoreapp3.1\publish\* -DestinationPath .\bin\release\netcoreapp3.1\package.zip |
or in Bash
1 | zip -r ./bin/release/netcoreapp3.1/package.zip ./bin/release/netcoreapp3.1/publish/ |
Finally, use the az functionapp deployment
command to deploy the function app.
1 | az functionapp deployment source config-zip -n davefuncapp123 --src ./bin/release/netcoreapp3.1/package.zip |
Okay, now that we have a functioning (pun intended) app deployed and running on a consumption plan, let's see what it takes to scale this thing up to a premium plan.
First, create a new Premium Hosting Plan with the parameters that make sense for the load you are expecting. The --sku
parameter refers to the size of the compute instance: EP1 is the smallest. The --min-instancs
parameter is the number of pre-warmed instances that will always be running for this hosting plan. The --max-burst
parameter is the upper bounds on the number of instances that the premium plan can elastically scale out if more instances are needed to handle load.
1 | az functionapp plan create -n davefuncapp123_premium_plan --sku EP1 --min-instances 4 --max-burst 12 |
Next, move the function app to that premium hosting plan.
1 | az functionapp update --plan davefuncapp123_premium_plan -n davefuncapp123 |
That's it! All it took was those 2 command and your function app is now running on a premium plan!
Of course, that premium plan isn't cheap. You might only want your function app running on the premium plan for a short period of time. Scaling back down is equally easy.
First, move the function app back to the consumption based plan. In my case, the name of the consumption plan is WestUS2Plan
. You should see a consumption plan in your resource group.
1 | az functionapp update --plan WestUS2Plan -n davefuncapp123 |
Next, delete the premium hosting plan.
1 | az functionapp plan delete -n davefuncapp123_premium_plan |
In this post, we saw how easy it is to move a function app between Premium and Consumption plans. A couple very simple az
commands can help you get the performance and features of the Premium plan only when you need it while taking advantages of the simplicity and cost savings of a Consumption plan the rest of the time.
First up, install the az cli.
Once installed, you'll need to login to your Azure Subscription.
1 | az login |
A browser window will popup, prompting you to log in to your Azure account. Once you've logged in, the browser window will close and the az cli will display a list of subscriptions available in your account. If you have more than one subscription, make sure you select the one you want to use.
1 | az account set --subscription YourSubscriptionId |
You will need a resource group for your Storage and CDN resources. If you don't already have one, create it here.
1 | az group create --name DavesFancyApp --location SouthCentralUs |
Most commands will require you to pass in a --resource-group
and --location
parameters. These parameters are -g
and -l
for short, but you can save yourself even more keystrokes by setting defaults for az
.
1 | az configure -d group=DavesFancyApp |
First, create a storage account:
1 | az storage account create --name davefancyapp123 |
Then, enable static site hosting for this account.
1 | az storage blob service-properties update --account-name davefancyapp123 --static-website --404-document 404.html --index-document index.html |
Your storage account will now have a blob container named $web
. That contents of that container will be available on the URL accountname.z21.web.core.windows.net/. For example, https://davefancyapp123.z21.web.core.windows.net/.
To deploy your app to the site, all you need to do is copy your app's static files to the $web
container in the storage account you created above. For my react app, that means running npm run build
and copying the build output to the $web
container.
1 | az storage blob upload-batch --account-name davefancyapp123 -s ./build -d '$web' |
Now your site should be available via the static hosting URL above. That was easy!
Next up, we are going to put a Content Delivery Network (CDN) endpoint in front of the blob storage account. We want to use a CDN for a couple reasons. First, it's going to provide much better performance overall. CDNs are optimized for delivering web content to user's devices and we should take advantage of that as much as possible. The second reason is that a CDN will allow us to configure SSL on a custom domain name.
First, we will need to create a CDN Profile. There are a few different of CDNs offerings available in Azure. You can read about them here. In this example, we will us the Standard Microsoft CDN.
1 | az cdn profile create -n davefancyapp123cdn --sku Standard_Microsoft |
Next, we will create the CDN endpoint. Here we need to set the origin to the static hosting URL from the previous step. Note that we don't include the protocol portion of the URL.
1 | az cdn endpoint create -n davefancyapp123cdnendpoint --profile-name davefancyapp123cdn --origin davefancyapp123.z21.web.core.windows.net --origin-host-header davefancyapp123.z21.web.core.windows.net --enable-compression |
Note: See the az cli docs for more information on the options available when creating a CDN endpoint.
Now your site should be available from endpointname.azureedge.net. In my case https://davefancyapp123cdnendpoint.azureedge.net/. Note that the endpoint is created quickly but it can take some time for the actual content to propagate through the CDN. You might initially get a 404 when you visit the URL.
These 2 steps are optional. The first one is highly recommended. The second is optional depending on the type of app your deploying.
First, create URL Redirect rule to redirect any HTTP requests to HTTPS.
1 | az cdn endpoint rule add -n davefancyapp123cdnendpoint --profile-name davefancyapp123cdn --rule-name enforcehttps --order 1 --action-name "UrlRedirect" --redirect-type Found --redirect-protocol HTTPS --match-variable RequestScheme --operator Equal --match-value HTTP |
Next, if you're deploying a Single Page Application (SPA) built in your favourite JavaScript framework (e.g. Vue, React, Angular), you will want a URL Rewrite rule that returns the app's root index.html
file for any request to a path that isn't an actual file. There are many variations on how to write this rule. I found this to be the simplest one that worked for me. Basically if the request path is not for a specific file with a file extension, rewrite to index.html
. This allows users to directly navigate to a route in my SPA and still have the CDN serve the index.html
that bootstraps the application.
1 | az cdn endpoint rule add -n davefancyapp123cdnendpoint --profile-name davefancyapp123cdn --rule-name sparewrite --order 2 --action-name "UrlRewrite" --source-pattern '/' --destination /index.html --preserve-unmatched-path false --match-variable UrlFileExtension --operator LessThan --match-value 1 |
The final step in configuring the CDN Endpoint is to configure a custom domain and enable HTTPS on that custom domain.
You will need access to update DNS records for the custom domain. Add a CNAME record for your subdomain that points to the CDN endpoint URL. For example, I created a CNAME record on my davepaquette.com domain:
1 | CNAME fancyapp davefancyapp123cdnendpoint.azureedge.net |
Once the CNAME record has been created, create a custom domain for your endpoint.
1 | az cdn custom-domain create --endpoint-name davefancyapp123cdnendpoint --profile-name davefancyapp123cdn -n fancyapp-domain --hostname fancyapp.davepaquette.com |
And finally, enable HTTPs. Unfortunately, this step fails due to a bug in the AZ CLI. There's a fix on it's way for this but it hasn't been merged into the CLI tool yet.
1 | az cdn custom-domain enable-https --endpoint-name davefancyapp123cdnendpoint --profile-name davefancyapp123cdn --name fancyapp-domain |
Due to the bug, this command returns InvalidResource - The resource format is invalid
. For now, you can do this step manually in the Azure Portal. When using CDN Managed Certificates, the process is full automated. Azure will verify your domain using the CNAME record above, provision a certificate and configure the CDN endpoint to use that certificate. Certificates are fully managed by Azure. That includes generating new certificates so you don't need to worry about your certificate expiring.
My biggest frustration with Azure CDN Endpoints is that CDN managed certificates are not supported for the apex/root domain. You can still use HTTPS but you need to bring your own certificate.
The same limitation exists for managed certificates on App Service. If you share my frustration, please upvote here.
The CDN will cache your files. That's great for performance but can be a royal pain when trying to deploy updates to your application. For SPA apps, I have found that simply telling the CDN to purge index.html
is enough to ensure updates are available very shortly after deploying a new version. This works because most JavaScript frameworks today use WebPack which does a good job of cache-busting your JavaScript and CSS assets. You just need to make sure the browser is able to get the latest version of index.html
and updates flow through nicely.
When you upload your latest files to blob storage, follow it with a purge command for index.html
on the CDN endpoint.
1 | az storage blob upload-batch --account-name davefancyapp123 -s ./build -d '$web' |
The purge command can take a while to complete. We pass the --no-wait
option so the command returns immediately.
Aside from the bug I ran in to with enabling HTTPS on the CDN endpoint, I've really enjoyed my experience with the az
cli. I was able to fully automate resource creation and deployments using the GitHub Actions az cli action. I can see az
becoming my preferred method of managing Azure resources.
For an ASP.NET Core process, the Application Insights SDK will automatically collect data about every request that the server process receives. This specific type of telemetry is called Request telemetry and it contains a ton of very useful data including: the request path, the HTTP verb, the response status code, the duration, the timestamp when the request was received.
The default data is great, but I often find myself wanting more information. For example, in a multi-tenant application, it would be very useful to track the tenant id as part of the request telemetry. This would allow us to filter data more effectively in the Application Insights portal and craft some very useful log analytics queries.
All types of telemetry in Application Insights provide an option to store custom properties. In the previous post, we saw how to create an ITelemetryInitializer
to set properties on a particular telemetry instance. We could easily add custom properties to our Request telemetry using a telemetry initializer.
1 | public class CustomPropertyTelemetryInitializer : ITelemetryInitializer |
Any custom properties you add will be listed under Custom Properties in the Application Insights portal.
But telemetry initializers are singletons and often don't have access to the useful data that we want to add to request telemetry. Typically the data we want is related in some way to the current request and that data wouldn't be available in a singleton service. Fortunately, there is another easy way to get an instance of the request telemetry for the current request.
1 | var requestTelemetry = HttpContext.Features.Get<RequestTelemetry>(); |
You can do it anywhere you have access to an HTTP Context. Some examples I have seen include: Middleware
, ActionFilters
, Controller
action methods, OnActionExecuting
in a base Controller
class and PageModel
classes in Razor Pages.
Once you've added custom properties to Request Telemetry, you can use those custom properties to filter data in the Application Insights portal. For example, you might want to investigate failures that are occurring for a specific tenant or investigate performance for a particular tenant.
This type of filtering can be applied almost anywhere in the portal and can help narrow things down when investigating problems.
Now this is where things get really interesting for me. What if we had one particular tenant complaining about performance. Wouldn't it be interesting to plot out the average request duration for all tenants? We can easily accomplish this using a log analytics query.
1 | requests |
This simple query will produce the following chart:
Small variations on this query can be extremely useful in comparing response times, failure rates, usage and pretty much anything else you can think of.
TenantId is just an example of a custom property. The custom properties that are useful for a particular application tend to emerge naturally as you're investigating issues and sifting through telemetry in Application Insights. You will eventually find yourself saying "I wish I knew what xxx
was for this request`. When that happens, stop and add that as a custom property to the request telemetry. You'll thank yourself later.
In any application that involves more than a single server process/service, the concept of Cloud Role becomes really important in Application Insights. A Cloud Role roughly represents a process that runs somewhere on a server or possibly on a number of servers. A cloud role made up of 2 things: a cloud role name and a cloud role instance.
The cloud role name is a logical name for a particular process. For example, I might have a cloud role name of "Front End" for my front end web server and a name of "Weather Service" for a service that is responsible for providing weather data.
When a cloud role name is set, it will appear as a node in the Application Map. Here is an example showing a Front End role and a Weather Service role.
However, when Cloud Role Name is not set, we end up with a misleading visual representation of how our services communicate.
By default, the application insights SDK attempts to set the cloud role name for you. For example, when you're running in Azure App Service, the name of the web app is used. However, when you are running in an on-premise VM, the cloud role name is often blank.
The cloud role instance tells us which specific server the cloud role is running on. This is important when scaling out your application. For example, if my Front End web server was running 2 instances behind a load balancer, I might have a cloud role instance of "frontend_prod_1" and another instance of "frontend_prod_2".
The application insights SDK sets the cloud role instance to the name of the server hosting the service. For example, the name of the VM or the name of the underlying compute instance hosting the app in App Service. In my experience, the SDK does a good job here and I don't usually need to override the cloud role instance.
Telemetry Initializers are a powerful mechanism for customizing the telemetry that is collected by the Application Insights SDK. By creating and registering a telemetry initializer, you can overwrite or extend the properties of any piece of telemetry collected by Application Insights.
To set the Cloud Role Name, create a class that implements ITelemetryInitializer
and in the Initialize
method set the telemetry.Context.Cloud.RoleName
to the cloud role name for the current application.
1 | public class CloudRoleNameTelemetryInitializer : ITelemetryInitializer |
Next, in the Startup.ConfigureServices
method, register that telemetry initializer as a singleton.
1 | services.AddSingleton<ITelemetryInitializer, CloudRoleNameTelemetryInitializer>(); |
For those who learn by watching, I have recorded a video talking about using telemetry initializers to customize application insights.
Creating a custom telemetry initializer to set the cloud role name is a simple enough, but it's something I've done so many times that I decided to publish a Nuget package to simplify it even further.
First, add the AspNetMonsters.ApplicationInsights.AspNetCore
Nuget package:
1 | dotnet add package AspNetMonsters.ApplicationInsights.AspNetCore |
Next, in call AddCloudRoleNameInitializer
in your application's Startup.ConfigureServices
method:
1 | services.AddCloudRoleNameInitializer("WeatherService"); |
Setting the Cloud Role Name / Instance is about a lot more than seeing your services laid out properly in the Application Map. It's also really important when you starting digging in to the performance and failures tabs in the Application Insights portal. In fact, on most of the sections of the portal, you'll see this Roles filter.
The default setting is all. When you click on it, you have the option to select any combination of your application's role names / instances. For example, maybe I'm only interested in the FrontEnd service and WeatherService that were running on the dave_yoga920 instance.
These filters are extremely useful when investigating performance or errors on a specific server or within a specific service. The more services your application is made up of, the more useful and essential this filtering become. These filters really help focus in on specific areas of an application within the Application Insights portal.
In this post, we saw how to customize telemetry data using telemetry initializers. Setting the cloud role name is a simple customization that can help you navigate the massive amount of telemetry that application insights collects. In the next post, we will explore a more in complex example of using telemetry initializers.
]]>Application Insights has built-in support for .NET, Java, Node.js, Python, and Client-side JavaScript based applications. This blog post is specifically about .NET applications. If you're application is built in another language, head over to the docs to learn more.
With codeless monitoring, you can configure a monitoring tool to run on the server (or service) that is hosting your application. The monitoring tool will monitor running processes and collect whatever information is available for that particular platform. There is built in support for Azure VM and scale sets, Azure App Service, Azure Cloud Services, Azure Functions, Kubernetes applications and On-Premises VMs. Codeless monitoring is a good option if you want to collect information for applications that have already been built and deployed, but you are generally going to get more information using Code-based monitoring.
With code-based monitoring, you add the Application Insights SDK. The steps for adding the SDK are well document for ASP.NET Core, ASP.NET, and .NET Console applications so I don't need to re-hash that here.
If you prefer, I have recorded a video showing how to add Application Insights to an existing ASP.NET Core application.
Once you've added the Application Insights SDK to your application, it will start collecting telemetry data at runtime and sending it to Application Insights. That telemetry data is what feeds the UI in the Application Insights portal. The SDK will automatically collection information about your dependencies calls to SQL Server, HTTP calls and calls to many popular Azure Services. It's the dependencies that often are the most insightful. In a complex system it's difficult to know exactly what dependencies your application calls in order to process an incoming request. With App Insights, you can see exactly what dependencies are called by drilling in to the End-to-End Transaction view.
In addition to dependencies, the SDK will also collect requests, exceptions, traces, customEvents, and performanceCounters. If your application has a web front-end and you add the JavaScript client SDK, you'll also find pageViews and browserTimings.
The SDK decides which Application Insights instance to send the collected telemetry based on the configured Instrumentation Key.
In the ASP.NET Core SDK, this is done through app settings:
1 | { |
When you're diagnosing an issue in production or investigating performance in your production systems, you don't want any noise from your development or staging environments. I always recommend creating an Application Insights resource per environment. In the Azure Portal, you'll find the instrumentation key in the top section of the Overview page for your Application Insights resource. Just grab that instrumentation key and add it to your environment specific configuration.
Consider a micro-services type architecture where your application is composed of a number of services, each hosted within it's own process. It might be tempting to have each service point to a separate instance of Application Insights.
Contrary to the guidance of separating your environments, you'll actually get the most value from Application Insights if you point all your related production services to a single Application Insights instance. The reason for this is that Application Insights automatically correlates telemetry so you can track a particular request across a series of separate services. That might sound a little like magic but it's not actually as complicated as it sounds.
It's this correlation that allows the Application Map in App Insights to show exactly how all your services interact with each other.
It also enables the end-to-end transaction view to show a timeline of all the calls between your services when you are drilling in to a specific request.
This is all contingent on all your services sending telemetry to the same Application Insights instance. The Application Insights UI in the Azure Portal has no ability to display this visualizations across multiple Application Insights instances.
I've often heard developers say "I can't use Application Insights because we're not on Azure". Well, you don't need to host your application on Azure to use Application Insights. Yes, you will need an Azure subscription for the Application Insights resource, but your application can be hosted anywhere. That includes your own on-premise services, AWS or any other public/private cloud.
Out of the box, Application Insights provides a tremendous amount of value but I always find myself having to customize a few things to really get the most out of the telemetry. Fortunately, the SDK provides some useful extension points. My plan is to follow up this post with a few more posts that go over those customizations in detail. I also have started to create a NuGet package to simplify those customizations so stay tuned!
Other posts in this series:Setting Cloud Role NameEnhancing Application Insights Request Telemetry
]]>After my recent misadventures attempting to use Noda Time with Entity Framework Core, I decided to see what it would take to use Dapper in a the same scenario.
In my app, I needed to model an Event
that occurs on a particular date. It might be initially tempting to store the date of the event as a DateTime in UTC, but that's not necessarily accurate unless the event happens to be held at the Royal Observatory Greenwich. I don't want to deal with time at all, I'm only interested in the date the event is being held.
NodaTime provides a LocalDate
type that is perfect for this scenario so I declared a LocalDate
property named Date
on my Event
class.
1 | public class Event |
I modified my app to query for the Event
entities using Dapper:
1 | var queryDate = new LocalDate(2019, 3, 26); |
The app started up just fine, but gave me an error when I tried to query for events.
System.Data.DataException: Error parsing column 1 (Date=3/26/19 12:00:00 AM - DateTime) ---> System.InvalidCastException: Invalid cast from 'System.DateTime' to 'NodaTime.LocalDate'.
Likewise, if I attempted to query for events using a LocalDate
parameter, I got another error:
1 | var queryDate = new LocalDate(2019, 3, 26); |
NotSupportedException: The member Date of type NodaTime.LocalDate cannot be used as a parameter value
Fortunately, both these problems can be solved by implementing a simple TypeHandler
.
Out of the box, Dapper already knows how to map to the standard .NET types like Int32, Int64, string and DateTime. The problem we are running into here is that Dapper doesn't know anything about the LocalDate
type. If you want to map to a type that Dapper doesn't know about, you can implement a custom type handler. To implement a type handler, create a class that inherits from TypeHandler<T>
, where T
is the type that you want to map to. In your type handler class, implement the Parse
and SetValue
methods. These methods will be used by Dapper when mapping to and from properties that are of type T
.
Here is an example of a type handler for LocalDate
.
1 | public class LocalDateTypeHandler : TypeHandler<LocalDate> |
Finally, you need to tell Dapper about your new custom type handler. To do that, register the type handler somewhere in your application's startup class by calling Dapper.SqlMapper.AddTypeHandler
.
1 | Dapper.SqlMapper.AddTypeHandler(new LocalDateTypeHandler()); |
As it turns out, someone has already created a helpful NuGet package containing TypeHandlers for many of the NodaTime types so you probably don't need to write these yourself. Use the Dapper.NodaTime package instead.
TypeHandlers are a simple extension point that allows for Dapper to handle types that are not already handled by Dapper. You can write your own type handlers but you might also want to check if someone has already published a NuGet package that handles your types.
]]>DateTime
class. For example, how to I represent a date in .NET when I don't care about the time. There is no type that represents a Date
on it's own. That's why the Noda Time library was created, billing itself as a better date and time API for .NET.Noda Time is an alternative date and time API for .NET. It helps you to think about your data more clearly, and express operations on that data more precisely.
In my app, I needed to model an Event
that occurs on a particular date. It might be initially tempting to store the date of the event as a DateTime in UTC, but that's not necessarily accurate unless the event happens to be held at the Royal Observatory Greenwich. I don't want to deal with time at all, I'm only interested in the date the event is being held.
NodaTime provides a LocalDate
type that is perfect for this scenario so I declared a LocalDate
property named Date
on my Event
class.
1 | public class Event |
This app was using Entity Framework Core and there was a DbSet
for the Event
class.
1 | public class EventContext : DbContext |
This is where I ran into my first problem. Attempting to run the app, I was greeted with a friendly InvalidOperationException
:
InvalidOperationException: The property 'Event.Date' could not be mapped, because it is of type 'LocalDate' which is not a supported primitive type or a valid entity type. Either explicitly map this property, or ignore it using the '[NotMapped]' attribute or by using 'EntityTypeBuilder.Ignore' in 'OnModelCreating'.
This first problem was actually easy enough to solve using a ValueConverter
. By adding the following OnModelCreating
code to my EventContext
, I was able to tell Entity Framework Core to store the Date
property as a DateTime
with the Kind set to DateTimeKind.Unspecified
. This has the effect of avoiding any unwanted shifts in the date time based on the local time of the running process.
1 | public class EventContext : DbContext |
With that small change, my application now worked as expected. The value conversions all happen behind the scenes so I can just use the Event
entity and deal strictly with the LocalDate
type.
I actually had this application running in a test environment for a week before I noticed a serious problem in the log files.
In my app, I was executing a simple query to retrieve the list of events for a particular date.
1 | var queryDate = new LocalDate(2019, 3, 25); |
In the app's log file, I noticed the following warning:
Microsoft.EntityFrameworkCore.Query:Warning: The LINQ expression 'where ([e].Date == __queryDate_0)' could not be translated and will be evaluated locally.
Uh oh, that sounds bad. I did a little more investigation and confirmed that the query was in fact executing SQL without a WHERE
clause.
1 | SELECT [e].[Id], [e].[Date], [e].[Description], [e].[Name] |
So my app was retrieving EVERY ROW from Events
table, then applying the where filter in the .NET process. That's really not what I intended to do and would most certainly cause me some performance troubles when I get to production.
So, the first thing I did was modified my EF Core configuration to throw an error when a client side evaluation like this occurs. I don't want this kind of thing accidently creeping in to this app again. Over in Startup.ConfigureServices
, I added the following option to ConfigureWarnings
.
1 | services.AddDbContext<EventContext>(options => |
Throwing an error by default is the correct behavior here and this is actually something that will be fixed in Entity Framework Core 3.0. The default behavior in EF Core 3 will be to throw an error any time a LINQ expression results in client side evaluation. You will then have the option to allow those client side evaluations.
Now that I had the app throwing an error for this query, I needed to find a way for EF Core to properly translate my simple e.Date == queryDate
expression to SQL. After carefully re-reading the EF Core documentation related for value converters, I noticed a bullet point under Limitations:
Use of value conversions may impact the ability of EF Core to translate expressions to SQL. A warning will be logged for such cases. Removal of these limitations is being considered for a future release.
Well that just plain sucks. It turns out that when you use a value converter for a property, Entity Framework Core just gives up trying to convert any LINQ expression that references that property. The only solution I found was to query for my entities using SQL.
1 | var queryDate = new LocalDate(2019, 3, 25); |
NodaTime is a fantastic date and time library for .NET and you should definitely consider using it in your app. Unfortunately, Entity Framework Core has some serious limitations when it comes to using value converters so you will need to be careful. I almost got myself into some problems with it. While there are work-arounds, writing custom SQL for any query that references a NodaTime type is less than ideal. Hopefully those will be addressed in Entity Framework Core 3.
]]>In today's post, we explore a pattern to prevent multiple users (or processes) from accidentally overwriting each other's change. Given our current implementation for updating the Aircraft
record, there is potential for data loss if there are multiple active sessions are attempting to update the same Aircraft
record at the same time. In the example shown below, Bob accidentally overwrites Jane's changes without even knowing that Jane made changes to the same Aircraft
record
The pattern we will use here is Optimistic Offline Lock, which is often also referred to as Optimistic Concurrency Control.
To implement this approach, we will use a rowversion column in SQL Server. Essentially, this is a column that automatically version stamps a row in a table. Any time a row is modified, the rowversion
column will is automatically incremented for that row. We will start by adding the column to our Aircraft
table.
1 | ALTER TABLE Aircraft ADD RowVer rowversion |
Next, we add a RowVer
property to the Aircraft
table. The property is a byte
array. When we read the RowVer
column from the database, we will get an array of 8 bytes.
1 | public class Aircraft |
Finally, we will modify the query used to load Aircraft
entities so it returns the RowVer
column. We don't need to change any of the Dapper code here.
1 | public async Task<Aircraft> Get(int id) |
Now that we have the row version loaded in to our model, we need to add the checks to ensure that one user doesn't accidentally overwrite another users changes. To do this, we simply need to add the RowVer
to the WHERE
clause on the UPDATE
statement. By adding this constraint to the WHERE
clause, we we ensure that the updates will only be applied if the RowVer
has not changed since this user originally loaded the Aircraft
entity.
1 | public async Task<IActionResult> Put(int id, [FromBody] Aircraft model) |
So, the WHERE
clause stops the update from happening, but how do we know if the update was applied successfully? We need to let the user know that the update was not applied due to a concurrency conflict. To do that, we add OUTPUT inserted.RowVer
to the UPDATE
statement. The effect of this is that the query will return the new value for the RowVer
column if the update was applied. If not, it will return null.
1 | public async Task<IActionResult> Put(int id, [FromBody] Aircraft model) |
Instead of calling ExecuteAsync
, we call ExecuteScalarAsync<byte[]>
. Then we can check if the returned value is null
and raise a DBConcurrencyException
if it is null. If it is not null, we can return the new RowVer
value.
Using SQL Server's rowversion
column type makes it easy to implement optimistic concurrency checks in a .NET app that uses Dapper.
If you are building as REST api, you should really use the ETag header to represent the current RowVer for your entity. You can read more about this pattern here.
]]>In today's post, we explore a more complex scenario that involves executing multiple write operations. In order to ensure consistency at the database level, these operations should all succeed / fail together as a single transaction. In this example, we will be inserting a new ScheduledFlight
entity along with an associated set of Flight
entities.
As a quick reminder, a Flight
represents a particular occurrence of a ScheduledFlight
on a particular day. That is, it has a reference to the ScheduledFlight
along with some properties indicating the scheduled arrival and departure times.
1 | public class Flight |
1 | public class ScheduledFlight |
Inserting the ScheduledFlight
and retrieving the database generated id is easy enough. We can use the same approach we used in the previous blog post.
1 | // POST api/scheduledflight |
According to the bosses at Air Paquette, whenever we create a new ScheduledFlight
entity, we also want to generate the Flight
entities for the next 12 months of that ScheduledFlight
. We can add a method to the ScheduledFlight
class to generate the flight entities.
NOTE: Let's just ignore the obvious bugs related to timezones and to flights that take off and land on a different day.
1 | public IEnumerable<Flight> GenerateFlights(DateTime startDate, DateTime endDate) |
Now in the controller, we can add some logic to call the GenerateFlight
method and then insert those Flight
entities using Dapper.
1 | // POST api/scheduledflight |
Note that we passed in an IEnumerable<Flight>
as the second argument to the ExecuteAsync
method. This is a handy shortcut in Dapper for executing a query multiple times. Instead of writing a loop and calling ExecuteAsync
for each flight entity, we can pass in a list of flights and Dapper will execute the query once for each item in the list.
So far, we have code that first inserts a ScheduledFlight
, next generates a set of Flight
entities and finally inserting all of those Flight
entities. That's the happy path, but what happens if something goes wrong along the way. Typically when we execute a set of related write operations (inserts, updates and deletes), we want those operations to all succeed or fail together. In the database world, we have transactions to help us with this.
The nice thing about using Dapper is that it uses standard .NET database connections and transactions. There is no need to re-invent the wheel here, we can simply use the transaction patterns that have been around in .NET since for nearly 2 decades now.
After opening the connection, we call connection.BeginTransaction()
to start a new transaction. Whenever we call ExecuteAsync
(or any other Dapper extension method), we need to pass in that transaction. At the end of all that work, we call transaction.Commit()
. Finally, we wrap the logic in a try / catch
block. If any exception is raised, we call transaction.Rollback()
to ensure that none of those write operations are committed to the database.
1 | [ ] |
Managing database transactions in .NET is a deep but well understood topic. We covered the basic pattern above and showed how Dapper can easily participate in a transaction. To learn more about managing database transactions in .NET, check out these docs:
Using transactions with Dapper is fairly straight forward process. We just need to tell Dapper what transaction to use when executing queries. Now that we know how to use transactions, we can look at some more advanced scenarios like adding concurrency checks to update operations to ensure users aren't overwriting each other's changes.
]]>In today's post, we explore how easy it is to perform basic Insert, Update and Delete operations using the same Aircraft
entity that we used in the first post in this series. Basically, instead of using Dapper's QueryAsync
extension method that we used to retrieve data, we will use the ExecuteAsync
method.
As a quick reminder, here is the Aircraft
class:
1 | public class Aircraft |
NOTE: In these examples, I am ignoring some important aspects like validation. I want to focus specifically on the Dapper bits here but validation is really important. In a real-world scenario, you should be validating any data that is passed in to the server. I recommend using Fluent Validation.
Inserting a single new record is really easy. All we need to do is write an INSERT
statement with parameters for each column that we want to set.
1 | [ ] |
The version of the ExecuteAsync
method we used here accepts two parameters: a string containing the SQL statement to execute and an object containing the parameter values to bind to the statement. In this case, it is an instance of the Aircraft
class which has properties with names matching the parameters defined in the INSERT
statement.
Our Aircraft
table's Id
column is an auto-incremented identity column. That means the primary key is generated by the database when the row is inserted. We will likely need to pass that value back to whoever called the API so they know how to retrieve the newly inserted Aircraft
.
An easy way to get the generated Id
is to add SELECT CAST(SCOPE_IDENTITY() as int)
after the INSERT
statement. The SCOPE_IDENTITY()
function returns the last identity value that was generated in any table in the current session and current scope.
Now, since the SQL statement we are executing will be returning a single value (the generated id), we need to call ExecuteScalarAsync<int>
. The ExecuteScalarAsync
method executes a SQL statement that returns a single value whereas the ExecuteAsync
method executes a SQL statement that does not return a value.
1 | [ ] |
Updating an existing entity is similar to inserting. All we need is a SQL statement containing an UPDATE
statement that sets the appropriate columns. We also want to make sure we include a WHERE
clause limiting the update only to the row with the specified Id
.
Again, the parameters in the SQL statement match the names of the properties in our Aircraft
class. All we need to do is call the ExecuteAsync
method passing in the SQL statement and the Aircraft
entity.
1 | // PUT api/aircraft/id |
Deleting an entity is the easiest of the three operations since it only requires a single parameter: the unique Id to identify the entity being deleted. The SQL statement is a simple DELETE
with a WHERE
clause on the Id
column. To execute the delete, call the ExecuteAsync
method passing in the SQL statement and an anonymous object containing the Id
to delete.
1 | // DELETE api/aircraft/id |
I really appreciate how simple delete is using Dapper. When using Entity Framework, delete requires you to first fetch the existing entity, then delete it. That requires 2 round trips to the database while the approach we used here only requires a single round trip.
Basic insert, update and delete operations are easy to implement using Dapper. Real world scenarios are often a little more complex and we will dig into some of those scenarios in future posts:
In today's post, we explore paging through large result sets. Paging is a common technique that is used when dealing with large results sets. Typically, it is not useful for an application to request millions of records at a time because there is no efficient way to deal with all those records in memory all at once. This is especially true when rendering data on a grid in a user interface. The screen can only display a limited number of records at a time so it is generally a bad use of system resources to hold everything in memory when only a small subset of those records can be displayed at any given time.
Source: AppStack Bootstrap Template
Modern versions of SQL Server support the OFFSET / FETCH clause to implement query paging.
In continuing with our airline theme, consider a Flight
entity. A Flight
represents a particular occurrence of a ScheduledFlight
on a particular day. That is, it has a reference to the ScheduledFlight
along with some properties indicating the scheduled arrival and departure times.
1 | public class Flight |
1 | public class ScheduledFlight |
As we learned in a previous post, we can load the Flight
entity along with it's related ScheduledFlight
entity using a technique called multi-mapping.
In this case, loading all the flights to or from a particular airport, we would use the following query.
1 | SELECT f.*, sf.* |
But this query could yield more results than we want to deal with at any given time. Using OFFSET/FETCH, we can ask for only a block of results at a time.
1 | SELECT f.*, sf.* |
Note that an ORDER BY clause is required when using OFFSET/FETCH.
1 | //GET api/flights |
Here we calculate the offset by based on the page
and pageSize
arguments that were passed in. This allows the caller of the API to request a particular number of rows and the starting point.
When dealing with paged result sets, it can be useful for the caller of the API to also know the total number of records. Without the total number of records, it would be difficult to know how many records are remaining which in turn makes it difficult to render a paging control, a progress bar or a scroll bar (depending on the use case).
A technique I like to use here is to have my API return a PagedResults<T>
class that contains the list of items for the current page along with the total count.
1 | public class PagedResults<T> |
To populate this using Dapper, we can add a second result set to the query. That second result set will simply be a count of all the records. Note that the same WHERE clause is used in both queries.
1 | SELECT f.*, sf.* |
Now in our code that executes the query, we will the QueryMultipleAsync
method to execute both SQL statements in a single round trip.
1 | //GET api/flights |
Paged result sets is an important technique when dealing with large amounts of data. When using a full ORM like Entity Framework, this is implemented easily using LINQ's Skip
and Take
methods. It's so easy in fact that it can look a little like magic. In reality, it is actually very simple to write your own queries to support paged result sets and execute those queries using Dapper.
The default MVC project templates already include jQuery, so you might use that. You'll probably end up writing a lot of code if you go down that path. Chances are, you will want to use a JavaScript framework that offers two-way data binding between the elements in the DOM to a your model data.
It seems that for many people, Knockout.js is the default library to use in these scenarios. I won't get into the specifics but I think that Knockout is a little dated and that there are better options these days. If you want to dig into some of the issues with Knockout, you can read Simon Timms' rant on the subject.
My fellow ASP.NET Monster James Chambers recently strongly recommended I take a look at Vue.js. I had been meaning to give Vue a try for some time now and I finally had a chance to use it on a recent project. Let me tell you...I love it.
I love it for a whole bunch of reasons. The documentation is great! It is super easy to get drop in to your existing project and it doesn't get in the way. For what I needed to do, it just did the job and allowed me to get on with my day. It is also designed to be "incrementally adoptable", which means you can start out with just using the core view layer, then start pulling in other things like routing and state management if/when you need them.
I won't go into great detail about how to use Vue. If you want a full tutorial, head on over to the Vue docs. What I want to show here is just how simple it is to drop Vue into an existing ASP.NET MVC project and add a bit of client side functionality.
The simplest example I can think of is a set of cascading dropdowns. Let's consider a form where a user is asked to enter their Country / Province. When the Country is selected, we would expect the Province dropdown to only display the valid Provinces/States for the selected Country. That probably involves a call to an HTTP endpoint that will return the list of valid values.
1 | public class ProvinceLookupController : Controller |
The easiest way to include Vue on a particular Razor view is to link to the vue.js
file from a CDN. You can add the following Script tag to your scripts
section.
1 | @section scripts { |
Be sure to check the docs to make sure you are referencing the latest version of Vue.
Now that you have included the core Vue library, you can start using Vue to bind DOM elements to model data.
Start by defining a Vue
object in JavaScript. You can add this in a new <script>
tag in your scripts
section.
1 | @section scripts { |
This Vue
object targets the DOM element with id vueApp
and contains some simple data. The currently selected country code and the list of countries.
Now, back in the HTML part of your csthml, wrap the form
in a div that has an id="vueApp"
.
1 | <div id="vueApp"> |
Next, bind the <select>
element to the data in your Vue
object. In Vue, data binding is done using a combination of custom attributes that start with v-
and the double curly bracket (aka. Mustache) syntax for text.
1 | <div class="form-group"> |
Now, when you run the app, you should see a dropdown containing Canada and United States.
Next, you will want to add some client side logic to get the list of valid provinces from the server whenever the selected country changes.
First, add an empty provinces
array and a selectedProvinceCode
property to the Vue
object's data.
Next, add a method called countryChanged
to the Vue
object. This method will call the ProvinceLookup
action method on the server, passing in the selectedCountryCode
as a parameter. Assign the response data to the provinces
array.
1 | var app = new Vue({ |
Here I used jQuery to make the call to the server. In the Vue community, Axios is a popular library for making HTTP requests.
Back in the HTML, bind the change
event from the country select element to the countryChanged
method using the v-on:change
attribute.
1 | <select id="CountryCode" name="CountryCode" class="form-control" |
Now you can add a select element for the provinces.
1 | <div class="form-group"> |
Voila! You now have a working set of cascading dropdowns.
You might want to disable the provinces dropdown whenever a request is being made to get the list of provinces for the selected country. You can do this by adding an isProvincesLoading
property to the Vue
object's data, then setting that property in the countryChanged
method.
1 | var app = new Vue({ |
In your HTML, bind the disabled
attribute to the isProvincesLoading
property.
1 | <div class="form-group"> |
Here is the entire cshtml file.
1 | @{ |
I hope this gives you a taste for how easy it is to work with Vue. My current thinking is that Vue should be the default choice for client side frameworks in existing ASP.NET MVC apps.
NOTE: This example uses ASP.NET MVC 5 to illustrate that Vue can be used with brownfield applications. It would be just as easy, if not easier, to use Vue in an ASP.NET Core project.
]]>I wanted to install the site extension that enables Application Insights Monitoring of a live website. After digging into existing arm templates, I found the name of that extension is Microsoft.ApplicationInsights.AzureWebSites
.
After searching for way too long, I eventually found PowerShell command I needed on a forum somewhere. I can't find it again so I'm posting this here in hopes that it will be easier for others to find in the future.
1 | New-AzureRmResource -ResourceType "Microsoft.Web/sites/siteextensions" -ResourceGroupName MyResourceGroup -Name "MyWebApp/SiteExtensionName" -ApiVersion "2018-02-01" -Force |
For example, given a resource group named Test
, a web app named testsite
and a site extension named Microsoft.ApplicationInsights.AzureWebSites
.
1 | New-AzureRmResource -ResourceType "Microsoft.Web/sites/siteextensions" -ResourceGroupName "Test" -Name "testsite/Microsoft.ApplicationInsights.AzureWebSites" -ApiVersion "2018-02-01" -Force |
The scenario I ran into was actually attempting to add this site extension to a deployment slot. When you create a deployment slot, it doesn't copy over any existing site extensions, which is a problem because when you swap your new slot to production, your new production slot ends up losing the site extensions that were in the old production slot.
1 | New-AzureRmResource -ResourceType "Microsoft.Web/sites/slots/siteextensions" -ResourceGroupName MyResourceGroup -Name "MyWebApp/SlotName/SiteExtensionName" -ApiVersion "2018-02-01" -Force |
Using the same example as above and a slot named Staging
:
1 | New-AzureRmResource -ResourceType "Microsoft.Web/sites/slots/siteextensions" -ResourceGroupName "Test" -Name "testsite/Staging/Microsoft.ApplicationInsights.AzureWebSites" -ApiVersion "2018-02-01" -Force |
Update: April 16, 2018 Something really cool happened in the comments. The amazing Phil Bolduc very kindly pointed out that the query I wrote was not optimal and as a result, my benchmarks were not showing the best results. He didn't stop there, he also submitted a pull request to the sample repo so I could rerun my benchmarks. Great job Phil and thanks a ton for being constructive in the comments section! I have updated the post to include Phil's superior query.
In today's post, we look at another option for how to load Many-to-One relationships. In the last post, we used a technique called Multi-Mapping to load related Many-to-One entities. In that post, I had a theory that maybe this approach was not the most efficient method for loading related entities because it duplicated a lot of data.
To recap, we would like to load a list of ScheduledFlight
entities. A ScheduleFlight
has a departure Airport
and an arrival Airport
.
1 | public class ScheduledFlight |
In the previous post, we loaded the ScheduledFlight
entities and all related Airport
entities in a single query. In this example we will use 2 separate queries: One for the ScheduledFlight
entities, one for the related arrival and departure Airport
entities. These 2 queries will all be executed as a single sql command that returns multiple result sets.
1 | SELECT s.Id, s.FlightNumber, s.DepartureHour, s.DepartureMinute, s.ArrivalHour, s.ArrivalMinute, s.IsSundayFlight, s.IsMondayFlight, s.IsTuesdayFlight, s.IsWednesdayFlight, s.IsThursdayFlight, s.IsFridayFlight, s.IsSaturdayFlight, |
Using Dapper's QueryMultipleAsync
method, we pass in 2 arguments: the query and the parameters for the query.
1 | public async Task<IEnumerable<ScheduledFlight>> GetAlt(string from) |
The QueryMultipleAsync
method returns a GridReader
. The GridReader
makes it very easy to map mutliple result sets to different objects using the Read<T>
method. When you call the Read<T>
method, it will read all the results from the next result set that was returned by the query. In our case, we call Read<ScheduledFlight>
to read the first result set and map the results into a collection of ScheduledFlight
entities. Next, we call Read<Airport>
to read the second result set. We then call ToDictionary(a => a.Id)
to populate those Airport
entities into a dictionary. This is to make it easier to read the results when setting the ArrivalAirport
and DepartureAirport
properties for each ScheduledFlight
.
Finally, we iterate through the scheduled flights and set the ArrivalAirport
and DepartureAirport
properties to the correct Airport
entity.
The big difference between this approach and the previous approach is that we no longer have duplicate instances for Airport
entities. For example, if the query returned 100 scheduled flights departing from Calgary (YYC), there would be a single instance of the Airport
entity representing YYC, whereas the previous approach would have resulted in 100 separate instances of the Airport
entity.
There is also less raw data returned by the query itself since the columns from the Airport
table are not repeated in each row from the ScheduleFlight
table.
I had a theory that the multi-mapping approach outlined in the previous blog post would be less efficient than the multiple result set approach outlined in this blog post, at least from a memory usage perspective. However, a theory is just theory until it is tested. I was curious and also wanted to make sure I wasn't misleading anyone so I decided to test things out using Benchmark.NET. Using Benchmark.NET, I compared both methods using different sizes of data sets.
I won't get into the details of Benchmark.NET. If you want to dig into it in more detail, visit the official site and read through the docs. For the purposes of this blog post, the following legend should suffice:
1 | Mean : Arithmetic mean of all measurements |
Method | Mean | Error | StdDev | Gen 0 | Allocated |
---|---|---|---|---|---|
MultiMapping | 397.5 us | 3.918 us | 4.192 us | 5.8594 | 6.77 KB |
MultipleResultSets | 414.2 us | 6.856 us | 6.077 us | 4.8828 | 6.69 KB |
As I suspected, the difference is minimal when dealing with small result sets. The results here are in microseconds so in both cases, executing the query and mapping the results takes less 1/2 a millisecond. The mutliple result sets approach takes a little longer, which I kind of expected because of the overhead of creating a dictionary and doing lookups into that dictionary when setting the ArrivalAirport
and DepartureAirport
properties. The difference is minimal and in a most real world scenarios, this won't be noticable. What is interesting is that even with this small amount of data, we can see that there is ~1 more Gen 0 garbage collection happening per 1,000 operations. I suspect we will see this creep up as the amount of data increases.
Method | Mean | Error | StdDev | Gen 0 | Gen 1 | Allocated |
---|---|---|---|---|---|---|
MultiMapping | 1.013 ms | 0.0200 ms | 0.0287 ms | 25.3906 | 5.8594 | 6.77 KB |
MultipleResultSets | 1.114 ms | 0.0220 ms | 0.0225 ms | 15.6250 | - | 6.69 KB |
Method | Mean | Error | StdDev | Gen 0 | Allocated |
---|---|---|---|---|---|
MultiMapping | 926.5 us | 21.481 us | 32.804 us | 25.3906 | 6.77 KB |
MultipleResultSets | 705.9 us | 7.543 us | 7.056 us | 15.6250 | 6.69 KB |
When mapping 100 results, the multiple result sets query is already almost 25% faster. Keep in mind though that both cases are still completing in less than 1ms so this is very much still a micro optimization (pun intented). Either way, less than a millsecond to map 100 records is crazy fast.
Method | Mean | Error | StdDev | Gen 0 | Gen 1 | Allocated |
---|---|---|---|---|---|---|
MultiMapping | 5.098 ms | 0.1135 ms | 0.2720 ms | 148.4375 | 70.3125 | 6.77 KB |
MultipleResultSets | 2.809 ms | 0.0549 ms | 0.0674 ms | 109.3750 | 31.2500 | 6.69 KB |
Here we go! Now the multiple result sets approach finally wins out, and you can see why. There are way more Gen 0 and Gen 1 garbage collections happening per 1,000 operations when using the multi-mapping approach. As a result, the multiple result sets approach is nearly twice as fast as the multi mapping approach.
Method | Mean | Error | StdDev | Gen 0 | Gen 1 | Gen 2 | Allocated |
---|---|---|---|---|---|---|---|
MultiMapping | 56.08 ms | 1.5822 ms | 1.4026 ms | 1687.5000 | 687.5000 | 187.5000 | 6.78 KB |
MultipleResultSets | 24.93 ms | 0.1937 ms | 0.1812 ms | 843.7500 | 312.5000 | 125.0000 | 6.69 KB |
One last test with 10,000 records shows a more substantial difference. The multiple result sets approach is a full 22ms faster!
I think that in most realistic scenarios, there is no discernable difference between the 2 approaches to loading many-to-one related entities. If you loading larger amounts of records into memory in a single query, then the multiple result sets approach will likely give you better performance. If you are dealing with < 100 records per query, then you likely won't notice a difference. Keep in mind also that your results will vary depending on the specific data you are loading.
]]>In today's post, we will start our journey into more complex query scenarios by exploring how to load related entities. There are a few different scenarios to cover here. In this post we will be covering the Many-to-One scenario.
Continuing with our sample domain for the ever expanding Air Paquette airline, we will now look at loading a list of ScheduledFlight
entities. A ScheduleFlight
has a departure Airport
and an arrival Airport
.
1 | public class ScheduledFlight |
Side Note: Let's ignore my poor representation of the arrival and departure times of the scheduled flights. In a future most we might look using Noda Time to properly represent these values.
Using Dapper, we can easily load a list of ScheduledFlight
using a single query. First, we need to craft a query that returns all the columns for a ScheduledFlight
, the departure Airport
and the arrival Airport
in a single row.
1 | SELECT s.Id, s.FlightNumber, s.DepartureHour, s.DepartureMinute, s.ArrivalHour, s.ArrivalMinute, s.IsSundayFlight, s.IsMondayFlight, s.IsTuesdayFlight, s.IsWednesdayFlight, s.IsThursdayFlight, s.IsFridayFlight, s.IsSaturdayFlight, |
We use the QueryAsync
method to load a list of ScheduledFlight
entities along with their related DepartureAirport
and ArrivalAirport
entities. The parameters we pass in are a little different from what we saw in our previous posts.
1 | [ ] |
First, instead of a single type parameter <ScheduledFlight>
, we need to provide a series of type parameters: <ScheduledFlight, Airport, Airport, ScheduledFlight>
. The first 3 parameters specify the types that are contained in each row that the query returns. In this example, each row contains columns that will be mapped to ScheduledFlight
and 2 Airports
. The order matters here, and Dapper assumes that when it seems a column named Id
then it is looking at columns for the next entity type. In the example below, the columns from Id
to IsSaturdayFlight
are mapped to a ScheduledFlight
entity. The next 5 columns Id, Code, City, ProvinceState, Country
are mapped to an Airport
entity, and the last 5 columns are mapped to a second Airport
entity. If you aren't using Id
, you can use the optional splitOn
argument to specify the column names that Dapper should use to identity the start of each entity type.
What's that last type parameter? Why do we need to specify ScheduledFlight
again? Well, I'm glad you asked. The thing about Dapper is that it doesn't actually know much about the structure of our entities so we need to tell it how to wire up the 3 entities that it just mapped from a row. That last ScheduledFlight
type parameter is telling Dapper that ScheduledFlight
is ultimately the entity we want to return from this query. It is important for the second argument that is passed to the QueryAsync
method.
That second argument is a function that takes in the 3 entities that were mapped back from that row and returns and entity of the type that was specified as the last type parameter. In this case, we assign the first Airport
to the flight's DepartureAirport
property and assign the second Airport
to the flight's ArrivalAiport
parameter, then we return the flight that was passed in.
1 | (flight, departure, arrival ) => { |
The first argument argument passed to the QueryAsync
method is the SQL query, and the third argument is an anonymous object containing any parameters for that query. Those arguments are really no different than the simple examples we saw in previous blog posts.
Dapper refers to this technique as Multi Mapping. I think it's called that because we are mapping multiple entities from each row that the query returns. In a fully featured ORM like Entity Framework, we call this feature Eager Loading. It is an optimization technique that avoids the need for multiple queries in order to load an entity and it's associated entities.
This approach is simple enough to use and it does reduce the number of round trips needed to load a set of entities. It does, however, come at a cost. Specifically, the results of the query end up causing some duplication of data. As you can see below, the data for the Calgary and Vancouver airports is repeated in each row.
This isn't a huge problem if the result set only contains 3 rows but it can become problematic when dealing with large result sets. In addition to creating somewhat bloated result sets, Dapper will also create new instances of those related entities for each row in the result set. In the example above, we would end up with 3 instances of the Airport
class representing YYC - Calgary and 3 instances of the Airport
class representing YVR - Vancouver. Again, this isn't necessarily a big problem when we have 3 rows in the result set but with larger result sets it could cause your application to use a lot more memory than necessary.
It is worth considering the cost associated with this approach. Given the added memory cost, this approach might be better suited to One-to-One associations rather than the Many-to-One example we talked about in this post. In the next post, we will explore an alternate approach that is more memory efficient but probably a little more costly on the CPU for the mapping.
]]>Let's just get this one out of the way early. Stored procedures are not my favorite way to get data from SQL Server but there was a time when they were extremely popular. They are still heavily used today and so this series would not be complete without covering how to use stored procedures with Dapper.
Let's imagine a simple stored procedure that allows us to query for Aircraft
by model.
1 | CREATE PROCEDURE GetAircraftByModel @Model NVARCHAR(255) AS |
To execute this stored procedure and map the results to a collection of Aircraft
objects, use the QueryAsync
method almost exactly like we did in the last post.
1 | //GET api/aircraft |
Instead of passing in the raw SQL statement, we simply pass in the name of the stored procedure. We also pass in an object that has properties for each of the stored procedures arguments, in this case new {Model = model}
maps the model
variable to the stored procedure's @Model
argument. Finally, we specify the commandType
as CommandType.StoredProcedure
.
That's all there is to using stored procedures with Dapper. As much as I dislike using stored procedures in my applications, I often do have to call stored procedures to fetch data from legacy databases. When that situation comes up, Dapper is my tool of choice.
Stay tuned for the next installment in this Dapper series. Comment below if there is a specific topic you would like covered.
]]>Today, we will start with the basics of loading a mapping and database table to a C# class.
Dapper calls itself a simple object mapper for .NET and is usually lumped into the category of micro ORM (Object Relational Mapper). When compared to a fully featured ORM like Entity Framework, Dapper lacks certain features like change-tracking, lazy loading and the ability to translate complex LINQ expressions to SQL queries. The fact that Dapper is missing these features is probably the single best thing about Dapper. While it might seem like you're giving up a lot, you are also gaining a lot by dropping those types of features. Dapper is fast since it doesn't do a lot of the magic that Entity Framework does under the covers. Since there is less magic, Dapper is also a lot easier to understand which can lead to lower maintenance costs and maybe even fewer bugs.
Throughout this series we will build on an example domain for an airline. All airlines need to manage a fleet of aircraft, so let's start there. Imagine a database with a table named Aircraft
and a C# class with property names that match the column names of the Aircraft
table.
1 | CREATE TABLE Aircraft |
1 | public class Aircraft |
Dapper is available as a Nuget package. To use Dapper, all you need to do is add the Dapper
package to your project.
.NET Core CLI: dotnet add package Dapper
Package Manager Console: Install-Package Dapper
Dapper provides a set of extension methods for .NET's IDbConnection
interface. For our first task, we want to execute a query to return the data for a single row from the Aircraft
table and place the results in an instance of the Aircraft
class. This is easily accomplished using Dapper's QuerySingleAsync
method.
1 | [ ] |
Before we can call Dapper's QuerySingleASync
method, we need an instance of an open SqlConnection
. If you are an Entity Framework user, you might not be used to working directly with the SqlConnection
class because Entity Framework generally manages connections for you. All we need to do is create a new SqlConnection
, passing in the connection string, then call OpenAsync
to open that connection. We wrap the connection in a using
statement to ensure that connection.Dispose()
is called when we are done with the connection. This is important because it ensures the connection is returned to the connection pool that is managed by .NET. If you forget to do this, you will quickly run into problems where your application is not able to connect to the database because the connection pool is starved. Check out the .NET Docs for more information on connection pooling.
We will use the following pattern throughout this series of blogs posts:
1 | using(var connection = new SqlConnection(_connectionString)) |
As @Disman pointed out in the comments, it is not necessary to call connection.OpenAsync()
. If the connection is not already opened, Dapper will call OpenAsync
for you. Call me old fashioned but I think that whoever created the connection should be the one responsible for opening it, that's why I like to open the connection before calling Dapper.
Let's get back to our example. To query for a single Aircraft
, we call the QuerySingleAsync
method, specifying the Aircraft
type parameter. The type parameter tells Dapper what class type to return. Dapper will take the results of the query that gets executed and map the column values to properties of the specified type. We also pass in two arguments. The first is the query that will return a single row based on a specified @Id
parameter.
1 | SELECT |
The next parameter is an anonymous class containing properties that will map to the parameters of the query.
1 | new {Id = id} |
Passing the parameters in this way ensures that our queries are not susceptible to SQL injection attacks.
That's really all there is to it. As long as the column names and data types match the property of your class, Dapper takes care of executing the query, creating an instance of the Aircraft
class and setting all the properties.
If the query doesn't contain return any results, Dapper will throw an InvalidOperationException
.
InvalidOperationException: Sequence contains no elements
If you prefer that Dapper returns null when there are no results, use the QuerySingleOrDefaultAsnyc
method instead.
Querying for a list of objects is just as easy as querying for a single object. Simply call the QueryAsync
method as follows.
1 | [ ] |
In this case, the query did not contain any parameters. If it did, we would pass those parameters in as an argument to the QueryAsync
method just like we did for the QuerySingleAsync
method.
This is just the beginning of what I expect will be a long series of blog posts. You can follow along on this blog and you can track the sample code on GitHub.
Leave a comment below if there is a topic you would like me to cover.
]]>Using the IAuthorizationService
in ASP.NET Core, it is easy to implement an authorization strategy that depends not only on properties of the User but also depends on the resource being accessed. To learn how resource-based authorization works, take a look at the well written offical documentation.
Once you have defined your authorization handlers and setup any policies in Startup.ConfigureServices
, applying resource-based authorization is a matter of calling one of two overloads of the AuthorizeAsync
method on the IAuthorizationService
.
1 | Task<AuthorizationResult> AuthorizeAsync(ClaimsPrincipal user, |
One method takes in a policy name while the other takes in an IAuthorizationRequirement
. The resulting AuthorizationResult
has a Succeeded
boolean that indicates whether or not the user meets the requirements for the specified policy. Using the IAuthorizationService
in a controller is easy enough. Simply inject the service into the controller, call the method you want to call and then check the result.
1 |
|
Using this approach, we can easily restrict which users can edit specific documents as defined by our EditDocument policy. For example, we might limit editing to only users who originally created the document.
Where things start to get a little ugly is if we want to render a UI element based on resource-based authorization. For example, we might only want to render the edit button for a document if the current user is actually authorized to edit that document. Out of the box, this would require us to inject the IAuthorizationService
in the Razor view and use it like we did in the controller action. The approach works, but the Razor code will get ugly really fast.
Similar to the Authorize Tag Helper from the last blog post, this Authorize Resource Tag Helper will make it easy to show or hide blocks of HTML by evaluating authorization rules.
Let's assume we have a named "EditDocument" that requires a user to be the original author of a Document
in order to edit the document. With the authorize resource tag helper, specify the resource instance using the asp-authorize-resource
attribute and the policy name using the asp-policy
attribute. Here is an example where Model
is an instance of a Document
1 | <a href="#" asp-authorize-resource="Model" |
If the user meets the requirments for the "EditDocument" policy and the specified resource, then the block of HTML will be sent to the browser. If the requirements are not met, the tag helper will suppress the output of that block of HTML. The tag helper can be applied to any HTML element.
Instead of specifying a policy name, authorization can be evaluated by specifying an instance of an IAuthorizationRequirement
. When using requirements directly instead of policies, specify the requirement using the asp-requirement
attribute.
1 | <a href="#" asp-authorize-resource="document" |
If the user meets Operations.Delete
requirement for the specified resource, then the block of HTML will be sent to the browser. If the requirement is not met, the tag helper will suppress the output of that block of HTML. The tag helper can be applied to any HTML element.
The authorize resource tag helper itself is fairly simple. The implementation will likely evolve after this blog post so you can check out the latest version here.
The tag helper needs an instance of the IHttpContextAccessor
to get access to the current user and an instance of the IAuthorizationService
. These are injected into the constructor. In the ProcessAsync
method, either the specified Policy
or the specified Requirement
are passed in to the IAuthorizationService
along with the resource.
1 | [ ] |
Note that either a policy or a requirement must be specified along with a resource, but you can't specify both a policy AND a requirement. Most of the code in the ProcessAsync
method is checking the argument values to make sure a valid combination was used.
You can see the authorize resource tag helper in action on my tag helper samples site here. The sample site contains the examples listed in this blog post and also provides a way to log in as different users to test different scenarios.
The authorize resource tag helper is also available on NuGet so you can use it in your own ASP.NET Core application.
1 | dotnet add package TagHelperSamples.Authorization |
Let me know what you think. Would you like to see this tag helper including in the next release of ASP.NET Core?
NOTE: If you choose to use the authorize resource tag helper in your application, you should remember that hiding a section of HTML is not enough to fully secure your application. You also need to make sure that resource-based authorization is applied to any related controllers and action methods.
There is one more authorization scenario related to supporting different authorization schemes that I hope to cover. Watch out for that in a future blog post. Also, this tag helper project is all open source so feel free to jump in on GitHub.
]]>[Authorize]
attribute. This attribute provides a simple way to ensure only authorized users are able to access certain parts of your application. While the [Authorize]
attribute makes it easy to control authorization for an entire page, the mechanism for controlling access to a section of a page is a little clumsy, involving the use of a the IAuthorizationService
and writing C# based if
blocks in your Razor code.In this blog post, we build an Authorize tag helper that makes it easy to control access to any block HTML in a Razor view.
The basic idea of this tag helper is to provide similar functionality to the [Authorize]
attribute and it's associated action filter in ASP.NET Core MVC. The authorize tag helper will provide the same options as the [Authorize]
attribute and the implementation will be based on the authorize filter. In the MVC framework, the [Authorize]
attribute provides data such as the names of roles and policies while the authorize filter contains the logic to check for roles and policies as part of the request pipeline. Let's walk through the most common scenarios.
In it's simplest form, adding the [Authorize]
attribute to a controller or action method will limit access to that controller or action method to users who are authenticated. That is, only users who are logged in will be able to access those controllers or action methods.
With the Authorize tag helper, we will implement a similar behaviour. Adding the asp-authorize
attribute to any HTML element will ensure that only authenticated users can see that that block of HTML.
1 | <div asp-authorize class="panel panel-default"> |
If a user is not authenticated, the tag helper will suppress the output of that entire block of HTML. That section of HTML will not be sent to the browser.
The [Authorize]
attribute provides an option to specify the role that a user must belong to in order to access a controller or action method. For example, if a user must belong to the Admin role, we would add the [Authorize]
attribute and specify the Roles
property as follows:
1 | [ ] |
The equivalent using the Authorize tag helper would be to add the asp-authorize
attribute to an HTML element and then also add the asp-roles
attribute specifying the require role.
1 | <div asp-authorize asp-roles="Admin" class="panel panel-default"> |
You can also specify a comma separated list of roles, in which case the HTML would be rendered if the user was a member of any of the roles specified.
The [Authorize]
attribe also provides an option to authorize users based on the requirements specified in a Policy. You can learn more about the specifics of this approach by reading the offical docs on Claims-Based Authorization and Custom-Policy Based Authorization. Policy based authorization is applied by specifying Policy
property for the [Authorize]
attribute as follows:
1 | [ ] |
This assumes a policy named Seniors was defined at startup. For example:
1 | services.AddAuthorization(o => |
The equivalent using the Authorize tag helper would be to add the asp-authorize
attribute to an HTML element and then also add the asp-policy
attribute specifying the policy name.
1 | <div asp-authorize asp-policy="Seniors" class="panel panel-default"> |
You can combine the role based and policy based approaches by specifying both the asp-roles
and asp-policy
attributes. This has the effect of requiring that the user meets the requiremnts for both the role and the policy. For example, the following would require that the usere were both a member of the Admin role and meets the requirements defined in the Seniors policy.
1 | <div asp-authorize asp-roles="Admin" asp-policy="Seniors" class="panel panel-default"> |
The Authorize tag helper itself is fairly simple. The implementation will likely evolve after this blog post so you can check out the latest version here.
The tag helper implements the IAuthorizeData
interface. This is the interface implemented by the Authorize attribute in ASP.NET Core. In the ProcessAsync
method, the properties of IAuthorizeData
are used to create an effective policy that is then evaluated against the current HttpContext
. If the policy does not succeed, then the output of the tag helper is supressed. Remember that supressing the output of a tag helper means that the HTML for that element, including it's children, will be NOT sent to the client.
1 | [ ] |
The code in the ProcessAsync
method is based on the AuthorizeFilter from ASP.NET Core MVC.
You can see the Authorize tag helper in action on my tag helper samples site here. The sample site contains the examples listed in this blog post and also provides a way to log in as different users to test different scenarios.
The Authorize tag helper is also available on NuGet so you can use it in your own ASP.NET Core application.
1 | dotnet add package TagHelperSamples.Authorization |
Let me know what you think. Would you like to see this tag helper included in the next release of ASP.NET Core?
If you choose to use the Authorize tag helper in your application, you should remember that hiding a section of HTML is not enough to fully secure your application. You also need to make sure that authorization is applied to any related controllers and action methods. The Authorize tag helper is meant to be used in conjugtion with the [Authorize]
attribute, not as a replacement for it.
There are a couple more scenarios I would like to go through and I will address those in a future post. One of those is supporting different Authorization Schemes and the other resource based authorization. Of course, this project is all open source so feel free to jump in on GitHub.
]]>While Entity Framework Core's in-memory store works great for many scenarios, there are some situations where it might be better to run our tests against a real relational database. Some examples include when loading entities using raw SQL or when using SQL Server specific features that can not be tested using the in-memory provider. In this case, the tests would be considered an integration test since we are no longer testing our Entity Framework context in isolation. We are testing how it will work in the real world when connected to SQL Server.
For this example, I used the following simple model and DbContext classes.
1 | public class Monster |
1 | public class MonsterContext : DbContext |
In an ASP.NET Core application, the context is configured to use SQL Server in the Startup.ConfigureServices
method.
1 | services.AddDbContext<MonsterContext>(options => |
The DefaultConnection
is defined in appsettings.json
which is loaded at startup.
1 | { |
The MonsterContext
is also configured to use Migrations which were initialized using the dotnet ef migrations add InitialCreate
command. For more on Entity Framework Migrations, see the official tutorial.
As a simple example, I created a query class that loads scary monsters from the database using a SQL query instead of querying the Monsters
DbSet
directly.
1 | public class ScaryMonstersQuery |
To be clear, a better way to write this query is _context.Monster.Where(m => m.IsScary == true)
, but I wanted a simple example. I also wanted to use FromSql
because it is inherently difficult to unit test. The FromSql
method doesn't work with the in-memory provider since it requires a relational database. It is also an extension method which means we can't simply mock the context using a tool like Moq
. We could of course create a wrapper service that calls the FromSql
extension method and mock that service but this only shifts the problem. The wrapper approach would allow us to ensure that FromSql
is called in the way we expect it to be called but it would not be able to ensure that the query will actually run successfully and return the expected results.
An integration test is a good option here since it will ensure that the query runs exactly as expected against a real SQL Server database.
I used xunit as the test framework in this example. In the constructor, which is the setup method for any tests in the class, I configure an instance of the MonsterContext
connecting to a localdb instance using a database name containing a random guid. Using a guid in the database name ensures the database is unique for this test. Uniqueness is important when running tests in parallel because it ensures these tests won't impact any other tests that aer currently running. After creating the context, a call to _context.Database.Migrate()
creates a new database and applies any Entity Framework migrations that are defined for the MonsterContext
.
1 | public class SimpleIntegrationTest : IDisposable |
The actual test itself happens in the QueryMonstersFromSqlTest
method. I start by adding some sample data to the database. Next, I create and execute the ScaryMonstersQuery
using the context that was created in the setup method. Finally, I verify the results, ensuring that the expected data is returned from the query.
The last step is the Dispose
method which in xunit is the teardown for any tests in this class. We don't want all these test databases hanging around forever so this is the place to delete the database that was created in the setup method. The database is deleted by calling _context.Database.EnsureDeleted()
.
These tests are slow! The very simple example above takes 13 seconds to run on my laptop. My advice here is to use this sparingly and only when it really adds value for your project. If you end up with a large number of these integration tests, I would consider splitting the integration tests into a separate test suite and potentially running them on a different schedule than my unit test suite (e.g. Nightly instead of every commit).
You can browse or download the source on GitHub.
]]>