Today my not problem was running a docker build wasn't copying the files I was expecting it to. In particular I had a themes
directory which was not ending up in the image and in fact the build was failing with something like
1 | ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref b1f3faa4-fdeb-41ed-b016-fac3862d370a::pjh3jwhj2huqmcgigjh9udlh2: "/themes": not found |
I was really confused because themes
absolutly did exist on disk. It was as if it wasn't being added to the build context. In fact it wasn't being added and, as it turns out, this was because my .dockerignore file contained
1 | ** |
Which ignores everything from the local directory. That seemed a bit extreme so I changed it to
1 | ** |
With this in place the build worked as expected.
]]>We run our builds inside a multi-stage docker build so we actually need to have a build container communicate with the database container during the build phase. This is easy enough in the run phase but in the build phase there is just a flag you can pass to the build called network
which takes an argument but the arguments it can take don't appear to be documented anywhere. After significant trial and error I found that the argument it takes that we want is host
. This will build the container using the host networking. As we surfaced the ports for postgres in our workflow file like so
1 | postgres: |
We are able to access the database from the build context with 127.0.0.1
. So we can pass in a variable to our container build
1 | docker build --network=host . --tag ${{ env.DOCKER_REGISTRY_NAME }}/${{ env.DOCKER_IMAGE_NAME }}:${{ github.run_number }} --build-arg 'DATABASE_CONNECTION_STRING=${{ env.DATABASE_CONNECTION_STRING }}' |
With all this in place the tests run nicely in the container during the build. Phew.
]]>)
Next in Keycloak set the credentials up in the realm settings under email. You'll want the host to be smtp.mailgun.org and the port to be 465. Enable all the encryptions and use the full email address as the username.)
Check both the SSL boxes and give it port 465.
]]>We have an API that we call which is super slow and super fragile. We were recently told by the team that maintains it that they'd made improvements and increased our rate limit from something like 200 requests per minute to 300 and could we test it. So sure, I guess we can do your job for you. For this we're going to use the load testing tool artillery.
Artillery is a node based tool so you'll need to have node installed. You can install artillery with npm install -g artillery
.
You then write a configuration file to tell artillery what to do. Here's the one I used for this test (with the names of the guilty redacted):
1 | config: |
As you can see this is graphql and it is a private API so we need to pass in a bearer token. The body I just stole from our postman collection so it isn't well formatted.
Running this is as simple as running artillery run <filename>
.
At the top you can see arrival rates and duration. This is saying that we want to ramp up to 1 requests per second over the course of 1 second. So basically this is just proving that our request works. The first time I ran this I only got back 400 errors. To get the body of the response to allow me to see why I was getting a 400 I set
1 | export DEBUG=http,http:capture,http:response |
Once I had the simple case working I was able to increase the rates to higher levels. To do this I ended up adjusting the phases to look like
1 | phases: |
This provisions 30 users a second up to a maximum of 150 users - so that takes about 5 seconds to saturate. I left the duration higher because I'm lazy and artillery is smart enough to not provision more. Then to ensure that I was pretty constantly hitting the API with the maximum number of users I added a loop to the scenario like so:
1 | scenarios: |
Pay attention to that count at the bottom.
I was able to use this to fire thousands of requests at the service and prove out that our rate limit was indeed higher than it was before and we could raise our concurrency.
]]>One of my clients has a fair bit of data stored in a file share hosted in Azure Storage. They do nightly processing on this data using a legacy IaaS system. We were concerned that we might saturate the blob storage account with our requests. Fortunately, there are metrics we can use to understand what's going on inside blob storage. Nobody wants to monitor these all the time so we set up some alerting rules for the storage account.
Alert rules can easily be created by going to the file share in the storage account and clicking on metrics. Then in the top bar click on New Alert Rule
The typical rules we applied were
However there was one additional metric we wanted to catch: when we have hit throttling. This was a bit trickier to set up because we've never actually hit this threshold. This means that the dimensions to filter on don't actually show up in the portal. They must be entered by hand.
These are the normal values we see)
By clicking on add custom value we were able to add 3 new response codes
)
With these in place we can be confident that should these ever occur we'll be alerted to it
]]>1 | <script type="importmap"> |
The import map is a way to map a module name to a URL. This is necessary because the Vuetify ESM module imports from Vue. Don't forget you'll also need to add in the CSS for Vuetify
]]>1 | Error: creating Service Plan: (Serverfarm Name "***devplan" / Resource Group "***_dev"): web.AppServicePlansClient#CreateOrUpdate: Failure sending request: StatusCode=401 -- Original Error: Code="Unauthorized" Message="This region has quota of 0 instances for your subscription. Try selecting different region or SKU." |
This was a pretty simple deployment to an S1 app service plan. I've run into this before and it's typically easy to request a bump in quota in the subscription. My problem today was that it isn't obvious what CPU quota I need to request. I Googled around and found some suggestion that S1 ran on A series VMs but that wasn't something I had any limits on.
Creating in the UI gave the same error
)
I asked around and eventually somebody in the know was able to look into the consumption in that region. The cloud was full! Well not full but creation of some resources was restricted. Fortunately this was just a dev deployment so I was able to move to a different region and get things working. It would have been pretty miserable if this was a production deployment or if I was adding onto an existing deployment.
]]>)
The managed identity on the app service had only GET access to the keyvault. I added LIST access and the reference started working. I'm not sure why this is but I'm guessing that the reference is doing a LIST to get the secret and then a GET to get the secret value.
]]>The content looked something like
1 | { |
This was going to an ExpressJS application which was parsing the body using body-parser
. These days we can just use express.json()
and avoid taking on that additional dependency. The JSON parsing in both these is too strict to allow for comments. Fortunately, we can use middleware to resolve the issue. There is a swell package called strip-json-comments
which does the surprisingly difficult task of stripping comments. We can use that.
The typical json paring middleware looks like
1 | app.use(express.json()) |
Instead we can do
1 | import stripJsonComments from 'strip-json-comments'; |
This still allows us to take advantage of the compression and character encoding facilities in the original parser while also intercepting and cleaning up the JSON payload.
]]>https://github.com/zdavatz/spreadsheet/
As the name suggests this library deals with Excel spreadsheets. It is able to both read and write them by using Spreadsheet::Excel Library and the ParseExcel Library. However it only supports the older XLS file format. While this is still widely used it is not the default format for Excel 2007 and later. I try to stay clear of the format as much as possible. There have not been any releases of this library in about 18 months but there haven't been any releases of the XLS file format for decades so it doesn't seem like a big deal.
The library can be installed using
1 | gem install spreadsheet |
Then you can use it like so
1 | require 'spreadsheet' |
There are some limitations around editing files such as cell formats not updating but for most things it should be fine.
https://github.com/weshatheleopard/rubyXL
This library works on the more modern XLSX file formats. It is able to read and write files with modifications. However there are some limitations such as being unable to insert images
1 | require 'rubyXL' |
https://github.com/caxlsx/caxlsx
This library is the community supported version of AXLSX. It is able to generate XLSX files but not read them or modify them. There is rich support for charts, images and other more advanced excel features. The
Install using
1 | gem install caxlsx |
And then a simple example looks like
1 | require 'axlsx' |
Of all the libraries mentioned here the documentation for this one is the best. It is also the most actively maintained. The examples directory https://github.com/caxlsx/caxlsx/tree/master/examples gives a plethora of examples of how to use the library.
https://github.com/Paxa/fast_excel
This library focuses on being the fastest excel library for ruby. It is actually written in C to speed it up so comes with all the caveats about running native code. Similar to CAXLSX it is only able to read and write files and not modify them.
1 | require 'fast_excel' |
As you can see here the library really excels at adding consistently shaped rows. You're unlikely to get a complex spreadsheet with headers and footers built using this tooling.
]]>Microsoft.SqlServer.Types
. This package is owned by the SQL server team so, as you'd expect, it is ridiculously behind the times. Fortunately, they are working on updating it and it is now available for Netstandard 2.1 in a preview mode.The steps I needed to take to update the app were:
Microsoft.SqlServer.Types
After that the tests we had for inserting polygons worked just great. This has been a bit of a challenge over the years but I'm delighted that we're almost there. We just need a non-preview version of the types package and we should be good to go.
When I'd only done step 1 I ran into errors like
1 | System.InvalidOperationException : The given value of type SqlGeometry from the data source cannot be converted to type udt of the specified target column. |
I went down a rabbit hole on that one before spotting a post from MVP Erik Jensen https://github.com/ErikEJ/EntityFramework6PowerTools/issues/103 which sent me in the right direction.
]]>The solution I'm using is to just go in an manually delete the backup from the terraform state to unblock my pipelines.
1 | terraform state list | grep <name of your backup> |
Editing Terraform state seems scary but it's not too bad after you do it a bit. Take backups!
]]>)
The warning is
1 | The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/ |
The history here without reading that link is basically that github are changing how we push variables to the pipeline for use in later steps. There were some security implications with the old approach and the new approach should be better
1 | - name: Save variable |
Problem was that the steps on which I was having trouble didn't obviously use the set-output
command.
1 | ... |
I had to dig a bit to find out that it was actually the terraform
command that was causing the problem. See as part of the build I install the terraform cli using the
1 | - name: HashiCorp - Setup Terraform |
Turns out that as of writing the latest version of the wrapper installed by the setup-terraform task makes use of an older version of the @actions/core
package. This package is what is used to set the output and before version 1.10 it did so using set-output
. A fix has been merged into the setup-terraform project but no update released yet.
For now I found that I had no need for the wrapper so I disabled it with
1 | - name: HashiCorp - Setup Terraform |
but for future readers if there is a more recent version of setup-terraform than 2.0.2 then you can update to that to remove the warnings. Now my build is clean
)
]]>What I'm looking to achieve is a number of things
To meet these requirements a build pipeline in GitHub actions (or Azure DevOps, for that matter) is an ideal fit. We maintain our Terraform scripts in a repository. Typically we use one repository per resource group but your needs may vary around that. There isn't any monetary cost to having multiple repositories but there can be some cognitive load to remembering where the right repository is (more on that later).
Changes to the infrastructure definition code are checked into a shared repository. Membership over this code is fairly relaxed. Developers and ops people can all make changes to the code. We strive to make use of normal code review approaches when checking in changes. We're not super rigorous about changes which are checked in because many of the people checking in changes have ops backgrounds and aren't all that well versed in the PR process. I want to make this as easy for them as possible so they aren't tempted to try to make changes directly in Azure.
In my experience there is a very strong temptation for people to abandon rigour when a change is needed at once to address a business need. We need to change a firewall rule - no time to review that let's just do it. I'm not saying that this is a good thing but it is a reality. Driving people to the Terraform needs to be easy. Having their ad-hoc changes overwritten by a Terraform deploy will also help drive the point home. Stick and carrot.
A typical build pipeline for us will include 3 stages.
)
The build step runs on a checkin trigger. This will run an initial build step which validates the terraform scripts are syntactically correct and well linted. A small number of our builds stop here. Unlike application deployments we typically want these changes to be live right away or at most during some maintenance window shortly after the changes have been authored. That deployments are run close to the time the changes were authored helps with our lack of rigour around code reviews.
The next stage is to preview what changes will be performed by Terraform. This stage is gated such that it need somebody to actually approve it. It is low risk because no changes to made - we run a terraform plan
and see what changes will be made. Reading over these changes is very helpful because we often catch unintended consequences here. Accidentally destroying and recreating a VM instead of renaming it? Caught here. Removing a tag that somebody manually applied to a resource and that should be preserved? Caught here.
)
The final stage in the pipeline is to run the Terraform changes. This step is also gated to prevent us from deploying it without proper approvals. Depending on the environment we might need 2 approvals or at least one approval that isn't the person writing the change. More eyes on a change will catch problems more easily and also socialize changes so that it isn't a huge shock to the entire ops team that we now have a MySQL server in the enviornment or whatever it may be.
Most Azure resources support tagging. These are basically just labels that you can apply to resources. We use tags to help us organize our resources. We have a tag called environment
which is used to indicate what environment the resource is in. We have a tag called owner
which is used to indicate who owns the resource. We have a tag called project
which is used to indicate what project the resource is associated with. But for these builds the most important tags are IaC Technology
and IaC Source
. The first is used to tell people that the resources are part of a Terraform deployment. The second is used to indicate where on GitHub the Terraform scripts are located. These tags make it really easy for people to find the Terraform scripts for a resource and get a change in place.
)
I mentioned earlier that we like to guide ops people to make enviornment changes in Terraform rather than directly in Azure. One of the approaches we take around that is to not grant owner or writer permissions to resources directly to people be they ops or dev. Instead we have a number of permission restricted service principals that are used to make changes to resources. These service principals are granted permissions to specific resource groups and these service principals are what's used in the pipeline to make the changes. This means that if somebody wants to make a change to a resource they need to go through the Terraform pipeline.
We keep the client id and secret in the secrets of the github pipeline
)
In this example we just keep a single repository wide key because we only have one enviornment. We'd make use of enviornment specific secrets if we had more than one environment.
This approach has the added bonus of providing rip stops in the event that we leak some keys somewhere. At worst that service principal has access only to one resource group so an attacker is limited to being able to mess with that group and not escape to the larger enviornment.
To my mind this approach is exactly how IaC was meant to be used. We have a single source of truth for our infrastructure. We can make changes to that infrastructure in a repeatable way. We can review those changes before they are applied. All this while keeping the barrier to entry low for people who are not familiar with the code review process.
We already make use of Terraform modules for most of our deployment but we're not doing a great job of reusing these modules from project to project. We're hoping to keep a library of these modules around which can help up standardize things. For instance our VM module doesn't just provision a VM - it sets up backups and uses a standardized source image.
I also really like the idea of using the build pipeline to annotate pull requests with the Terraform changes using https://github.com/marketplace/actions/terraform-pr-commenter. Surfacing this directly on the PR would save the reviewers the trouble of going through the pipeline to see what changes are being made. However it would be added friction for our ops team as they'd have to set up PRs.
]]>1 |
|
The account running terraform in my github actions pipeline is restricted to only have contributor over the resource group into which I'm deploying so it's unable to properly set up providers. Two things needed to fix it:
For 1 the provider block in the terraform file needs to be updated to look like
1 | provider "azurerm" { |
For 2 it requires logging into the azure portal and registering the providers manually. Go to the subscription and select Resource Providers
then search for the one you need, select it and hit Register
. In my case the provider was already registered and the problem was just Terraform's attempt to register it without sufficient permission.
)
]]>On the command line running
1 | az container logs -g <resource group name> -n <container group name> --container <container name> |
Just gave me an output of None
. Not useful either.
Fortunately, you can attach directly to the logs streams coming out of the container which will give you a better idea of what is going on.
1 | az container attach -g <resource group name> -n <container group name> --container <container name> |
This was able to give me output like
1 | Start streaming logs: |
Looks like I missed adding a DB_PORT
to the environmental variables
Taking a step back for you youngsters out there SOAP was the service communication technology that existed before REST showed up with its JSON and ate everybody's lunch. SOAP is really just the name for the transport mechanism but I think most of us would refer to the whole method of invoking remote procedures over the web as SOAP Web Services
. SOAP, or Simple Object Access Protocol, is an XML-based standard for serializing objects from various different languages in a way that Java could talk to .NET could talk to Python. Unlike JSON it was a pretty well thought out protocol and had standard representations of things like dates which JSON just kind of ignores.
Web services were closer to a remote method invocation in that you would call something like GetUserId
rather than the RESTful approach of hitting an endpoint like /Users/7
to get a user with Id 7. The endpoints which were provided by a Web Service were usually written down in a big long XML document called a WSDL which stands for Web Service Definition Language.
Web services gained a reputation for being very enterprisy and complex. There were a large number of additional standards defined around it which are commonly known as ws-*. These include such things as WS-Discovery, WS-Security, WS-Policy and, my personal favorite, the memorably named Web Single Sign-On Metadata Exchange Protocol.
In the last month we've seen the 1.0 release of Core WCF which I'm pretty certain I mocked at being a silly thing in which to invest resources. Tables have turned now I'm the one who needs it so thank to Scott Hunter or whoever it was that allocated resources to developing this.
To get started I needed to find the WSDLs for the services I wanted. This required a call to the support department of the company providing the services. The had a .NET library they pointed me to but it was compiled against .NET 4.5 so I wanted to refresh it. Fortunately the Core WCF release includes an updated svcutil
. This tool will read a WSDL and generate service stubs in .NET for you.
I started with a new console project
1 | dotnet new console |
Then installed the dotnet-svcutil tool globally (you only need to do this once) and generated a service reference
1 | dotnet tool install --global dotnet-svcutil |
This updated my project's csproj file to include a whole whack of new library references
1 |
|
It also generated a 13 000 line long service reference file in the project. Wowzers. I'm glad I don't have to write that fellow myself.
With that all generated I'm now able to call methods in that service by just doing
1 | using ServiceReference; |
This example really only scratches the surface of what the new Core WCF brings to .NET Core. I certainly wouldn't want to develop new WCF services but for consuming existing ones or even updating existing ones then this library is going to be a great boost to productivity.
]]>1 | A host error has occurred during startup operation 'b59ba8b8-f264-4274-a9eb-e17ba0e02ed8'. |
This is the sort of error that terrifies me. Something is wrong but who knows what. No changes in git and an infinity of generic errors on google for Could not load file or assembly
. Eventually after some digging it seems like I might be suffering from some corrupted tooling (some hints about that here: https://github.com/Azure/azure-functions-core-tools/issues/2232). I was able to fix mine by downloading the latest version of the tooling from https://docs.microsoft.com/en-us/azure/azure-functions/functions-run-local?tabs=v4%2Cwindows%2Ccsharp%2Cportal%2Cbash
The 3 options for hosting we're considering are
I've listed these in my order of preference. I'd rather push people to a more managed solution than a less managed one. There is a huge shortage of SQL server skills out there so if you can take a more managed approach then you're less likely to run into problems that require you finding an SQL expert. I frequently say to companies that they're not in the business of managing SQL server but in the business of building whatever widgets they build. Unless there is a real need don't waste company resources building custom solutions when you can buy a 90% solution off the shelf.
When I talk with companies about migrating their existing workloads to the cloud from on premise SQL servers I find myself asking these questions:
CERTIFICATE
, ASYMMETRIC KEY
or SID
?If the answer to any of the first 3 questions is yes
then they can't easily use SQL Azure* and should set the baseline to a managed instance. If the answer to any of the rest of the questions is yes
then they should set the baseline to a VM running a full on version of SQL Server. Only if the answer to all these questions is no
is SQL Azure the best solution.
Quickly get started with
1 | docker run --name somename -p 6379:6379 redis |
The simplest connection string is to use localhost
which just connects to the localhost on port 6379.
Redis is a really simple server to which you can just telnet (or use the redis-cli) to run commands.
List All Keys
This might not be a great idea against a prod server with lots of keys
1 | keys * |
Get key type
Redis supports a bunch of different data primatives like a simple key value, a list, a hash, a zset, ... to find the type of a key use type
1 | type HereForElizabethAnneSlovak |
Set key value
1 | set blah 7 |
This works for updates too
Get key value
1 | get blah |
Get everything in a zset
1 | ZRANGE "id" 0 -1 |
Count everything in a zset
1 | zcount HereForjohnrufuscolginjr. -inf +inf |
Get the first thing in a zset
1 | ZRANGE "id" 0 0 |
Get everything in a set
1 | SMEMBERS HereFeedHashTags |
Get the first member of a list
1 | LPOP somekey |
Get the last member of a list
1 | RPOP somekey |
Get the contents of a hash
1 | HGETALL somekey |
Clear out the whole database
1 | FLUSHALL |
Clear just the current db
1 | FLUSHDB |
Stats on keys
1 | INFO keyspace |
Get an idea of db size
1 | INFO Memory |