November 12, 2020
June 7, 2021
In a previous blog post we successfully demonstrated that it’s possible to deploy isolated test environments of a service automatically whenever a new pull request is opened. In this blog post we will extend that proof of concept (POC) by also deploying the service to a production environment. This will happen automatically every time we merge something to the master branch. As described in the previous post, there are at least two different ways of achieving this.The first option is to have a dedicated (mutable) production stack which is updated each time a pull request is merged. Another alternative is to promote a previous test environment (i.e., a stack that was automatically launched from a development branch when opening a pull request) to become the new production stack. For this POC we decided to go with the second option, mainly due to the following reasons.
There are several different ways to implement traffic shifting, but for this POC we decided to try out a DNS based approach. In our organization’s AWS account there was already a hosted zone in place for the organization’s domain name. So, in the stack with less frequently changed resources we added a hosted zone for a new subdomain. For the scope of the POC, we performed a manual setup step as well, by adding the NS records to connect the two hosted zones.The main change introduced compared to the POC from part 1 is the addition of a workflow in our Github Actions-based pipeline that will be triggered upon every merge to the master branch. The workflow basically does the following:
A downside of using DNS based traffic shifting is that we have limited control over how quickly changes propagate to clients. In our implementation the time-to-live (TTL) of the DNS record is set to 60 seconds (since we use a DNS record of type alias record the TTL is “inherited” by the DNS record of the underlying resource), so changes should be relatively quickly propagated, assuming that the TTL is respected by clients and intermediary hosts. Still, bear in mind to always be backwards-compatible when performing database migrations or API changes, since both the old and the new web clients will be reachable by your end users for a short period of time (due to this potential delay in DNS propagation).As this is just a POC to demonstrate traffic shifting in practice, the are several future improvements that might need to be addressed before using this pipeline in a real application:
As with the previous blog post, feel free to fork the example repo and try it out yourself. Thanks for reading!