AWS load balancing service is a core component of many application deployments. It distributes incoming application traffic across multiple backend pool members, thus providing high availability and automatic scaling. Three types of load balancers are offered by WS, namely Classic load balancer/ELB, Application load balancer/ALB and Network load balancer/NLB.
Challenges from Traditional Firewall Rules and Dynamic Load Balancer IP addresses
Of the tree types load balancers, ELB and ALB can scale up/down dynamically as application traffic volume changes. This creates challenges in case there is a traditional firewall between the application client and AWS load balancers. Most firewalls still use rules based on tuple of IP addresses and ports. Some vendors offer capabilities similar to AWS security group where AWS resource tags can be used to construct firewall rules. Behind the scene, FW is configured to retrieve from AWS account the mapping between tag and IPs. This approach only works if AWS and firewall are under the control of same organization.
FQDN based Firewall Rules
FQDN based firewall rule is another option supported by some vendors. In this approach, firewall rules can be constructed using FQDN directly instead of IP addresses. The firewall device then performs DNS queries at certain internals (typically FQDN’s TTL or configured refresh value) to fetch IP addresses associated with FQDN. Figure 1 illustrates high-level setups using ELB as an example. Internal facing ELB will be used as a primary example in this blog post where ELB is deployed in front of an autoscaling group of EC2. The ELB has an FQDN name of bardef-098765432d1.us-west-1.elb.amazonaws.com. A user-friendly FQDN of bar.internal.example.com is registered in Route53 private hosted zone. On customer’s on-prem firewall device, an FQDN rule is configured using bar.internal.example.com.
The application becomes very popular after launch and starts to pick up more traffic. Additional EC2 instances are added automatically as well as ELB instances. Checking again on firewall device, all newly added ELB IP addresses are picked up automatically through DNS queries. The solution works perfectly! Until one day, some users call in and report intermittent access issues to the application. Firewall logs also confirm some flows are blocked. So what could have gone wrong??
Limitation of standard AWS Load Balancer FQDN Resolution – Eight IP Addresses
The issue could be due to the success of your application. As application traffic continues to grow, it reaches a hard limit of ELB (or Route53) that is not well documented. An FQDN of ELB can only resolve to a maximum of eight IP addresses. In Figure 2, it is assumed ELB has scaled up to 18 IP addresses (IP 1-18). Subsequent DNS queries of ELB FQDN, even from same client, can return sets of eight addresses with different values. On firewall device, FQDN bardef-098765432d1.us-west-1.elb.amazonaws.com or bar.internal.example.com resolves to IP 1,2,3,4,6,8,10,12 whereas end-user machine resolves to IP 1,2,3,5,7,9,11,13. Should user machine OS choose IP 1, firewall would allow the flow from/to application as it has IP 1 in its cache. However if the user’s machine OS were to pick IP 7, firewall would block the traffic. As firewall device and user machines resolve DNS independently, they get different answers of eight IP addresses. Firewall drops flow at random times.
Undocumented all.* DNS record
To work around this issue, an undocumented ELB feature can be utilized. It turns out that ELB offers another FQDN in the format of all.existing_elb_fqdn that resolves to all IP addresses of an ELB. In Figure 2, this new FQDN is all.bardef-098765432d1.us-west-1.elb.amazonaws.com. A quick dig or nslookup of this FQDN should reveal all 18 IP addresses in Figure 2. Implementation of AWS ELB is proprietary. My educated guess is that the regular FQDN of bardef-098765432d1.us-west-1.elb.amazonaws.com is registered in Route53 using multivalue answer routing policy. From AWS’s own document about multivalue answer policy, “Route 53 responds to DNS queries with up to eight healthy records selected at random for the particular domain name”. The FQDN of all.bardef-098765432d1.us-west-1.elb.amazonaws.com could use simple routing policy with all ELB IP addresses registered in its A record (for IPv4).
Using the above ELB feature, Figure 4 shows the enhanced design of firewall rule. A new user-friendly FQDN, all.bar.internal.example.com, is created on Route53 private hosted zone. A CNAME is used to all.bardef-098765432d1.us-west-1.elb.amazonaws.com because alias does not work in this case. On firewall, the FW rule is changed to refer to all.bar.internal.example.com. In this way, all 18 IP addresses of ELB are built into firewall cache. Note FW rule can also use reference to all.bardef-098765432d1.us-west-1.elb.amazonaws.com directly, which saves a small cost on Route53 CNAME record. The advantages of user friendly CNAME are mainly management-related. In case Application/ELB is redeployed, no change is needed on the FW rule. Also it is much easier to identify the purpose of the FW rule from the FQDN name itself. One important note. Neither all.bardef-098765432d1.us-west-1.elb.amazonaws.com nor all.bar.internal.example.com should be used by end user to access the application. Not all ELB instances are ready to serve traffic. Some of them could be in the process of warming up and not ready to take requests. Recall Figure 3 above, only ELB’s regular FQDN returns healthy ELB IP addresses.
Now coming to the public-facing ELB, AWS also offers what is called “stable IP” feature. It allows the creation of a pool with fixed number of ELB instances. For details, you can check with your AWS account team or open a support case.
Use case for NLB
The all.* FQDN is also available for NLB. However, NLB normally doesn’t need to scale to multiple IPs per AZ. DNS resolution from its regular FQDN should include the complete list of IP addresses (< 8). The all.* FQDN is not necessary for ELB/ALB use cases discussed above. If it’s possible for your application to switch from ELB to NLB, that’s a preferred workaround than the all.* record.
