AWS VPC Flow Logs reports IP traffic in and out ENI interfaces. I consider it a similar tool as Netflow of 100% sample rate. I use it extensively when troubleshooting reachability issues between AWS resources. AWS network and security services become increasingly complex. It’s very common that traffic passes through security group, network ACL, routing components (VPC, TGW, Peering, IGW), load balancer (including Private Link), and sometimes my own Virtual Network Functions (VNFs). A hop by hop analysis using AWS VPC flow logs at each ENI along the path could offer quick insight. AWS VPC Reachability Analyzer is another option. However, I find its functions still limited. Besides, I still have more trust in data-plane tools that work on real traffic flows. Tools that utilize configurations and routing data alone can’t show the real route, especially when there are multiple paths (ECMP, LB).
Network Troubleshooting Scenario
Figure 1 illustrates a very typical inter-vpc traffic pattern through AWS TGW. EC2-a and EC2-b in VPC1 need to communicate with EC2-c in VPC2. After all EC2 instances are up and running, it’s observed EC2-a is not able to ping EC2-c whereas EC2-b works fine. EC2-a and EC2-b share the same security group and route table. ACLs for subnet1 and subnet2 allow all inbound/outbound traffic.
As the next step, flow log data on EC2-a’s ENI are checked. An interesting finding is no icmp traffic to EC2-c is reported in flow log fl-aaa. Now the doubt becomes whether traffic actually leaves EC2-a. The next logical place to suspect is OS of EC2-a. OS-level route table on EC2-a is properly configured. Disabling OS-level firewall on EC2-a doesn’t help either. Ping to internet addresses (ie 8.8.8.8) works fine. Ping to an arbitrary internal address (ie 10.3.0.1) is sent. At least corresponding icmp requests show up in fl-aaa flow log.
Does AWS VPC Flow Logs Report All Traffic?
Where does traffic to EC2-c go? Shouldn’t VPC flow log report all EC2 traffic leaving OS? VPC Flow Logs user guide mentions limitations where some traffic will be captured. It doesn’t include any scenario that leads to the absence of traffic data in this case.
In this simple setup, a quick comparison between the related setup for EC2-a and EC2-b quickly reveals the culprit. As shown in Figure 2, VPC TWG attachment is only configured in AZ2 for VPC1. Once TGW attachment is added to AZ1, ping starts to work!!
I am the type of person who normally has a short attention span on fine prints. Reading below limit from AWS guide again, it states traffic is not forwarded to TGW if no VPC attachment is configured in the same AZ. I’m guessing ENI (or even somewhere before ENI) silently drops the packets. Still, I wish VPC Flow Logs would flag them.
“When you attach a VPC to a transit gateway, any resources in Availability Zones where there is no transit gateway attachment cannot reach the transit gateway. If there is a route to the transit gateway in a subnet route table, traffic is forwarded to the transit gateway only when the transit gateway has an attachment in a subnet in the same Availability Zone.”
I fault myself for not properly configuring VPC attachment following AWS’s instructions. This proves one more time the importance of relying on automation to build/scale AWS infrastructure.
Additional Observations
It’s mentioned above Flow logs reports Ping traffic to arbitrary Internet and other private addresses except 10.1.0.0/16, whether those addresses respond or not. It starts to make sense now. As there is a default route and TGW is not the next hop, AWS has a working next hop for ENI to forward traffic to. So I would draw a broader rule here. If ENI does not know where to forward traffic, either there’s no route or no local VPC attachment as in this case, Flow Logs will not report such traffic. However, there seem to be some exceptions. I am seeing that traffic to IP ranges belonging to AWS is forwarded properly without explicit routing rules.
How about ping from EC2-c to EC2-a? Without VPC attachment configured in AZ1 of VPC1, it’s interesting EC2-a can receive icmp requests from EC2-c. So traffic comes in from attachment in another AZ into VPC1 and gets forwarded across-zone to EC2-a. EC2-a sends icmp responses back to EC2-c. However, due to the lack of local VPC attachment in AZ1, the responses are dropped. The asymmetrical behavior could easily confuse you more about where the root cause is. 🙂
It’s also observed that VPC Traffic Mirroring does not capture/forward the dropped packets to the Mirror target. All above observations also apply when TGW appliance mode is enabled on VPC attachment.
Conclusions
This is a made-up case to better illustrate the idea. In real life, the network setup is typically more complicated. You might end up searching all over the places for RCA. Next time you don’t see traffic shown up in VPC flow logs of an ENI, you should start at vpc attachment configuration of the AZ, besides verifying your OS configurations. Normally missing routes in RT is much easier to spot. Believe me, not all AWS support folks are aware of the fact AWS Flow Logs could miss to report some traffic. Personally, I can not find it documented.
This also brings up another point in case Flow Logs data are used for audit purposes. It’s likely suspicious traffic will not be detected early enough because it is silently dropped.
If there are other scenarios that Flow logs fails to report traffic, please share your findings in the comments.
