VMC Connected VPC is a feature that allows customers to consume native AWS services over high bandwidth and low latency paths. In this blog, I’m sharing some lessons learned in deployments.
General VMC Connected VPC recommendations from VMware
- The VPC subnet the ENI is attached to should be in same AWS availability zone as VMC SDDC
- Only primary VPC CIDR is advertised into SDDC T0 route table. So factor in future growth when sizing it
- VMware would only insert/delete SDDC subnets in VPC’s main route table
- This is not documented explicitly. Connected VPC can’t be used for transit purpose. It’s possible to place LB/proxy in it to extend the scope for some use cases.
VPC route table size
When a new SDDC subnet is configured on VMC side, VMware makes API call toward your AWS account behind the scene and add the prefix to connected VPC’s main route table. The next hop points to ENI of the corresponding SDDC. Figure 1 shows an example SDDC segment of 192.168.1.0/24 in VPC’s main RT. Note VPC route table has default size of 50 for non-propagated routes. Once the total number of NSX-T segments reaches 50 in all SDDCs combined (ie SDDC1 and SDDC2 in Figure 1) and you try to configure any new segment on SDDC side, it’s added to NSX-T successfully but failed to insert into connected VPC’s RT. No error is reported on VMC side. Digging into AWS cloudtrail, you can find events with errorMessage stating “The maximum number of routes has been reached”.
To make it worse, VMC has yet to have a mechanism to periodically sync NSX-T route table to VPC’s route table at the time of this writing. Even if AWS quota of routes per RT is increased, it is still your responsibility to recreate those missing NSX-T segments to manually trigger the synchronization. To avoid the pain, my suggestion is to increase VPC route table size to the proper value ( up to 1000 is allowed) before you run into issues.
Firewall Firewall Firewall
Now you have VMware VM in SDDC and EC2 in AWS up and running. For beginners, it’s very common the two machines can’t reach each other. You check T0 route table and VPC’s main RT and both have prefixes of the other side. The next step is normally to look into various firewall checkpoints along the path.
The starting point is VMC CGW firewall. Make sure you have proper src/dest/service defined. To be precise, apply the rule to VPC interface. In case you utilize NSX-T DFW, that’s another place to investigate on VMC side.
The next stop is the security group (SG) on ENI. Note at the time of this writing, VMC does not provide a way to specify SG when linking connected VPC. The default SG of VPC is used. My recommendation is to apply fairly broad rules to this SG. This reduces management effort every time new traffic is added to this pipe. However, since this is default SG, make sure the rules are still restrictive enough if other workloads in the VPC pick it up unintentionally. Also, pay attention to the directions of inbound and outbound for this ENI/SG. Flow from SDDC to VPC is considered outbound!
Finally, we come a long way to the last firewall along the path, SG for the workload. Nothing special here.
Another tip that comes handy for troubleshooting is VPC flow log. Bear in mind in case flow log is configured with default format. You won’t see traffic with IP of VMware VM. Instead, the flows are reported as IP of the ENI. Using setup in Figure 1 as an example. Flow log list records between 10.1.20.10 (EC1) and 10.1.10.20 (ENI-1), not 192.168.1.10 under default format!
MTU
Yes. Jumbo is supported over this path. I didn’t get 9K but pretty close. Go ahead bump up interface MTU on your workloads on both VMC and VPC sides. Large packet size helps NSX-T edge performance!
VMC Multi-edge support
One nice feature of connected VPC is that traffic over ENI is FREE, compared to 2cent/GB over VMC Transit Connect for example. In case you shift heavy traffic over this path, the new multi-edge support can be utilized as well.
Plan connected VPC carefully
It does not take much effort to link to VPC in your AWS account. However, VMC lacks self-service support when it comes to making changes. If you need to switch to another VPC, a support ticket is needed.
