思科ACI架構(gòu)詳解

1、思科ACI架構(gòu)詳解Application Centric Infrastructure提綱ACI的控制平面及轉(zhuǎn)發(fā)平面ACI的EcosystemACI的應(yīng)用案例ACI網(wǎng)絡(luò)架構(gòu)Infra VRF Used for inband APIC to switch node communication, non routable outside the fabricInband Management Network tenant VRF created for inband access to switch nodesOOB Management Network APIC and switch node

2、dedicated mgmt portsOOB Management NetworkAPIC will have: 2 attached to fabric for data2 for mgmt (OOB)1 console ethernet port (can be only used for direct laptop hookup)CIMC/IPMI portsInband Management VRFInfrastructure VRFSwitch nodes will have: Inband access to Infra & Mgmt VRFMgmt Port (OOB)Cons

3、ole portAPICAPICAPICACI架構(gòu)的初始化APIC bootstrap configurationAPIC Cluster ConfigurationFabric NameTEP Address space (Infra-VRF)Leaf switch discovers attached APIC via LLDP, requests TEP address and boot file via DHCP21Spine switch discovers attached Leaf via LLDP, requests TEP address and boot file via

4、DHCP3All nodes in the same APIC cluster should contain same bootstrap information if they are intended to form a cluster4Fabric can be discovered and initialized from multiple sources concurrently56Fabric will self assemble starting from multiple APIC sourcesAPIC Cluster7APIC Cluster will form when

5、members discovery each other via Appliance Vector (AV)APICAPICAPICFabric leverages IS-IS for infrastructure topologyAdvertises loopback and VTEP addressesResponsible for generating the multicast FTAG trees in the fabric using vendor TLVsIS-IS is responsible for identifying the TEPs and announce the

6、creation of tunnels from every leaf node to all other nodes in the fabricIS-IS has been tuned for a densely connected fabricIS-IS Level 1 IS-IS LSPsIP un-numberedCisco ACI Fabric利用ISIS路由協(xié)議通告VTEP地址ACI控制平面：分布式映射數(shù)據(jù)庫支持高達(dá)1,000,000以上的主機(jī)數(shù)量本地轉(zhuǎn)發(fā)表項(xiàng)包含本Leaf節(jié)點(diǎn)直連的所有主機(jī)的路由信息1fe80:462a:60ff:fef7:8e5e5fe80:62c5:47ff:

7、fe0a:5b1aProxy A15Port 9Leaf 3Proxy A*全局轉(zhuǎn)發(fā)表項(xiàng)本Leaf節(jié)點(diǎn)Cache下來的部分其他主機(jī)路由信息5Leaf 31Leaf 1Leaf 4Leaf 6fe80:8e5efe80:5b1a代理主機(jī)表項(xiàng)包含了連接到交換架構(gòu)的所有主機(jī)路由信息Leaf節(jié)點(diǎn)上的轉(zhuǎn)發(fā)表項(xiàng)可以分為本地轉(zhuǎn)發(fā)表（本地直連主機(jī)信息）和全局轉(zhuǎn)發(fā)表項(xiàng)（基于Cache）如果目的IP在Leaf節(jié)點(diǎn)本地沒有查到路由，則轉(zhuǎn)發(fā)到Spine節(jié)點(diǎn)上去查詢Proxy AProxy BProxy BProxy ScalingScaled based on number of Fabric NFEs per ch

8、assisSpine ProxyTotal Host Entries in the Mapping DB9336200K9504 (6 fabrics)300K9508 (6 fabrics)600K9516 (6 fabrics)1M+Chassis TypeNexus 9504Nexus 9508Nexus 9516NFEs per Fabric Module124NFEFabric Module for Nexus 9504NFENFEFabric Module for Nexus 9508NFENFEFabric Module for Nexus 9516NFENFEACI控制平面：地

9、址學(xué)習(xí)及發(fā)布End Point Location and Identity mapping can be learned in multiple waysAPIC statically programs (location, identity) mappingVMM programs (location, identity) mappingDHCP forwarding can be used to learn (location, identity) mapping entryARP/GARP learningData plane learning creates (location, id

10、entity) mapping at an Leaf Leaf communicates the (location, identity) mapping information to the spine proxyvDSPort GroupVMM will update APIC on creation of a new VM with endpoint information and locationARP/DHCPData plane LearningDMEAPIC(location, identity) advertised from Leaf to spine proxy using

11、 ZeroMQ (Zero Message Queue)COOPOverview of End Point Identity/Location LearningvSwitchvSwitchPayloadIPVXLANVTEPOn ingress the iLeaf will learn the MAC & IP address of the arriving frame along with the ingress port + VLAN/VXLAN/NVGRE (optionally this can be programmed by APIC) 2PayloadIPVXLANVTEPEgr

12、ess iLeaf will learn internal IP and MAC address and cache the source VTEP address binding5PayloadIPNVGREGRE IPiLeaf forwards frame to vSwitch or directly to physical server6PayloadIPPacket transmitted on vSwitch port7PayloadIPvSwitch will be programmed with VM address to port and VTEP mapping by th

13、e VMM1iLeaf will forward end point address information to Spine Oracle using ZeroMQ (Zero Message Queue) 3The Council of Oracles Protocol (COOP) running on the Spine nodes will ensure all spine nodes maintain a consistent copy of end point address and location information and additional maintain the

14、 DHT repository of end point identity to location mapping database4ACI leaf learning Learn all endpoints in VLAN 10and VLAN 20VLAN 10VMVMVMVMVLAN 20VMVMVMVMLocal Station Table15Port 9Leaf 3Proxy A*Global Station TableACI leaf learns MAC and IP(when routing is enabled under Bridge Domain) on non-fabr

15、ic portThe MAC and IP addresses behind non-fabric ports are kept in ACI leaf Local Station Table.Be mindful of Local Station Table when connect ACI leaf to large L2 networkNon-fabric portsMP-BGPPeeringProtocol Peeringfor VRF1RouteRedistributionBGP RRBGP RRVMVMProtocol Peeringfor VRF2MP-BGP1. User as

16、signs BGP ASN and specify spine as BGP RR2. APIC provision MP-BGP within ACI fabric3.Configure L3 outside. Border leaf learns external routes4. BL redistribute external routes to MP-BGP5. External routes are propagated to leaves where private network is instantiated6.BL injects tenant public subnets

17、 to external routersAPIC provisions MP-BGP sessionsAPIC provisions redistribution policy on BLACI控制平面：外部路由采用BGP分發(fā)Internal EPG to External EPGForwarding and Policy Lookup15External EPG Mapping Table(border leaf)WEB EPGVM/24 Leaf 6 /24 Leaf 6 External LPM Table Global Station Table5Leaf 3*Proxy A 1 Po

18、rt 9Local Station Table /24ExtEPG1/24 ExtEPG2 VMExtEPG1/24/24ExtEPG2 /24/241. Derive source EPG. Set source EPG in VXLAN header2. External LPM table lookup with destination IP. Find border leaf VTEP IP3. Derive destination EPG by checking destination IP against this table4. Apply policy based on sou

19、rce, destination EPG and configured contractExternal EPG is derived from IP prefixesSupport multiple external EPGs. One could be remote branch or DC. Another could be InternetDifferent policy for different external EPGs分布式三層網(wǎng)關(guān)及ARP直接轉(zhuǎn)發(fā)015015ACI Fabric supports full Layer 2 and Layer 3 forwarding sema

20、ntics; no changes required to applications or endpoint IP stacksACI Fabric provides optimal forwarding for Layer 2 and Layer 3 Fabric provides a pervasive SVI, which allows for a distributed default gatewayLayer 2 and Layer 3 traffic are directly forwarded to the destination endpointIP ARP and GARP

21、packets are forwarded directly to the target endpoint address contained within ARP or GARP header (elimination of flooding)Distributed Default GatewayDirected ARP ForwardingAPICAPIC分布式三層網(wǎng)關(guān)實(shí)現(xiàn)原理VM1 sends ARP request for Default Gateway The ARP request will be received at TOR and punted to the Supervis

22、or, where MAC and IP is learned and distributed TOR acts as regular Default Gateway and sends ARP response with GW_MAC to VM1VXLAN L3 GatewayVXLAN L3 GatewayL3 FabricVMOSVMOSSVI IP Address (VRF Blue)MAC: 0000.dead.beefIP: VM10PM201CPU23VM1 ARP Cache 0 - GW_MACSVI IP Address (VRF Blue)MAC: 0000.dead.

23、beefIP: ribMACIPL2 VNIL3 VNI VM1_MAC01000050000VM1 generates a data packet destined to PM2 IP (0) with GW_MAC as destination MACTOR receives the packet and performs Layer-3 lookup for the destination (known) TOR adds VXLAN-Header information (Destination VTEP, VNI, etc) and forwards the packet acros

24、s the Layer-3 fabric, picking one of the equal cost paths available via the multiple SpinesThe destination TOR receives the packet, strips off the VXLAN header and performs lookup and forwarding toward PM2VXLAN L3 GatewayVXLAN L3 GatewayL3 FabricVMOSVMOSVM10PM20DMAC: GW_MACSMAC: VM1_MACDIP: 0SIP : 0

25、VLAN 123 4VNI 50000DMAC: DTOR_MACDIP: 0SIP : 0SMAC: STOR_MACDVTEP: DTOR_L0SVTEP : STOR_L060 - DTOR_L0, 500005SIP : 0DIP: 0SMAC: GW_MACDMAC: PM2_MACVLAN 321 70 - PM2_MACPM2_MAC - eth1/32分布式三層網(wǎng)關(guān)實(shí)現(xiàn)原理交換架構(gòu)內(nèi)無ARP泛洪ARP直接轉(zhuǎn)發(fā)ARPMACVXLANVTEP2ARPMACARP幀從終端發(fā)出1ARP PayloadARP幀轉(zhuǎn)發(fā)給目標(biāo)終端3Leaf節(jié)點(diǎn)使用ARP包頭中的目標(biāo)IP地址，查詢該主機(jī)位于哪個(gè)

26、Leaf節(jié)點(diǎn)，然后建立到該Leaf節(jié)點(diǎn)的VXLAN隧道所有的二層攻擊在Fabric里都不能奏效，比如ARP攻擊、生成樹攻擊、組播IGMP欺騙、DHCP欺騙等Proxy-GatewayAnycast-GatewayACI AnycastVLANs/Subnets stretched between leavesYesYesYesCommon Anycast GWY IP across leavesYesYesYesCommon Anycast GWY MAC across leavesYesYesYesUse of Proxy ARP/NDYes (respond to ARP/ND only

27、if the destination is available in the RIB)NoNoARP flooding in L2 domain NoYes (floods also across VXLAN Fabric)No - HW UnicastIntra-subnet forwardingAlways routed (TTL decrement)BridgedBridgedSilent Host DiscoveryNoYesYesVXLAN Forwarding Modes ComparisonProxy-Gateway, Anycast-Gateway & ACI-Anycast子

28、網(wǎng)內(nèi)或子網(wǎng)間主機(jī)之間的IP轉(zhuǎn)發(fā)均基于目的IP地址，在交換架構(gòu)內(nèi)構(gòu)建eVXLAN隧道進(jìn)行轉(zhuǎn)發(fā)，二層非IP流量基于目的MAC地址采用eVXLAN隧道進(jìn)行轉(zhuǎn)發(fā)交換架構(gòu)相對于應(yīng)用完全是透明的，管理人員無需關(guān)注交換架構(gòu)的情況，可以把更過精力放在應(yīng)用和數(shù)據(jù)上。采用分布式映射數(shù)據(jù)庫在Leaf節(jié)點(diǎn)和Spine節(jié)點(diǎn)管理內(nèi)部主機(jī)IP地址與位置的關(guān)系VTEPVTEPVTEPVTEPVTEPVTEPPayloadIPeVXLANVTEPAPICACI模式下的數(shù)據(jù)轉(zhuǎn)發(fā)過程ACI VXLAN (eVXLAN) HeaderACI VXLAN (eVXLAN) header provides a tagging mecha

29、nism to identify properties associated with frames forwarded through an ACI capable fabric. It is an extension of the Layer 2 LISP protocol (draft-smith-lisp-layer2-01) with the additional of policy group, load and path metric, counter and ingress port and encapsulation information. The eVXLAN heade

30、r is not associated with a specific L2 segment or L3 domain but provides a multi-function tagging mechanism used in ACI Application Defined Networking enabled fabric.EthernetHeaderPayloadFCSOuterIPOuterUDPeVXLANOuterEthernetInnerEthernetPayloadNewFCSVXLAN Instance ID (VNID)M/LB/SPSource GroupFlagsRs

31、vd8 Bytes1 ByteN L Rsvd IN: The N bit is the nonce-present bitL: The L bit is the Locator-Status-Bits field enabled bitI: The I bit is the Instance ID bit, Indicates the presence of the VXLAN Network ID (VNID) field. When set, it indicates that the VNID field is valid IPHeaderInner IPHeaderFlags/DRE

32、Virtual eXtensible LAN (VXLAN)Virtual eXtensible LAN (VXLAN) is a Layer 2 overlay scheme over a Layer 3 network.A 24-bit VXLAN Segment ID or VXLAN Network Identifier (VNI) is included in the encapsulation to provide up to 16M VXLAN segments for traffic isolation / segmentation, in contrast to 4K seg

33、ments achievable with VLANs.Each of these segments represents a unique Layer 2 broadcast domain, and can be administered in such a way that it can uniquely identify a given tenants address space or subnet.EthernetHeaderPayloadFCSOuterIPOuterUDPVXLANOuterEthernetInnerEthernetPayloadNewFCSInstance ID1

34、ReservedReservedFlagsRsvdRsvd8 Bytes1 ByteOuter UDP Destination Port = VXLAN (originally 8472, recently updated to 4789)Outer UDP Source Port = Hash of Inner Frame Headers (optional)ACI Fabric VNID的不同作用VNID = Bridge Domain M/LB/SPSource GroupFlagsFlags/DREVNID = VRFM/LB/SPSource GroupFlagsFlags/DREV

35、NID = EPGM/LB/SPSource GroupFlagsFlags/DREIP Forwarding is enabled for the Bridge Domain (default)Any packet is routed (forwarded to the default GW MAC) when the BD is configured for non IP forwardingAny IP packet is forwarded to the spine proxy for address resolutionAny Unicast ARP packet is forwar

36、ded to the target host (technically not a routed frame but within the fabric forwarded according to the ARP IP address)A multicast packet is forwardedAny packet is forwarded when IPv6 is enabled in the VRF/Context (IPv6 will follow IPv4 behaviour with support of IPv6 tenant forwarding) A frame is fo

37、rwarded to a service appliance in standard/legacy IP Services chaining modeVNID identifies the VRF when:VNID identifies the BD when:VNID identifies the EPG when:vSwitch (VMWare)vSwitch (MSFT)PayloadIPPacket Sourced from VM attached to Ingress Port Group or directly from physical server1PayloadIPVXLA

38、NVTEPvSwitch encapsulates frame and forwards to Leaf VTEP2If Leaf has learned the Inner IP to egress VTEP binding it will set required VTEP address and forward directly to egress Leaf4aPayloadIPeVXLANaVTEPLeaf swaps ingress encapsulation with eVXLAN and performs any required policy functions3Payload

39、IPeVXLANaVTEPEgress Leaf will swap outer iVXLAN with correct egress encapsulation and perform any required policy5PayloadIPNVGREGRE IPLeaf forwards frame to vSwitch or directly to physical server6PayloadIPPacket transmitted on vSwitch port7PayloadIPeVXLANaVTEPIf ingress iLeaf does not contain cache

40、entry for IP to egress VTEP binding set VTEP address as anycast VTEP which will perform inline HW lookup and perform egress VTEP rewrite. No additional latency nor any decrease in throughput due to lookup 4bVTEPVTEPVTEPACI單播報(bào)文轉(zhuǎn)發(fā)流程Multicast Forwarding FTAG TopologiesTo improve load balancing Multicas

41、t traffic is distributed across 16 FTAG topologies in the FabricDestination groups are hashed across the FTAG topologiesFTAG trees are rooted at spine switches (roots determined by IFC) FTAG tree calculation is performed by IS-IS and will create the FTAG trees as maximally redundant graphs FTAG node

42、s are advertised using using GM-LSP with martian addresses 0.0.0. representing the FTAG (the last 4 bits indicate the FTAG)FTAG Root for Tree 0FTAG Root for Tree 1FTAG topologies are advertised via IS-IS using martian addresses with the final 4 bits set to the FTAG, 0.0.0.FTAG topologies are limited

43、 to spine to leaf links and only if a downlink is lost will the FTAG topology pass through a leafvSwitchFTAG Root for Tree 0, 4, 8, 12FTAG Root for Tree 2, 6, 10, 14FTAG Root for Tree 3, 7, 11, 15FTAG Root for Tree 1, 5, 9, 13vSwitchSpine Switches maintain a table of GIPo (Multicast IP Overlay Group

44、) to Leaf binding. A Leaf will receive traffic for a GIPo if the EPG BD exists on that Leaf. The GIPo represents a multicast TEP.vSwitchPayloadMCAST IPTraffic to locally attached destinations is replicated locally2PayloadIPVXLANGIPo3For destinations on remote leaves the ingress Leaf maps the multica

45、st traffic to a GIPo and hashes flow to an FTAG tree PayloadMCAST IPMulticast frame is received from a server when attached to the unique EPG Bridge Domain6PayloadMCAST IPMulticast frame is received from a server attached to a unique EPG Bridge Domain either on a VXLAN, VLAN, NVGRE Segment or Port1P

46、ayloadIPVXLANGIPo4Spine replicates and forwards the frame to the Leafs based on the GIPo address5Leaf replicates to all downstream servers, vSwitches, ACI組播報(bào)文轉(zhuǎn)發(fā)流程vPC在ACI架構(gòu)中的數(shù)據(jù)轉(zhuǎn)發(fā)HostTraffic arriving on vPC port is sourced from vPC anycast VTEP address to avoid flapping of source VTEP on egress LeafTr

47、affic arriving to a vPC anycast address is forwarded based on a symmetrical hardware hash to avoid duplicationVTEPHostHostHostVTEPVTEPVTEPHostUnicast frame arrives from vPC attached source, local 802.3ad hashing on host choses which Leaf to forward frame to12Ingress Leaf identifies ingress frame is

48、arriving on a vPC port and sources traffic in the fabric using the local vPC VTEP IP address3Packet is forwarded in the fabric in a manner identical to any other encapsulated unicast packet. If the destination VTEP IP address is also a vPC anycast address copies are sent to both Leaves4Traffic recei

49、ved destined to a vPC VTEP is hashed at the spine (ECMP), hash will result in one of the two Leaves forwarding the frameACI的基本術(shù)語EP.EPEPEPG WEBBDEPG APP SERVEREPG BDsubnetsubnetsubnetL3 context(isolated tenant VRF)With or without flooding semanticsnetwork profileTenantoutside26Tenant “University”PN “

50、Engineering”PN “Business”Subnet /24Subnet /24Subnet /24EPG WebEPG AppBridge Domain 172Subnet /24EPG DBBridge Domain 10ContractPolicy “HTTP”ContractPolicy “SQL”Subnet /24Subnet /24Bridge Domain 100EPG AppEPG WebEPG DBContractPolicy “HTTP”ContractPolicy “SQL”InfrastructureAppsACI中的一些基本概念配置ACI的轉(zhuǎn)發(fā)模式Unic

51、ast Routing: The forwarding method based on predefined forwarding criteria (IP or MAC address). The default is layer 3 forwarding (IP address)L2 Unknown Unicast: forwarding method for unknown layer 2 destinations. The method can be flood or proxy (default)ARP Flooding: Specifies whether ARP flooding

52、 is enabled. If flooding is disabled, unicast routing will be performed on the target IP address. Can be on or off (default)ACI Interaction with STPBPDUBPDUBPDUSTP Root SwitchSame EPGNo STP running within ACI fabricBPDU frames are flooded within EPG. No Configuration requiredExternal switches break

53、any potential loop upon receiving the flooded BPDU frame fabricBPDU filter and BPDU guard can be enabled with interface policyACI Interaction with STPVLAN StitchingVLAN 20Bridge DomainEPG1BPDUDataEPG2BPDUDataDataDataVLAN 10Access port VLAN 10BPDU guard enableAccess port VLAN 20BPDU guard enableACI F

54、abric allows VLAN stitchingMake sure no external loop and the BPDU guard is enabled on access portsData is flooded within BD and BPDU is flooded within EPG.Connecting Two ACI FabricsARP Resolution with Same GW MACvCenter ServerESXESXvCenter ServerESXESXACI Fabric 1ACI Fabric 20000Anycast GW IPMAC: M

55、AC-AAnycast GW IPMAC: MAC-A1. 003. 0 is unknown. Generate ARP request2. Routing. 0 is unknown. Send to spine proxy4. ARP request is flooded to Fabric 25. Send ARP reply. Destination IP Destination MAC: MAC-A6. ARP reply destining to me. Leaf consumes ARP reply and learn IP and MAC of 07. ARP reply n

56、ever makes to Fabric 1. Fabric 1 is not aware of 0Make sure the GW MAC on two fabrics are differentAPICAPICConnecting Two ACI FabricsARP Resolution with Different GW MACvCenter ServerESXESXvCenter ServerESXESXACI Fabric 1ACI Fabric 20000Anycast GW IPMAC: MAC-AAnycast GW IPMAC: MAC-B1. 003. 0 is unkn

57、own. Generate ARP request2. Routing. 0 is unknown. Send to spine proxy4. ARP request is flooded to Fabric 25. Send ARP reply. Destination IP Destination MAC: MAC-A6. ARP reply is encapsulated to vxlan destining to border leaf7. ARP reply is sent over to Fabric 1 border leaf8. ARP reply is received b

58、y spine of Fabric 1APICAPICConnecting Multiple ACI FabricsBridge Domain ConfigurationChange GW MAC address. By default, All fabric and all BD share same GW MACEnable routing and ARP floodingConnecting Two ACI FabricsConsiderationsvCenter ServerVLAN100ESXVLAN100ESXVMM Domain: DC1EPG WEB /24vCenter Se

59、rverVLAN200ESXVLAN200ESXVMM Domain: DC2EPG WEB /24One L2 DomainOne subnetVLAN300ACI Fabric 1ACI Fabric 2Two separate ACI FabricsAnycast GW with same GW IP but different MACTwo separate vCenterL2 Outside connectionAnycast GW with same GW IP but different MACTwo separate/independent ACI fabric. No con

60、trol plan interaction between two fabricsTwo vCenter without live migration or one vCenter without VMM integrationNeed to turn on ARP flooding. No ARP suppression. Two fabrics are in same flooding zoneSupport anycast GW on both sides. Same GW IP but different MACBorder leaf learns all the MAC and IP