Switch/Router Architectures: Review of Shared-Bus and Shared-Memory Based Systems

**A practicing engineer's inclusive review of communication systems based on shared-bus and shared-memory switch/router architectures** This book delves into the inner workings of router and switch design in a comprehensive manner that is accessible to a broad audience. It begins by describing the role of switch/routers in a network, then moves on to the functional composition of a switch/router. A comparison of centralized versus distributed design of the architecture is also presented. The author discusses use of bus versus shared-memory for communication within a design, and also covers Quality of Service (QoS) mechanisms and configuration tools. Written in a simple style and language to allow readers to easily understand and appreciate the material presented, __Switch/Router Architectures: Shared-Bus and Shared-Memory Based Systems__ discusses the design of multilayer switches—starting with the basic concepts and on to the basic architectures. It describes the evolution of multilayer switch designs and highlights the major performance issues affecting each design. It addresses the need to build faster multilayer switches and examines the architectural constraints imposed by the various multilayer switch designs. The book also discusses design issues including performance, implementation complexity, and scalability to higher speeds. This resource also: * Summarizes principles of operation and explores the most common installed routers * Covers the design of example architectures (shared bus and memory based architectures), starting from early software based designs * Provides case studies to enhance reader comprehension __Switch/Router Architectures: Shared-Bus and Shared-Memory Based Systems__ is an excellent guide for advanced undergraduate and graduate level students, as well for engineers and researchers working in the field. Cover 1 Switch/Router Architectures: Shared-Bus and Shared-Memory Based Systems 4 Copyright, © 2018 5 Table of Contents 6 About the Author 8 Preface 10 1: Introduction to Switch/Router Architectures 12 1.1 Introducing the Multilayer Switch 12 1.1.1 Control and Data Planes in the Multilayer Switch 13 1.1.2 Control Engine 15 1.1.3 Forwarding Engine 22 1.2 Evolution of Multilayer Switch Architectures 24 1.2.1 Centralized Forwarding versus Distributed Forwarding Architectures 25 2: Understanding Shared-Bus AndShared-Memory SwitchFabrics 27 2.1 Introduction 27 2.2 Switch Fabric Design Fundamentals 29 2.3 Types of Blocking in Switch Fabrics 31 2.4 Emerging Requirements for High-Performance Switch Fabrics 32 2.5 Shared Bus Fabric 34 2.6 Hierarchical Bus-Based Architecture 37 2.7 Distributed Output Buffered Fabric 37 2.8 Shared Memory Switch Fabric 39 2.8.1 Shared Memory Switch Fabric with Write and Read Controls 42 2.8.2 Generic Shared-Memory-Based Switch/Router or Router 44 2.8.3 Example Shared Memory Architectures 45 2.8.3.1 ``One-Way ́ ́ Switch Fabric Implementation Using NetRAM 47 2.8.3.2 ``Snoop ́ ́ Switch Fabric Using NetRAMs 48 2.8.3.3 Design Rationale of the NetRAM 48 2.9 Shared Ring Fabric 49 2.10 Electronic Design Problems 51 3: Shared-Bus and Shared-Memory-Based Switch/Router Architectures 53 3.1 Architectures With Bus-Based Switch Fabrics and Centralized Forwarding Engines 53 3.1.1 Traditional Bus-Based Architecture with Software-Based Routing and Forwarding Engines in a Centralized Processor 54 3.1.2 Bus-Based Architecture with Routing and Forwarding Engines in Separate Processors 56 3.1.3 Bus-Based Architecture with Forwarding Using a Flow/Route Cache in Centralized Processor 57 3.1.4 Bus-Based Architecture with Forwarding Using an Optimized Lookup System in Centralized Processor 59 3.2 Architectures With Bus-Based Switch Fabrics and Distributed Forwarding Engines 61 3.2.1 Bus-Based Architecture with Multiple Parallel Forwarding Engines 61 3.2.2 Bus-Based Architecture with Forwarding Engine and Flow Cache in Line Cards 64 3.2.3 Bus-Based Architecture with Fully Distributed Forwarding Engines in Line Cards 65 3.3 Architectures With Shared-Memory-Based Switch Fabrics And Distributed Forwarding Engines 67 3.4 Relating Architectures to Multilayer Switch Types 68 4: Software Requirements for Switch/Routers 70 4.1 Introduction 70 4.2 Switch/Router Software Development Methods 71 4.2.1 Kernel Software 71 4.2.2 Software Implementation 71 4.2.3 Switch/Router Software Design Issues 72 4.3 Stability of the Routing Protocols 73 4.3.1 Requirements on Processing Power 75 4.3.2 Queuing Requirements 76 4.3.3 Requirements on Memory Allocation 77 4.4 Network Management 78 4.4.1 Installation and Loading of Software 79 4.4.2 Router/Software Configuration 79 4.4.3 Network Monitoring 80 4.4.4 System and Access Control 81 4.4.5 Problem Solving and Debugging 82 4.5 Switch/Router Performance 85 4.5.1 Performance Metrics 85 4.5.2 Packet Throughput/Forwarding Rate 86 4.5.3 Packet Transit Latency 87 4.6 Interaction between Layer 3 (Routing) and Layer 2 (Bridging) Functions in Switch/Routers 89 4.6.1 Switch/Router Design with Protocol Split 91 4.6.2 Integrated Switch/Router Design with Shared External Interface 92 4.6.3 Integrated Switch/Router with Multiple External Interfaces 92 4.7 Control and Management of Line Cards 92 4.7.1 Watchdog Polling of the Line Cards 92 4.7.2 Statistics Counters in the Line Cards 93 4.7.3 Control of the Line Cards 93 4.8 Distributed Forwarding 93 5: Architectures With Bus-Based Switch Fabrics: Case Study-Decnis 500/600 Multiprotocol Bridge/Router 95 5.1 Introduction 95 5.2 In-Place Packet Forwarding in Line Cards 96 5.3 Main Architectural Features of the Decnis 500/600 98 5.4 Decnic 500/600 Forwarding Philosophy 99 5.5 Detail System Architecture 100 5.5.1 Centralized System Resources 101 5.5.1.1 Management Processor Card (MPC) 102 5.5.1.2 Packet Random-Access Memory 104 5.5.1.3 Address Resolution Engine 104 5.5.2 Backplane and Interface Logic 105 5.5.3 Line Cards 107 5.5.3.1 Ethernet and FDDI Adapters 107 5.5.3.2 Synchronous Communications Interfaces 108 5.5.4 Buffer System 109 5.5.5 DECNIS 500/600 Software Architecture 112 5.6 Unicast Packet Reception in a Line Card 114 5.7 Unicast Packet Transmission in a Line Card 116 5.8 Multicast Packet Transmission in a Line Card 118 6: Architectures With Bus-Based Switch Fabrics: Case Study-Fore Systems Powerhub Multilayer Switches 119 6.1 Introduction 119 6.2 PowerHub 7000 and 6000 Architectures 121 6.2.1 PowerHub 7000 Architecture 123 6.2.1.1 The Packet Engine: Combined Route Processor and Forwarding Engine 125 6.2.1.2 Network Interface Modules 127 6.2.2 PowerHub 6000 Architecture 129 6.3 Powerhub Software Architecture 129 6.3.1 PowerHub Polling Architecture 131 6.4 Packet Processing in the Powerhub 132 6.5 Looking Beyond the First-Generation Architectures 134 7: Architectures With Bus-Based Switch Fabrics: Case Study-Cisco Catalyst 6000 Series Switches 137 7.1 Introduction 137 7.2 Main Architectural Features of the Catalyst 6000 Series 138 7.3 High-Level Architecture of the Catalyst 6000 140 7.3.1 32 Gb/s Switching Bus Operating Modes 141 7.3.1.1 Pipelining Mode 141 7.3.1.2 Burst Mode 142 7.3.2 Control Plane and Data Plane Functions in the Catalyst 6000 142 7.4 Catalyst 6000 Control Plane Implementation and Forwarding Engines: Supervisor Engines 143 7.4.1 Supervisor Engine 1A Architecture 144 7.4.1.1 Supervisor Engine 1A with Only an EARL Switching System 145 7.4.1.2 Supervisor Engine 1A with an EARL Switching System Plus PFC1 145 7.4.1.3 Supervisor Engine 1A with an EARL Switching System Plus PFC1/MSFC1 (or 2) 145 7.4.1.4 Details of Packet Processing in the Supervisor Engine 1A 146 7.4.2 Supervisor Engine 2 Architecture 149 7.4.2.1 Supervisor Engine 2 with EARL Switching System Integrated with PFC2 150 7.4.2.2 Supervisor Engine 2 with EARL Switching System Integrated with PFC2/MSFC2 150 7.4.2.3 Details of Packet Processing in the Supervisor Engine 2 151 7.4.3 Multilayer Switching Highlights in Cisco Catalyst Switches 151 7.4.3.1 Front-End Processor Approach with Flow-Based Forwarding 151 7.4.3.2 Distributed Forwarding Approach (aka Cisco Express Forwarding) 152 7.5 Catalyst 6000 Line Card Architectures 152 7.6 Packet Flow in the Catalyst 6000 With Centralized Flow Cache-Based Forwarding 154 8: Architectures With Shared-Memory-Based Switch Fabrics: Case Study-Cisco Catalyst 3550 Series Switches 158 8.1 Introduction 158 8.2 Main Architectural Features of the Catalyst 3550 Series 160 8.3 System Architecture 161 8.3.1 Packet Forwarding Subsystem 161 8.3.1.1 Switching and Forwarding Engines 161 8.3.1.2 Shared Memory Switch Fabric ASIC 162 8.3.1.3 Shared Data Buffer 162 8.3.1.4 Network Satellites 163 8.3.1.5 Notify Ring 163 8.3.1.6 Radial Channels 164 8.3.1.7 Format Conversions 164 8.3.1.8 Destination Address Lookup 164 8.3.2 Supervisor Subsystem 165 8.3.2.1 Control CPU 165 8.3.2.2 Supervisor Interface Satellite 165 8.3.2.3 Flash Memory 166 8.3.2.4 System I/O Interface 166 8.4 Packet Forwarding 166 8.4.1 Catalyst 3550 Packet Flow 167 8.4.2 Catalyst 3550 QoS and Security ACL Support 168 8.5 Catalyst 3550 Software Features 172 8.6 Catalyst 3550 Extended Features 174 8.6.1 EtherChannel and Link Aggregation 174 8.6.2 Port Security 174 8.6.3 Switch Clustering 176 8.6.4 Channel Multiplexing and Frame Stitching 176 8.6.5 Switched Port Analyzer 177 9: Architectures With Bus-Based Switch Fabrics: Case Study-Cisco Catalyst 6500 Series Switches With Supervisor Engine 32 178 9.1 Introduction 178 9.2 Cisco Catalyst 6500 32&Thinsp;Gb/S Shared Switching Bus 180 9.2.1 Main Features of the Catalyst 6500 Shared Bus 181 9.2.1.1 Pipelining Mode 181 9.2.1.2 Burst Mode 181 9.3 Supervisor Engine 32 182 9.3.1 Supervisor Engine 32 Architecture 182 9.3.2 Multilayer Switch Feature Card 2a (MSFC2a) 183 9.3.3 Policy Feature Card 3B (PFC3B) 185 9.3.4 Supervisor Engine 32 as a ``Classic ́ ́ Module 187 9.3.5 Fabric ASIC and Replication Engine 187 9.4 Catalyst 6500 Line Cards Supported By Supervisor Engine 32 188 9.4.1 Classic Line Card Architecture 188 9.4.2 CEF256 Line Card Architecture 188 9.5 Cisco Catalyst 6500 32&Thinsp;Gb/S Shared Switching Bus Modes 190 9.5.1 Flow-Through Mode 190 9.5.2 Compact Mode 190 9.5.3 Truncated Mode 191 9.6 Supervisor Engine 32 Qos Features 191 9.6.1 Uplink Port Queues and Buffering 191 9.6.2 DSCP Transparency 192 9.6.3 Traffic Scheduling Mechanisms 192 9.7 Packet Flow Through Supervisor Engine 32 193 10: Architectures With Shared-Memory-Based Switch Fabrics: Case Study-Cisco Catalyst 8500 Csr Series 197 10.1 Introduction 197 10.2 Main Architectural Features of the Catalyst 8500 Series 197 10.2.1 Catalyst 8510 198 10.2.2 Catalyst 8540 199 10.3 The Switch-Route and Route Processors 201 10.4 Switch Fabric 203 10.5 Line Cards 205 10.5.1 Internal Memory 205 10.5.2 Microcontroller 206 10.5.3 CAM and Search Engine 206 10.5.4 Fabric Interface 207 10.6 Catalyst 8500 Forwarding Technology and Operations 208 10.6.1 Forwarding Philosophy 208 10.6.2 Forwarding Operations 210 10.6.2.1 Layer 3 Forwarding in the Catalyst 8500 211 10.6.2.2 Layer 2 Forwarding in the Catalyst 8500 213 10.7 Catalyst 8500 Quality-Of-Service Mechanisms 214 11: Quality of Service Mechanisms in the Switch/Routers 218 11.1 Introduction 218 11.2 QoS Forwarding Operations Within a Typical Layer 2 Switch 219 11.3 QoS Forwarding Operations Within a Typical Multilayer Switch 221 11.4 QoS Features in the Catalyst 6500 225 11.4.1 Packet Classification 225 11.4.2 Queuing 226 11.4.3 Congestion Avoidance 227 11.4.3.1 Tail-Drop Thresholds 228 11.4.3.2 WRED Thresholds 228 11.4.3.3 Queue Configuration Information 228 11.4.4 Traffic Policing 229 11.4.5 Rewrite 229 11.4.6 Scheduling 230 11.4.6.1 Input Queue Scheduling 231 11.4.6.2 Output Queue Scheduling 231 12: Quality of Service Configuration Tools InSwitch/Routers 232 12.1 Introduction 232 12.2 Ingress Qos and Port Trust Settings 234 12.3 Ingress and Egress Port Queues 235 12.4 Ingress and Egress Queue Thresholds 235 12.4.1 Queue Utilization Thresholds 236 12.4.2 Priority Discard Thresholds 236 12.5 Ingress and Egress Qos Maps 236 12.5.1 Mapping a Packet Priority Value to a Queue and Threshold 237 12.5.2 Mapping Packet Priority Values to Internal Switch Priority Values 238 12.5.3 Policing Map 239 12.5.4 Egress DSCP Mutation Map 239 12.5.5 Ingress IEEE 802.1p CoS Mutation Map 240 12.6 Ingress and Egress Traffic Policing 240 12.6.1 Aggregate Policing 241 12.6.2 Microflow Policing 241 12.6.3 User-Based Rate Limiting 242 12.7 Weighted Tail-Drop: Congestion Avoidance With Tail-Drop and Multiple Thresholds 243 12.8 Congestion Avoidance With Wred 244 12.9 Scheduling With Wrr 246 12.10 Scheduling With Deficit Weighted Round-Robin (Dwrr) 247 12.10.1 Deficit Weighted Round-Robin 248 12.10.2 Modified Deficit Round-Robin (MDRR) 248 12.11 Scheduling With Shaped Round-Robin (Srr) 250 12.12 Scheduling With Strict Priority Queuing 250 12.13 Netflow and Flow Entries 250 12.13.1 NetFlow Entry and Flow Mask 250 12.13.2 NetFlow Table 252 13: Case Study: Quality of Service Processing in the Cisco Catalyst 6000 And 6500 Series Switches 254 13.1 Introduction 254 13.2 Policy Feature Card (Pfc) 254 13.2.1 Policing in the PFC 255 13.2.2 Access Control Entries and QoS ACLs 256 13.3 Distributed Forwarding Card (Dfc) 257 13.4 Port-Based Asics 257 13.4.1 Original 10/100 Mb/s Ethernet Line Cards (WS-X6348-RJ45) 258 13.4.2 Newer Fabric 10/100 Mb/s Ethernet Line Cards (WS-X6548-RJ45) 259 13.4.3 Gigabit Ethernet Line Cards (WS-X6408A, WS-X6516, WS-X6816) 260 13.4.4 10 Gigabit Ethernet Line Cards (WS-X6502-10GE) 260 13.5 QoS Mappings 260 13.6 QoS Flow in the Catalyst 6000 and 6500 Family 261 13.6.1 IP Precedence and IEEE 802.1p/Q CoS Tagging 261 13.7 Configuring Port Asic-Based QoS on the Catalyst 6000 and 6500 Family 263 13.7.1 Trust States of Ports: Trusted and Untrusted Ports 263 13.7.1.1 Untrusted Ports (Default Setting for Ports) 264 13.7.1.2 Trusted Ports 264 13.7.2 Input Classification and Setting Port-Based CoS 264 13.7.3 Configure Receive (Rx) Drop Threshold 264 13.7.4 Configure Transmit (Tx) Drop Threshold 265 13.7.5 Mapping CoS to Thresholds 265 13.7.6 Configure Bandwidth on Tx Queues 265 13.7.7 DSCP to CoS Mapping 266 13.8 IP Precedence and Ieee 802.1P Cos Processing Steps 266 13.8.1 Input Port Processing 267 13.8.2 Forwarding Engine (PFC) Processing 268 13.8.3 Output Port Processing 270 Appendix A: Ethernet Frame 272 A.1 Introduction 272 A.2 Ethernet Frame Format 272 A.2.1 MAC Address 277 A.2.1.1 MAC-48 and EUI-48 279 A.2.1.2 EUI-64 280 A.2.1.3 Written Address Conventions 281 A.2.2 IEEE 802.1Q Tagging Option - VLAN Tagging Option 281 A.2.3 Ethernet Byte and Bit Ordering 286 A.2.4 Legal Ethernet Frames: Untagged Frames 288 A.2.4.1 Illegal and Ethernet Frames 289 Appendix B: IPv4 Packet 290 B.1 Introduction 290 B.2 Ipv4 Packet Format 291 B.2.1 IPv4 Header 291 B.3 Ipv4 Addressing 295 B.3.1 Original ARPANET Addressing Scheme 297 B.3.2 Classful Addressing 298 B.3.3 IPv4 Variable-Length Subnet Masks (VLSM) 299 B.3.4 IPv4 CIDR 302 B.3.5 Reserved IPv4 Addresses 305 B.4 Address Resolution 307 B.5 Ipv4 Address Exhaustion 307 B.6 Ipv4 Options 309 B.7 Ipv4 Packet Fragmentation and Reassembly 310 B.8 Ip Packets Encapsulated Into Ethernet Frames 313 B.9 Forwarding Ipv4 Packets 313 B.9.1 CIDR and Routing/Forwarding Table Entries 313 B.9.2 TTL Update 315 B.9.3 IPv4 Header Checksum Computation 315 References 317 Index 321 WILEY 326 A practicing engineer's inclusive review of communication systems based on shared-bus and shared-memory switch/router architectures This book delves into the inner workings of router and switch design in a comprehensive manner that is accessible to a broad audience. A practicing engineer's inclusive review of communication systems based on shared-bus and shared-memory switch/router architecturesThis book delves into the inner workings of router and switch design in a comprehensive manner that is accessible to a broad audience. It begins by describing the role of switch/routers in a network, then moves on to the functional composition of a switch/router. A comparison of centralized versus distributed design of the architecture is also presented. The author discusses use of bus versus shared-memory for communication within a design, and also covers Quality of Service (QoS) mechanisms and configuration tools.Written in a simple style and language to allow readers to easily understand and appreciate the material presented, Switch/Router Architectures: Shared-Bus and Shared-Memory Based Systems discusses the design of multilayer switches-starting with the basic concepts and on to the basic architectures. It describes the evolution of multilayer switch designs and highlights the major performance issues affecting each design. It addresses the need to build faster multilayer switches and examines the architectural constraints imposed by the various multilayer switch designs. The book also discusses design issues including performance, implementation complexity, and scalability to higher speeds. This resource also:Summarizes principles of operation and explores the most common installed routers Covers the design of example architectures (shared bus and memory based architectures), starting from early software based designs Provides case studies to enhance reader comprehensionSwitch/Router Architectures: Shared-Bus and Shared-Memory Based Systems is an excellent guide for advanced undergraduate and graduate level students, as well for engineers and researchers working in the field