The networking choice inside an AI cluster is not a simple bandwidth comparison. Training jobs repeatedly move parameters, gradients, and activations between accelerators. If the network stalls, expensive GPUs wait. The result can be lower utilization even when every link’s headline speed looks impressive.

InfiniBand and Ethernet can both build large clusters. They differ in how much of the transport, congestion control, topology, and software stack is standardized or integrated for the target workload.

Why AI traffic is unusual

AI training produces synchronized bursts. Many accelerators finish a step and communicate at nearly the same time. That creates a traffic pattern that is sensitive to tail latency, congestion, packet loss, and collective-operation performance.

A general enterprise network can tolerate variable paths and occasional retransmission. A large training cluster may lose efficiency if a small number of links become hot and delay an all-reduce operation. The network is part of the compute system, not just plumbing.

InfiniBand’s case

InfiniBand was designed for high-performance computing. It supports low-latency communication, remote direct memory access, and features intended to keep data moving between nodes with limited CPU intervention.

NVIDIA’s April 2025 description of a DGX SuperPOD deployment cited Quantum-2 InfiniBand networking and a digital-twin validation process. The company said a SoftBank deployment of 510 DGX B200 systems was completed in three weeks, compared with a typical six-plus months, and that NVIDIA estimated the avoided downtime at roughly $1 million per day for a 1,000-server, 8,000-GPU data center. That estimate is NVIDIA’s analysis, not an independent industry average, but it illustrates why network deployment speed can have financial value.

InfiniBand’s strength is a validated stack. Hardware, drivers, collective libraries, and topology can be designed together. That can reduce the integration burden for a customer that wants a high-performing training appliance.

Its weaknesses are ecosystem and flexibility. It can require specialized adapters, switches, expertise, and operational tools. A company with a large Ethernet organization may prefer a fabric it already knows.

Ethernet’s case

Ethernet is widely deployed across data centers and connects servers, storage, offices, and the public network. It offers a large supplier base, familiar management tools, and multiple implementation choices.

AI Ethernet systems commonly use lossless or low-loss designs, remote direct memory access over Converged Ethernet, explicit congestion notification, and careful buffer and topology planning. The performance depends heavily on configuration. “Ethernet” by itself does not say whether a cluster will perform well.

Intel’s Gaudi 3 materials emphasize standard Ethernet and a 32-node all-Ethernet reference design. The pitch is not merely lower equipment cost. It is the ability to use existing operational skills and to avoid a completely closed networking path.

Speed changes the economics

Arista announced new 800G platforms in October 2025 and said 800GbE port shipments more than tripled sequentially in the second quarter of 2025, according to cited market research. The company described 3.2-terabit-per-second HyperPorts and AI uses including lossless training and inference.

800G matters because an AI rack can have many high-bandwidth links. If each accelerator needs 800 gigabits per second of east-west bandwidth, a 72-accelerator rack creates enormous switching demand. Moving to 1.6T transceivers can reduce port counts or support larger bisection bandwidth, but it also raises optical, thermal, and signal-integrity requirements.

The comparison should be made at job completion time

A network that delivers higher link speed but poor congestion behavior may lose to a slower network with better collective performance. The useful test is end-to-end: time to train, tokens per second, scaling efficiency from one node to 32 or 128 nodes, tail latency, and behavior during failures.

The test should include the actual model, precision, batch size, topology, and software release. A vendor’s ideal benchmark can be informative, but it is not a deployment guarantee.

Operations may decide the outcome

A cluster can fail financially through poor utilization. If the team cannot troubleshoot the fabric, the expensive accelerators sit idle. A familiar Ethernet operation may be worth more than a theoretical performance advantage. Conversely, a dedicated AI service with a specialized team may benefit from the tighter InfiniBand stack.

A hybrid architecture is common. InfiniBand or specialized Ethernet can carry accelerator-to-accelerator traffic, while ordinary Ethernet handles management, storage, and user access. The boundary must be clear so a failure in one network does not quietly degrade the other.

The decision rule

Choose InfiniBand when predictable scale-out training, validated deployment, and low-latency collectives dominate. Choose Ethernet when supplier choice, operational familiarity, interoperability, and mixed workloads matter more. Measure both on the workload that pays the bills.

The procurement decision should include the people who will operate the fabric. Ask how many engineers are trained on the selected switches, how firmware is qualified, how telemetry is exported, and how a bad optic is isolated. Also ask whether the vendor can support a mixed-generation cluster during a refresh. A network that is technically fast but difficult to diagnose can reduce accelerator utilization for weeks. In that sense, operational simplicity is a performance feature, not an administrative preference.

At very large scale, fabric choice also affects the power budget. More ports, higher-speed optics, and additional switch tiers consume electricity and occupy rack space. That overhead should be included in the same total-cost model as accelerator power. A lower-cost accelerator that needs a larger network may not be cheaper at the cluster level. The fair comparison is the amount of useful work delivered by the complete system after networking, cooling, software, and operations are included.

The old link between switch and optics

In a conventional data-center switch, the switch ASIC sits on a board and connects electrically to pluggable optical transceivers at the front panel. The transceiver converts electrical signals to light and back again. This architecture is serviceable and familiar, but the electrical path gets harder to manage as SerDes speeds rise.

Co-packaged optics, or CPO, moves the optical engines closer to the switch ASIC on a common package or substrate. The shorter path reduces electrical loss and the power required to compensate for that loss. It can also increase bandwidth density around the switch.

Broadcom describes CPO as heterogeneous integration of optics and silicon on one packaged substrate. On its CPO materials, Broadcom claims more than 3.5 times power-consumption savings, 40% lower optics cost per bit and more than 1 terabit per millimeter of bandwidth density. Those are vendor claims tied to specific designs, so buyers should evaluate the test conditions rather than treat them as universal results.

Why AI makes the issue urgent

AI clusters create large scale-out and scale-up fabrics. Thousands of accelerators must exchange model parameters, gradients, activations and control traffic. A network link can become a performance limit if it cannot keep up with the compute fleet.

As electrical lane speeds move toward 200 gigabits per second and beyond, the board traces between a switch ASIC and front-panel modules consume more power and leave less signal margin. The optical module must compensate for the channel, often using a high-power digital signal processor. CPO reduces that electrical distance.

The benefit is not just headline bandwidth. Lower power per bit can reduce the heat that must be removed from the switch and the facility. A lower-loss path can also improve signal integrity and reduce the complexity of equalization.

The cost of integration

CPO turns optics into part of the switch package. That can make the switch more efficient, but it changes how the system is built and repaired. A failed optical engine may not be replaced like a front-panel module. The manufacturer must qualify the optical engine, silicon photonics, fiber attach, thermal path and package together.

The packaging line therefore matters. The same advanced assembly ecosystem used for AI accelerators may be relevant to optical switches. TSMC has described CPO work that integrates CoWoS modules with its photonics engine technology for high-end network switches. That is a reminder that compute packaging and network packaging are beginning to overlap.

The service model

Pluggable optics allow operators to replace an individual module when it fails or when a link standard changes. CPO can make the optical engine more tightly coupled to the switch, but it may require replacing a larger field-replaceable unit or sending the unit to a service center.

That can be acceptable in a tightly controlled hyperscale fleet. It is less straightforward in a multi-tenant colocation environment with varied optics, longer service contracts and customer-specific network designs.

CPO also raises questions about supply diversity. A pluggable ecosystem can mix modules from multiple vendors if they meet the standard. A CPO design may bind optics, switch silicon, package and firmware more closely. The efficiency gain can therefore come with greater vendor concentration.

Where CPO fits

CPO is most attractive where bandwidth density, power and scale are critical. It may be used in high-radix Ethernet switches, scale-up fabrics, or systems where front-panel power and electrical reach are limiting growth.

It is not automatically the right answer for every link. Short links may use copper or active electrical cables. Longer links may still favor pluggable optics because they are easier to upgrade. The architecture can be mixed inside one campus.

What to watch

For each CPO announcement, track:

• Switch bandwidth and lane rate.
• Optical engine technology and reach.
• Power per port or per terabit.
• Whether the product is sampling or shipping.
• Repair and replacement procedure.
• Package and test partner.
• Interoperability with the target network.
• Fiber-attach and yield constraints.

Broadcom’s 2025 optical-interconnect announcement described a 3-nanometer 200-gigabit-per-lane DSP and a 6.4-terabit-per-second optics attach for a custom AI accelerator. The important point is not just the number. It is the combination of optics, DSP, package, ASIC and network architecture.

BOSSTOX takeaway

CPO may solve an electrical-distance problem by creating a packaging and service problem. That does not make it a bad technology. It means the investment case should include the whole lifecycle: factory yield, optics qualification, power savings, field replacement and upgrade path.

The best CPO disclosure will state what is shipping, at what scale, with what power per bit and how the operator handles a failed optical engine. The package is becoming part of the network.

There is also a thermal tradeoff at the switch. CPO can reduce the electrical path and the power spent recovering signal quality, but it concentrates optics near a high-power ASIC. That creates a shared thermal environment in which the optical engine must operate reliably beside the switch silicon. The cooling system, package materials and optical alignment all become part of the network’s uptime story.

This is why early CPO deployments are likely to be selective. A hyperscale operator can standardize one switch, one optic and one service procedure across a large fleet. A smaller operator may prefer pluggables because the operational flexibility is worth more than the maximum power savings. The technology’s adoption curve will reflect service economics as much as raw bandwidth.

CPO also has an interoperability question. The optical engine may support an open electrical or optical interface, but the complete switch still depends on firmware, thermal controls, management software and diagnostic tooling. A buyer should ask whether the system can be monitored with the same telemetry used for pluggable optics and whether failed engines can be replaced without replacing the entire switch.

The adoption signal is therefore a combination of shipments, customer deployment and service data. Production CPO is meaningful. Production CPO with published power, failure and replacement statistics is much stronger evidence.

For operators, the question is not whether CPO will replace every pluggable optic. It is whether the package delivers a measurable advantage at the specific switch bandwidth and deployment scale. A smaller fleet may value service flexibility; a hyperscale fleet may value watts saved across thousands of ports.

In practice, that comparison should be made per rack and per deployment phase.

Bandwidth is a construction input

An AI data center can have enough floor space and enough GPUs but still fail to deliver useful throughput if the network cannot move data between them. Optical transceivers connect switches, servers and, in some cases, buildings or campuses. As the compute fleet grows, the network needs more links and each link needs more bandwidth.

An 800G transceiver can move 800 gigabits per second under the relevant interface specification. A 1.6T transceiver doubles that nominal rate. The numbers matter because they reduce the number of ports, fibers and switch stages needed for a given traffic load. They do not eliminate the need for careful topology, optics reach and congestion control.

What changed from 400G

AI clusters increase traffic in both scale-up and scale-out networks. Scale-up connects accelerators within a tightly coupled system. Scale-out connects racks and pods across a larger fabric. The optical requirements differ by distance, protocol, fiber type and error-correction scheme.

The move to 800G commonly uses multiple optical lanes. Broadcom’s 800G product materials describe 200-gigabit-per-lane PAM-4 technology and support for 800G DR4 and 1.6T DR8 pluggable designs. Coherent describes 800G as production-ready and 1.6T as the next step, with optical lanes moving from 100 gigabits per second toward 200 gigabits per second.

Higher speed can lower the amount of equipment required for a target bandwidth, but it can increase the engineering difficulty of each link. Signal integrity, laser performance, digital signal processing, fiber connectors, thermal limits and error correction all become more important.

The port-count math

Suppose a network needs 51.2 terabits per second of aggregate switching bandwidth. At a nominal 800G per link, it takes 64 links if the design were perfectly divisible and had no overhead. At 1.6T, it takes 32. Real systems include protocol overhead, redundant paths, uplinks, breakouts and spare capacity, so the final bill of materials is larger.

The example shows why transceiver speed can influence the whole system. Fewer physical links can mean fewer switch ports and less cabling. It can also mean fewer opportunities for link failure. But each transceiver may be more expensive, more power-hungry or harder to source.

Reach is part of the product

An 800G transceiver is not a single product. A short-reach multimode module, a 500-meter single-mode module and a multi-kilometer data-center-interconnect module use different lasers, fiber and thermal designs.

Coherent says AI-fabric Level 0 links are often below 50 meters, while links from top-of-rack to spine can reach up to 500 meters and longer router connections can run from 2 to 10 kilometers. The same 800G label therefore hides different products and supply chains.

Why 1.6T is not just a faster 800G

A 1.6T module often uses eight 200-gigabit-per-second lanes. That raises the demands on the host electrical interface, optical engine, DSP, thermal design and fiber connector. The module must also maintain an acceptable bit-error rate across the target reach.

The network may need new switch ASICs, new NICs, new cables and new test equipment. A data center cannot simply replace every 800G module with a 1.6T unit unless the switch, port, firmware and fiber infrastructure are compatible.

Power matters

Broadcom’s optical-interconnect material compares example power levels of 14 watts for an 800G pluggable and 25 watts for a 1.6T pluggable, while showing lower example values for CPO. Actual power depends on reach, temperature, DSP and form factor. Even so, optical power becomes a facility issue when a campus deploys hundreds of thousands of modules.

If a network has 100,000 800G modules at 14 watts each, the optics alone draw 1.4 megawatts before switch ASICs, servers and cooling. At 25 watts per module, the same count would draw 2.5 megawatts. The arithmetic is simple, but the planning implication is large: optics can move from a component line item to a measurable load.

What buyers should track

Track shipment dates, not demos. Broadcom announced a 3-nanometer 200-gigabit-per-lane PHY on August 27, 2025 for 800G DR4 and FR4 pluggable transceivers. Coherent’s product data for a 1.6T DR8 OSFP module lists an operating distance of 500 meters and an 8-by-200G PAM-4 architecture.

The relevant diligence questions are:

• Is the module in volume production?
• Which switch and NIC interfaces support it?
• What is the reach and fiber type?
• What is power at the target temperature?
• Is the module interoperable across vendors?
• How many modules are required for the planned cluster?
• Are lasers, DSPs and optical engines supply constrained?

BOSSTOX takeaway

800G and 1.6T transceivers matter because AI infrastructure is a bandwidth system. Faster links can reduce port count and cabling, but they also concentrate cost, heat and supply-chain risk into more complex components.

The best project model connects transceiver speed to rack count, switch count, fiber length, port power and commissioning dates. “The network is ready” should mean those links have been tested at the target protocol and workload, not merely that a product was announced.

The link budget should be modeled at the cluster level. An operator needs to know whether the module count grows with every accelerator, every switch, every redundant path or every building-to-building connection. It should also reserve spare modules and account for failures during commissioning. A 1.6T roadmap can reduce ports in the steady state but create an interim period where 800G and 1.6T equipment must interoperate.

The procurement risk is concentrated in specialized inputs. Lasers, DSPs, optical engines, connectors and test systems may each have different lead times. A network can therefore be “designed” months before the optics are actually available. Shipping the first module is a milestone; shipping enough tested modules to fill a production fabric is the milestone that changes capacity.