CARDINALThe A100 40GB versus 80GB decision starts with the memory needed by one useful unit of work. Then choose PCIe or SXM for the server platform. More capacity can remove a constraint, but it does not make every workload twice as fast.
A100 is a family, not one hardware configuration
Separate two decisions: memory capacity and physical integration. A100 has 40 GB and 80 GB configurations in both PCIe and SXM4 forms. Ask for the complete description and OEM part number. A seller's shorthand of “A100 80G” is not enough for an order.
| Variant | Memory | Peak memory bandwidth | Accelerator power |
|---|---|---|---|
| A100 PCIe 40GB | 40 GB HBM2 | 1,555 GB/s | 250 W |
| A100 SXM4 40GB | 40 GB HBM2 | 1,555 GB/s | 400 W |
| A100 PCIe 80GB | 80 GB HBM2e | 1,935 GB/s | 300 W |
| A100 SXM4 80GB | 80 GB HBM2e | 2,039 GB/s | 400 W standard |
Sources: NVIDIA's 40 GB PCIe brief, Ampere architecture whitepaper and 80 GB specification table. The 80 GB SXM custom thermal solution has a separate up-to-500 W footnote. Do not apply that rating to every 80 GB module.
Make the physical form explicit when comparing prices. A PCIe card and a bare SXM module have different installation requirements. If one offer includes a server and the other includes only accelerators, request a component breakdown before comparing their totals.
Start with working memory, not just model weights
For inference, calculate weight storage first, then add the allocations needed to serve requests. A hypothetical 13-billion-parameter model at two bytes per parameter has 26 decimal GB of idealized weights. That calculation leaves other memory requirements unresolved.
A hypothetical 30-billion-parameter model at the same storage precision has 60 decimal GB of weights. That exceeds one 40 GB A100's stated capacity before cache or runtime memory. An 80 GB card gives more room, but the same caveat remains: actual peak memory must be established.
Scroll across the diagram to read every label
Download diagramThe right question is how much memory your selected deployment needs at its peak. Record context length and concurrent sequences for an autoregressive service. Include batch settings for an offline job. If the application changes those settings dynamically, test the largest supported combination.
Weight quantization can reduce one component of the budget, but it does not remove the need to check quality and runtime behavior. Keep the exact model artifact in the test record. A nominal four-bit format may still contain metadata or higher-precision tensors.
The KV cache has its own format and growth behavior. It should not automatically inherit the weight precision in a spreadsheet. Cache strategies differ, so use the actual serving configuration when estimating and measuring memory. Source: Transformers cache strategies
Training needs a different calculation
Do not reuse an inference weights-only estimate for full-parameter training. Training also stores optimizer state, gradients and activations. Those allocations depend on the optimizer and implementation, while activations depend on the workload configuration.
Hugging Face's current memory anatomy describes a mixed-precision Adam example with six bytes of weights, eight bytes of optimizer state and four bytes of gradients per parameter. Activations and temporary tensors come on top. That is one documented assumption set, not a universal constant for every training method. Source: GPU memory usage
For a buying comparison, ask your team to provide measured activation memory or a justified estimate for the intended configuration. Record any checkpointing or sharding strategy. A memory saving that relies on a software change belongs in the proposal, together with its measured tradeoffs.
Choose around the shape of the work
Several independent jobs already fit within 40 GB
Compare how many useful jobs the complete server can finish within the required time. Larger memory per card may not improve that result if capacity is not the constraint. Evaluate scheduling efficiency and host resources before assuming that 80 GB creates twice the useful capacity.
Also consider whether the jobs can share devices safely through the chosen infrastructure. NVIDIA's Multi-Instance GPU technology partitions supported GPUs into isolated instances. The usable profiles and software support should be checked for the exact configuration, rather than inferred from dividing the nameplate capacity. Source: MIG introduction
Use the actual job mix for the evaluation. A server dedicated to one large model may make a different tradeoff from a shared batch cluster. Record the operational objective so the purchasing decision remains connected to how the hardware will be used.
One job exceeds a 40 GB device
Evaluate an 80 GB device against a distributed 40 GB configuration that can actually run the workload. Include any replication and communication overhead in the distributed case. Two 40 GB cards have 80 GB in aggregate, but the application must distribute its allocations correctly.
A single larger device may simplify the arrangement if the complete working set fits. That is a reason to test it, not a promise of higher throughput. A workload can fit in one device and still need more compute capacity than that device provides.
You expect the workload to grow
Describe the expected change rather than asking for vague future-proofing. It might be a larger context, a different model artifact or higher concurrency. Estimate and test the effect of that change. Then decide how much additional capacity is worth purchasing today.
Keep a fallback plan if the forecast does not arrive. The value of spare memory depends on alternative workloads and how the system is operated. Avoid treating unused capacity as either automatically wasteful or automatically valuable.
More memory does not mean twice the bandwidth
The table shows why the capacity label should not substitute for a specification review. Both 80 GB variants double the stated capacity of their 40 GB counterparts. Their published memory bandwidth does not double. PCIe and SXM also have different bandwidth figures within the 80 GB family.
Use those figures to form a workload hypothesis. If an application is limited by memory transfers, bandwidth may influence its performance. If the workload waits elsewhere, the same bandwidth increase may have little effect. A theoretical number is not a measured service result.
Run the same model or dataset with the same numerical format and software image for the first comparison. Record the power setting and GPU count. For inference, compare latency and throughput together. For training, compare work completed toward the same target rather than only iteration time.
Check dense versus sparse notation whenever you compare tensor figures. Also distinguish tensor operations from ordinary floating-point execution. A spreadsheet that omits these labels can create a performance difference that exists only in the source formatting.
PCIe and SXM4 need separate platform checks
A PCIe A100 remains a server integration task. Confirm the exact card with the server OEM, including slot arrangement, cooling and auxiliary power. Ask whether the offered bracket and mounting parts suit the chassis.
The 40 GB PCIe brief describes a passive thermal solution. Passive does not mean that cooling is unnecessary; the system provides the airflow. Do not use a photograph of a card in an arbitrary chassis as evidence that the operating conditions are suitable. Source: A100 40GB PCIe brief
For 80 GB PCIe, NVIDIA lists air-cooled dual-slot and liquid-cooled single-slot forms. Request the cooling configuration and part number explicitly. Those options should not be treated as interchangeable accessories during procurement. Source: A100 form factors
SXM4 modules require their compatible baseboard and thermal assembly. Ask whether the offer is a module, a populated baseboard or a complete server. Confirm the host system's service procedure if your plan involves replacing or adding modules.
For a multi-GPU proposal, request the interconnect topology and bridge inventory. An aggregate memory figure does not describe communication paths. Record the proposed layout before comparing application results, because the server arrangement is part of the tested configuration.
Software evidence matters on a mature GPU family
A100's age does not remove the need to validate the software image. Confirm driver support for the intended operating system and framework. Record any application extensions or custom kernels that must be rebuilt.
NVIDIA's Ampere tuning guide distinguishes architecture-specific tuning from general CUDA practices. Use it as a technical reference when investigating performance, rather than assuming every workload exploits the hardware in the same way. Source: Ampere tuning guide
Keep software entitlement separate from hardware ownership. If an offer includes virtualization or enterprise software, ask for the exact entitlement and transfer basis. Do not attach a value to an unspecified subscription simply because a product page mentions software support.
Before changing an existing production image, save its version information and reproduce a baseline run. This makes it easier to distinguish a hardware effect from an updated framework or kernel. A fair comparison should make those changes visible.
What to check when buying a used A100
Begin with identity and scope. Ask for readable labels from the actual units and a list of identifiers. Confirm memory capacity and form factor against the part number. Request an explicit list of included cooling, bridging and mounting components.
Then request the seller's condition statement and dated test evidence. Ask which units were tested, which tools were used and which host system held them. A report for a sample unit should not be presented as a record for every card in a larger lot.
Diagnostic coverage has limits. NVIDIA does not describe DCGM as comprehensive hardware diagnosis. Treat a passing result as evidence of the reported checks under their stated conditions. It does not establish a remaining lifetime. Source: DCGM diagnostic scope
Compare offers using the same commercial scope. Separate hardware cost from shipping, installation and any explicitly documented service. Keep taxes and currency treatment visible. A headline card price can be misleading when another offer includes components you would otherwise need to source.
Agree on the acceptance process before the hardware leaves the seller. Record how your team will identify the delivered units and report discrepancies. Cardinal reviews seller-supplied documents; it does not claim to perform hardware testing or assign a remaining service life.
Common A100 buying questions
Is an 80 GB A100 twice as fast as a 40 GB A100?
No such conclusion follows from the capacity. The larger variant doubles stated memory, while bandwidth and other configuration details differ separately. Measure the application under comparable settings.
Are two 40 GB cards equivalent to one 80 GB card?
No. Their aggregate capacity is the same number, but the memory belongs to different devices. The application needs a distribution strategy, and its communication and replication behavior affect the result.
Can I replace a 40 GB SXM4 module with an 80 GB module?
Ask the server OEM to confirm the exact replacement. Shared form-factor naming does not establish firmware, thermal or platform qualification. Use the proposed part numbers and server configuration.
Does a 40 GB A100 still make sense for inference?
It can be a candidate when the full workload fits and measured performance meets the service target. Evaluate the current complete offer against alternatives. Neither the memory label nor the product's age decides its value alone.
What information makes an A100 quote useful?
Specify 40 GB or 80 GB, PCIe or SXM4, unit count, desired condition and destination. Include the server part number if applicable. Describe the workload and evidence needed for acceptance.
Review the exact A100 variants or continue with the used hardware buying guide before requesting an offer.