Modernization Monday Deep Dive: Rubber, Please Meet Road. Two GPU Sizing Problems, Worked.
A companion to the Modernization Monday series written for a technical audience, this Deep Dive covers two GPU sizing examples.
https://delivery-p155402-e1860468.adobeaemcloud.com/adobe/assets/urn:aaid:aem:4f3babbf-098a-4c20-99e2-ec1b48daf0c8/as/deep%20dive%20blog%20car%20and%20circuit%20road%20AdobeStock_2223107668.avif
GPU
2026-10-28T00:00:00.000Z
15
Adam Stone
Field CTO
blog author portrait Adam Stone

Special Guest: Nathan Bray, Enterprise Architect, ePlus Technology, inc.

And one colorful analogy by, Son Nguyen, Systems Engineer, ePlus Technology, inc.

Deep Dives are the technical companions to Modernization Monday, written for the architects and engineers who want the arithmetic behind the main series. They publish between installments and do not count toward the twelve. I just made the Deep Dive series up as I kept going over my approved word count on the main series. As promised, this article works through two GPU sizing examples.

The Modernization Monday series is kind of like swimming in the shallow end of a pool. Don’t get me wrong, it is a nice swim in a great pool, but sometimes some people just want to jump into the deep end. Off the diving board we go in this one. Now, keep in mind, AI has true depth. Think more along the lines of Crater Lake than the Olympic pool with the high diving boards. Let’s grab our goggles and get to it.

mountain lake

For me to present this topic with accuracy and depth, Nathan Bray, Enterprise Architect of ePlus, is going to be joining us for this one. He is one of the people that I mentioned in the last post that keeps me from running with scissors. I sent the first revision of this one over to him for review and I think he may have hurt his neck from shaking his head from side to side.

GPUs, Part Two gave you the four numbers that decide which GPU fits a model: memory, precision, interconnectivity, and power. This companion puts them to work on an open-weights model and two use cases, inference for a hundred users and a full fine-tuning of the same model. Red Alert! Shields Up! We are going to talk bits and bytes here. If you want the plain-terms version, rewind to the last post. The target audience for this one is the engineers, architects, technologists, and the curious computer nerds. *Proudly adjusts glasses with index finger* I’m serious here. Our poor marketing team has to approve these and there is absolutely no way they get to the bottom of this. I should just insert random sentences in the middle just to see if they catch them and strike them out. (I just said the quiet part out loud. Gah!)

Ok, so sizing sounds mystical. It is mostly detailed arithmetic, which may seem mystical to people who hate math and never quite got the hang of Excel. Our builder from the tokenomics post never priced a house by guessing. They counted nails and studs. We are going to count dials.

Sizing for inference

Part One called this “inferencing,” because that is what you hear in conference rooms. The engineers call it inference, because that is what the papers and the spec sheets say. Same activity. This post is for the engineers, so inference it is. That was one of the things that Nathan was shaking his head about.

In sizing for inference, we must address capacity and performance. In this context the size of a GPU’s memory is a major capacity measurement. A GPU’s performance is heavily influenced by the memory’s bandwidth. Are those the only two sizing measurements? Nope. There is also the attached compute, batching, input length, scheduling, the serving stack, etc. We’ll get there, but we need to get the foundation down first.

As we go through this, remember that both capacity and performance (typically seen as model responsiveness) are closely related sizing measurements, but they aren’t the same thing and there are other measurements that need to be taken in to get it juuuuust right.

We are going to start off with capacity.

Each model that you use, whether through a website, inside an application, or running on your own, has been trained by its maker. In most cases these models have also already been fine-tuned into the chat version users interact with. That fine-tuning is typically done with mixed precision (16 bits is used for heavy weightlifting, and 32 bits is used for the bookkeeping). The finished product of the model is typically published at 16-bit precision, and that is the starting point quantization shrinks from. When you see a model release that shows FP16, this is what it is referencing. Floating-Point, 16-bits. (Still there marketing? Did “quantization” do the trick to bore you to sleep? Jodi! You awake?! We’ll see if this gets cut.) FP16 is known as “half precision.” FP32 is known as “single precision.” FP64 is known as, you guessed it, “double precision” and is what makes HPC people do the happy dance. BF16 stands for “brain floating point,” named for Google Brain, and it is what most open models, Qwen3 included, actually ship in. Same two bytes per dial. Are there more? Yep. Will we go through them all? Nope. We don’t want to spend all day making lists.

So, let’s download an open-weights model called Qwen3-32B from Alibaba Cloud and let’s take a look at the spec sheet the way a cautious parent looks at the back of a cereal box. (For the OpenSource community out there… yes, I should have done with a newer model like Qwen3.8-27B or something, but I got lazy and wanted the math to be easier and I ran out of room on the images below. *spoiler alert*)

Here is the back of the box, lightly trimmed from the model card (taken from Hugging Face):

Qwen3-32B is a dense 32.8-billion-parameter open-weights large language model developed by Alibaba Cloud's Qwen team.

Core Architecture & Parameters

Context Window & Capabilities

Performance Benchmarks

GPU Memory Requirements (Local Deployment)

That… is a lot to take in and, if you aren’t an expert in this space, it is reasonable to just give up right there. But wait. Don’t quit on us just yet…

Let’s take all those nonsensical-sounding ingredients in the mystery cereal and put them into a format we can all relate to.

Time to read the nutrition label of a cereal box. The comic flips the usual question. Instead of asking which GPU fits the model, the GPU is the kid and the model is the cereal it wants. Does the box fit in the bowl? Also, I love analogies. Some of them just land perfectly. This one needed a little bending to work: the bowl is the GPU’s memory, even though the GPU is the one holding the spoon. It will make sense in a minute. Hang tight.

Cereal

Cereal

Overhead? There is always overhead. Always.

Makes a lot more sense when put into real-world context. How do you like this analogy? Great, right?! Thanks! That wasn’t my idea. Nathan came up with that.

One reading tip for the back of the box: the three weight lines are alternatives, not a stack. You pick one precision and pay that line once per GPU. The KV cache line is separate, and you pay it once per conversation.

Let’s take a real parent looking at the back of a cereal box. What are their concerns? Even though there are a lot of lines on the nutrition table, if they cared enough to read it, they aren’t going to read the details if the amount of sugar is too high. They aren’t likely to read past the calorie count if that is too high. If sugar and calories look ok, they are going to take a look at what makes up those calories. Is my child going to get enough protein? What is the serving size they are showing here? Is it misleading?

Now put an adorable GPU in the cart and walk by a box of Qwen3-32B on the shelf. The cereal aisle is about to get… awkward.

Cereal

Cereal

Here is how that parent reads the back of our box. Serving size comes first, because it decides whether every other number is honest. On a cereal box, a small serving size makes the calories look innocent. On a model spec sheet, memory quoted for one user at a short context does the same thing. Our serving is one conversation of about 8,000 tokens, and we are feeding 100 of them at once. Think of 8k tokens as about 12 pages of single-spaced text for reference.

Calories are gigabytes, and they come from different ingredients. The weights, which are fixed at 8-bit in our calorie counter, with the 16-bit and 4-bit alternatives listed right under it, take up the largest percentage of a single serving’s total. That flips once 100 conversations show up, as we will see.

The KV cache is the next ingredient taking up space. It is variable and grows with every token the conversation eats. We can’t forget the rest of the breakfast. In this analogy, you can’t eat the cereal right out of the box. The activations, the working memory the model uses while it computes, sit in the bowl next to the cereal. The milk is the runtime overhead, the serving engine and its bookkeeping, and you have to leave room for it before you pour. Everyone forgets about the overhead. Well, almost everyone. Not Nathan. Not sure how he always remembers it. *Shakes head*

Servings per container is the context window, 32K tokens natively and 128K with YaRN, which is how long one conversation can run before the bowl is full. Overfill the bowl and you are wiping up spilled milk, which in model terms is truncated context and a confused answer. Amnesia, anyone? That limit lives inside one conversation. How many conversations can run at once is a different number, and it is waiting for us in the enterprise aisle.

The license line is the allergen warning. If it says you cannot run the model commercially, nothing else on the label matters. This one says Apache 2.0. Commercial use permitted. So, we can keep reading without worry.

Think of benchmarks like counting grams of protein when figuring out a diet. It tells you whether the model can do the job, and that is a question for the use case owner, not the sizing architect. Yes, those are typically different people. Sometimes they don’t get along well.

The ingredients list, SwiGLU, RoPE, QK-Norm, is the disodium guanylate of the model. It is real, but you don’t always need to know what every single item of the ingredients is to make a reasonable decision.

Also, some ingredients you have to pay attention to more than others. Let’s take grouped-query attention (GQA). Those numbers are why a conversation on this model costs 2 GB per 8,000 tokens instead of 16 GB. More on this to come. Two models with the same parameter count can have very different appetites for notes, which is why the name on the front of the box is a poor sizing guide.

Let’s complete the analogy.

Calories: the weights

Cereal

Qwen3-32B’s roughly 32 billion dials (parameters, on the spec sheet) take two bytes each at 16-bit precision, so 32 billion × 2 bytes = roughly 65 GB. Our poor GPU only has 24 GB. This won’t fit. Readers, don’t worry. No real GPU feelings were hurt in the making of this comic. (It’s ok, Jodi.)

Cereal

Most models are trained at 16 bits, so getting a smaller box means shrinking every dial to a coarser setting. This process is called quantization. Every shrunken dial needs a label saying how far it was shrunk. These labels also take up room. More overhead. So the final footprint lands a bit above the headline number. At 8 bits the model is reduced to about 33 GB. At 4 bits it is about 20 GB, which fits the available 24 GB with about 4 GB to spare.

Look at that cute GPU perk up seeing that model quantized down to 4 bits.

Quantization deserves its own Deep Dive. Let’s call it a byte per dial and some change at 8 bits, and about half of that at 4.

Looks like we can fit that model at 4 bits with 4 GB of spare room! Are we good? Maybe.

Second helpings: the KV cache

Cereal

Every live conversation also needs memory for the model’s notes about everything said so far, the working memory from Part Two. The technical name is the key-value (KV) cache. For this particular model, with grouped-query attention and a 16-bit KV cache, about 8,000 tokens use roughly 2 GB per active conversation. That is a planning assumption, not a universal per-user number. Actual usage depends on context length, cache precision, and serving configuration. Crater Lake… this is the deepest lake in the United States at 1,949 feet deep. Just throwing that out there.

The GPU’s 4 GB of spare room is two short chats. A 32K-token document is 8 GB, and a 128K context with friends over is 32 GB, more than the whole GPU. The bowl does not get bigger. It gets fuller.

Here comes that important ingredient. Without grouped-query attention, this model would keep eight times the notes, and the same conversation would cost 16 GB instead of 2.

Don’t forget the milk: activations and overhead

Cereal

Real deployments reserve GPU memory for activations (the working memory in the bowl) and for the serving engine (the software that loads the model and schedules requests), its runtime buffers, fragmentation, and other overhead. That overhead is the milk, and the empty rim above it is your headroom. We will use the arithmetic to narrow our choices, then benchmark the candidates. Which is why “about 20 GB fits in 24” is tighter than it sounds. When sizing GPUs for models, bet the over. Don’t make it close unless you absolutely have to.

Let’s move on to performance for a bit.

How fast can you eat?

Cereal

Memory bandwidth is the speed of chewing and swallowing. Two GPUs can hold the same model and the same 100 conversations while producing very different performance results. To produce a token of output, the GPU must read every dial. Then it does the same for the next token. When a real person is in a prompt, waiting on a result, the chewing speed is what they feel while they are tapping their finger on the table. Whether they get a seat at the table at all, and how many people can eat at once, is the size of the bowl. How fast the words arrive is chewing.

Let’s also keep this in mind. Each GPU only has one mouth and one bowl. Every conversation working through a GPU shares the same memory size and bandwidth. That bandwidth is the main value that sets the chewing speed.

One more imperfection: chewing implies that the cereal gets destroyed. The weights don’t. They stay loaded and get read again for every token. Analogies aren’t perfect.

Memory capacity (the bowl) determines whether the model and active KV caches fit. Memory bandwidth (chewing speed) influences token generation. Latency also depends on compute, batching, input length, scheduling, and the serving stack (I got lazy and ran out of analogies). Reading the prompt in the first place, called prefill, leans on compute rather than bandwidth, which is why uploading a long document feels different from waiting on a long answer. Capacity and responsiveness are related sizing questions, not interchangeable measurements.

Cereal

Here’s the deal. You can get one box, the 4-bit version, breakfast for one, and no friends are coming over.

Here is the problem. At work, in a real scenario, the friends are the whole point unless you really are deploying a model to serve only one conversation at a time. The label was written for one serving, and production has 100 people at the table. The guest list belongs in the sizing discussion from the start.

So let’s take this label at enterprise scale. The boxes are bigger. We’ll keep the precision at 8-bit because quality matters. Our questions change a bit in this scenario. Now we are shopping for how many we can feed, how fast, and what the bill is going to be.

Cereal

Boxes big enough for 100

Let’s get to a real example and grab a GPU known for inference off the shelf. Let’s check out an NVIDIA L40S. Its 48 GB of memory cannot hold our roughly 64 GB model at 16-bit precision on one card. At 8-bit precision, the roughly 32 GB model plus eight conversations at our assumed 2 GB each would nominally fill all 48 GB, and the extra gigabyte of quantization labels is already eating into that. In practice, that leaves no room for serving overhead, so fewer than eight such conversations may fit, and even those get tight when conversations run long. Fitting conversations into memory also says nothing yet about how quickly they get answers. If your pilot tests with eight users but production has 100, you must measure peak simultaneous requests, not just count registered users. The green light problem shows its ugly face again.

Let’s put that L40S back on the shelf and read the spec sheets for other modern GPUs in the NVIDIA family. The H100 has 80 GB. The H200 gets you to 141 GB. The B200 lands at 180 GB. The B300? 288 GB. Now we are getting somewhere. Baby got bytes! BabyGotBytes is the name of the WiFi network on my travel router. Well, not exactly that, but I am security conscious, so I’m not posting it here.

Ok, where were we?

If 100 users are all active at once, our simplified memory budget is 200 GB of KV cache plus a copy of the roughly 32 GB 8-bit model on each GPU, since each GPU runs its own replica, meaning its own complete copy of the model. On an 80 GB H100, the theoretical balance after model weights is 48 GB, or 24 of our assumed 2 GB conversations per GPU. Five separate H100 replicas would nominally accommodate 120 conversations, before overhead. That is a memory-only calculation, not proof that five GPUs will meet the service-level target (the response time you promised the business).

The same arithmetic gives about 54 conversations per 141 GB H200, 74 per 180 GB B200, or 128 per 288 GB B300, before overhead. Two H200s or two B200s could nominally fit 100 conversations; one B300 has sufficient raw memory on paper. But GPU packaging, actual usable memory, request mix, and throughput all affect what can be deployed and delivered. The shelf tag on the family-size box is exactly that kind of estimate. It says what fits. It promises nothing about how fast anyone gets fed.

Chewing speed, by the numbers

In factory terms, the GPU walks the entire assembly line, all 32 billion stations, for every token, and then walks it again for the next one. That is why GPU spec sheets list a total amount of memory, a memory type, and then a memory bandwidth number. The L40S shows 48 GB at 864 GB/s. The PCIe version of the H100 gets you 2 TB/s. A B300 shows about 8 TB/s. For a single conversation in the decoding phase (the token-by-token output, as opposed to reading the prompt), where the job is bandwidth-bound (limited by how fast memory can be read, not by arithmetic), a rough upper-bound thought experiment is memory bandwidth divided by model-weight bytes read per generated token. For our approximately 32 GB 8-bit model, the L40S at 864 GB/s gives about 27 tokens per second (864 ÷ 32). This is a simplified ceiling, not a measured serving benchmark. Real execution has additional work and overhead. But, if you are sizing safely and save room for overhead, educated estimates can get you good results.

An H100 PCIe at roughly 2 TB/s gives a similar back-of-the-envelope estimate near 60 tokens per second, and a B300 near 8 TB/s gives about 250. Those are single-stream, weight-bandwidth thought experiments, not limits on aggregate batched throughput. Continuous batching, where many conversations share each pass over the dials and new requests join mid-stream instead of waiting for a batch to finish (one chew, a spoonful for every bowl on the table), increases total tokens per second, while compute and KV-cache traffic also rise.

To put this in perspective, people typically read at roughly four to five words a second, about six tokens, so 10 to 20 tokens per second per user feels responsive. For example, if 100 active users each need 20 output tokens per second, the serving system must sustain roughly 2,000 aggregate output tokens per second, plus an acceptable time to first token (the pause before the first word appears, which is the prompt-reading phase). We cannot establish that from memory capacity or advertised bandwidth alone. Benchmark realistic prompt lengths, output lengths, concurrency, and latency targets before deciding how many GPUs to buy.

What do you do if your simultaneous user count doubles? Simple answer, not-as-simple execution. Set more places at the table. You need more GPUs. GPUs need servers. Servers need networking, racks, power, and cooling. Which is why the main series is a long one.

Don’t worry, power. We didn’t forget you. That is the fourth of our four numbers.

Every one of those GPUs has its own rated power draw, and so do the server it goes into, the networking, the storage, and the supporting devices. An H100 pulls roughly 700 watts (W) in its SXM form (NVIDIA’s socketed module for data center GPUs that mounts on a baseboard instead of plugging into a PCIe slot), and about half that in PCIe form. An H200 draws about the same. A B200 draws roughly 1,000 W (1 kW), and a B300 roughly 1.4 kW. Five H100 SXM GPUs would total about 3.5 kW of rated GPU power, compared with 1.4 kW for one B300, but those are not equivalent, benchmarked production configurations, so compare deployable systems rather than isolated GPU figures.

NVIDIA’s HGX platform, the eight-GPU baseboard behind the NVL8 node from Part Two, is how these GPUs are commonly sold, and the rest of the server, the networking, and the cooling consume power too. A defensible comparison is an actual HGX H100 system versus an HGX B300 system, measuring delivered tokens per second, latency, system power, and cost per million tokens. That is tokenomics with the chassis included.

Why can’t we simply order a single B300 GPU and put it in a standalone server? B300s ship eight to a board (HGX B300 or NVL8) or 72 to a rack (GB300 NVL72), never one at a time. In the HGX configuration discussed here, B300 GPUs are integrated into a purpose-built, multi-GPU platform with NVLink and NVSwitch. Packaging is a core part of the design and build, not just a purchasing footnote. You could go through a detailed sizing exercise and come up with a need of only 3 x B300s, but they come in 8-packs. The theoretical versus deployment reality.

Can you split a model across GPUs, say, Qwen3 32B at 16-bit precision across two L40S GPUs? Sure. Model parallelism, splitting one model across GPUs so they trade partial results on every token, introduces communication and synchronization costs. NVLink can help where supported; the L40S has no NVLink, so communication between those cards uses PCIe. That is not automatically disqualifying for production, but it deserves workload-specific benchmarking. Prefer a single-GPU fit when it meets the requirement. Distribute the model when capacity or throughput makes it worthwhile.

The verdict

Memory said a pair of H200s, a pair of B200s, or a single B300 could seat 100. Bandwidth said the B300 chews fastest, and the benchmark decides whether that matters at your table. Power said compare whole systems, not GPUs. Interconnect said keep the model on one GPU when it fits. Precision gets the last word. The B300 is built for inference, and it gave up nearly all of its 64-bit math to get there, roughly a thirtieth of a B200’s double-precision rate and well below an H100’s. If its label were honest, it would say “Not a significant source of FP64.” Newer GPUs aren’t always better for a workload. Feed it the climate model down the hall and the greedy pig from Part Two goes hungry. Sure, kiddo. We are getting the 8-bit version, and all 100 friends are coming to breakfast.

That interconnect discussion matters even more when we move from inference to fine-tuning.

Can you feel the foreshadowing? I try to not ruin the endings of things, but sometimes I can’t help myself. Dumbledore dies.

Sizing for fine-tuning

Fine-tuning sizes to memory, interconnect, and time-to-result. No, that isn’t the same as latency, and remember that correct terminology is cheap insurance. Let’s take that same Qwen3 32B model and fine-tune every dial with our own proprietary data.

Wait. Wait. Wait. Why fine-tune a model? Think of an off-the-shelf open-weights model like a brilliant new hire with the right background and certifications. Fine-tuning can adapt how that employee performs a specialized task, follows a format, or responds to domain-specific patterns. But if the employee just needs the latest product documents, policies, or customer records, retrieval-augmented generation (RAG) may be a better choice. RAG hands the new hire the company wiki at question time instead of making them memorize it, so it supplies current information without repeatedly changing model weights. Fine-tuning and RAG solve different problems, and they can work together.

One of my coworkers, Son, said he had an analogy that was better than mine. We always have room for two analogies in the deep dives. Think of fine-tuning like *reading* a book verses RAG like *having* a book. If person A has read the entire book, that person can answer a question about the book very rapidly with not a lot of effort (token usage). If you have a person B, that only has access to the book, that person will need to go reference the book to give you an answer, thus consuming a lot of energy (tokens) and increasing time-to-answer (latency).

Did you like his analogy better? I did too.

Back to Qwen3 32B. Full fine-tuning adjusts every dial. Inference reads fixed weights; training also tracks gradients (which way to turn each dial) and optimizer states (how far, and how confidently). Think of a clipboard of notes for every one of the 32 billion dials. With one common mixed-precision configuration using Adam, the optimizer most teams reach for, a useful planning rule is about 16 bytes of model-related training state per parameter: 32 billion × 16 bytes = roughly 512 GB. That is not the total training memory requirement. Activations (the intermediate results from each training example, held until the backward pass), temporary buffers, and other overhead add to the budget, and the precise number changes with optimizer, precision, activation checkpointing (recomputing intermediate results instead of storing them, unrelated to the training-run checkpoints in the storage post), and sharding (slicing the training state across GPUs so no single GPU holds all of it).

A 512 GB model-state estimate already exceeds the memory of any single GPU on our list, but it does not dictate one fixed GPU count. Distributed approaches such as FSDP or ZeRO, the two sharding frameworks most teams use, can shard weights, gradients, and optimizer states; activation memory and desired time-to-result then help determine the number of GPUs. Fast GPU-to-GPU communication is valuable, especially within an NVLink domain, although larger jobs can also span nodes over a suitable network. We should size the complete training configuration, not simply divide 512 GB by GPU memory.

To make it concrete anyway, because this is the Deep Dive and the title promised arithmetic: with 512 GB of model state alone, two B300s at 576 GB are already tight before a single activation. Four B200s at 720 GB or an eight-GPU H100 node at 640 GB, one NVL8 from Part Two, is a realistic starting point for this model, and time-to-result sets the final count from there.

I just got a new rider mower that is so much fun, so now you must suffer through a lawn analogy. The desired time-to-result (how long it takes to cut every blade of grass in the yard) determines the count of physical GPUs (lawn mowers and trimmers) working together as one to finish on time (the happy hour deadline). This is why lots and lots of GPUs get purchased together for fine-tuning initiatives. Back in the tokenomics post we said that designing a system for inference when you need fine-tuning is an expensive mistake. Now you can see it in the arithmetic. A single GPU that comfortably serves 100 users of a model can’t comfortably fine-tune it on its own. It is simply a more demanding job. The mower I got for my yard just wouldn’t do for a ranch.

There is a lower-cost path to fine-tuning, with plenty of pluses and minuses. Instead of turning all 32 billion dials, you can freeze most of them and train a much smaller set of adapter parameters. This is parameter-efficient fine-tuning, and LoRA is the name you’ll hear most. Back to the new hire: full fine-tuning is a substantial onboarding program; LoRA is targeted coaching for a specific job. For many tasks, that is enough. Training-state memory can drop dramatically, although the base model, activations, and runtime still need space, so memory does not necessarily fall all the way to inference levels. Choose the lightest method that achieves the required quality and benchmark it before buying the NVLink domain.

I have enough to make my decision now, right?

Nope. Still not even close.

You now know what the GPU is and how to size and pick the right one. That is a good portion of the work but, alas, more must be done. You know what engines you want, but you don’t have the fleet of cars or trucks to put them in. As the tokenomics post warned, buying “a vehicle” can land you a muscle car when you needed a cement truck. In the next two installments of the main series, we pick the fleet, meet the builders, and settle whether you should own it or rent. After that, we check out the roads and the garages.

Think big. Start small. Fail fast. Discipline matters.

If your team is working through this right now, it is exactly the kind of conversation we have every day. Get in touch and we can talk through where you are.

Back to the main series next: Compute, part one. What the GPU goes in: nodes, racks, rows, and who builds them.

false
Blog
Artificial Intelligence,Networking
3
technology-area
true
related-cards