WattGPU

What the energy figures cover

The models predict the GPU board's power draw. The figures shown are scaled up from there to roughly what the wall socket sees:

facility watts = GPU watts ÷ 0.7 × 1.2

The GPU is taken as 70% of the host server's draw — the rest is CPU, memory, storage, NICs and fans — and a PUE of 1.2 is applied on top for cooling and power distribution. Both are typical values rather than measurements, so your own hardware and site will differ. The same scaling applies in every scenario, so the per-token and per-hour numbers describe the same system boundary.

Why per token offline, but per hour under server load

Offline the GPU is saturated: it is generating essentially all of the time, so every joule is attributable to a token and energy per token is a stable property of the model and the hardware.

Under server load it is not. The GPU spends much of its time waiting for the next request, so energy per token would mostly reflect how busy the server happens to be rather than anything about the model or the card. What an operator can act on there is the draw of keeping the deployment up, so the figures are reported per hour instead.

Grid carbon intensity

The demo reads model architectures from the Hugging Face Hub anonymously, which is enough almost all of the time. A token helps in two cases:

• Licence-gated models. Most of the well-known gated models are already bundled, but for one that is not, a token from an account that has accepted the licence will read it.
• Rate limiting. The Hub limits anonymous requests per IP address, and every visitor to this demo shares one. Your own token gives you your own allowance.

Create a read-only token at huggingface.co/settings/tokens. It is used for your request and then discarded — never stored, never logged, and never included in a shared link. Even so, prefer a fine-grained read-only token over one with write access.

Measured and bundled models need no token at all: they are answered without contacting the Hub.

Generalising to hardware that was never measured is consistently harder than generalising to an unmeasured model architecture, which is why an unknown GPU ranks below an unknown LLM.

The first three protocols are the paper's. The double holdout is an extension added for this demo: for every measured pair, the model is retrained with that GPU and that LLM removed entirely, then asked to predict the pair. It is what lets the hardest case carry a measured error instead of an unvalidated guess.

That measurement is also why the hardest case appears in two different bands. Under server load, dropping the LLM as well as the GPU barely hurts — 15.1% against 13.1% for power, 8.6% against 8.5% for latency — so it sits in the same band as an unknown GPU alone. Offline it does hurt: latency goes from 24.9% to 36.2%, which is a genuinely weaker estimate and gets its own band. Whichever band is shown, the error quoted is always the one measured for the actual situation.

Some requests are refused outright rather than given a tier — mixture-of-experts and quantised models, embedding and reranking models, and pairs whose weights do not fit in GPU memory. Those are outside what the models were fitted on, so an estimate would be misleading rather than merely uncertain.

The typical error shown beside each number is the median absolute percentage error (MdAPE) of the model, measured during training and looked up when you ask — it is not computed from your particular query.

How it is measured

Every row of the dataset is predicted by a model that never saw it (out-of-fold prediction), then:

MdAPE = median( |actual − predicted| ÷ actual ) × 100

The median rather than the mean, following the paper: it is not distorted by the few pairs the model gets badly wrong — but that is also why it describes a typical case rather than a worst case.

Why it changes with the tier

The data is split in whatever way reproduces the situation the tier describes, so the error you are shown was measured under exactly that kind of generalisation:

The band shown and the protocol behind it are not always the same. A pair where neither side was measured is backed by the double holdout in every scenario, but under server load that protocol performs closely enough to leave-one-GPU-out to share its band. The number beside the estimate always comes from the protocol, never from the band.

Grouping on the pair for the green tier keeps the repeated runs of one configuration from landing on both sides of the split, which would otherwise flatter the result.

The figures are also split between offline and server operation, because the two regimes differ a lot — especially for latency, where offline behaviour depends on batching that the features only partly capture.

What it does not tell you

• It is a median across a whole tier, so half the held-out cases in that tier were worse. It is not a bound, and not a confidence interval for your pair.
• There is no per-prediction uncertainty: two queries in the same tier and regime always show the same figure.
• It measures the model against the paper's setup — vLLM, FP16, one GPU, 512-token prompts. It says nothing about how well that setup matches yours.

These are the numbers reported in WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMs, reproduced by scripts/train_models.py in this repository. MdAPE is the median absolute percentage error. Pearson r measures how well predictions track the true values, and GPU τ / LLM τ are Kendall rank correlations — how reliably the model puts GPUs, and LLMs, in the right order. Ranking matters as much as absolute error when the question is "which of these should I deploy on?".

Mean power draw (Table 2)

Assuming a GPU simply draws its TDP is roughly adequate offline, where a saturated GPU really does sit near its power limit, and badly wrong under server load (190% error) where utilisation is far below peak. That is where the learned model earns its place.

Inter-token latency (Table 3)

Latency is harder offline than under server load: offline throughput depends on how vLLM batches requests, which the features capture only partly. The roofline baseline — the time to stream the model's FP16 weights from memory once per token — tracks the ordering well (τ up to 0.91) but is off by 80% in magnitude offline, because it ignores batching entirely.

How to read this

• CV rows are the green tier, LOLO the yellow, LOGO the orange.
• Generalising to an unseen GPU (LOGO) is consistently harder than to an unseen LLM (LOLO). That is why an unprofiled GPU is ranked below an unprofiled LLM in the certainty tiers.
• All errors are for the paper's setup: vLLM, FP16 weights, a single GPU, 512-token prompts. They are not warranties for other configurations.

If this demo or the models behind it are useful in your work, please cite the paper:

Fadel Argerich, M., Fürst, J., and Patiño-Martínez, M. (2026). WattGPU: Predicting Inference Power and Latency on Unseen GPUs and LLMs. 1st Workshop on Sustainability and Resource-Efficiency of AI @ IJCAI 2026. CEUR Workshop Proceedings, ISSN 1613-0073. arxiv.org/abs/2607.02391

@inproceedings{fadelargerich2026wattgpu,
  title     = {{WattGPU}: Predicting Inference Power and Latency on
               Unseen GPUs and LLMs},
  author    = {Fadel Argerich, Mauricio and F{\"u}rst, Jonathan and
               Pati{\~n}o-Mart{\'i}nez, Marta},
  booktitle = {Proceedings of the 1st Workshop on Sustainability and
               Resource-Efficiency of Artificial Intelligence (SuRE) @ IJCAI},
  series    = {CEUR Workshop Proceedings},
  publisher = {CEUR-WS.org},
  issn      = {1613-0073},
  year      = {2026},
  eprint    = {2607.02391},
  archivePrefix = {arXiv},
  url       = {https://arxiv.org/abs/2607.02391}
}

The code, the trained models and the measurement subset are available under Apache 2.0 at github.com/maufadel/wattgpu.

GPUs (8): A100 SXM4 40 GB, A30 PCIe, H100 NVL 94 GB, H200 NVL, L4, L40S, Tesla T4, Tesla V100 SXM2 32 GB

LLMs (42): 01-ai/Yi-1.5-6B-Chat, 01-ai/Yi-1.5-9B-Chat, EleutherAI/gpt-j-6b, EleutherAI/gpt-neox-20b, HuggingFaceTB/SmolLM2-1.7B-Instruct, Qwen/Qwen2.5-0.5B-Instruct, Qwen/Qwen2.5-1.5B-Instruct, Qwen/Qwen2.5-3B-Instruct, Qwen/Qwen2.5-7B-Instruct, Qwen/Qwen3-14B, allenai/OLMo-2-1124-13B-Instruct, allenai/OLMo-2-1124-7B-Instruct, berkeley-nest/Starling-LM-7B-alpha, deepseek-ai/deepseek-llm-7b-chat, google/gemma-2-2b-it, google/gemma-3-12b-it, google/gemma-3-27b-it, ibm-granite/granite-3.0-2b-instruct, ibm-granite/granite-3.0-8b-instruct, internlm/internlm2-7b, internlm/internlm2_5-20b-chat, meta-llama/Llama-2-13b-chat-hf, meta-llama/Llama-3.1-8B-Instruct, meta-llama/Llama-3.2-3B-Instruct, microsoft/Phi-3-medium-4k-instruct, microsoft/Phi-3-mini-4k-instruct, microsoft/phi-2, microsoft/phi-4, mistralai/Mistral-7B-Instruct-v0.3, mistralai/Mistral-Small-24B-Instruct-2501, mistralai/Mistral-Small-Instruct-2409, mlabonne/AlphaMonarch-7B, mlabonne/NeuralBeagle14-7B, nvidia/AceMath-1.5B-Instruct, nvidia/AceMath-7B-Instruct, nvidia/Llama-3.1-Minitron-4B-Width-Base, nvidia/Mistral-NeMo-Minitron-8B-Instruct, nvidia/Nemotron-Mini-4B-Instruct, openai-community/gpt2, tiiuae/Falcon3-3B-Instruct, upstage/SOLAR-10.7B-Instruct-v1.0, upstage/solar-pro-preview-instruct

All from the subset of Watt Counts used in the paper: vLLM, FP16 weights, one GPU, 512-token prompts. Quantised, multi-GPU or long-context deployments will differ.

Licence-gated models that were not measured (some Llama and Gemma variants) need an HF_TOKEN environment variable from an account that has accepted their licences.