OpenAI says its first custom inference chip, Jalapeño, can deliver substantially more AI work per watt while reducing response latency across several large language models. The company says the system is designed to avoid the usual tradeoff between throughput and responsiveness, a claim that could matter as AI agents increase the number of model calls required to complete a task.

The results are based on OpenAI’s testing with the public InferenceX benchmark and comparisons against commercially available accelerator systems. OpenAI plans to begin deploying Jalapeño inside its compute infrastructure by the end of the year, while continuing production qualification, software development, and testing across additional models.

OpenAI’s reported performance gains

According to OpenAI, Jalapeño produced between 1.5 and 1.9 times more AI work per watt at peak throughput than comparison systems across GPT‑OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T. The company also reports 1.7 to 3.6 times lower end-to-end latency across the tested workloads.

For highly interactive workloads, OpenAI says performance was 2.1 to 4.1 times higher. On Kimi K2.5, described by the company as the largest public model in the test set, Jalapeño reportedly delivered about 1.5 times higher peak performance per watt and 3.4 times lower end-to-end latency than the comparison system.

Those figures are OpenAI-reported results, not an independent validation presented in the available evidence. The company says testing used InferenceX, a public benchmark from SemiAnalysis that measures the complete process of serving an AI request rather than focusing only on chip-level specifications.

OpenAI normalized the comparisons using each accelerator’s published chip power rating. Jalapeño is rated at 700 watts, although OpenAI says measured sustained power was at or below 550 watts during the tested workloads. The comparisons listed by OpenAI include systems built around NVIDIA’s GB200 and GB300 platforms, with package thermal design power figures of 1,200 and 1,400 watts respectively.

A system designed around inference workloads

OpenAI’s argument is that Jalapeño’s advantage comes from more than the accelerator itself. The company says it designed the chip, memory, networking, software, and rack-scale system together around the behavior of modern language-model inference.

That behavior is uneven. The prefill phase, when a prompt is processed, tends to be compute-intensive. Decode, when the model generates a response token by token, is more constrained by memory bandwidth. Communication between chips can add another source of delay, particularly when model state must move between separate resources.

Jalapeño is intended to keep data, including the key-value cache used during generation, close to the computing resources that need it. OpenAI says its network architecture allows a workload to remain within one connected system, reducing data movement and helping the system handle both prefill and decode efficiently.

This design target is especially relevant to AI agents. Agents often make several sequential model calls, use tools, and evaluate intermediate results. A small delay in each step can accumulate into a noticeably slower task. Higher throughput can also help infrastructure operators serve more simultaneous requests, but only if latency remains acceptable for interactive use.

AI helped build and program the chip

OpenAI says AI tools were involved directly in Jalapeño’s development. The company reports moving from initial design to tapeout in nine months by using AI to explore implementations, shorten design and verification cycles, and iterate against model workloads. It also says AI-assisted work helped optimize arithmetic circuits and fit additional compute performance into the chip’s schedule.

The programming effort provides another part of OpenAI’s case for custom hardware. Using Codex with GPT‑Astra, engineers brought three open-weight models that were not included in Jalapeño’s original production plan to high performance within two months, according to the company.

OpenAI says AI-generated implementations for selected GPT‑OSS attention and mixture-of-experts blocks ran 1.5 to 1.8 times faster than existing implementations written by human experts. The company explicitly limits that claim to selected blocks rather than complete models, so it should not be read as a full-model speedup.

Supporting new model families still requires new kernels and model-specific optimization. That qualification is important for buyers and infrastructure teams assessing whether a custom accelerator can adapt as model architectures change.

What Jalapeño means for AI infrastructure

For OpenAI, the immediate benefit is operating leverage. More useful model output from the same power and hardware capacity could reduce the cost of serving requests or allow the company to handle greater demand without adding infrastructure at the same rate. Lower latency could also make more interactive agent workflows practical.

The chip does not represent a withdrawal from commercial hardware suppliers. OpenAI says it will continue deploying accelerators from NVIDIA and other partners for both training and inference. Jalapeño therefore appears positioned as an additional platform within a mixed infrastructure strategy, rather than an immediate replacement for external silicon.

For AI builders and enterprise buyers, the central question will be whether the reported gains survive production conditions. Benchmark performance can differ from deployed performance because of software maturity, model compilation, memory capacity, networking overhead, utilization patterns, and the difficulty of supporting new model families.

OpenAI’s full-stack approach could give it tighter control over those variables for its own workloads. It also creates a tradeoff: custom hardware may deliver strong results for the models and serving patterns it was designed around, but its long-term value depends on how quickly the software stack can accommodate models that were not anticipated during design.

What to watch next

The first signal will be deployment. OpenAI says Jalapeño is scheduled to enter its compute infrastructure by the end of the year, but the evidence does not specify the initial scale, products, or customer workloads that will use it.

The market should also watch for independent or customer-verifiable measurements beyond OpenAI’s InferenceX results. Useful details would include sustained performance under production utilization, software overhead, cost per completed request, model coverage, and comparisons that account for complete system power rather than published package ratings alone.

OpenAI says Gen 2 is already in development and Gen 3 is taking shape. Progress on those generations, along with the pace of kernel development for new models, will indicate whether Jalapeño is becoming a durable platform or primarily an optimization for OpenAI’s internal stack.

Creati.ai perspective

Jalapeño is significant because OpenAI is moving from buying inference capacity to shaping more of the hardware-software stack that determines how efficiently its models are served. The reported combination of throughput and latency is strategically more useful than a single peak-speed number, particularly for products built around AI agents.

Still, the strongest claims remain vendor-reported, and the chip has not yet been shown in broad production deployment in the supplied evidence. The next test is operational: whether OpenAI can translate benchmark efficiency into reliable, flexible, and cost-effective service across changing models and real customer demand.