IBM, a company that spent decades building its own hardware, just signed a $240M agreement with a startup to run its AI inference. That's not a partnership — it's an admission. The deal, first reported by Crypto Briefing, states IBM and Together AI will collaborate on a "large-scale inference cluster." Two hundred forty million dollars. No model training. No proprietary chips. Just raw, optimized inference capacity.
Code doesn't lie — but contracts do. The $240M figure is a headline, not a blueprint. The real story hides in the missing details: GPU type, cluster size, ownership structure, and service term. Without those, any analysis is a hypothesis. But the deal's existence alone tells us more about the enterprise AI racket than any whitepaper.
Context: Why Now?
Enterprise AI deployment has hit a bottleneck. Training is a solved problem — throw GPUs at it, and you get a model. Inference is the messy part: latency, cost, compliance, and scale. IBM's watsonx platform, launched in 2023, was designed to be the enterprise AI workbench. But it lacked one critical element: a dedicated, high-performance inference engine. IBM's cloud — unlike AWS, Azure, or GCP — never built a massive GPU fleet. It relied on partnerships and third-party compute.
Together AI, founded in 2022, is a inference-focused cloud built on open-source models. Its stack uses vLLM, SGLang, and custom optimizations like PagedAttention and continuous batching. It's not a chip company. It's a software layer that makes inference cheaper and faster. The $240M deal is Together AI's first major enterprise contract — a leap from serving startups to powering a Fortune 100.
Core: The Technical Anatomy of the Cluster
Let's do the math. A $240M inference cluster can be structured two ways: pure hardware purchase or a multi-year service contract.
Scenario A: All-in hardware. If every dollar goes to GPUs, servers, networking, and storage, the average cost per H100 GPU (with supporting infrastructure) is about $30,000. That yields 8,000 H100s. At 700W per GPU, peak power consumption hits 5.6 MW — plus cooling and networking, total ~7 MW. That's a medium-sized data center.
Scenario B: Service contract. If $240M covers 3-5 years of operations, the hardware capex is likely 30-40% of the total. Assume $80M for hardware: that's 2,667 H100s. The remaining $160M covers software, support, and profit. This is more plausible for a startup like Together AI, which lacks the balance sheet to buy 8,000 GPUs upfront.
Either way, the cluster is in the thousand-to-ten-thousand GPU range. That's significant but not unprecedented. CoreWeave and Lambda Labs run clusters of similar size. The difference is the target: inference, not training. Inference clusters are optimized for low latency and high throughput. They use InfiniBand or RoCE for job-level parallelism, not the all-to-all connectivity needed for training. They also require sophisticated KV cache management to handle long-context models.
Code doesn't lie — the inference engine matters more than the hardware. Together AI's advantage is its software stack. vLLM's PagedAttention can reduce memory fragmentation by 60-80%. Speculative decoding cuts latency by 2x. Combined, these optimizations double effective throughput per GPU. That's why IBM chose them over building in-house.
Based on my audits of cloud infrastructure deals during the 2020 DeFi Summer, I've seen the pattern before: a legacy company buys a startup's technology to avoid the time and cost of internal R&D. In 2020, it was DeFi protocols outsourcing their oracle feeds. In 2024, it's IBM outsourcing its inference engine.
Contrarian: The Unreported Risks
The press release paints this as a win-win. But the hidden risks are substantial.
Risk 1: GPU supply. The H100 is still constrained. NVIDIA allocates chips based on strategic relationships. Together AI is a small customer compared to hyperscalers. If the cluster requires 5,000+ H100s, delivery could slip 6-12 months. IBM's enterprise clients won't wait.
Risk 2: Utilization. Inference demand is spiky — not like training, which runs 24/7. A 10,000-GPU cluster idling at 30% utilization is a money pit. Together AI's contract may include minimum usage commitments, but those are hard to enforce. The deal could become a white elephant.
Risk 3: Operational maturity. Together AI has ~100 employees. Running a multi-thousand-GPU cluster at 99.9% uptime requires a Site Reliability Engineering (SRE) team, 24/7 monitoring, and disaster recovery. IBM's brand is on the line. If the cluster goes down during a client's production inference, it's IBM that gets blamed.
Code doesn't lie — but stability does. I've seen this in the 2021 NFT smart contract audits: the flashiest features hide the weakest infrastructure. Together AI's software is elegant, but its operations are unproven at scale.
Regulatory landmines. The cluster likely serves regulated industries — finance, healthcare, government. Data residency requirements mean the cluster must be deployed in multiple regions (US, EU, maybe Asia). That multiplies complexity and cost. Export controls on NVIDIA GPUs to certain countries could also limit deployment options.
The real contrarian angle: IBM is not building an AI moat — it's renting one. By signing a $240M deal with a startup, IBM admits it cannot compete with hyperscalers on AI infrastructure itself. This is a defensive move, not an offensive one. The question is whether the rental is worth the price.
Takeaway: What to Watch Next
This deal is a signal, not a destination. It marks the point where enterprise AI inference shifts from "build your own" to "buy from specialists." But the signal is ambiguous. It could herald a new era of AI-as-a-service — or it could be a cautionary tale of a legacy giant overpaying for a startup's hype.
Watch for three signals: 1. Together AI's next funding round. If they raise a Series B at $1B+ valuation within 12 months, the deal is validating. If they struggle, it's a red flag. 2. IBM's Q1 2025 earnings call. Listen for mentions of "AI inference revenue" and "watsonx adoption." 3. Competitor responses. If Oracle or SAP sign similar deals with Fireworks AI or Anyscale, the market is consolidating. If they don't, it's a one-off.
The bottom line: The $240M deal is a bet on the thesis that enterprise AI will be won by the team that can combine legacy trust with startup agility. I've seen this bet before — in 2017 ICOs, in 2020 DeFi, in 2022 Luna. Sometimes the bet pays off. Sometimes it's a rug pull. The difference is in the code, the contracts, and the execution.
Code doesn't lie. But the market often does. Watch the execution, not the headline.
