Home›Products›Serverless Inference

Serverless Inference

Give a model, get a URL. A vLLM-based, OpenAI-compatible endpoint; when traffic stops it drops to zero, and so does its bill. This page is a plan: the product is not live yet.

Phase 2 · soon Not live yet The figures below are targets, not measurements
Join the waitlist Look at what works today

The email address is a placeholder; the mailbox opens soon. No form, no tracking pixel — if you write, we keep the queue by hand.

The intended product

Three promises, and all three are still targets.

What follows are design targets. The day we measure them, we will replace this page with the measured figures; until then it should be read as a target.

< 2 s

Cold-start target

The route: pre-pulling the image onto the host and caching the model weights on the host's NVMe. This is not a commitment but an engineering target — when we ship it, we will publish the real distribution (p50, p95).

0

Scale to zero

When traffic stops the endpoint sleeps; a sleeping endpoint accrues no compute charge. It wakes on the first request. Not requiring a standby replica is what "you don't pay for seconds that don't work" means on the inference side.

1

The one line that changes

An OpenAI-compatible API: bring your current client, your agent framework, your eval scripts as they are. We want the only changes to be base_url and the key — the cost of moving code should be close to zero.

The planned interface

Change base_url, touch nothing else.

The example beside this is the call shape we are aiming for. It is not a working endpoint today; when the contract is settled it will be published in the docs and this page will be updated.

Chat and completion endpoints will follow the OpenAI schema; streaming is a target from day one.
The model name will come from the Hugging Face repo; the weights will be cached on the host's NVMe.
The same budget cap will apply here too: endpoints attach to the org budget, and no surprise-invoice category opens up.
Python · planned
# This endpoint DOES NOT WORK YET — intended usage from openai import OpenAI client = OpenAI( base_url="https://api.kaldera.ai/v1", api_key="kld_...", ) resp = client.chat.completions.create( model="meta-llama/Llama-3.1-8B-Instruct", messages=[{"role": "user", "content": "hello"}], stream=True, ) # The only line that changes: base_url

The honest status

Why it doesn't exist yet, and when it will.

The ordering is not arbitrary: an inference product cannot work honestly without a reliable marketplace and our own supply underneath it. The phase order in the technical plan runs like this.

Phase 0 — Broker MVPWe are here today · one account, one API, one invoice
Phase 1 — Our own supplyHost agent, community GPUs, our own metering
Phase 2 — Serverless InferenceThis product · vLLM, scale to zero, model cache
Phase 3 — Sovereignty and VerifiedRegion pinning package, audited host tier
Metering first, endpoint second

The invoice for an endpoint that scales to zero cannot be built honestly without per-second, health-gated metering underneath it. That infrastructure is being built in Phase 0 and Phase 1; inference rides on top of it.

Cold start depends on the cache

Under two seconds is only possible if the model weights are already on the host's local disk. That in turn requires the host agent and our own supply — that is, Phase 1.

The definition of done is explicit

A user pastes a Hugging Face model name and has a production endpoint within ten minutes. Until that holds we will not say "live"; we don't ship half a screen.

The phase schedule rests on the month ranges in our technical plan and is a plan, not a commitment. If it slips, we will write it in the changelog. If you want to run inference work today the road is open: spin up a GPU pod and run vLLM yourself — it is described on the inference solution page.

Get notified

Be the first to hear when it opens.

Write one line and tell us which model you want to run. We will lock the scope around the work of the people in the queue.

Join the waitlist Changelog

The mailbox opens soon. The waitlist is an email list; you are not added to a marketing sequence.