Home/Solutions/Fine-tuning

Fine-tuning: checkpoints that survive, a bill you can see.

From LoRA and QLoRA all the way to full training. Your checkpoint sits on a persistent volume independent of the pod, you don't pay for seconds that don't work, and the hard budget cap stops the job — not the invoice.

You pay per second and stop whenever you want. No commitment.

Your work probably looks like this

The model is yours. What you lose is hours and money.

Fine-tuning takes a long time, and every long job hits something eventually. The items below are not problems we invented; they are the three places anyone renting cheap GPUs already knows.

The checkpoint leaves with the pod

A container's disk is ephemeral. When the pod goes down, nine hours of training go with it; all you are left with is your terminal history.

The node dies mid-run

The price of a cheap card is usually uncertain reliability. The driver locks up at 03:00, and in the morning you find both a stopped job and a bill that kept running.

The invoice arrives as a surprise

A stopped pod's disk keeps billing, and a forgotten experiment stays open for weeks. At the end of the month the number is twice your estimate.

The Kaldera answer

Three promises, three mechanisms.

These are not marketing lines but three separate pieces running inside the product: a health gate, a budget enforcer and per-second metering.

You don't pay for seconds that don't work

The health probe runs every 15 seconds. Seconds that fail it never enter the bill — not as a discount, but as a line item that does not exist. A correction found after the period closes shows up as a refund line.

Hard budget cap

The cap is not a warning bar but an enforcement mechanism. Notifications arrive at 80% and 95%; when the cap is reached, pods are paused, not deleted. The disk stays, the checkpoint stays, you raise the limit and carry on.

Per-second billing

If training took 9 hours 14 minutes you do not pay for 10 hours. If you cancelled a trial run after 4 minutes you pay for 4 minutes. The live counter in the console ticks up second by second.

The flow

A training job from start to finish

Create a persistent volume, put the data and the checkpoint folder on it

The volume lives independently of the pod. Even if the pod is deleted, /vol stays; a new pod attaches to the same volume. Writing checkpoints to the container disk is the one real mistake.

Pick an offer by VRAM

The marketplace sorts by price, scorecard and region. The default sort is not cheapest but best-scored: price, reliability and network are weighed together. On a long training run that choice matters.

Set a separate budget cap for the pod

Alongside the org cap you can define a per-pod cap. That is how a trial run is stopped from accidentally spreading over a week.

Start training, watch the live cost

Log stream, GPU/VRAM/temperature telemetry and the spend counter are on the same screen. The intervals where the health check failed are marked on the chart — unbilled time is visible to the eye.

If the node goes down, continue from the checkpoint

On an unhealthy node the metering stops, an incident record is written and the job can be moved to another host. Because the volume is unchanged, you resume from the last checkpoint rather than from zero.

Download the weights, shut the pod down, keep the volume

When you delete the pod the compute charge ends. If you keep the volume only storage keeps billing, and that row appears separately on the invoice — it is not hidden.

Fictional scenario

A typical job looks like this.

The flow below is not a real customer: it is an illustrative example constructed to show how the product works. The figures are examples too.

The job: QLoRA on an 8B model with your own Turkish support data, over one night.
The card: 1× RTX 4090, 24 GB, Istanbul — example price $0.42/hr.
Duration: 9 hours 14 minutes; roughly $3.88 of compute on this illustrative calculation.
The incident: at 06:12 the node fails its health check; those seconds are not billed and the job resumes from the last checkpoint.
CLIPythoncurl
# Sort cards above 24 GB by score $ kaldera offers --min-vram 24 --region eu_tr GPU VRAM REGION $/HR SCORECARD SCORE RTX 4090 24 Istanbul 0.42 4.9 0.90 RTX A6000 48 Ankara 0.88 4.6 0.63 H100 SXM 80 Helsinki 2.19 5.0 0.55 # Create a pod with a persistent volume and a cap $ kaldera pod create --offer kld-4090-ist \ --image pytorch/2.4-cuda12.4 \ --volume ckpt-8b:200 --budget-cap 12.00 ✓ pod ready → /vol mounted, cap $12.00 # Example output: unhealthy seconds never enter the bill $ kaldera billing usage --pod last compute 9h 14m $3.88 unbilled 6m 20s $0.00

Choosing a GPU

VRAM first, price second.

In fine-tuning, picking a card is a question of fitting before it is a question of speed: do the model, the optimizer state and the activations fit in VRAM? What follows is a rough direction; quantization, LoRA rank, context length and gradient checkpointing all change the table.

GPUVRAMWork it handles comfortably (example)Example starting price
RTX 409024 GB7B–8B QLoRA, 13B LoRA (short context), diffusion fine-tuning$0.42/hr
RTX A600048 GB13B LoRA comfortably, 7B full fine-tune, long-context experiments$0.88/hr
H100 SXM80 GB70B QLoRA, 13B full training, serious speed with bf16$2.19/hr

These are example prices and are illustrative; they move with supply and demand in the marketplace. The pool also holds intermediate steps such as the L40S (48 GB) and the A100 PCIe (80 GB) — the full list is on the Pricing page.

How does persistent storage work?

The volume is independent of the pod, stored encrypted, and stays when the pod is deleted. Details: Persistent Storage.

Pods and the marketplace

Offer search, scorecard score, region pinning and image selection: GPU Pods.

Honest limits

What exists today, what does not.

The product is in Phase 0: one account, one API and one invoice on top of provider pools. The list below is today's reality, not an advertisement for the roadmap.

Multi-GPU training on a single machinePhase 0 — available today
Persistent volume and checkpoint folderPhase 0 — available today
Hard budget cap, live cost counterPhase 0 — available today
Health gate: unbilled secondsPhase 0 — available today
Supply from our own host network (community GPUs)Phase 1 — soon
Ready-made training templates (model catalog)Soon — no date fixed
Multi-node cluster trainingSoon — no date fixed

Our phase schedule is written in our technical plan: Phase 0 broker MVP, Phase 1 our own supply, Phase 2 serverless inference, Phase 3 sovereignty and the Verified tier. If a feature says "soon" here, it does not exist today; we do not describe things as present when they are not.

FAQ

If my training crashes because of my own mistake, are those hours unbilled too?

No, they are billed. The guarantee covers the health of the infrastructure: if the GPU, driver, network or host goes down, those seconds never enter the bill. If your code OOMs or your script errors out, the GPU is up and running and the charge applies. The exact definition of the boundary will be written in the documentation; rather than hiding this distinction, we write it down.

Where do checkpoints live, and can the host see them?

On the persistent volume, encrypted. Secrets and environment variables never land on the host disk in plain text, images are signed, and the host cannot see inside the container. An audited "Verified" tier for sensitive workloads is a Phase 3 matter.

Do I lose my checkpoint when the budget cap is reached?

You do not. When the cap is reached the pod is paused, not deleted: the disk and volume stay in place and you get a notification. You raise the limit and continue where you left off. Storage for the paused pod keeps billing as a separate line item, and that row is visible on the billing screen.

Can I do multi-node training?

Not today. Multi-GPU training on a single machine works; for distributed training across nodes, the "Clusters" product is on the roadmap and carries a "soon" badge in the nav. We do not make promises without fixing a date.

Why are the prices so variable?

Because it is a marketplace price: supply is set by hosts and integrated providers, and competition pushes it down. We take a 15% commission on top and write that everywhere. The figures on this site are illustrative; the live ones appear in the console.

Get started

Start with a single overnight run; the downside is one coffee break.

Create a persistent volume, set your cap, pick your card. You don't pay for seconds that don't work — behind that sentence sit a health gate, a refund line item and an incident record.

The console runs on the local development address: localhost:3000