DigitalOcean
Machines that do one job and then stop existing
A cloud host. Plain virtual machines you create from an API, pay for by the hour, and destroy when the work is finished.
Some jobs are too big for one machine and too short-lived to deserve real infrastructure. Scraping a few hundred thousand domains is the clearest example: it needs a lot of parallel work for a day or two, and then nothing at all.
DigitalOcean is what I use for that. A script creates a set of machines, hands each one a slice of the work, collects the results, and destroys them. The whole fleet exists for as long as the job does.
What it is actually good at
The API is the product. Creating a machine, giving it a startup script and tearing it down are all one HTTP call each, which means a fleet is something a program can decide to build rather than something a person clicks together.
Hourly billing is what makes that worth doing. A machine that runs for six hours costs six hours. Fifteen machines for six hours costs the same as one machine for ninety, so the choice between “fast” and “cheap” mostly stops being a choice.
What it taught me that the documentation does not
A fleet you can create, you can forget. The failure here is not technical, it is that nothing reminds you. Machines left running after a job finished have cost me more than every job they ever ran. What fixed it was not discipline: it was a scheduled task that looks for idle machines and destroys them, so the default state is off.
Never write a teardown that takes “everything” as its target. The command that destroys a whole fleet is one typo away from destroying a fleet that is still working, or one that belongs to something else. Teardown scripts should name what they are allowed to touch and refuse anything else.
The bottleneck is rarely the machine. Doubling the fleet does nothing if the thing being scraped rate-limits by IP, or if the work is queued through one process. It is worth measuring where the time actually goes before paying for more machines, because the obvious answer is usually wrong.
Resources
This is one part of a stack. The rest is on the tools page, and what I build with it is on work.