Todays newsletter continues chapter 5 from Bitcoin: A Work in Progress.
I wrote a little Twitter thread about the problem of hiding your IP from spy nodes, especially if you make many transactions. It covers both the initial transaction broadcast to your peers, as well as subsequent rebroadcasting if it hasn’t confirmed yet.
One solution to the huge blockchain download problem was the use of checkpoints: Developers would put the hash and height of several known valid blocks in the source code, and any new block that didn’t descend from one of these checkpoints would be ignored. This didn’t completely undo dead-end branches, but it limited their maximum length.
The downside of checkpoints is that they potentially give a lot of power to developers. A malicious cabal of developers, or a benevolent dictator doing what’s best for the community — whichever perspective you prefer — could decree that a certain block is valid. Even if an alternative branch with more proof-of-work exists, nodes wouldn’t consider this branch.
Perhaps a developer loses their Bitcoin in a hack; they could then introduce a malicious checkpoint right before the hacked coins moved and move their coins to safety in the revised history. Such an attack can’t happen in secret, and if it ever really happened, users might simply refuse to install new node software with the checkpoint. But prevention would be better.
The last checkpoint was added in late 2014. Checkpoints were made mostly unnecessary by various means, including the introduction of nMinimumChainWork in 2016 (pull request). This parameter states how much proof-of-work any chain of headers must demonstrate before even being considered. But for this to work, it requires nodes to be less myopic; they need to consider where a given trail of blocks leads before spending lots of computer power chasing it. And that’s where Headers First comes in.
Addendum 2023: Bitcoin Core v24 introduced a feature that paves the way to completely getting rid of checkpoints, see episode 65
Given enough time — and if it doesn’t crash — your node would compare all blockchain branches and eventually pick the one with the most difficulty. But because it needs to verify every branch first, it could take a very long time to determine the correct one.
So rather than downloading and verifying entire blocks, the new approach is to download and verify just the headers, which are much smaller. In particular, headers are the only thing you need to determine the cumulative proof-of-work difficulty in any given branch.
Once your node knows which branch has the most proof-of-work, it downloads the blocks for it and starts verifying. This step can’t be skipped, because it’s still possible there’s an invalid block in the chain with the most proof-of-work. Should your node run into such an invalid block, it discards the branch and repeats the process for whichever branch had the second most proof-of-work.