Who cares about political tweets from some random country's president when payment channels are a much more interesting and are actually capable of carrying value?
So let's have a short history of various payment channel techs!
Generation 0: Satoshi's Broken nSequence Channels
Because Satoshi's Vision included payment channels, except his implementation sucked so hard we had to go fix it and added RBF as a by-product.
Originally, the plan for nSequence was that mempools would replace any transaction spending certain inputs with another transaction spending the same inputs, but only if the nSequence field of the replacement was larger.
Since 0xFFFFFFFF was the highest value that nSequence could get, this would mark a transaction as "final" and not replaceable on the mempool anymore.
In fact, this "nSequence channel" I will describe is the reason why we have this weird rule about nLockTime and nSequence. nLockTime actually only works if nSequence is not 0xFFFFFFFF i.e. final. If nSequence is 0xFFFFFFFF then nLockTime is ignored, because this if the "final" version of the transaction.
So what you'd do would be something like this:
- You go to a bar and promise the bartender to pay by the time the bar closes. Because this is the Bitcoin universe, time is measured in blockheight, so the closing time of the bar is indicated as some future blockheight.
- For your first drink, you'd make a transaction paying to the bartender for that drink, paying from some coins you have. The transaction has an nLockTime equal to the closing time of the bar, and a starting nSequence of 0. You hand over the transaction and the bartender hands you your drink.
- For your succeeding drink, you'd remake the same transaction, adding the payment for that drink to the transaction output that goes to the bartender (so that output keeps getting larger, by the amount of payment), and having an nSequence that is one higher than the previous one.
- Eventually you have to stop drinking. It comes down to one of two possibilities:
- You drink until the bar closes. Since it is now the nLockTime indicated in the transaction, the bartender is able to broadcast the latest transaction and tells the bouncers to kick you out of the bar.
- You wisely consider the state of your liver. So you re-sign the last transaction with a "final" nSequence of 0xFFFFFFFF i.e. the maximum possible value it can have. This allows the bartender to get his or her funds immediately (nLockTime is ignored if nSequence is 0xFFFFFFFF), so he or she tells the bouncers to let you out of the bar.
Now that of course is a payment channel. Individual payments (purchases of alcohol, so I guess buying coffee is not in scope for payment channels). Closing is done by creating a "final" transaction that is the sum of the individual payments. Sure there's no routing and channels are unidirectional and channels have a maximum lifetime but give Satoshi a break, he was also busy inventing Bitcoin at the time.
Now if you noticed I called this kind of payment channel "broken". This is because the mempool rules are not consensus rules, and cannot be validated (nothing
about the mempool can be validated onchain: I sigh every time somebody proposes "let's make block size dependent on mempool size", mempool state cannot be validated by onchain data). Fullnodes can't see all of the transactions you signed, and then validate that the final one with the maximum nSequence is the one that actually is used onchain. So you can do the below:
- Become friends with Jihan Wu, because he owns >51% of the mining hashrate (he totally reorged Bitcoin to reverse the Binance hack right?).
- Slip Jihan Wu some of the more interesting drinks you're ordering as an incentive to cooperate with you. So say you end up ordering 100 drinks, you split it with Jihan Wu and give him 50 of the drinks.
- When the bar closes, Jihan Wu quickly calls his mining rig and tells them to mine the version of your transaction with nSequence 0. You know, that first one where you pay for only one drink.
- Because fullnodes cannot validate nSequence, they'll accept even the nSequence=0 version and confirm it, immutably adding you paying for a single alcoholic drink to the blockchain.
- The bartender, pissed at being cheated, takes out a shotgun from under the bar and shoots at you and Jihan Wu.
- Jihan Wu uses his mystical chi powers (actually the combined exhaust from all of his mining rigs) to slow down the shotgun pellets, making them hit you as softly as petals drifting in the wind.
- The bartender mutters some words, clothes ripping apart as he or she (hard to believe it could be a she but hey) turns into a bear, ready to maul you for cheating him or her of the payment for all the 100 drinks you ordered from him or her.
- Steely-eyed, you stand in front of the bartender-turned-bear, daring him to touch you. You've watched Revenant, you know Leonardo di Caprio could survive a bear mauling, and if some posh actor can survive that, you know you can too. You make a pose. "Drunken troll logic attack!"
- I think I got sidetracked here.
- Bears are bad news.
- You can't reasonably invoke "Satoshi's Vision" and simultaneously reject the Lightning Network because it's not onchain. Satoshi's Vision included a half-assed implementation of payment channels with nSequence, where the onchain transaction represented multiple logical payments, exactly what modern offchain techniques do (except modern offchain techniques actually work). nSequence (the field, but not its modern meaning) has been in Bitcoin since BitCoin For Windows Alpha 0.1.0. And its original intent was payment channels. You can't get nearer to Satoshi's Vision than being a field that Satoshi personally added to transactions on the very first public release of the BitCoin software, like srsly.
- Miners can totally bypass mempool rules. In fact, the reason why nSequence has been repurposed to indicate "optional" replace-by-fee is because miners are already incentivized by the nSequence system to always follow replace-by-fee anyway. I mean, what do you think those drinks you passed to Jihan Wu are, other than the fee you pay him to mine a specific version of your transaction?
- Satoshi made mistakes. The original design for nSequence is one of them. Today, we no longer use nSequence in this way. So diverging from Satoshi's original design is part and parcel of Bitcoin development, because over time, we learn new lessons that Satoshi never knew about. Satoshi was an important landmark in this technology. He will not be the last, or most important, that we will remember in the future: he will only be the first.
Incentive-compatible time-limited unidirectional channel; or, Satoshi's Vision, Fixed (if transaction malleability hadn't been a problem, that is).
Now, we know the bartender will turn into a bear and maul you if you try to cheat the payment channel, and now that we've revealed you're good friends with Jihan Wu, the bartender will no longer accept a payment channel scheme that lets one you cooperate with a miner to cheat the bartender.
Fortunately, Jeremy Spilman proposed a better way that would not let you cheat the bartender.
First, you and the bartender perform this ritual:
- You get some funds and create a transaction that pays to a 2-of-2 multisig between you and the bartender. You don't broadcast this yet: you just sign it and get its txid.
- You create another transaction that spends the above transaction. This transaction (the "backoff") has an nLockTime equal to the closing time of the bar, plus one block. You sign it and give this backoff transaction (but not the above transaction) to the bartender.
- The bartender signs the backoff and gives it back to you. It is now valid since it's spending a 2-of-2 of you and the bartender, and both of you have signed the backoff transaction.
- Now you broadcast the first transaction onchain. You and the bartender wait for it to be deeply confirmed, then you can start ordering.
The above is probably vaguely familiar to LN users. It's the funding process of payment channels! The first transaction, the one that pays to a 2-of-2 multisig, is the funding transaction that backs the payment channel funds.
So now you start ordering in this way:
- For your first drink, you create a transaction spending the funding transaction output and sending the price of the drink to the bartender, with the rest returning to you.
- You sign the transaction and pass it to the bartender, who serves your first drink.
- For your succeeding drinks, you recreate the same transaction, adding the price of the new drink to the sum that goes to the bartender and reducing the money returned to you. You sign the transaction and give it to the bartender, who serves you your next drink.
- At the end:
- If the bar closing time is reached, the bartender signs the latest transaction, completing the needed 2-of-2 signatures and broadcasting this to the Bitcoin network. Since the backoff transaction is the closing time + 1, it can't get used at closing time.
- If you decide you want to leave early because your liver is crying, you just tell the bartender to go ahead and close the channel (which the bartender can do at any time by just signing and broadcasting the latest transaction: the bartender won't do that because he or she is hoping you'll stay and drink more).
- If you ended up just hanging around the bar and never ordering, then at closing time + 1 you broadcast the backoff transaction and get your funds back in full.
Now, even if you pass 50 drinks to Jihan Wu, you can't give him the first transaction (the one which pays for only one drink) and ask him to mine it: it's spending a 2-of-2 and the copy you have only contains your own signature. You need the bartender's signature to make it valid, but he or she sure as hell isn't going to cooperate in something that would lose him or her money, so a signature from the bartender validating old state where he or she gets paid less isn't going to happen.
So, problem solved, right? Right? Okay, let's try it. So you get your funds, put them in a funding tx, get the backoff tx, confirm the funding tx...
Once the funding transaction confirms deeply, the bartender laughs uproariously. He or she summons the bouncers, who surround you menacingly.
"I'm refusing service to you," the bartender says.
"Fine," you say. "I was leaving anyway;" You smirk. "I'll get back my money with the backoff transaction, and posting about your poor service on reddit so you get negative karma, so there!"
"Not so fast," the bartender says. His or her voice chills your bones. It looks like your exploitation of the Satoshi nSequence payment channel is still fresh in his or her mind. "Look at the txid of the funding transaction that got confirmed."
"What about it?" you ask nonchalantly, as you flip open your desktop computer and open a reputable blockchain explorer.
What you see shocks you.
"What the --- the txid is different! You--- you changed my signature
?? But how? I put the only copy of my private key in a sealed envelope in a cast-iron box inside a safe buried in the Gobi desert protected by a clan of nomads who have dedicated their lives and their childrens' lives to keeping my private key safe in perpetuity!"
"Didn't you know?" the bartender asks. "The components of the signature are just very large numbers. The sign of one of the signature components can be changed, from positive to negative, or negative to positive, and the signature will remain valid. Anyone can do that, even if they don't know the private key. But because Bitcoin includes the signatures in the transaction when it's generating the txid, this little change also changes the txid." He or she chuckles. "They say they'll fix it by sep
arating the sig
natures from the transaction body. They're saying that these kinds of signature malleability won't affect transaction ids anymore after they do this, but I bet I can get my good friend Jihan Wu to delay this 'SepSig' plan for a good while yet. Friendly guy, this Jihan Wu, it turns out all I had to do was slip him 51 drinks and he was willing to mine a tx with the signature signs flipped." His or her grin widens. "I'm afraid your backoff transaction won't work anymore, since it spends a txid that is not existent and will never be confirmed. So here's the deal. You pay me 99% of the funds in the funding transaction, in exchange for me signing the transaction that spends with the txid that you see onchain. Refuse, and you lose 100% of the funds and every other HODLer, including me, benefits from the reduction in coin supply. Accept, and you get to keep 1%. I lose nothing if you refuse, so I won't care if you do, but consider the difference of getting zilch vs. getting 1% of your funds." His or her eyes glow. "GENUFLECT RIGHT NOW."
- Payback's a bitch.
- Transaction malleability is a bitchier bitch. It's why we needed to fix the bug in SegWit. Sure, MtGox claimed they were attacked this way because someone kept messing with their transaction signatures and thus they lost track of where their funds went, but really, the bigger impetus for fixing transaction malleability was to support payment channels.
- Yes, including the signatures in the hash that ultimately defines the txid was a mistake. Satoshi made a lot of those. So we're just reiterating the lesson "Satoshi was not an infinite being of infinite wisdom" here. Satoshi just gets a pass because of how awesome Bitcoin is.
CLTV-protected Spilman Channels
Using CLTV for the backoff branch.
This variation is simply Spilman channels, but with the backoff transaction replaced with a backoff branch in the SCRIPT you pay to. It only became possible after OP_CHECKLOCKTIMEVERIFY (CLTV) was enabled in 2015.
Now as we saw in the Spilman Channels discussion, transaction malleability means that any pre-signed offchain transaction can easily be invalidated by flipping the sign of the signature of the funding transaction while the funding transaction is not yet confirmed.
This can be avoided by simply putting any special requirements into an explicit branch of the Bitcoin SCRIPT. Now, the backoff branch is supposed to create a maximum lifetime for the payment channel, and prior to the introduction of OP_CHECKLOCKTIMEVERIFY this could only be done by having a pre-signed nLockTime transaction.
With CLTV, however, we can now make the branches explicit in the SCRIPT that the funding transaction pays to.
Instead of paying to a 2-of-2 in order to set up the funding transaction, you pay to a SCRIPT which is basically "2-of-2, OR this singlesig after a specified lock time".
With this, there is no backoff transaction that is pre-signed and which refers to a specific txid. Instead, you can create the backoff transaction later, using whatever txid the funding transaction ends up being confirmed under. Since the funding transaction is immutable once confirmed, it is no longer possible to change the txid afterwards.
Todd Micropayment Networks
The old hub-spoke model (that isn't how LN today actually works).
One of the more direct predecessors of the Lightning Network was the hub-spoke model discussed by Peter Todd. In this model, instead of payers directly having channels to payees, payers and payees connect to a central hub server. This allows any payer to pay any payee, using the same channel for every payee on the hub. Similarly, this allows any payee to receive from any payer, using the same channel.
Remember from the above Spilman example? When you open a channel to the bartender, you have to wait around for the funding tx to confirm. This will take an hour at best
. Now consider that you have to make channels for everyone you want to pay to. That's not very scalable.
So the Todd hub-spoke model has a central "clearing house" that transport money from payers to payees. The "Moonbeam" project takes this model. Of course, this reveals to the hub who the payer and payee are, and thus the hub can potentially censor transactions. Generally, though, it was considered that a hub would more efficiently censor by just not maintaining a channel with the payer or payee that it wants to censor (since the money it owned in the channel would just be locked uselessly if the hub won't process payments to/from the censored user).
In any case, the ability of the central hub to monitor payments means that it can surveill the payer and payee, and then sell this private transactional data to third parties. This loss of privacy would be intolerable today.
Peter Todd also proposed that there might be multiple hubs that could transport funds to each other on behalf of their users, providing somewhat better privacy.
Another point of note is that at the time such networks were proposed, only unidirectional (Spilman) channels were available. Thus, while one could be a payer, or payee, you would have to use separate channels for your income versus for your spending. Worse, if you wanted to transfer money from your income channel to your spending channel, you had to close both and reshuffle the money between them, both onchain activities.
Poon-Dryja Lightning Network
Bidirectional two-participant channels.
The Poon-Dryja channel mechanism has two important properties:
- No time limit.
Both the original Satoshi and the two Spilman variants are unidirectional: there is a payer and a payee, and if the payee wants to do a refund, or wants to pay for a different service or product the payer is providing, then they can't use the same unidirectional channel.
The Poon-Dryjam mechanism allows channels, however, to be bidirectional instead: you are not a payer or a payee on the channel, you can receive or send at any time as long as both you and the channel counterparty are online.
Further, unlike either of the Spilman variants, there is no time limit for the lifetime of a channel. Instead, you can keep the channel open for as long as you want.
Both properties, together, form a very powerful scaling property
that I believe most people have not appreciated. With unidirectional channels, as mentioned before, if you both earn and spend over the same network of payment channels, you would have separate channels for earning and spending. You would then need to perform onchain operations to "reverse" the directions of your channels periodically. Secondly, since Spilman channels have a fixed lifetime, even if you never used either channel, you would have to periodically "refresh" it by closing it and reopening.
With bidirectional, indefinite-lifetime channels, you may instead open some channels when you first begin managing your own money, then close them only after your lawyers have executed your last will and testament on how the money in your channels get divided up to your heirs: that's just two onchain transactions in your entire lifetime. That is the potentially very powerful scaling property that bidirectional, indefinite-lifetime channels allow.
I won't discuss the transaction structure needed for Poon-Dryja bidirectional channels --- it's complicated and you can easily get explanations with cute graphics elsewhere.
a weakness of Poon-Dryja that people tend to gloss over (because it was fixed very well by RustyReddit
- You have to store all the revocation keys of a channel. This implies you are storing 1 revocation key for every channel update, so if you perform millions of updates over your entire lifetime, you'd be storing several megabytes of keys, for only a single channel. RustyReddit fixed this by requiring that the revocation keys be generated from a "Seed" revocation key, and every key is just the application of SHA256 on that key, repeatedly. For example, suppose I tell you that my first revocation key is SHA256(SHA256(seed)). You can store that in O(1) space. Then for the next revocation, I tell you SHA256(seed). From SHA256(key), you yourself can compute SHA256(SHA256(seed)) (i.e. the previous revocation key). So you can remember just the most recent revocation key, and from there you'd be able to compute every previous revocation key. When you start a channel, you perform SHA256 on your seed for several million times, then use the result as the first revocation key, removing one layer of SHA256 for every revocation key you need to generate. RustyReddit not only came up with this, but also suggested an efficient O(log n) storage structure, the shachain, so that you can quickly look up any revocation key in the past in case of a breach. People no longer really talk about this O(n) revocation storage problem anymore because it was solved very very well by this mechanism.
Another thing I want to emphasize is that while the Lightning Network paper and many of the earlier presentations developed from the old Peter Todd hub-and-spoke model, the modern Lightning Network takes the logical conclusion of removing a strict separation between "hubs" and "spokes". Any node on the Lightning Network can very well work as a hub for any other node. Thus, while you might operate as "mostly a payer", "mostly a forwarding node", "mostly a payee", you still end up being at least partially a forwarding node ("hub") on the network, at least part of the time. This greatly reduces the problems of privacy inherent in having only a few hub nodes: forwarding nodes cannot get significantly useful data from the payments passing through them, because the distance between the payer and the payee can be so large that it would be likely that the ultimate payer and the ultimate payee could be anyone on the Lightning Network.
- We can decentralize if we try hard enough!
- "Hubs bad" can be made "hubs good" if everybody is a hub.
- Smart people can solve problems. It's kinda why they're smart.
After LN, there's also the Decker-Wattenhofer Duplex Micropayment Channels (DMC). This post is long enough as-is, LOL. But for now, it uses a novel "decrementing nSequence channel", using the new
relative-timelock semantics of nSequence (not the broken one originally by Satoshi). It actually uses multiple such "decrementing nSequence" constructs, terminating in a pair of Spilman channels, one in both directions (thus "duplex"). Maybe I'll discuss it some other time.
The realization that channel constructions could actually hold more channel constructions inside them (the way the Decker-Wattenhofer puts a pair of Spilman channels inside a series of "decrementing nSequence channels") lead to the further thought behind Burchert-Decker-Wattenhofer channel factories. Basically, you could host multiple two-participant channel constructs inside a larger multiparticipant "channel" construct (i.e. host multiple channels inside a factory).
Further, we have the Decker-Russell-Osuntokun or "eltoo" construction. I'd argue that this is "nSequence done right". I'll write more about this later, because this post is long enough.
- Bitcoin offchain scaling is more powerful than you ever thought.
On May 6th, 2017, Bitcoin hit an all-time high in transactions processed on the network in a single day: it moved 375,000 transactions which accounted for a nominal output of about $2.5b. Average fees on the Bitcoin network had climbed over a dollar for the first time a couple days prior. And they kept climbing: by early June average fees hit an eye-watering $5.66. This was quite unprecedented. In the three-year period from Jan. 1 2014 to Jan. 1 2017, per-transaction fees had never exceeded 31 cents on a weekly average. And the hits kept coming. Before 2017 was over, average fees would top out at $48 on a weekly basis. When the crypto-recession set in, transaction count collapsed and fees crept back below $1.
During the most feverish days of the Bitcoin run-up, when normal users found themselves with balances that would cost more to send than they were worth, cries for batching — the aggregation of many outputs into a single transaction — grew louder than ever. David Harding had written a blog post on the cost-savings of batching at the end of August and it was reposted to the Bitcoin subreddit on a daily basis.
The idea was simple: for entities sending many transactions at once, clustering outputs into a single transaction was more space- (and cost-) efficient, because each transaction has a fixed data overhead. David found that if you combined 10 payments into one transaction, rather than sending them individually, you could save 75% of the block space. Essentially, batching is one way to pack as many transactions as possible into the finite block space available on Bitcoin.
When fees started climbing in mid-2017, users began to scrutinize the behavior of heavy users of the Bitcoin blockchain, to determine whether they were using block space efficiently. By and large, they were not — and an informal lobbying campaign began, in which these major users — principally exchanges — were asked to start batching transactions and be good stewards of the scarce block space at their disposal. Some exchanges had been batching for years, others relented and implemented it. The question faded from view after Bitcoin’s price collapsed in Q1 2018 from roughly $19,000 to $6000, and transaction load — and hence average fee — dropped off.
But we remained curious. A common refrain, during the collapse in on-chain usage, was that transaction count was an obfuscated method of apprehending actual usage. The idea was that transactions could encode an arbitrarily large (within reason) number of payments, and so if batching had become more and more prevalent, those payments were still occurring, just under a regime of fewer transactions.
Some sites popped up to report outputs and payments per day rather than transactions, seemingly bristling at the coverage of declining transaction count. However, no one conducted an analysis of the changing relationship between transaction count and outputs or payments. We took it upon ourselves to find out.
Table Of Contents:
Introduction to batching
Bonus content: UTXO consolidation
- Introduction to batching
Bitcoin uses a UTXO model, which stands for Unspent Transaction Output. In comparison, Ripple and Ethereum use an account/balance model. In bitcoin, a user has no balances, only UTXOs that they control. If they want to transfer money to someone else, their wallet selects one or more UTXOs as inputs that in sum need to add up to the amount they want to transfer. The desired amount then goes to the recipient, which is called the output, and the difference goes back to the sender, which is called change output. Each output can carry a virtually unlimited amount of value in the form of satoshis. A satoshi is a unit representing a one-hundred-millionth of a Bitcoin. This is very similar to a physical wallet full of different denominations of bills. If you’re buying a snack for $2.50 and only have a $5, you don’t hand the cashier half of your 5 dollar bill — you give him the 5 and receive some change instead.
Unknown to some, there is no hardcoded limit to the number of transactions that can fit in a block. Instead, each transaction has a certain size in megabytes and constitutes an economic incentive for miners to include it in their block. Because miners have limited space of 2 MB to sell to transactors, larger transactions (in size, not bitcoin!) will need to pay higher fees to be included. Additionally, each transaction can have a virtually unlimited number of inputs or outputs — the record stands at transactions with 20,000 inputs and 13,107 outputs.
So each transaction has at least one input and at one output, but often more, as well as some additional boilerplate stuff. Most of that space is taken up by the input (often 60% or more, because of the signature that proves they really belong to the sender), while the output(s) account for 15–30%. In order to keep transactions as small as possible and save fees, Bitcoin users have two major choices:
Use as few inputs as possible. In order to minimize inputs, you can periodically send your smaller UTXOs to yourself in times when fees are very low, getting one large UTXO back. That is called UTXO consolidation or consolidating your inputs.
Users who frequently make transfers (especially within the same block) can include an almost unlimited amount of outputs (to different people!) in the same transaction. That is called transaction batching. A typical single output transaction takes up 230 bytes, while a two output transaction only takes up 260 bytes, instead of 460 if you were to send them individually.
This is something that many casual commentators overlook when comparing Bitcoin with other payment systems — a Bitcoin transaction can aggregate thousands of individual economic transfers! It’s important to recognize this, as it is the source of a great deal of misunderstanding and mistaken analysis.
We’ve never encountered a common definition of a batched transaction — so for the purposes of this study we define it in the loosest possible sense: a transaction with three or more outputs. Commonly, batching is understood as an activity undertaken primarily by mining pools or exchanges who can trade off immediacy for efficiency. It is rare that a normal bitcoin user would have cause to batch, and indeed most wallets make it difficult to impossible to construct batched transactions. For everyday purposes, normal bitcoiners will likely not go to the additional effort of batching transactions.
We set the threshold at three for simplicity’s sake — a normal unbatched transaction will have one transactional output and one change output — but the typical major batched transaction from an exchange will have dozens if not hundreds of outputs. For this reason we are careful to provide data on various different batch sizes, so we could determine the prevalence of three-output transactions and colossal, 100-output ones.
We find it helpful to think of a Bitcoin transaction as a mail truck full of boxes. Each truck (transaction) contains boxes (outputs), each of contains some number of letters (satoshis). So when you’re looking at transaction count as a measure of the performance and economic throughput of the Bitcoin network, it’s a bit like counting mail trucks to discern how many letters are being sent on a given day, even though the number of letters can vary wildly. The truck analogy also makes it clear why many see Bitcoin as a settlement layer in the future — just as mail trucks aren’t dispatched until they’re full, some envision that the same will ultimately be the case for Bitcoin.
- A timeline
So what actually happened in the last six months? Let’s look at some data. Daily transactions on the Bitcoin network rose steadily until about May 2017, when average fees hit about $4. This precipitated the first collapse in usage. Then began a series of feedback loops over the next six months in which transaction load grew, fees grew to match, and transactions dropped off. This cycle repeated itself five times over the latter half of 2017.
more like this on coinmetrics.io
The solid red line in the above chart is fees in BTC terms (not USD) and the shaded red area is daily transaction count. You can see the cycle of transaction load precipitating higher fees which in turn cause a reduction in usage. It repeats itself five or six times before the detente in spring 2018. The most notable period was the December-January fee crisis, but fees were actually fairly typical in BTC terms — the rising BTC price in USD however meant that USD fees hit extreme figures.
In mid-November when fees hit double digits in USD terms, users began a concerted campaign to convince exchanges to be better stewards of block space. Both Segwit and batching were held up as meaningful approaches to maximize the compression of Bitcoin transactions into the finite block space available. Data on when exchanges began batching is sparse, but we collected information where it was available into a chart summarizing when exchanges began batching.
Batching adoption at selected exchanges
We’re ignoring Segwit adoption by exchanges in this analysis; as far as batching is concerned, the campaign to get exchanges to batch appears to have persuaded Bitfinex, Binance, and Shapeshift to batch. Coinbase/GDAX have stated their intention to begin batching, although they haven’t managed to integrate it yet. As far as we can tell, Gemini hasn’t mentioned batching, although we have some mixed evidence that they may have begun recently. If you know about the status of batching on Gemini or other major exchanges please get in touch.
So some exchanges have been batching all along, and some have never bothered at all. Did the subset of exchanges who flipped the switch materially affect the prevalence of batched transactions? Let’s find out.
3.1 How common is batching?
We measured the prevalence of batching in three different ways, by transaction count, by output value and by output count.
Batching accounts for roughly 12% of all transactions, 40% of all outputs, and 30–60% of all raw BTC output value. Not bad.
3.2 Have batched transactions become more common over time?
From the chart in 3.1, we can already see a small, but steady uptrend in all three metrics, but we want to dig a little deeper. So we first looked at the relationship of payments (all outputs that actually pay someone, so total outputs minus change outputs) and transactions.
More at transactionfee.info/charts
The first thing that becomes obvious is that the popular narrative — that the drop in transactions was caused by an increase in batching — is not the case; payments dropped by roughly the same proportion as well.
Dividing payment count by transaction count gives us some insight into the relationship between the two.
In our analysis we want to zoom into the time frame between November 2017 and today, and we can see that payments per transactions have actually been rallying, from 1.5 payments per transaction in early 2017 to almost two today.
3.3 What are popular batch sizes?
In this next part, we will look at batch sizes to see which are most popular. To determine which transactions were batched, we downloaded a dataset of all transactions on the Bitcoin network between November 2017 and May 2018from Blockchair.
We picked that period because the fee crisis really got started in mid-November, and with it, the demands for exchanges to batch. So we wanted to capture the effect of exchanges starting to batch. Naturally a bigger sample would have been more instructive, but we were constrained in our resources, so we began with the six month sample.
We grouped transactions into “batched” and “unbatched” groups with batched transactions being those with three or more outputs.
We then divided batched transactions into roughly equal groups on the basis of how much total output in BTC they had accounted for in the six-month period. We didn’t select the batch sizes manually — we picked batch sizes that would split the sample into equal parts on the basis of transaction value. Here’s what we ended up with:
All of the batch buckets have just about the same fraction of total BTC output over the period, but they account for radically different transaction and output counts over the period. Notice that there were only 183,108 “extra large” batches (with 41 or more outputs) in the six-month period, but between them there were 23m outputs and 30m BTC worth of value transmitted.
Note that output value in this context refers to the raw or unadjusted figure — it would have been prohibitively difficult for us to adjust output for change or mixers, so we’re using the “naive” estimate.
Let’s look at how many transactions various batch sizes accounted for in the sample period:
Batched transactions steadily increased relative to unbatched ones, although the biggest fraction is the small batch with between 3 and 5 outputs. The story for output counts is a bit more illuminating. Even though batched transactions are a relatively small fraction of overall transaction count, they contain a meaningful number of overall outputs. Let’s see how it breaks down:
Lastly, let’s look at output value. Here we see that batched transactions represent a significant fraction of value transmitted on Bitcoin.
As we can see, even though batched transactions make up an average of only 12% of all transactions, they move between 30%-60% of all Bitcoins, at peak times even 70%. We think this is quite remarkable. Keep in mind, however that the ‘total output’ figure has not been altered to account for change outputs, mixers, or self-churn; that is, it is the raw and unadjusted figure. The total output value is therefore not an ideal approximation of economic volume on the Bitcoin network.
3.4 Has transaction count become an unreliable measure of Bitcoin’s usage because of batching?
Yes. We strongly encourage any analysts, investors, journalists, and developers to look past mere transaction count from now on. The default measure of Bitcoin’s performance should be “payments per day” rather than transaction count. This also makes Bitcoin more comparable with other UTXO chains. They generally have significantly variable payments-per-transaction ratios, so just using payments standardizes that. (Stay tuned: Coinmetrics will be rolling out tools to facilitate this very soon.)
More generally, we think that the economic value transmitted on the network is its most fundamental characteristic. Both the naive and the adjusted figures deserve to be considered. Adjusting raw output value is still more art than science, and best practices are still being developed. Again, Coinmetrics is actively developing open-source tools to make these adjustments available.
We started by revisiting the past year in Bitcoin and showed that while the mempool was congested, the community started looking for ways to use the blockspace more efficiently. Attention quickly fell on batching, the practice of combining multiple outputs into a single transaction, for heavy users. We showed how batching works on a technical level and when different exchanges started implementing the technique.
Today, around 12% of all transactions on the Bitcoin network are batched, and these account for about 40% of all outputs and between 30–60% of all transactional value. The fact such that a small set of transactions carries so much economic weight makes us hopeful that Bitcoin still has a lot of room to scale on the base layer, especially if usage trends continue.
Lastly, it’s worth noting that the increase in batching on the Bitcoin network may not be entirely due to deliberate action by exchanges, but rather a function of its recessionary behavior in the last few months. Since batching is generally done by large industrial players like exchanges, mixers, payment processors, and mining pools, and unbatched transactions are generally made by normal individuals, the batched/unbatched ratio is also a strong proxy for how much average users are using Bitcoin. Since the collapse in price, it is quite possible that individual usage of Bitcoin decreased while “industrial” usage remained strong. This is speculation, but one explanation for what happened.
Alternatively, the industrial players appear to be taking their role as stewards of the scarce block space more seriously. This is a significant boon to the network, and a nontrivial development in its history. If a culture of parsimony can be encouraged, Bitcoin will be able to compress more data into its block space and everyday users will continue to be able to run nodes for the foreseeable future. We view this as a very positive development. Members of the Bitcoin community that lobbied exchanges to add support for Segwit and batching should be proud of themselves.
- Bonus content: UTXO consolidation
Remember that we said that a second way to systematically save transaction fees in the Bitcoin network was to consolidate your UTXOs when fees were low? Looking at the relationship between input count and output count allows us to spot such consolidation phases quite well.
Typically, inputs and outputs move together. When the network is stressed, they decouple. If you look at the above chart carefully, you’ll notice that when transactions are elevated (and block space is at a premium), outputs outpace inputs — look at the gaps in May and December 2017. However, prolonged activity always results in fragmented UTXO sets and wallets full of dust, which need to be consolidated. For this, users often wait until pressure on the network has decreased and fees are lower. Thus, after transactions decrease, inputs become more common than outputs. You can see this clearly in February/March 2017.
Here we’ve taken the ratio of inputs to outputs (which have been smoothed on a trailing 7 day basis). When the ratio is higher, there are more inputs than outputs on that day, and vice versa. You can clearly see the spam attack in summer 2015 in which thousands (possibly millions) of outputs were created and then consolidated. Once the ratio spikes upwards, that’s consolidation. The spike in February 2018 after the six weeks of high fees in December 2017 was the most pronounced sigh of relief in Bitcoin’s history; the largest ever departure from the in/out ratio norm. There were a huge number of UTXOs to be consolidated.
It’s also interesting to note where inputs and outputs cluster. Here we have histograms of transactions with large numbers of inputs or outputs. Unsurprisingly, round numbers are common which shows that exchanges don’t publish a transaction every, say, two minutes, but instead wait for 100 or 200 outputs to queue up and then publish their transaction. Curiously, 200-input transactions were more popular than 100-input transactions in the period.
We ran into more curiosities when researching this piece, but we’ll leave those for another time.
Future work on batching might focus on:
Determining batched transactions as a portion of (adjusted) economic rather than raw volume
Looking at the behavior of specific exchanges with regards to batching
Investigating how much space and fees could be saved if major exchanges were batching transactions
Lastly, we encourage everyone to run their transactions through the service at transactionfee.info
to assess the efficiency of their transactions and determine whether exchanges are being good stewards of the block space.
Antoine Le Calvez has created a series of live-updated charts to track batching and batch sizes, which you can find here.
We’d like to thank 0xB10C for their generous assistance with datasets and advice, the people at Blockchair for providing the core datasets, and David A. Harding for writing the initial piece and answering our questions.
Bitcoin Price (BTC USD): Get all information on the Bitcoin to US-Dollar Exchange Rate including Charts, News and Realtime Price. Bitcoin may now have a fee market, but that reality doesn't mean the network has optimized for this environment, developer Jameson Lopp argues. News Learn Videos Research. Trending. Binance ... With Bitcoin, once a miner has authenticated 1 megabyte worth of transactions, they become eligible to win 12.5 BTC. This megabyte limit was set by the founder of Bitcoin, Satoshi Nakamoto, has been a point of contention, as some people believe that the block size should be enlarged to handle more data. It’s important to note that 1 MB of data makes a single miner eligible to earn some ... how many satoshi equal a bitcoin - The Transaction fee in BTC does not matter to the miner. satoshi/Byte matters to him. In the above example we can calculate, 0.01 BTC 512 KB = 0.00001950 BTC/KB = 1950 satoshi/KB. Satoshi per Byte in this case will be. 1950 sat 1024 = 1.904 satoshi/Byte. For the other example with 512KB transaction size and 0.015 BTC fee, 0.015 BTC 512 KB x 10 ... Learn about bitcoin fees... Bitcoin is made up of blocks.Blocks are a set of transactions, and currently restricted to be less than or equal to 1,000,000 bytes and designed so that on average only 1 block per ~10 minutes can be created. The groups the create blocks are known as bitcoin miners.These miners can pick which ever transactions they want in the block they create. bitcoin mining hashrate calculator - Find out what your expected return is depending on your hash rate and electricity cost. Find out if it's profitable to mine Bitcoin, Ethereum, Litecoin, DASH or Monero. Do you think you've got what it takes to join the tough world of cryptocurrency mining? - Hashrates.com is the leading reference site for checking GPU and CPU hash rates. Binance = One the fastest growing exchange for trading cryptocurrencies. Binance Coin = Binance Coin is the digital currency on the Binance ecosystem. The asset is an ERC-20 token which means it is based on the Ethereum network. BIP = "Bitcoin Improvement Proposal" is a standard to submit potential changes or improvements that will have a positive effect on the Bitcoin protocol as a whole. BIP ... The mebibyte is a multiple of the unit byte for digital information. The binary prefix mebi means 220, therefore one mebibyte is equal to 1048576bytes = 1024 kibibytes. The unit symbol for the mebibyte is MiB. Technically a megabyte (MB) is a power of ten, while a mebibyte (MiB) is a power of two, appropriate for binary machines. The argument is a false one, it’s a straw man used by people in CoreCoin, or SegueCoin, or anarchist, loser coin, you can use any of those names, they’re all equal. But anarchist, loser coin, with their 1Mg cap, that’s the one that trades as BTC falsely…so, what they do is if you get that 1Mg cap full, your transaction may not get in. That’s it. So Bitcoin wasn’t originally ... In Bitcoin terms, simultaneous answers occur frequently, but at the end of the day there can only be one winning answer. When multiple simultaneous answers are presented that are equal to or less than the target number, the Bitcoin network will decide by a simple majority--51%--which miner to honor. Typically, it is the miner who has done the most work, i.e. verifies the most transactions. The ...
Binance SG Singapore Exchange has extended their referral promotion for Singaporean / people living in Singapore to get a chance to get an easy entry into the cryptocurrency space! This promotion ... Crypto is going through another hard correction, also lets have a look at a Achain staking guide Check out the first episode of the series here: https://goo.gl/LNPt88 Come join the Alphcrypto Army ... Results Not Typical – This proof of earnings is not a guarantee that you would earn the same, but it is possible to earn this much or more with an equal or greater strategy and work ethic. Mine ... Ábrete una cuenta en Binance con este link y empieza a hacer trading ya: ... Que uso para comprar Bitcoin con Fiat: Coinbase (Utiliza el código y recibe $10 en tu primera compra): https://www ... What is Bitcoin? Bitcoin is a cryptocurrency. It is a decentralized digital currency without a central bank or single administrator that can be sent from user to user on the peer-to-peer bitcoin ... If you want to create an additional source of income or to trade Forex professionally to replace your job, this Professional Forex Trading course will give y... 🏆 Buy Bitcoin to Trade at Coinbase and get $15 discount: https://bit.ly/2Us4Qto 🏆 Get $30 off a TradingView account: https://bit.ly/3bBGOSs 🏆 Get 10% reduction in fees for 6 months by ... NOTICE: All Paid Reviews And Features on my channel were paid for by the crypto companies in the form of Bitcoin ranging from .2 to 1 BTC or in some cases equal or double value in the project's ... As long as the current weeks locked tokens are equal to or more than the previous weeks locked tokens, you will be entitled to the additional 0.3% rewards. The 0.3% rewards will be compounded on a ... I JUST GOT FREE BITCOIN DIAMONDS HAHA! W/ only 5yr old mobilephone apps no fees. All apps for free W/ only 5yr old mobilephone apps no fees. All apps for free - Duration: 7:31.