Discussion:
[bitcoin-dev] Transaction Merging (bip125 relaxation)
Rhavar via bitcoin-dev
2018-01-22 17:40:31 UTC
Permalink
So my half-baked idea is very simple:

Allow users to merge multiple unconfirmed transactions, stripping extraneous inputs and change as they go.

This is currently not possible because of the bip125 rule:
"The replacement transaction pays an absolute fee of at least the sum paid by the original transactions."

Because the size of the merged transaction is smaller than the original transactions, unless there is a considerable feerate bump, this rule isn't possible to observe.

I my question is: is it possible or reasonable to relax this rule? If this rule was removed in its entirety, does it introduce any DoS vectors? Or can it be changed to allow my use-case?

---
Full backstory: I have been trying to use bip125 (Opt-in Full Replace-by-Fee) to do "transaction merging" on the fly. Let's say that I owe John 1 bitcoin, and have promised to pay him immediately: Instead of creating a whole new transaction if I have an in-flight (unconfirmed) transaction, I can follow the rules of bip125 to create a replacement that accomplishes this goal.

From a "coin selection" point of view, this was significantly easier than
I had anticipated. I was able to encode the rules in my linear model and
feed in all my unspent and in-flight transactions and it can solve it without difficulty.

However, the real problem is tracking the mess. Consider this sequence of events:
1) I have unconfirmed transaction A
2) I replace it with B, which pays John 1 BTC
3) Transaction A gets confirmed

So now I still owe John 1 BTC, however it's not immediately clear if
it's safe to send to him without waiting $n transactions. However even
for a small $n, this breaks my promise to pay him immediately.

One possible solution is to only consider a transaction "replaceable" if it has change, so if the original transaction confirms -- payments can immediately be made that source the change, and provide safety in a reorg.

However, this will only work <50% of the time for me (most transactions
don't have change) and opens a pandora's box of complexity.

There's a few other hacks you can do to make it work in a few more cases, but nothing that is realistic to expect anyone to implement any time soon.

However, if there was a straight foward way to merge N unconfirmed transactions, it would be easy get into production, and potentially offer some pretty nice savings for everyone.
Rhavar via bitcoin-dev
2018-01-22 18:18:14 UTC
Permalink
If you spent your change from transaction A, that would be safe. There'd be no way you John could end up with 2 BTC from you then.
Yes, that's what the following paragraph says -- along with it's limitations =)

-Ryan

-------- Original Message --------
So now I still owe John 1 BTC, however it's not immediately clear if it's safe to send to him
If you spent your change from transaction A, that would be safe. There'd be no way you John could end up with 2 BTC from you then.
Allow users to merge multiple unconfirmed transactions, stripping extraneous inputs and change as they go.
"The replacement transaction pays an absolute fee of at least the sum paid by the original transactions."
Because the size of the merged transaction is smaller than the original transactions, unless there is a considerable feerate bump, this rule isn't possible to observe.
I my question is: is it possible or reasonable to relax this rule? If this rule was removed in its entirety, does it introduce any DoS vectors? Or can it be changed to allow my use-case?
---
Full backstory: I have been trying to use bip125 (Opt-in Full Replace-by-Fee) to do "transaction merging" on the fly. Let's say that I owe John 1 bitcoin, and have promised to pay him immediately: Instead of creating a whole new transaction if I have an in-flight (unconfirmed) transaction, I can follow the rules of bip125 to create a replacement that accomplishes this goal.
From a "coin selection" point of view, this was significantly easier than
I had anticipated. I was able to encode the rules in my linear model and
feed in all my unspent and in-flight transactions and it can solve it without difficulty.
1) I have unconfirmed transaction A
2) I replace it with B, which pays John 1 BTC
3) Transaction A gets confirmed
So now I still owe John 1 BTC, however it's not immediately clear if
it's safe to send to him without waiting $n transactions. However even
for a small $n, this breaks my promise to pay him immediately.
One possible solution is to only consider a transaction "replaceable" if it has change, so if the original transaction confirms -- payments can immediately be made that source the change, and provide safety in a reorg.
However, this will only work <50% of the time for me (most transactions
don't have change) and opens a pandora's box of complexity.
There's a few other hacks you can do to make it work in a few more cases, but nothing that is realistic to expect anyone to implement any time soon.
However, if there was a straight foward way to merge N unconfirmed transactions, it would be easy get into production, and potentially offer some pretty nice savings for everyone.
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Moral Agent via bitcoin-dev
2018-01-22 18:50:31 UTC
Permalink
Along the same lines, I wonder if unrelated people with tx that are not
confirming could cooperate to merge their disparate tx into a CoinJoin tx
with a higher fee rate?

Perhaps they could even replace old tx with economically equivalent summary
transactions?

The mempool seems like nature's accumulator for pre-mining compression
opportunities.

On Mon, Jan 22, 2018 at 1:18 PM, Rhavar via bitcoin-dev <
If you spent your change from transaction A, that would be safe. There'd
be no way you John could end up with 2 BTC from you then.
Yes, that's what the following paragraph says -- along with it's limitations =)
-Ryan
-------- Original Message --------
So now I still owe John 1 BTC, however it's not immediately clear if it's
safe to send to him
If you spent your change from transaction A, that would be safe. There'd
be no way you John could end up with 2 BTC from you then.
On Mon, Jan 22, 2018 at 1:40 PM, Rhavar via bitcoin-dev <
Allow users to merge multiple unconfirmed transactions, stripping
extraneous inputs and change as they go.
"The replacement transaction pays an absolute fee of at least the sum
paid by the original transactions."
Because the size of the merged transaction is smaller than the original
transactions, unless there is a considerable feerate bump, this rule isn't
possible to observe.
I my question is: is it possible or reasonable to relax this rule? If
this rule was removed in its entirety, does it introduce any DoS vectors?
Or can it be changed to allow my use-case?
---
Full backstory: I have been trying to use bip125 (Opt-in Full
Replace-by-Fee) to do "transaction merging" on the fly. Let's say that I
owe John 1 bitcoin, and have promised to pay him immediately: Instead of
creating a whole new transaction if I have an in-flight (unconfirmed)
transaction, I can follow the rules of bip125 to create a replacement that
accomplishes this goal.
From a "coin selection" point of view, this was significantly easier than
I had anticipated. I was able to encode the rules in my linear model and
feed in all my unspent and in-flight transactions and it can solve it without difficulty.
1) I have unconfirmed transaction A
2) I replace it with B, which pays John 1 BTC
3) Transaction A gets confirmed
So now I still owe John 1 BTC, however it's not immediately clear if
it's safe to send to him without waiting $n transactions. However even
for a small $n, this breaks my promise to pay him immediately.
One possible solution is to only consider a transaction "replaceable" if
it has change, so if the original transaction confirms -- payments can
immediately be made that source the change, and provide safety in a reorg.
However, this will only work <50% of the time for me (most transactions
don't have change) and opens a pandora's box of complexity.
There's a few other hacks you can do to make it work in a few more cases,
but nothing that is realistic to expect anyone to implement any time soon.
However, if there was a straight foward way to merge N unconfirmed
transactions, it would be easy get into production, and potentially offer
some pretty nice savings for everyone.
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Rhavar via bitcoin-dev
2018-01-22 18:59:34 UTC
Permalink
Perhaps they could even replace old tx with economically equivalent summary transactions?
I imagine with schnorr signatures, the incentives will emerge for that to make sense. But right now if I want to merge my transaction with an untrusted party in general we're only really going to be saving like 12 bytes of overhead or something. But if I'm merging my own transactions, I can get that fixed overhead, strip extraneous inputs and merge my change outputs (which also means in the future it's cheaper to spend).

Although it's obviously a lot worse for privacy, I do like the pattern of broadcast the transaction standalone and then merge it for savings. It helps keep the more or less fire-and-forget style, without a ridiculous amount of complexity "if this happens, do this, if this, then this, ..."

-Ryan

-Ryan

-------- Original Message --------
Along the same lines, I wonder if unrelated people with tx that are not confirming could cooperate to merge their disparate tx into a CoinJoin tx with a higher fee rate?
Perhaps they could even replace old tx with economically equivalent summary transactions?
The mempool seems like nature's accumulator for pre-mining compression opportunities.
Post by Rhavar via bitcoin-dev
If you spent your change from transaction A, that would be safe. There'd be no way you John could end up with 2 BTC from you then.
Yes, that's what the following paragraph says -- along with it's limitations =)
-Ryan
-------- Original Message --------
So now I still owe John 1 BTC, however it's not immediately clear if it's safe to send to him
If you spent your change from transaction A, that would be safe. There'd be no way you John could end up with 2 BTC from you then.
Allow users to merge multiple unconfirmed transactions, stripping extraneous inputs and change as they go.
"The replacement transaction pays an absolute fee of at least the sum paid by the original transactions."
Because the size of the merged transaction is smaller than the original transactions, unless there is a considerable feerate bump, this rule isn't possible to observe.
I my question is: is it possible or reasonable to relax this rule? If this rule was removed in its entirety, does it introduce any DoS vectors? Or can it be changed to allow my use-case?
---
Full backstory: I have been trying to use bip125 (Opt-in Full Replace-by-Fee) to do "transaction merging" on the fly. Let's say that I owe John 1 bitcoin, and have promised to pay him immediately: Instead of creating a whole new transaction if I have an in-flight (unconfirmed) transaction, I can follow the rules of bip125 to create a replacement that accomplishes this goal.
From a "coin selection" point of view, this was significantly easier than
I had anticipated. I was able to encode the rules in my linear model and
feed in all my unspent and in-flight transactions and it can solve it without difficulty.
1) I have unconfirmed transaction A
2) I replace it with B, which pays John 1 BTC
3) Transaction A gets confirmed
So now I still owe John 1 BTC, however it's not immediately clear if
it's safe to send to him without waiting $n transactions. However even
for a small $n, this breaks my promise to pay him immediately.
One possible solution is to only consider a transaction "replaceable" if it has change, so if the original transaction confirms -- payments can immediately be made that source the change, and provide safety in a reorg.
However, this will only work <50% of the time for me (most transactions
don't have change) and opens a pandora's box of complexity.
There's a few other hacks you can do to make it work in a few more cases, but nothing that is realistic to expect anyone to implement any time soon.
However, if there was a straight foward way to merge N unconfirmed transactions, it would be easy get into production, and potentially offer some pretty nice savings for everyone.
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Peter Todd via bitcoin-dev
2018-01-22 20:00:23 UTC
Permalink
Post by Rhavar via bitcoin-dev
Allow users to merge multiple unconfirmed transactions, stripping extraneous inputs and change as they go.
"The replacement transaction pays an absolute fee of at least the sum paid by the original transactions."
Because the size of the merged transaction is smaller than the original transactions, unless there is a considerable feerate bump, this rule isn't possible to observe.
I my question is: is it possible or reasonable to relax this rule? If this rule was removed in its entirety, does it introduce any DoS vectors? Or can it be changed to allow my use-case?
It would definitely introduce DoS vectors by making it much cheaper to use
relay bandwidth. You'd also be able to push others' txs out of the mempool.
Post by Rhavar via bitcoin-dev
---
Full backstory: I have been trying to use bip125 (Opt-in Full Replace-by-Fee) to do "transaction merging" on the fly. Let's say that I owe John 1 bitcoin, and have promised to pay him immediately: Instead of creating a whole new transaction if I have an in-flight (unconfirmed) transaction, I can follow the rules of bip125 to create a replacement that accomplishes this goal.
From a "coin selection" point of view, this was significantly easier than
I had anticipated. I was able to encode the rules in my linear model and
feed in all my unspent and in-flight transactions and it can solve it without difficulty.
1) I have unconfirmed transaction A
2) I replace it with B, which pays John 1 BTC
3) Transaction A gets confirmed
So now I still owe John 1 BTC, however it's not immediately clear if
it's safe to send to him without waiting $n transactions. However even
for a small $n, this breaks my promise to pay him immediately.
One possible solution is to only consider a transaction "replaceable" if it has change, so if the original transaction confirms -- payments can immediately be made that source the change, and provide safety in a reorg.
However, this will only work <50% of the time for me (most transactions
don't have change) and opens a pandora's box of complexity.
Most transactions don't have change?! Under what circumstance? For most
use-cases the reverse is true: almost all all transactions have change, because
it's rare for the inputs to exactly math the requested payment.
--
https://petertodd.org 'peter'[:-1]@petertodd.org
Rhavar via bitcoin-dev
2018-01-22 20:09:20 UTC
Permalink
Post by Peter Todd via bitcoin-dev
Most transactions don't have change?! Under what circumstance? For most
use-cases the reverse is true: almost all all transactions have change, because
it's rare for the inputs to exactly math the requested payment.
It's actually a common misconception. With good coin selection, I am able to avoid change about ~75% of the time in my simulations (on my real world data). In practice it's a bit lower, probably about 40-50% of the time because of the need to keep the majority of my funds offline where they can't be used for coin selection, and I have not been able to accurate simulate how I consolidate.

Also the other misconception is that inputs don't need to match exactly the requested payment, it's totally fine to do something I call a "miner sacrifice" where you overpay txfees up to the amount that that would otherwise be the total cost (immediate + consolidation) of creating change.

Also another trick I use, is something I call "output selection". If I have N queued non-time sensitive payments, I don't really need to send them all at the same time. So I can pick the best combination of inputs+outputs.

Obviously none of this applies to consumer wallets, who typically have less than a handful of options. But for a service, avoiding change can be the norm with good coin selection.

---

-Ryan

-------- Original Message --------
Post by Peter Todd via bitcoin-dev
Post by Rhavar via bitcoin-dev
Allow users to merge multiple unconfirmed transactions, stripping extraneous inputs and change as they go.
"The replacement transaction pays an absolute fee of at least the sum paid by the original transactions."
Because the size of the merged transaction is smaller than the original transactions, unless there is a considerable feerate bump, this rule isn't possible to observe.
I my question is: is it possible or reasonable to relax this rule? If this rule was removed in its entirety, does it introduce any DoS vectors? Or can it be changed to allow my use-case?
It would definitely introduce DoS vectors by making it much cheaper to use
relay bandwidth. You'd also be able to push others' txs out of the mempool.
Post by Rhavar via bitcoin-dev
---------------------------------------------------------------
Full backstory: I have been trying to use bip125 (Opt-in Full Replace-by-Fee) to do "transaction merging" on the fly. Let's say that I owe John 1 bitcoin, and have promised to pay him immediately: Instead of creating a whole new transaction if I have an in-flight (unconfirmed) transaction, I can follow the rules of bip125 to create a replacement that accomplishes this goal.
From a "coin selection" point of view, this was significantly easier than
I had anticipated. I was able to encode the rules in my linear model and
feed in all my unspent and in-flight transactions and it can solve it without difficulty.
- I have unconfirmed transaction A
- I replace it with B, which pays John 1 BTC
- Transaction A gets confirmed
So now I still owe John 1 BTC, however it's not immediately clear if
it's safe to send to him without waiting $n transactions. However even
for a small $n, this breaks my promise to pay him immediately.
One possible solution is to only consider a transaction "replaceable" if it has change, so if the original transaction confirms -- payments can immediately be made that source the change, and provide safety in a reorg.
However, this will only work <50% of the time for me (most transactions
don't have change) and opens a pandora's box of complexity.
Most transactions don't have change?! Under what circumstance? For most
use-cases the reverse is true: almost all all transactions have change, because
it's rare for the inputs to exactly math the requested payment.
Rhavar via bitcoin-dev
2018-01-23 16:31:36 UTC
Permalink
Post by Peter Todd via bitcoin-dev
It would definitely introduce DoS vectors by making it much cheaper to use
relay bandwidth.
I think I'm missing something, as I don't really understand this DoS vector. Relay bandwidth is already very cheap and easy to use by repeatedly fee bumping. And it's not obvious to me that requiring an absolute higher fee actually makes such an attack more expensive.

I can see that my "proposed" change would make it cheaper to evict low-fee transactions from other node's mempool. Maybe I'm being naive, but I don't really see why this would be such a big deal.

But what about a compromise, and require that the absolute fee must be >= half the original fees. I know everyone hates magic values, but I think in practice it will allow legitimate and useful use of "retroactive transaction merging" without much downside.

And really the great thing about "retroactive transaction merging" is just how easy it is to implement. In fact, right now it's quite possible to do -- but because of the "higher absolute fee" rule the benefits are pretty muted (although if you can compress 2 change into 1, that's still likely worthwhile)

-Ryan

-------- Original Message --------
Post by Peter Todd via bitcoin-dev
Post by Rhavar via bitcoin-dev
Allow users to merge multiple unconfirmed transactions, stripping extraneous inputs and change as they go.
"The replacement transaction pays an absolute fee of at least the sum paid by the original transactions."
Because the size of the merged transaction is smaller than the original transactions, unless there is a considerable feerate bump, this rule isn't possible to observe.
I my question is: is it possible or reasonable to relax this rule? If this rule was removed in its entirety, does it introduce any DoS vectors? Or can it be changed to allow my use-case?
It would definitely introduce DoS vectors by making it much cheaper to use
relay bandwidth. You'd also be able to push others' txs out of the mempool.
Post by Rhavar via bitcoin-dev
---------------------------------------------------------------
Full backstory: I have been trying to use bip125 (Opt-in Full Replace-by-Fee) to do "transaction merging" on the fly. Let's say that I owe John 1 bitcoin, and have promised to pay him immediately: Instead of creating a whole new transaction if I have an in-flight (unconfirmed) transaction, I can follow the rules of bip125 to create a replacement that accomplishes this goal.
From a "coin selection" point of view, this was significantly easier than
I had anticipated. I was able to encode the rules in my linear model and
feed in all my unspent and in-flight transactions and it can solve it without difficulty.
- I have unconfirmed transaction A
- I replace it with B, which pays John 1 BTC
- Transaction A gets confirmed
So now I still owe John 1 BTC, however it's not immediately clear if
it's safe to send to him without waiting $n transactions. However even
for a small $n, this breaks my promise to pay him immediately.
One possible solution is to only consider a transaction "replaceable" if it has change, so