[bitcoin-dev] Proposed new policy for transactions that depend on other unconfirmed transactions

Discussion:

Alex Morcos via bitcoin-dev

2015-08-14 19:33:31 UTC

Hi everyone,

I'd like to propose a new set of requirements as a policy on when to accept
new transactions into the mempool and relay them. This policy would affect
transactions which have as inputs other transactions which are not yet
confirmed in the blockchain.

The motivation for this policy is 6470
<https://github.com/bitcoin/bitcoin/pull/6470> which aims to limit the size
of a mempool. As discussed in that pull
<https://github.com/bitcoin/bitcoin/pull/6470#issuecomment-125324736>, once
the mempool is full a new transaction must be able to pay not only for the
transaction it would evict, but any dependent transactions that would be
removed from the mempool as well. In order to make sure this is always
feasible, I'm proposing 4 new policy limits.

All limits are command line configurable.

The first two limits are required to make sure no chain of transactions
will be too large for the eviction code to handle:

Max number of descendant txs : No transaction shall be accepted if it would
cause another transaction in the mempool to have too many descendant
transactions (all of which would have to be evicted if the ancestor
transaction was evicted). Default: 1000

Max descendant size : No transaction shall be accepted if it would cause
another transaction in the mempool to have the total size of all its
descendant transactions be too great. Default : maxmempool / 200 = 2.5MB

The third limit is required to make sure calculating the state required for
sorting and limiting the mempool and enforcing the first 2 limits is
computationally feasible:

Max number of ancestor txs: No transaction shall be accepted if it has too
many ancestor transactions which are not yet confirmed (ie, in the
mempool). Default: 100

The fourth limit is required to maintain the pre existing policy goal that
all transactions in the mempool should be mineable in the next block.

Max ancestor size: No transaction shall be accepted if the total size of
all its unconfirmed ancestor transactions is too large. Default: 1MB

(All limits include the transaction itself.)

For reference, these limits would have affected less than 2% of
transactions entering the mempool in April or May of this year. During the
period of 7/6 through 7/14, while the network was under stress test, as
many as 25% of the transactions would have been affected.

The code to implement the descendant package tracking and new policy limits
can be found in 6557 <https://github.com/bitcoin/bitcoin/pull/6557> which
is built off of 6470.

Thanks,
Alex

Matt Corallo via bitcoin-dev

2015-08-21 19:22:39 UTC

Permalink

I dont see any problem with such limits. Though, hell, if you limited
entire tx dependency trees (ie transactions and all required unconfirmed
transactions for them) to something like 10 txn, maximum two levels
deep, I also wouldnt have a problem.

Matt

Post by Alex Morcos via bitcoin-dev
Hi everyone,
I'd like to propose a new set of requirements as a policy on when to
accept new transactions into the mempool and relay them. This policy
would affect transactions which have as inputs other transactions which
are not yet confirmed in the blockchain.
The motivation for this policy is 6470
<https://github.com/bitcoin/bitcoin/pull/6470> which aims to limit the
size of a mempool. As discussed in that pull
<https://github.com/bitcoin/bitcoin/pull/6470#issuecomment-125324736>,
once the mempool is full a new transaction must be able to pay not only
for the transaction it would evict, but any dependent transactions that
would be removed from the mempool as well. In order to make sure this
is always feasible, I'm proposing 4 new policy limits.
All limits are command line configurable.
The first two limits are required to make sure no chain of transactions
Max number of descendant txs : No transaction shall be accepted if it
would cause another transaction in the mempool to have too many
descendant transactions (all of which would have to be evicted if the
ancestor transaction was evicted). Default: 1000
Max descendant size : No transaction shall be accepted if it would cause
another transaction in the mempool to have the total size of all its
descendant transactions be too great. Default : maxmempool / 200 = 2.5MB
The third limit is required to make sure calculating the state required
for sorting and limiting the mempool and enforcing the first 2 limits is
Max number of ancestor txs: No transaction shall be accepted if it has
too many ancestor transactions which are not yet confirmed (ie, in the
mempool). Default: 100
The fourth limit is required to maintain the pre existing policy goal
that all transactions in the mempool should be mineable in the next block.
Max ancestor size: No transaction shall be accepted if the total size of
all its unconfirmed ancestor transactions is too large. Default: 1MB
(All limits include the transaction itself.)
For reference, these limits would have affected less than 2% of
transactions entering the mempool in April or May of this year. During
the period of 7/6 through 7/14, while the network was under stress test,
as many as 25% of the transactions would have been affected.
The code to implement the descendant package tracking and new policy
limits can be found in 6557
<https://github.com/bitcoin/bitcoin/pull/6557> which is built off of 6470.
Thanks,
Alex
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Danny Thorpe via bitcoin-dev

2015-08-21 19:52:35 UTC

Permalink

The limits Alex proposed are generous (bordering on obscene!), but dropping
that down to allowing only two levels of chained unconfirmed transactions
is too tight.

Use case: Brokered asset transfers may require sets of transactions with a
dependency tree depth of 3 to be published together. ( N seller txs, 1
broker bridge tx, M buyer txs )

If the originally proposed depth limit of 100 does not provide a sufficient
cap on memory consumption or loop/recursion depth, a depth limit of 10
would provide plenty of headroom for this 3 level use case and similar
patterns.

-Danny

On Fri, Aug 21, 2015 at 12:22 PM, Matt Corallo via bitcoin-dev <

Post by Matt Corallo via bitcoin-dev
I dont see any problem with such limits. Though, hell, if you limited
entire tx dependency trees (ie transactions and all required unconfirmed
transactions for them) to something like 10 txn, maximum two levels
deep, I also wouldnt have a problem.
Matt

2.5MB

Post by Alex Morcos via bitcoin-dev
The third limit is required to make sure calculating the state required
for sorting and limiting the mempool and enforcing the first 2 limits is
Max number of ancestor txs: No transaction shall be accepted if it has
too many ancestor transactions which are not yet confirmed (ie, in the
mempool). Default: 100
The fourth limit is required to maintain the pre existing policy goal
that all transactions in the mempool should be mineable in the next

block.

Post by Alex Morcos via bitcoin-dev
Max ancestor size: No transaction shall be accepted if the total size of
all its unconfirmed ancestor transactions is too large. Default: 1MB
(All limits include the transaction itself.)
For reference, these limits would have affected less than 2% of
transactions entering the mempool in April or May of this year. During
the period of 7/6 through 7/14, while the network was under stress test,
as many as 25% of the transactions would have been affected.
The code to implement the descendant package tracking and new policy
limits can be found in 6557
<https://github.com/bitcoin/bitcoin/pull/6557> which is built off of

6470.

Post by Alex Morcos via bitcoin-dev
Thanks,
Alex
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Alex Morcos via bitcoin-dev

2015-09-21 15:02:20 UTC

Permalink

Thanks for everyone's review. These policy changes have been merged in to
master in 6654 <https://github.com/bitcoin/bitcoin/pull/6654>, which just
implements these limits and no mempool limiting yet. The default ancestor
package size limit is 900kb not 1MB.

Yes I think these limits are generous, but they were designed to be as
generous as was computationally feasible so they were unobjectionable
(since the existing policy was no limits). This does not preclude future
changes to policy that would reduce these limits.

Post by Danny Thorpe via bitcoin-dev
The limits Alex proposed are generous (bordering on obscene!), but
dropping that down to allowing only two levels of chained unconfirmed
transactions is too tight.
Use case: Brokered asset transfers may require sets of transactions with a
dependency tree depth of 3 to be published together. ( N seller txs, 1
broker bridge tx, M buyer txs )
If the originally proposed depth limit of 100 does not provide a
sufficient cap on memory consumption or loop/recursion depth, a depth limit
of 10 would provide plenty of headroom for this 3 level use case and
similar patterns.
-Danny
On Fri, Aug 21, 2015 at 12:22 PM, Matt Corallo via bitcoin-dev <

2.5MB

block.

6470.

Post by Alex Morcos via bitcoin-dev
Thanks,
Alex
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Alex Morcos via bitcoin-dev

2015-10-05 18:45:57 UTC

Permalink

I'd like to propose updates to the new policy limits on unconfirmed
transaction chains.

The existing limits in master and scheduled for release in 0.12 are:
Ancestor packages = 100 txs and 900kb total size
Descendant packages = 1000 txs and 2500kb total size

Before 0.12 is released I would like to propose a significant reduction in
these limits. In the course of analyzing algorithms for mempool limiting,
it became clear that large packages of unconfirmed transactions were the
primary vector for mempool clogging or relay fee boosting attacks. Feedback
from the initial proposed limits was that they were too generous anyway.

The proposed new limits are:
Ancestor packages = 25 txs and 100kb total size
Descendant packages = 25 txs and 100kb total size

Based on historical transaction data, the most restrictive of these limits
is the 25 transaction count on descendant packages. Over the period of
April and May of this year (before stress tests), 5.8% of transactions
would have violated this limit alone. Applying all the limits together
would have affected 6.1% of transactions.

Please keep in mind these are policy limits that affect transactions which
depend on other unconfirmed transactions only. They are not a change to
consensus rules and do not affect how many chained txs a valid block may
contain. Furthermore, any transaction that was unable to be relayed due to
these limits need only wait for some of its unconfirmed ancestors to be
included in a block and then it could be successfully broadcast. This is
unlikely to affect the total time from creation to inclusion in a block.
Finally, these limits are command line arguments that can easily be changed
on an individual node basis in Bitcoin Core.

Please give your feedback if you know of legitimate use cases that would be
hindered by these limits.

Thanks,
Alex

Post by Alex Morcos via bitcoin-dev
Thanks for everyone's review. These policy changes have been merged in to
master in 6654 <https://github.com/bitcoin/bitcoin/pull/6654>, which just
implements these limits and no mempool limiting yet. The default ancestor
package size limit is 900kb not 1MB.
Yes I think these limits are generous, but they were designed to be as
generous as was computationally feasible so they were unobjectionable
(since the existing policy was no limits). This does not preclude future
changes to policy that would reduce these limits.

Post by Danny Thorpe via bitcoin-dev
The limits Alex proposed are generous (bordering on obscene!), but
dropping that down to allowing only two levels of chained unconfirmed
transactions is too tight.
Use case: Brokered asset transfers may require sets of transactions with
a dependency tree depth of 3 to be published together. ( N seller txs, 1
broker bridge tx, M buyer txs )
If the originally proposed depth limit of 100 does not provide a
sufficient cap on memory consumption or loop/recursion depth, a depth limit
of 10 would provide plenty of headroom for this 3 level use case and
similar patterns.
-Danny
On Fri, Aug 21, 2015 at 12:22 PM, Matt Corallo via bitcoin-dev <

cause

Post by Alex Morcos via bitcoin-dev
another transaction in the mempool to have the total size of all its
descendant transactions be too great. Default : maxmempool / 200 =

2.5MB

Post by Alex Morcos via bitcoin-dev
The third limit is required to make sure calculating the state required
for sorting and limiting the mempool and enforcing the first 2 limits

Post by Alex Morcos via bitcoin-dev
Max number of ancestor txs: No transaction shall be accepted if it has
too many ancestor transactions which are not yet confirmed (ie, in the
mempool). Default: 100
The fourth limit is required to maintain the pre existing policy goal
that all transactions in the mempool should be mineable in the next

block.

Post by Alex Morcos via bitcoin-dev
Max ancestor size: No transaction shall be accepted if the total size

Post by Alex Morcos via bitcoin-dev
all its unconfirmed ancestor transactions is too large. Default: 1MB
(All limits include the transaction itself.)
For reference, these limits would have affected less than 2% of
transactions entering the mempool in April or May of this year. During
the period of 7/6 through 7/14, while the network was under stress

test,

Post by Alex Morcos via bitcoin-dev
as many as 25% of the transactions would have been affected.
The code to implement the descendant package tracking and new policy
limits can be found in 6557
<https://github.com/bitcoin/bitcoin/pull/6557> which is built off of

6470.

Post by Alex Morcos via bitcoin-dev
Thanks,
Alex
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Danny Thorpe via bitcoin-dev

2015-10-05 18:51:23 UTC

Permalink

What does "package" mean here?

When you say 25 txs, does that mean maximum linked chain depth, or total
number of dependent transactions regardless of chain depth?

Thanks,
-Danny

On Mon, Oct 5, 2015 at 11:45 AM, Alex Morcos via bitcoin-dev <

Post by Alex Morcos via bitcoin-dev
I'd like to propose updates to the new policy limits on unconfirmed
transaction chains.
Ancestor packages = 100 txs and 900kb total size
Descendant packages = 1000 txs and 2500kb total size
Before 0.12 is released I would like to propose a significant reduction in
these limits. In the course of analyzing algorithms for mempool limiting,
it became clear that large packages of unconfirmed transactions were the
primary vector for mempool clogging or relay fee boosting attacks. Feedback
from the initial proposed limits was that they were too generous anyway.
Ancestor packages = 25 txs and 100kb total size
Descendant packages = 25 txs and 100kb total size
Based on historical transaction data, the most restrictive of these limits
is the 25 transaction count on descendant packages. Over the period of
April and May of this year (before stress tests), 5.8% of transactions
would have violated this limit alone. Applying all the limits together
would have affected 6.1% of transactions.
Please keep in mind these are policy limits that affect transactions which
depend on other unconfirmed transactions only. They are not a change to
consensus rules and do not affect how many chained txs a valid block may
contain. Furthermore, any transaction that was unable to be relayed due to
these limits need only wait for some of its unconfirmed ancestors to be
included in a block and then it could be successfully broadcast. This is
unlikely to affect the total time from creation to inclusion in a block.
Finally, these limits are command line arguments that can easily be changed
on an individual node basis in Bitcoin Core.
Please give your feedback if you know of legitimate use cases that would
be hindered by these limits.
Thanks,
Alex

Post by Alex Morcos via bitcoin-dev
Thanks for everyone's review. These policy changes have been merged in
to master in 6654 <https://github.com/bitcoin/bitcoin/pull/6654>, which
just implements these limits and no mempool limiting yet. The default
ancestor package size limit is 900kb not 1MB.
Yes I think these limits are generous, but they were designed to be as
generous as was computationally feasible so they were unobjectionable
(since the existing policy was no limits). This does not preclude future
changes to policy that would reduce these limits.

Post by Danny Thorpe via bitcoin-dev
The limits Alex proposed are generous (bordering on obscene!), but
dropping that down to allowing only two levels of chained unconfirmed
transactions is too tight.
Use case: Brokered asset transfers may require sets of transactions with
a dependency tree depth of 3 to be published together. ( N seller txs, 1
broker bridge tx, M buyer txs )
If the originally proposed depth limit of 100 does not provide a
sufficient cap on memory consumption or loop/recursion depth, a depth limit
of 10 would provide plenty of headroom for this 3 level use case and
similar patterns.
-Danny
On Fri, Aug 21, 2015 at 12:22 PM, Matt Corallo via bitcoin-dev <

which

Post by Alex Morcos via bitcoin-dev
are not yet confirmed in the blockchain.
The motivation for this policy is 6470
<https://github.com/bitcoin/bitcoin/pull/6470> which aims to limit

the

Post by Alex Morcos via bitcoin-dev
size of a mempool. As discussed in that pull
<https://github.com/bitcoin/bitcoin/pull/6470#issuecomment-125324736
,
once the mempool is full a new transaction must be able to pay not

only

Post by Alex Morcos via bitcoin-dev
for the transaction it would evict, but any dependent transactions

that

Post by Alex Morcos via bitcoin-dev
would be removed from the mempool as well. In order to make sure this
is always feasible, I'm proposing 4 new policy limits.
All limits are command line configurable.
The first two limits are required to make sure no chain of

transactions

Post by Alex Morcos via bitcoin-dev
Max number of descendant txs : No transaction shall be accepted if it
would cause another transaction in the mempool to have too many
descendant transactions (all of which would have to be evicted if the
ancestor transaction was evicted). Default: 1000
Max descendant size : No transaction shall be accepted if it would

cause

Post by Alex Morcos via bitcoin-dev
another transaction in the mempool to have the total size of all its
descendant transactions be too great. Default : maxmempool / 200 =

2.5MB

Post by Alex Morcos via bitcoin-dev
The third limit is required to make sure calculating the state

required

Post by Alex Morcos via bitcoin-dev
for sorting and limiting the mempool and enforcing the first 2 limits

Post by Alex Morcos via bitcoin-dev
Max number of ancestor txs: No transaction shall be accepted if it

has

Post by Alex Morcos via bitcoin-dev
too many ancestor transactions which are not yet confirmed (ie, in the
mempool). Default: 100
The fourth limit is required to maintain the pre existing policy goal
that all transactions in the mempool should be mineable in the next

block.

Post by Alex Morcos via bitcoin-dev
Max ancestor size: No transaction shall be accepted if the total size

During

Post by Alex Morcos via bitcoin-dev
the period of 7/6 through 7/14, while the network was under stress

test,

6470.

Post by Alex Morcos via bitcoin-dev
Thanks,
Alex
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Alex Morcos via bitcoin-dev

2015-10-05 20:02:40 UTC

Permalink

Yes, total number of dependent transactions regardless of chain depth.

A descendant package means all the transactions that can not be included in
a block before the transaction in question.

An ancestor package means all the transactions that are required to be
included in a block before the transaction in question can be.

Post by Danny Thorpe via bitcoin-dev
What does "package" mean here?
When you say 25 txs, does that mean maximum linked chain depth, or total
number of dependent transactions regardless of chain depth?
Thanks,
-Danny
On Mon, Oct 5, 2015 at 11:45 AM, Alex Morcos via bitcoin-dev <

Post by Alex Morcos via bitcoin-dev
I'd like to propose updates to the new policy limits on unconfirmed
transaction chains.
Ancestor packages = 100 txs and 900kb total size
Descendant packages = 1000 txs and 2500kb total size
Before 0.12 is released I would like to propose a significant reduction
in these limits. In the course of analyzing algorithms for mempool
limiting, it became clear that large packages of unconfirmed transactions
were the primary vector for mempool clogging or relay fee boosting attacks.
Feedback from the initial proposed limits was that they were too generous
anyway.
Ancestor packages = 25 txs and 100kb total size
Descendant packages = 25 txs and 100kb total size
Based on historical transaction data, the most restrictive of these
limits is the 25 transaction count on descendant packages. Over the period
of April and May of this year (before stress tests), 5.8% of transactions
would have violated this limit alone. Applying all the limits together
would have affected 6.1% of transactions.
Please keep in mind these are policy limits that affect transactions
which depend on other unconfirmed transactions only. They are not a change
to consensus rules and do not affect how many chained txs a valid block may
contain. Furthermore, any transaction that was unable to be relayed due to
these limits need only wait for some of its unconfirmed ancestors to be
included in a block and then it could be successfully broadcast. This is
unlikely to affect the total time from creation to inclusion in a block.
Finally, these limits are command line arguments that can easily be changed
on an individual node basis in Bitcoin Core.
Please give your feedback if you know of legitimate use cases that would
be hindered by these limits.
Thanks,
Alex

Post by Alex Morcos via bitcoin-dev
Thanks for everyone's review. These policy changes have been merged in
to master in 6654 <https://github.com/bitcoin/bitcoin/pull/6654>, which
just implements these limits and no mempool limiting yet. The default
ancestor package size limit is 900kb not 1MB.
Yes I think these limits are generous, but they were designed to be as
generous as was computationally feasible so they were unobjectionable
(since the existing policy was no limits). This does not preclude future
changes to policy that would reduce these limits.

Post by Danny Thorpe via bitcoin-dev
The limits Alex proposed are generous (bordering on obscene!), but
dropping that down to allowing only two levels of chained unconfirmed
transactions is too tight.
Use case: Brokered asset transfers may require sets of transactions
with a dependency tree depth of 3 to be published together. ( N seller txs,
1 broker bridge tx, M buyer txs )
If the originally proposed depth limit of 100 does not provide a
sufficient cap on memory consumption or loop/recursion depth, a depth limit
of 10 would provide plenty of headroom for this 3 level use case and
similar patterns.
-Danny
On Fri, Aug 21, 2015 at 12:22 PM, Matt Corallo via bitcoin-dev <

which

Post by Alex Morcos via bitcoin-dev
are not yet confirmed in the blockchain.
The motivation for this policy is 6470
<https://github.com/bitcoin/bitcoin/pull/6470> which aims to limit

the

only

Post by Alex Morcos via bitcoin-dev
for the transaction it would evict, but any dependent transactions

that

Post by Alex Morcos via bitcoin-dev
would be removed from the mempool as well. In order to make sure

this

Post by Alex Morcos via bitcoin-dev
is always feasible, I'm proposing 4 new policy limits.
All limits are command line configurable.
The first two limits are required to make sure no chain of

transactions

cause

Post by Alex Morcos via bitcoin-dev
another transaction in the mempool to have the total size of all its
descendant transactions be too great. Default : maxmempool / 200

= 2.5MB

Post by Alex Morcos via bitcoin-dev
The third limit is required to make sure calculating the state

required

Post by Alex Morcos via bitcoin-dev
for sorting and limiting the mempool and enforcing the first 2

limits is

Post by Alex Morcos via bitcoin-dev
Max number of ancestor txs: No transaction shall be accepted if it

has

Post by Alex Morcos via bitcoin-dev
too many ancestor transactions which are not yet confirmed (ie, in

the

Post by Alex Morcos via bitcoin-dev
mempool). Default: 100
The fourth limit is required to maintain the pre existing policy goal
that all transactions in the mempool should be mineable in the next

block.

Post by Alex Morcos via bitcoin-dev
Max ancestor size: No transaction shall be accepted if the total

size of

During

Post by Alex Morcos via bitcoin-dev
the period of 7/6 through 7/14, while the network was under stress

test,

of 6470.

Post by Alex Morcos via bitcoin-dev
Thanks,
Alex
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Matt Corallo via bitcoin-dev

2015-10-08 03:33:07 UTC

Permalink

There is a PR up for this change at https://github.com/bitcoin/bitcoin/pull/6771, which is getting some discussion for those following along.

Post by Alex Morcos via bitcoin-dev
Yes, total number of dependent transactions regardless of chain depth.
A descendant package means all the transactions that can not be
included in
a block before the transaction in question.
An ancestor package means all the transactions that are required to be
included in a block before the transaction in question can be.

Post by Danny Thorpe via bitcoin-dev
What does "package" mean here?
When you say 25 txs, does that mean maximum linked chain depth, or

total

Post by Danny Thorpe via bitcoin-dev
number of dependent transactions regardless of chain depth?
Thanks,
-Danny
On Mon, Oct 5, 2015 at 11:45 AM, Alex Morcos via bitcoin-dev <

reduction