[bitcoin-dev] An implementation of BIP102 as a softfork.

[joe2015--- via bitcoin-dev]

Below is a proof-of-concept implementation of BIP102 as a softfork:

https://github.com/ZoomT/bitcoin/tree/2015_2mb_blocksize
https://github.com/jgarzik/bitcoin/compare/2015_2mb_blocksize...ZoomT:2015_2mb_blocksize?diff=split&name=2015_2mb_blocksize

BIP102 is normally a hardfork. The softfork version (unofficial
codename BIP102s) uses the idea described here:
http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-December/012073.html

The basic idea is that post-fork blocks are constructed in such a way
they can be mapped to valid blocks under the pre-fork rules. BIP102s
is a softfork in the sense that post-fork miners are still creating a
valid chain under the old rules, albeit indirectly.

From the POV of non-upgraded clients, BIP102s circumvents the
block-size limit by moving transaction validation data "outside" of
the block. This is a similar trick used by Segregated Witness and
Extension Blocks (both softfork proposals).

From the POV of upgraded clients, the block layout is unchanged,
except:
- A larger 2MB block-size limit (=BIP102);
- The header Merkle root has a new (backwards compatible)
interpretation;
- The coinbase encodes the Merkle root of the remaining txs.
Aside from this, blocks maintain their original format, i.e. a block
header followed by a vector of transactions. This keeps the
implementation simple, and is distinct from SW and EB.

Since BIP102s is a softfork it means that:
- A miner majority (e.g. 75%, 95%) force miner consensus (100%). This
is not true for a hardfork.
- Fraud risk is significantly reduced (6-conf unlikely depending on
activation threshold).
This should address some of the concerns with deploying a block-size
increase using a hardfork.

Notes:

- The same basic idea could be adapted to any of the other proposals
(BIP101, 2-4-8, BIP202, etc.).
- I used Jeff Garzik's BIP102 implementation which is incomplete (?).
The activation logic is left unchanged.
- I am not a Bitcoin dev so hopefully no embarrassing mistakes in my
code :-(

--joe

[Jonathan Toomim via bitcoin-dev]

As a first impression, I think this proposal is intellectually interesting, but crufty and hackish and should never actually be deployed. Writing code for Bitcoin in a future in which we have deployed a few generalized softforks this way sounds terrifying.

Instead of this:

CTransaction GetTransaction(CBlock block, unsigned int index) {
return block->vtx[index];
}

We might have this:

CTransaction GetTransaction(CBlock block, unsigned int index) {
if (!IsBIP102sBlock(block)) {
return block->vtx[index];
} else {
if (!IsOtherGeneralizedSoftforkBlock(block)) {
// hooray! only one generalized softfork level to deal with!
return LookupBlock(GetGSHashFromCoinbase(block->vtx[0].vin[0].scriptSig))->vtx[index];
} else {
throw NotImplementedError; // I'm too lazy to write pseudocode this complicated just to argue a point
}
}

It might be possible to make that a bit simpler with recursion, or by doing subsequent generalized softforks in a way that doesn't have multi-levels-deep block-within-a-block-within-a-block stuff. Still: ugh.

Post by joe2015--- via bitcoin-dev
https://github.com/ZoomT/bitcoin/tree/2015_2mb_blocksize
https://github.com/jgarzik/bitcoin/compare/2015_2mb_blocksize...ZoomT:2015_2mb_blocksize?diff=split&name=2015_2mb_blocksize
BIP102 is normally a hardfork. The softfork version (unofficial
http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-December/012073.html
The basic idea is that post-fork blocks are constructed in such a way
they can be mapped to valid blocks under the pre-fork rules. BIP102s
is a softfork in the sense that post-fork miners are still creating a
valid chain under the old rules, albeit indirectly.
From the POV of non-upgraded clients, BIP102s circumvents the
block-size limit by moving transaction validation data "outside" of
the block. This is a similar trick used by Segregated Witness and
Extension Blocks (both softfork proposals).
From the POV of upgraded clients, the block layout is unchanged,
- A larger 2MB block-size limit (=BIP102);
- The header Merkle root has a new (backwards compatible)
interpretation;
- The coinbase encodes the Merkle root of the remaining txs.
Aside from this, blocks maintain their original format, i.e. a block
header followed by a vector of transactions. This keeps the
implementation simple, and is distinct from SW and EB.
- A miner majority (e.g. 75%, 95%) force miner consensus (100%). This
is not true for a hardfork.
- Fraud risk is significantly reduced (6-conf unlikely depending on
activation threshold).
This should address some of the concerns with deploying a block-size
increase using a hardfork.
- The same basic idea could be adapted to any of the other proposals
(BIP101, 2-4-8, BIP202, etc.).
- I used Jeff Garzik's BIP102 implementation which is incomplete (?).
The activation logic is left unchanged.
- I am not a Bitcoin dev so hopefully no embarrassing mistakes in my
code :-(
--joe
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[Peter Todd via bitcoin-dev]

Post by Jonathan Toomim via bitcoin-dev
As a first impression, I think this proposal is intellectually interesting, but crufty and hackish and should never actually be deployed. Writing code for Bitcoin in a future in which we have deployed a few generalized softforks this way sounds terrifying.

<snip>

Post by Jonathan Toomim via bitcoin-dev
It might be possible to make that a bit simpler with recursion, or by doing subsequent generalized softforks in a way that doesn't have multi-levels-deep block-within-a-block-within-a-block stuff. Still: ugh.

Your fear is misplaced: it's trivial to avoid recursion with a bit of
planning.

For instance, if Bitcoin was redesigned to incorporate the forced fork
concept, instead of block headers committing to just a merkle root,
they could instead commit to H(version + digest)

For version == 0, digest would be a merkle root of all transactions. If
the version was > 0, any digest would be allowed and the block would be
interpreted as a NOP with no effect on the UTXO set.

In the event of a major change - e.g. what would otherwise be a
hard-forking change to the way the merkle root was calculated - a
soft-fork would change the block validity rules to make version == 0
invalid, and verison == 1 blocks would interpret the digest according to
the new merkle root rules. Again, version > 1 blocks would be treated as
NOPs.

A good exercise is to apply the above to the existing Bitcoin ecosystem
as a soft-fork - it certainely can be done, and done right is
technically very simple.


Regardless of how it's done - existing Bitcoin compatible or clean sheet
redesign - you get the significant safety advantages soft-forks have
over hard-forks in nearly all situations where you'd have to do a
hard-fork. OTOH, it's kinda scary how this institutionalizes what could
be seen as 51% attacks, possibly giving miners significantly more
control over the system politically. I'm not sure I agree with that
viewpoint - miners can do this anyway - but that has made people shy
away from promoting this idea in the past. (previously it's been often
referred to as an "evil" soft-fork)

[Peter Todd via bitcoin-dev]

Post by Peter Todd via bitcoin-dev

Post by Jonathan Toomim via bitcoin-dev
As a first impression, I think this proposal is intellectually interesting, but crufty and hackish and should never actually be deployed. Writing code for Bitcoin in a future in which we have deployed a few generalized softforks this way sounds terrifying.

<snip>

Post by Jonathan Toomim via bitcoin-dev
It might be possible to make that a bit simpler with recursion, or by doing subsequent generalized softforks in a way that doesn't have multi-levels-deep block-within-a-block-within-a-block stuff. Still: ugh.

Your fear is misplaced: it's trivial to avoid recursion with a bit of
planning.
For instance, if Bitcoin was redesigned to incorporate the forced fork

Actually, a better name is probably "forced soft-fork", making this
clear we're using the soft-fork mechanism to force everyone to upgrade.

[Jonathan Toomim via bitcoin-dev]

Post by Peter Todd via bitcoin-dev
Your fear is misplaced: it's trivial to avoid recursion with a bit of
planning...

That makes some sense. I downgrade my emotions from "a future in which we have deployed a few generalized softforks this way sounds terrifying" to "the idea of a future in which we have deployed at least one generalized softfork this way gives me the heebie jeebies."

[Marcel Jamin via bitcoin-dev]

I guess the same could be said about the softfork flavoured SW
implementation. In any case, the strategy pattern helps with code structure
in situations like this.

2015-12-30 14:29 GMT+01:00 Jonathan Toomim via bitcoin-dev <

Post by Jonathan Toomim via bitcoin-dev
As a first impression, I think this proposal is intellectually
interesting, but crufty and hackish and should never actually be deployed.
Writing code for Bitcoin in a future in which we have deployed a few
generalized softforks this way sounds terrifying.
CTransaction GetTransaction(CBlock block, unsigned int index) {
return block->vtx[index];
}
CTransaction GetTransaction(CBlock block, unsigned int index) {
if (!IsBIP102sBlock(block)) {
return block->vtx[index];
} else {
if (!IsOtherGeneralizedSoftforkBlock(block)) {
// hooray! only one generalized softfork level to deal with!
return
LookupBlock(GetGSHashFromCoinbase(block->vtx[0].vin[0].scriptSig))->vtx[index];
} else {
throw NotImplementedError; // I'm too lazy to write
pseudocode this complicated just to argue a point
}
}
It might be possible to make that a bit simpler with recursion, or by
doing subsequent generalized softforks in a way that doesn't have
multi-levels-deep block-within-a-block-within-a-block stuff. Still: ugh.
On Dec 29, 2015, at 9:46 PM, joe2015--- via bitcoin-dev <

Post by joe2015--- via bitcoin-dev
https://github.com/ZoomT/bitcoin/tree/2015_2mb_blocksize

https://github.com/jgarzik/bitcoin/compare/2015_2mb_blocksize...ZoomT:2015_2mb_blocksize?diff=split&name=2015_2mb_blocksize

Post by joe2015--- via bitcoin-dev
BIP102 is normally a hardfork. The softfork version (unofficial

http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-December/012073.html

Post by joe2015--- via bitcoin-dev
The basic idea is that post-fork blocks are constructed in such a way
they can be mapped to valid blocks under the pre-fork rules. BIP102s
is a softfork in the sense that post-fork miners are still creating a
valid chain under the old rules, albeit indirectly.
From the POV of non-upgraded clients, BIP102s circumvents the
block-size limit by moving transaction validation data "outside" of
the block. This is a similar trick used by Segregated Witness and
Extension Blocks (both softfork proposals).
From the POV of upgraded clients, the block layout is unchanged,
- A larger 2MB block-size limit (=BIP102);
- The header Merkle root has a new (backwards compatible)
interpretation;
- The coinbase encodes the Merkle root of the remaining txs.
Aside from this, blocks maintain their original format, i.e. a block
header followed by a vector of transactions. This keeps the
implementation simple, and is distinct from SW and EB.
- A miner majority (e.g. 75%, 95%) force miner consensus (100%). This
is not true for a hardfork.
- Fraud risk is significantly reduced (6-conf unlikely depending on
activation threshold).
This should address some of the concerns with deploying a block-size
increase using a hardfork.
- The same basic idea could be adapted to any of the other proposals
(BIP101, 2-4-8, BIP202, etc.).
- I used Jeff Garzik's BIP102 implementation which is incomplete (?).
The activation logic is left unchanged.
- I am not a Bitcoin dev so hopefully no embarrassing mistakes in my
code :-(
--joe
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[Marco Falke via bitcoin-dev]

This is an interesting approach but I don't see how this is a soft
fork. (Just because something is not a hard fork, doesn't make it a
soft fork by definition)
Softforks don't require any nodes to upgrade. [1]
Nonetheless, as I understand your approach, it requires nodes to
upgrade. Otherwise they are missing all transactions but the coinbase
transactions. Thus, they cannot update their utxoset and are easily
susceptible to double spends...

Am I missing something obvious?

-- Marco


[1] https://en.bitcoin.it/wiki/Softfork#Implications

[joe2015--- via bitcoin-dev]

Post by Marco Falke via bitcoin-dev
This is an interesting approach but I don't see how this is a soft
fork. (Just because something is not a hard fork, doesn't make it a
soft fork by definition)
Softforks don't require any nodes to upgrade. [1]
Nonetheless, as I understand your approach, it requires nodes to
upgrade. Otherwise they are missing all transactions but the coinbase
transactions. Thus, they cannot update their utxoset and are easily
susceptible to double spends...
Am I missing something obvious?
-- Marco
[1] https://en.bitcoin.it/wiki/Softfork#Implications

It just depends how you define "softfork". In my original write-up I
called it a "generalized" softfork, Peter suggested a "firm" fork, and
there are some suggestions for other names. Ultimately what you call it
is not very important.

--joe.

[joe2015--- via bitcoin-dev]

Post by joe2015--- via bitcoin-dev

Post by Marco Falke via bitcoin-dev
This is an interesting approach but I don't see how this is a soft
fork. (Just because something is not a hard fork, doesn't make it a
soft fork by definition)
Softforks don't require any nodes to upgrade. [1]
Nonetheless, as I understand your approach, it requires nodes to
upgrade. Otherwise they are missing all transactions but the coinbase
transactions. Thus, they cannot update their utxoset and are easily
susceptible to double spends...
Am I missing something obvious?
-- Marco
[1] https://en.bitcoin.it/wiki/Softfork#Implications

It just depends how you define "softfork". In my original write-up I
called
it a "generalized" softfork, Peter suggested a "firm" fork, and there
are
some suggestions for other names. Ultimately what you call it is not
very
important.
--joe.

joe, indeed it is not important how you call it, but please, let's not
call it "soft fork".

This kind of fork (whatever it is called) has all the traditional
properties of a softfork except meaningful backwards compatibility for
non-upgraded clients. So I think it is reasonable to call it a softfork
with some qualification.

Besides my initial question about the coinbase
tx, I was also wondering how non-updated nodes would verify the
collected fees without the actual txs at hand. (They only have the
coinbase tx, don't they?)

Yes this appears to be an oversight in my proof-of-concept
implementation. The unintended consequence being that all transactions
would have to be zero-fee...

The simplest fix would be make the new rules add the fees implicitly.
There are other solutions.

Moreover, I can't see the benefits over a hard fork. A hard fork is
much cleaner in regard to code changes. As one of the intends of
"generalized soft forks" is to force user to update, at least a hard
fork doesn't lie about the fact. Am I missing any obvious advantages
of a "generalized soft fork" over a "clean" hard fork?

A "firm soft fork" also does not lie about that fact -- you must
upgrade. I don't see it dishonest if it was never claimed otherwise.

I agree that hardforks can be "cleaner".

However the obvious disadvantage of a hardfork is the risk of the
network splitting between upgraded and non-upgraded clients. This is
not a problem if there is 100% consensus behind the hardfork, but I am
not sure if 100% is realistically achievable for contentious issues such
as the blocksize limit.

If 100% consensus is never achieved, then the options are:
1. Never upgrade and keep the blocksize limit unchanged forever.
2. Use a firm softfork to resolve the deadlock.
3. Hardfork anyway and split the network.

My argument is simply that 2 is better than 3 and possibly 1.

--joe

[Nick ODell via bitcoin-dev]

How are you collecting fees from the transactions in the block?

On Sat, Jan 2, 2016 at 8:51 PM, joe2015--- via bitcoin-dev

Post by joe2015--- via bitcoin-dev

Post by Marco Falke via bitcoin-dev
This is an interesting approach but I don't see how this is a soft
fork. (Just because something is not a hard fork, doesn't make it a
soft fork by definition)
Softforks don't require any nodes to upgrade. [1]
Nonetheless, as I understand your approach, it requires nodes to
upgrade. Otherwise they are missing all transactions but the coinbase
transactions. Thus, they cannot update their utxoset and are easily
susceptible to double spends...
Am I missing something obvious?
-- Marco
[1] https://en.bitcoin.it/wiki/Softfork#Implications

It just depends how you define "softfork". In my original write-up I
called
it a "generalized" softfork, Peter suggested a "firm" fork, and there are
some suggestions for other names. Ultimately what you call it is not
very
important.
--joe.

joe, indeed it is not important how you call it, but please, let's not
call it "soft fork".

This kind of fork (whatever it is called) has all the traditional properties
of a softfork except meaningful backwards compatibility for non-upgraded
clients. So I think it is reasonable to call it a softfork with some
qualification.

Besides my initial question about the coinbase
tx, I was also wondering how non-updated nodes would verify the
collected fees without the actual txs at hand. (They only have the
coinbase tx, don't they?)

Yes this appears to be an oversight in my proof-of-concept implementation.
The unintended consequence being that all transactions would have to be
zero-fee...
The simplest fix would be make the new rules add the fees implicitly. There
are other solutions.

Moreover, I can't see the benefits over a hard fork. A hard fork is
much cleaner in regard to code changes. As one of the intends of
"generalized soft forks" is to force user to update, at least a hard
fork doesn't lie about the fact. Am I missing any obvious advantages
of a "generalized soft fork" over a "clean" hard fork?

A "firm soft fork" also does not lie about that fact -- you must upgrade. I
don't see it dishonest if it was never claimed otherwise.
I agree that hardforks can be "cleaner".
However the obvious disadvantage of a hardfork is the risk of the network
splitting between upgraded and non-upgraded clients. This is not a problem
if there is 100% consensus behind the hardfork, but I am not sure if 100% is
realistically achievable for contentious issues such as the blocksize limit.
1. Never upgrade and keep the blocksize limit unchanged forever.
2. Use a firm softfork to resolve the deadlock.
3. Hardfork anyway and split the network.
My argument is simply that 2 is better than 3 and possibly 1.
--joe
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[joe2015--- via bitcoin-dev]

Post by Nick ODell via bitcoin-dev
How are you collecting fees from the transactions in the block?

Probably the simplest way to do this is to map the new-rules coinbase tx
(which collects the block reward and fees) into an old-rules legacy
coinbase tx (which collects the block reward only). Care must be taken
to ensure the mapping is not reversible. I will update my
implementation in due course.

--joe.

[joe2015--- via bitcoin-dev]

Post by joe2015--- via bitcoin-dev

Post by Nick ODell via bitcoin-dev
How are you collecting fees from the transactions in the block?

Probably the simplest way to do this is to map the new-rules coinbase
tx (which collects the block reward and fees) into an old-rules legacy
coinbase tx (which collects the block reward only). Care must be
taken to ensure the mapping is not reversible. I will update my
implementation in due course.

The redesigned implementation is here:

https://github.com/ZoomT/bitcoin/tree/2015_2mb_blocksize
https://github.com/jgarzik/bitcoin/compare/2015_2mb_blocksize...ZoomT:2015_2mb_blocksize

The new version maps the Merkle root onto a 'legacy' coinbase
transaction, solving the problem with fees.

--joe.

[Martijn Meijering via bitcoin-dev]

That looks very interesting. But is effectively blocking old clients from
seeing transactions really safe? After all, such transactions are still
confirmed on the new chain. A person might try to send a similar
transaction several times, perhaps with increasing fees in an attempt to
get it to confirm and end up paying someone several times.

Maybe we could require the tx version number to be increased as well so
transactions sent from old clients would never confirm? Perhaps your code
already includes this idea, I need to look at it more closely.

[Peter Todd via bitcoin-dev]

Post by Martijn Meijering via bitcoin-dev
That looks very interesting. But is effectively blocking old clients from
seeing transactions really safe? After all, such transactions are still
confirmed on the new chain. A person might try to send a similar
transaction several times, perhaps with increasing fees in an attempt to
get it to confirm and end up paying someone several times.

It's very dangerous to simply send multiple transactions in such a way
that they don't double-spend each other; you have no good way of knowing
for sure that you're seeing the longest block chain with software alone.

Competently designed software with fee-bumping wouldn't allow that
mistake to be made; the UX should make it clear that txs sent are still
pending until confirmed or clearly double-spent.

Post by Martijn Meijering via bitcoin-dev
Maybe we could require the tx version number to be increased as well so
transactions sent from old clients would never confirm? Perhaps your code
already includes this idea, I need to look at it more closely.

That can mess up pre-signed transations, e.g. refunds.