Discussion:
[bitcoin-dev] Requesting BIP assignment; Flexible Transactions.
Tom via bitcoin-dev
2016-09-20 17:15:45 UTC
Permalink
As the title suggests, I would like to formally request the assignment of a
BIP number for my FT spec.

Thank you!


Source;

https://github.com/zander/bips/blob/FlexTrans/bip-9999.mediawiki

<pre>
BIP: ??
Title: Flexible Transactions
Author: Tom Zander <***@freedommail.ch>
Status: Draft
Type: Standards Track
Created: 2016-07-27
</pre>

==Abstract==

This BIP describes the next step in making Bitcoin's most basic element,
the transaction, more flexible and easier to extend. At the same time this
fixes all known cases of malleability and resolves significant amounts of
technical debt.

==Summary==

Flexible Transactions uses the fact that the first 4 bytes in a transaction
determine the version and that the majority of the clients use a
non-consensus rule (a policy) to not accept transaction version numbers
other than those specifically defined by Bitcoin.
This BIP chooses a new version number, 4, and defines that the data
following the bytes for the version is in a format called Compact Message
Format (CMF). CMF is a flexible, token based format where each token is a
combination of a name, a format and a value. Because the name is added we
can skip unused tokens and we can freely add new tokens in a simple manner
in future. Soft fork upgrades will become much easier and cleaner this
way.

This protocol upgrade cleans up past soft fork changes like BIP68 which
reuse existing fields and do them in a much better to maintain and easier
to parse system. It creates the building blocks to allow new features to be
added much cleaner in the future.

It also shows to be possible to remove signatures from transactions with
minimal upgrades of software and still maintain a coherent transaction
history. Tests show that this can reduce space usage to about 75%.

==Motivation==

Token based file-formats are not new, systems like XML and HTMl use a
similar system to allow future growth and they have been quite successful
for decades in part because of this property.

Bitcoin needs a similar way of making the transaction future-proof because
re-purposing not used fields for new features is not good for creating
maintainable code.

Next to that this protocol upgrade will re-order the data-fields which
allows us to cleanly fix the malleability issue which means that future
technologies like Lightning Network will depend on this BIP being deployed.

At the same time, due to this re-ordering of data fields, it becomes very
easy to remove signatures from a transaction without breaking its tx-id,
which is great for future pruning features.


=== Tokens ===

In the compact message format we define tokens and in this specification we
define how these tokens are named, where they can be placed and which are
optional. To refer to XML, this specification would be the schema of
a transaction.

CMF tokens are triplets of name, format (like PositiveInteger) and value.
Names in this scope are defined much like an enumeration where the actual
integer value (id, below) is equally important to the written name.
If any token found that is not covered in the next table will make the
transaction that contains it invalid.

{| class="wikitable"
|-
! Name !! id !! Format !! Default Value !! Description
|-
|TxEnd || 0 ||BoolTrue || Required ||A marker that is the last
byte in the txid calculation
|-
|TxInPrevHash || 1 ||ByteArray|| Required ||TxId we are spending
|-
|TxPrevIndex || 2 ||Integer || 0 ||Index in prev tx we are
spending (applied to previous TxInPrevHash)
|-
|TxInScript || 3 ||ByteArray|| Required ||The 'input' part of the
script
|-
|TxOutValue || 4 ||Integer || Required ||Amount of satoshi to
transfer
|-
|TxOutScript || 5 ||ByteArray|| Required ||The 'output' part of the
script
|-
|LockByBlock || 6 ||Integer || Optional ||BIP68 replacement
|-
|LockByTime || 7 ||Integer || Optional ||BIP68 replacement
|-
|ScriptVersion || 8 ||Integer || 2 ||Defines script version for
outputs following
|-
|NOP_1x || 1x || . || Optional ||Values that will be ignored by
anyone parsing the transaction
|}


=== Scripting changes ===

In the current version of Bitcoin-script, version 1, there are various
opcodes that are used to validate the cryptographic proofs that users have
to provide in order to spend outputs.

The OP_CHECKSIG is the most well known and, as its name implies, it
validates a signature.
In the new version of 'script' (version 2) the data that is signed is
changed to be equivalent to the transaction-id. This is a massive
simplification and also the only change between version 1 and version 2 of
script.

=== Serialization order===

The tokens defined above have to be serialized in a certain order for the
transaction to be well-formatted. Not serializing transactions in the
order specified would allow multiple interpretations of the data which
can't be allowed.
There is still some flexibility and for that reason it is important for
implementors to remember that the actual serialized data is used for the
calculation of the transaction-id. Reading and writing it may give you a
different output and when the txid changes, the signatures will break.

At a macro-level the transaction has these segments. The order of the
segments can not be changed, but you can skip segments.

{| class="wikitable"
!Segment !! Description
|-
| Inputs || Details about inputs.
|-
| Outputs || Details and scripts for outputs
|-
| Additional || For future expansion
|-
| Signatures || The scripts for the inputs
|-
| TxEnd || End of the transaction
|}

The TxId is calculated by taking the serialized transaction without the
Signatures and the TxEnd and hashing that.


{| class="wikitable"
!Segment !! Tags !! Description
|-
|Inputs||TxInPrevHash and TxInPrevIndex||Index can be skipped, but in any
input the PrevHash always has to come first
|-
|Outputs||TxOutScript, TxOutValue||Order is not relevant
|-
|Additional||LockByBlock LockByTime NOP_1x
|-
|Signatures||TxInScript||Exactly the same amount as there are inputs
|-
|TxEnd||TxEnd
|}

TxEnd is there to allow a parser to know when one transaction in a stream
has ended, allowing the next to be parsed.

Notice that the token ScriptVersion is currently not allowed because we
don't have any valid value to give it. But if we introduce a new script
version it would be placed in the outputs segment.

=== Script v2 ===

The default value of ScriptVersion is number 2, as opposed to the version 1
of script that the is in use today. The version 2 is mostly identical
to version one, including upgrades made to it over the years and in the
future. The only exception is that the OP_CHECKSIG is made dramatically
simpler. The input-type for OP_CHECKSIG is now no longer configurable, it is
always '1' and the content that will be signed is the txid.

TODO: does check-multisig need its own mention?


=== Block-malleability ===

The effect of leaving the signatures out of the calculation of the
transaction-id implies that the signatures are also not used for the
calculation of the merkle tree. This means that changes in signatures
would not be detectable. Except naturally by the fact that missing or
broken signatures breaks full validation. But it is important to detect
modifications to such signatures outside of validating all transactions.

For this reason the merkle tree is extended to include (append) the hash of
the v4 transactions (and those alone) where the hash is taken over a
data-blob that is build up from:

1. the tx-id
2. the CMF-tokens 'TxInScript'


=== Future extensibility ===

The NOP_1x wildcard used in the table explaining tokens is actually a list
of 10 values that currently are specified as NOP (no-operation) tags.

Any implementation that supports the v4 transaction format should ignore
this field in a transaction. Interpreting and using the transaction as if
that field was not present at all.

Future software may use these fields to decorate a transaction with
additional data or features. Transaction generating software should not
trivially use these tokens for their own usage without cooperation and
communication with the rest of the Bitcoin ecosystem as miners certainly
have the option to reject transactions that use unknown-to-them tokens.


==Reference Implementation==

Bitcoin Classic includes this in its beta releases and a reference
implementation can be found at;

https://github.com/bitcoinclassic/bitcoinclassic/pull/186


==Deployment==

To be determined

==References==

[https://github.com/bitcoinclassic/documentation/blob/master/spec/compactmessageformat.md]
CMF
Luke Dashjr via bitcoin-dev
2016-09-20 21:31:47 UTC
Permalink
Post by Tom via bitcoin-dev
As the title suggests, I would like to formally request the assignment of a
BIP number for my FT spec.
Please open a pull request on the bitcoin/bips repo after this has been
discussed a bit on the ML.

Note that at least a basic backwards compatibility section is required for
assignment, especially since this appears to be a hard-fork proposal (if it is
meant as a soft-fork, please explain how that would work).
Post by Tom via bitcoin-dev
==Motivation==
Token based file-formats are not new, systems like XML and HTMl use a
similar system to allow future growth and they have been quite successful
for decades in part because of this property.
There is already a binary-safe format called EBML. Why not use that as the
basis for this BIP?
Post by Tom via bitcoin-dev
Next to that this protocol upgrade will re-order the data-fields which
allows us to cleanly fix the malleability issue which means that future
technologies like Lightning Network will depend on this BIP being deployed.
Note this won't fix malleability entirely, only third-party malleability (much
like segwit). The sender can always modify the transaction with more
inputs/outputs.
Post by Tom via bitcoin-dev
|TxEnd || 0 ||BoolTrue || Required ||A marker that is the last
byte in the txid calculation
The last byte in the txid calculation, or the last byte of the entire
transaction?

It seems from the later comments, that it is the end of the transaction as a
whole. Yet a separator between the txid and non-txid data would probably be
valuable, rather than hard-coding txid to skip signature types (which may be
unknown to old nodes, when extended).
Post by Tom via bitcoin-dev
The OP_CHECKSIG is the most well known and, as its name implies, it
validates a signature.
In the new version of 'script' (version 2) the data that is signed is
changed to be equivalent to the transaction-id. This is a massive
simplification and also the only change between version 1 and version 2 of
script.
This seems to be a major regression. What is the replacement for
SIGHASH_SINGLE and SIGHASH_ANYONECANPAY?

When revising OP_CHECKSIG, it would also be nice to add the ability to use
*only* a hash of the prevout's scriptPubKey in the input, so that *when* the
prevtx is malleated, the spending one remains valid. (This use case is
currently not supported.)
Post by Tom via bitcoin-dev
=== Serialization order===
The tokens defined above have to be serialized in a certain order for the
transaction to be well-formatted. Not serializing transactions in the
order specified would allow multiple interpretations of the data which
can't be allowed.
What happens if nodes encounter a different order in a block?
Post by Tom via bitcoin-dev
Notice that the token ScriptVersion is currently not allowed because we
don't have any valid value to give it. But if we introduce a new script
version it would be placed in the outputs segment.
What happens if I put ScriptVersion=1 here?
Post by Tom via bitcoin-dev
=== Script v2 ===
TODO: does check-multisig need its own mention?
Does multisig still require a dummy item on the stack?
Post by Tom via bitcoin-dev
=== Block-malleability ===
For this reason the merkle tree is extended to include (append) the hash of
the v4 transactions (and those alone) where the hash is taken over a
How should nodes know where in the merkle-tree the txids end, and the v4hashes
begin?

Luke
Tom via bitcoin-dev
2016-09-21 09:32:30 UTC
Permalink
Post by Luke Dashjr via bitcoin-dev
Post by Tom via bitcoin-dev
As the title suggests, I would like to formally request the assignment of a
BIP number for my FT spec.
Please open a pull request on the bitcoin/bips repo after this has been
discussed a bit on the ML.
It seems from the later comments, that it is the end of the transaction as a
whole. Yet a separator between the txid and non-txid data would probably be
valuable, rather than hard-coding txid to skip signature types (which may
be unknown to old nodes, when extended).
Post by Tom via bitcoin-dev
The OP_CHECKSIG is the most well known and, as its name implies, it
validates a signature.
In the new version of 'script' (version 2) the data that is signed is
changed to be equivalent to the transaction-id. This is a massive
simplification and also the only change between version 1 and version 2 of
script.
This seems to be a major regression. What is the replacement for
SIGHASH_SINGLE and SIGHASH_ANYONECANPAY?
How is this a regression? Can you explain what functionality is lost please?
Post by Luke Dashjr via bitcoin-dev
When revising OP_CHECKSIG, it would also be nice to add the ability to use
*only* a hash of the prevout's scriptPubKey in the input, so that *when* the
prevtx is malleated, the spending one remains valid. (This use case is
currently not supported.)
Maybe for the next version of script :)
Post by Luke Dashjr via bitcoin-dev
Post by Tom via bitcoin-dev
Notice that the token ScriptVersion is currently not allowed
What happens if I put ScriptVersion=1 here?
The transaction is invalid...
Post by Luke Dashjr via bitcoin-dev
Post by Tom via bitcoin-dev
=== Block-malleability ===
For this reason the merkle tree is extended to include (append) the hash of
the v4 transactions (and those alone) where the hash is taken over a
How should nodes know where in the merkle-tree the txids end, and the
v4hashes begin?
Because the txid based ones are not going away. So the number of transactions
in the block can be used to determine when the pure tx-id segment stops and
when the v4 hashes begin. Then its up to the client to rebuild the tree from
that list based on the larger input set to get the same root-node.

I clarified many little things on my clone of the bips, check there if you
want to see the details.
Peter Todd via bitcoin-dev
2016-09-20 21:56:44 UTC
Permalink
Post by Tom via bitcoin-dev
=== Serialization order===
The tokens defined above have to be serialized in a certain order for the
transaction to be well-formatted. Not serializing transactions in the
order specified would allow multiple interpretations of the data which
can't be allowed.
If the order of the tokens is fixed, the tokens themselves are redundant
information when tokens are required; when tokens may be omitted, a simple
"Some/None" flag to mark whether or not the optional data has been omitted is
appropriate.


Also, if you're going to break compatibility with all existing software, it
makes sense to use a format that extends the merkle tree down into the
transaction inputs and outputs.
--
https://petertodd.org 'peter'[:-1]@petertodd.org
Tom via bitcoin-dev
2016-09-21 09:32:33 UTC
Permalink
Thanks for your email Peter!
Post by Peter Todd via bitcoin-dev
Post by Tom via bitcoin-dev
=== Serialization order===
The tokens defined above have to be serialized in a certain order for the
transaction to be well-formatted. Not serializing transactions in the
order specified would allow multiple interpretations of the data which
can't be allowed.
If the order of the tokens is fixed, the tokens themselves are redundant
information when tokens are required; when tokens may be omitted, a simple
"Some/None" flag to mark whether or not the optional data has been omitted
is appropriate.
This is addressed in the spec;
https://github.com/bitcoinclassic/documentation/blob/master/spec/transactionv4.md

«The way towards that flexibility is to use a generic concept made popular
various decades ago with the XML format. The idea is that we give each
field a name and this means that new fields can be added or optional fields
can be omitted from individual transactions»
Post by Peter Todd via bitcoin-dev
Also, if you're going to break compatibility with all existing software, it
makes sense to use a format that extends the merkle tree down into the
transaction inputs and outputs.
Please argue your case.
Peter Todd via bitcoin-dev
2016-09-22 18:26:18 UTC
Permalink
Post by Tom via bitcoin-dev
Thanks for your email Peter!
Post by Peter Todd via bitcoin-dev
Post by Tom via bitcoin-dev
=== Serialization order===
The tokens defined above have to be serialized in a certain order for the
transaction to be well-formatted. Not serializing transactions in the
order specified would allow multiple interpretations of the data which
can't be allowed.
If the order of the tokens is fixed, the tokens themselves are redundant
information when tokens are required; when tokens may be omitted, a simple
"Some/None" flag to mark whether or not the optional data has been omitted
is appropriate.
This is addressed in the spec;
https://github.com/bitcoinclassic/documentation/blob/master/spec/transactionv4.md
«The way towards that flexibility is to use a generic concept made popular
various decades ago with the XML format. The idea is that we give each
field a name and this means that new fields can be added or optional fields
can be omitted from individual transactions»
That argument is not applicable to required fields: the code to get the fields
from the extensible format is every bit as complex as the very simple code
required to deserialize/serialize objects in the current system.

In any case your BIP needs to give some explicit examples of hypothetical
upgrades in the future, how they'd take advantage of this, and what the code to
do so would look like.
Post by Tom via bitcoin-dev
Post by Peter Todd via bitcoin-dev
Also, if you're going to break compatibility with all existing software, it
makes sense to use a format that extends the merkle tree down into the
transaction inputs and outputs.
Please argue your case.
See my arguments re: segwit a few months ago, e.g. the hardware wallet txin
proof use-case.
--
https://petertodd.org 'peter'[:-1]@petertodd.org
Tom via bitcoin-dev
2016-09-22 18:47:50 UTC
Permalink
Post by Peter Todd via bitcoin-dev
Post by Tom via bitcoin-dev
«The way towards that flexibility is to use a generic concept made
popular various decades ago with the XML format. The idea is that we
give each field a name and this means that new fields can be added or
optional fields can be omitted from individual transactions»
The argument that optional fields can be omitted is not applicable to
required fields, you are correct. That should be rather obvious because
required fields are not optional fields.
Post by Peter Todd via bitcoin-dev
the code to get the
fields from the extensible format is every bit as complex as the very
simple code required to deserialize/serialize objects in the current
system.
Probably a tiny bit more complex as the current format assumes a lot more.

You may have misread my email because there was no argument made towards
complexity. The argument was towards flexibility.
Post by Peter Todd via bitcoin-dev
In any case your BIP needs to give some explicit examples of hypothetical
upgrades in the future, how they'd take advantage of this, and what the
code to do so would look like.
Why?
Post by Peter Todd via bitcoin-dev
Post by Tom via bitcoin-dev
Post by Peter Todd via bitcoin-dev
Also, if you're going to break compatibility with all existing
software, it makes sense to use a format that extends the merkle
tree down into the transaction inputs and outputs.
Please argue your case.
See my arguments re: segwit a few months ago, e.g. the hardware wallet
txin proof use-case.
Please consider that I'm not going to search for something based on a vague
reference like that, if you want to convince me you could you at least
provide a URL?
You want me to see the value of your idea, I think you should at least
provide the argument. Isn't that fair?

Thanks for your email Peter, would love you to put a bit more time into
understanding flexible transactions and we can have a proper discussion
about it.
Andreas Schildbach via bitcoin-dev
2016-09-21 12:00:23 UTC
Permalink
Just glancing over your BIP, I wonder if we should use Protobuf. It uses
this "flexible" format already and is quite compact/binary. We use
Protobuf already for the payment protocol, and there is very good tool
support.
Tom via bitcoin-dev
2016-09-21 12:58:02 UTC
Permalink
Post by Andreas Schildbach via bitcoin-dev
Just glancing over your BIP, I wonder if we should use Protobuf. It uses
this "flexible" format already and is quite compact/binary. We use
Protobuf already for the payment protocol, and there is very good tool
support.
There is a lot of overlap between different binary formats. Looking through
the on-the-wire protocol you'll see that my spec is very similar. Practically
all the advantages of protobuf are present in CMF. I can write you a java
parser if you want, it should be easy to port from Qt/C++ code :)
https://github.com/bitcoinclassic/transactions

CMF:
https://github.com/bitcoinclassic/documentation/blob/master/spec/compactmessageformat.md

There is no tool support needed, just one or two classes. Which personally I
think is an advantage.


Some advantages of CMF over protobuf from the top of my head;

* It reuses the var-int parsing that Bitcoin uses (which is itself slightly
different from others).

* zero-copy support (not relevant for this bip, though).

* Additional values addition (i.e. adding new data) is .. tricky in protobuf.
https://developers.google.com/protocol-buffers/docs/proto#updating

* In my experience parsing a message manually (like a SOX parser) is much
better in reporting errors and detecting wrong usages than auto-generated code
(but personally I'm not much a fan of auto-generated APIs) at all...

* Generated parsing/writing code will not be as fast as we can make it.

* CMF is more compact (uses less bytes) for its messages.


Protobuf is something I've used before and I think we can do better. I think
that CMF together with some support classes can do this better.
Gregory Maxwell via bitcoin-dev
2016-09-21 18:01:30 UTC
Permalink
On Tue, Sep 20, 2016 at 5:15 PM, Tom via bitcoin-dev
Post by Tom via bitcoin-dev
BIP number for my FT spec.
This document does not appear to be concretely specified enough to
review or implement from it.

For example, it does not specify the serialization of "integer" (is it
a 32 bit word in network byte order or?) nor does it specify how the
presence of the optional fields are signaled nor the cardinality of
the inputs or outputs. For clearly variable length elements
('bytearray') no mention is made of their length encoding. etc.

Without information like this, I don't see how someone could
realistically begin reviewing this proposal.

The motivation seems unclear to me as well: The scheme is described as
'flexible' but it appears to remove flexibility from the existing
system. The "schema" appears to be hardcoded and never communicated.
If the goal is to simply have a more compact on the wire
representation, this could be done without changing the serialization
used for hashing or the serialization used for costing.
Tom via bitcoin-dev
2016-09-22 08:56:31 UTC
Permalink
Post by Gregory Maxwell via bitcoin-dev
On Tue, Sep 20, 2016 at 5:15 PM, Tom via bitcoin-dev
Post by Tom via bitcoin-dev
BIP number for my FT spec.
This document does not appear to be concretely specified enough to
review or implement from it.
For example, it does not specify the serialization of "integer"
It refers to the external specification which is linked at the bottom.
In that spec you'll see that "Integer" is the standard var-int that Bitcoin
has used for years.
Post by Gregory Maxwell via bitcoin-dev
nor does it specify how the
presence of the optional fields are signaled
How does one signals an optional field except of in the spec? Thats the job of
a specification.
Post by Gregory Maxwell via bitcoin-dev
nor the cardinality of
the inputs or outputs.
Did you miss this in the 3rd table ? I suggest clicking on the github bips
repo link as tables are not easy to read in mediawiki plain format that the
email contained.
Post by Gregory Maxwell via bitcoin-dev
For clearly variable length elements
('bytearray') no mention is made of their length encoding. etc.
Also in the external CMF spec.
Post by Gregory Maxwell via bitcoin-dev
Without information like this, I don't see how someone could
realistically begin reviewing this proposal.
I agree, that would be bad. Luckily that you just missed the link :)
Here it is;
https://github.com/bitcoinclassic/documentation/blob/master/spec/compactmessageformat.md
Post by Gregory Maxwell via bitcoin-dev
The motivation seems unclear to me as well: The scheme is described as
'flexible' but it appears to remove flexibility from the existing
system. The "schema" appears to be hardcoded and never communicated.
Being hardcoded and never communicated is what the current format does to. How
does that "remove flexibility".

Also read my reply to Peter Todd on why this is flexible.
Post by Gregory Maxwell via bitcoin-dev
If the goal is to simply have {snip}
It is not.

Thanks for asking, I understand that the CMF spec is useful to see as well.
Hopefully you can now review it properly since I linked to it above.

Cheers!
Christian Decker via bitcoin-dev
2016-09-22 11:10:49 UTC
Permalink
Post by Tom via bitcoin-dev
Post by Gregory Maxwell via bitcoin-dev
On Tue, Sep 20, 2016 at 5:15 PM, Tom via bitcoin-dev
Post by Tom via bitcoin-dev
BIP number for my FT spec.
This document does not appear to be concretely specified enough to
review or implement from it.
For example, it does not specify the serialization of "integer"
It refers to the external specification which is linked at the bottom.
In that spec you'll see that "Integer" is the standard var-int that Bitcoin
has used for years.
I think BIPs should be self-contained, or rely on previous BIPs,
whenever possible. Referencing an external formatting document should
be avoided and requiring readers to reverse engineer a reference
implementation doesn't seem too user friendly either. Publishing a BIP
with CMF would certainly help, and completing this spec with the
details that are missing, or only "defined" in the implementation,
would be better.
Post by Tom via bitcoin-dev
Post by Gregory Maxwell via bitcoin-dev
nor does it specify how the
presence of the optional fields are signaled
How does one signals an optional field except of in the spec? Thats the job of
a specification.
So the presence is signaled by encountering the tag, which contains
both token type and name-reference. The encoder and decoder operations
could be described better.
Post by Tom via bitcoin-dev
Post by Gregory Maxwell via bitcoin-dev
nor the cardinality of
the inputs or outputs.
Did you miss this in the 3rd table ? I suggest clicking on the github bips
repo link as tables are not easy to read in mediawiki plain format that the
email contained.
Minor nit: that table is not well-formed. As was pointed out in the
normalized transaction ID BIP, your proposal only addresses
third-party malleability, since signers can simply change the
transaction and re-sign it. This is evident from the fact that inputs
and outputs do not have a canonical order and it would appear that
tokens can be re-ordered in segments. Dependencies of tokens inside a
segment are also rather alarming (TxInPrevHash <-> TxInPrevIndex,
TxOutScript <-> TxOutValue).

Finally, allowing miners to reject transactions with unknown fields
makes the OP_NOPs unusable since they'd result in forks: non-upgraded
nodes would reject blocks from upgraded nodes.

Regards,
Christian
Tom via bitcoin-dev
2016-09-22 12:09:38 UTC
Permalink
Post by Christian Decker via bitcoin-dev
Post by Tom via bitcoin-dev
Post by Gregory Maxwell via bitcoin-dev
On Tue, Sep 20, 2016 at 5:15 PM, Tom via bitcoin-dev
Post by Tom via bitcoin-dev
BIP number for my FT spec.
This document does not appear to be concretely specified enough to
review or implement from it.
For example, it does not specify the serialization of "integer"
It refers to the external specification which is linked at the bottom.
In that spec you'll see that "Integer" is the standard var-int that Bitcoin
has used for years.
I think BIPs should be self-contained, or rely on previous BIPs,
whenever possible. Referencing an external formatting document should
be avoided
If luke-jr thinks I should submit CMF as a BIP, I can certainly do that.
Luke, what do you think?

I don't have a preference either way.
Post by Christian Decker via bitcoin-dev
Post by Tom via bitcoin-dev
Post by Gregory Maxwell via bitcoin-dev
nor does it specify how the
presence of the optional fields are signaled
How does one signals an optional field except of in the spec? Thats the
job of a specification.
So the presence is signaled by encountering the tag, which contains
both token type and name-reference. The encoder and decoder operations
could be described better.
I'm sorry, I'm not following you here. Is there a question?
Post by Christian Decker via bitcoin-dev
Post by Tom via bitcoin-dev
Post by Gregory Maxwell via bitcoin-dev
nor the cardinality of
the inputs or outputs.
Did you miss this in the 3rd table ? I suggest clicking on the github bips
repo link as tables are not easy to read in mediawiki plain format that the
email contained.
Minor nit: that table is not well-formed.
I am not very well versed in mediawiki tables, and I found github has some
incompatibilities too.
The markdown one looks better;
https://github.com/bitcoinclassic/documentation/blob/master/spec/transactionv4.md
Post by Christian Decker via bitcoin-dev
As was pointed out in the
normalized transaction ID BIP, your proposal only addresses
third-party malleability, since signers can simply change the
transaction and re-sign it.
I have to disagree. That is not malleability. Creating a new document and re-
signing it is not changing anything. Its re-creating. Something that the owner
of the coin has every right to do.
Post by Christian Decker via bitcoin-dev
This is evident from the fact that inputs
and outputs do not have a canonical order and it would appear that
tokens can be re-ordered in segments.
Sorry, what is evident? You seem to imply that it is uncommon that you can
create two transactions of similar intent but using different bytes.
You would be wrong with this implication as this is very common. You can just
alter the order of the inputs, for instance.

I am unable to see what the point is you are trying to make. Is there a
question or a suggestion for improvement here?
Post by Christian Decker via bitcoin-dev
Dependencies of tokens inside a
segment are also rather alarming (TxInPrevHash <-> TxInPrevIndex,
TxOutScript <-> TxOutValue).
Maybe you missed this line;
«TxInPrevHash and TxInPrevIndex
Index can be skipped, but in any input the PrevHash always has
to come first»

If you still see something alarming, let me know.
You can look at the code in Bitcoin Classic and notice that it really isn't
anything complicated or worrying.
Post by Christian Decker via bitcoin-dev
Finally, allowing miners to reject transactions with unknown fields
makes the OP_NOPs unusable
Hmm, it looks like you are mixing terminology and abstraction-levels. OP_NOP
is a field from script and there is no discussion about any rejection based on
script in this BIP at all.

Rejection of transactions is done on there being unrecognised tokens in the
transaction formatting itself.

Thank you for your email to my BIP, I hope you got the answers you were
looking for :)
Christian Decker via bitcoin-dev
2016-09-23 11:42:36 UTC
Permalink
Post by Tom via bitcoin-dev
Post by Christian Decker via bitcoin-dev
I think BIPs should be self-contained, or rely on previous BIPs,
whenever possible. Referencing an external formatting document should
be avoided
If luke-jr thinks I should submit CMF as a BIP, I can certainly do that.
Luke, what do you think?
I don't have a preference either way.
Post by Christian Decker via bitcoin-dev
So the presence is signaled by encountering the tag, which contains
both token type and name-reference. The encoder and decoder operations
could be described better.
I'm sorry, I'm not following you here. Is there a question?
Nope, just clarifying how presence or absence is indicated :-)
Post by Tom via bitcoin-dev
Post by Christian Decker via bitcoin-dev
Minor nit: that table is not well-formed.
I am not very well versed in mediawiki tables, and I found github has some
incompatibilities too.
The markdown one looks better;
https://github.com/bitcoinclassic/documentation/blob/master/spec/transactionv4.md
It's just some rows have 3 columns, others have 2. It's a minor nit
really.
Post by Tom via bitcoin-dev
Post by Christian Decker via bitcoin-dev
As was pointed out in the
normalized transaction ID BIP, your proposal only addresses
third-party malleability, since signers can simply change the
transaction and re-sign it.
I have to disagree. That is not malleability. Creating a new document and re-
signing it is not changing anything. Its re-creating. Something that the owner
of the coin has every right to do.
Same thing I was arguing back then, however Luke pointed out that
malleability just refers to the possibility of modifying a transaction
after the fact. Always referring to "third-party malleability" avoids
this ambiguity.
Post by Tom via bitcoin-dev
Post by Christian Decker via bitcoin-dev
This is evident from the fact that inputs
and outputs do not have a canonical order and it would appear that
tokens can be re-ordered in segments.
Sorry, what is evident? You seem to imply that it is uncommon that you can
create two transactions of similar intent but using different bytes.
You would be wrong with this implication as this is very common. You can just
alter the order of the inputs, for instance.
I am unable to see what the point is you are trying to make. Is there a
question or a suggestion for improvement here?
Post by Christian Decker via bitcoin-dev
Dependencies of tokens inside a
segment are also rather alarming (TxInPrevHash <-> TxInPrevIndex,
TxOutScript <-> TxOutValue).
Maybe you missed this line;
«TxInPrevHash and TxInPrevIndex
Index can be skipped, but in any input the PrevHash always has
to come first»
Nope, that is exactly the kind of dependency I was talking
about. Instead of nesting a construct like the current transactions
do, you rely on the order of tokens to imply that they belong
together.
Post by Tom via bitcoin-dev
If you still see something alarming, let me know.
You can look at the code in Bitcoin Classic and notice that it really isn't
anything complicated or worrying.
Post by Christian Decker via bitcoin-dev
Finally, allowing miners to reject transactions with unknown fields
makes the OP_NOPs unusable
Hmm, it looks like you are mixing terminology and abstraction-levels. OP_NOP
is a field from script and there is no discussion about any rejection based on
script in this BIP at all.
Rejection of transactions is done on there being unrecognised tokens in the
transaction formatting itself.
Ah, thanks for clearing that up. However, the problem persists, if we
add new fields that a non-upgraded node doesn't know about and it
rejects transactions containing it, we'll have a hard-fork. It should
probably not reject transactions with unknown fields if the
transaction is included in a block.
Post by Tom via bitcoin-dev
Thank you for your email to my BIP, I hope you got the answers you were
looking for :)
Cheers,
Christian
Tom via bitcoin-dev
2016-09-23 13:17:52 UTC
Permalink
Post by Christian Decker via bitcoin-dev
Post by Tom via bitcoin-dev
I have to disagree. That is not malleability. Creating a new document
and re- signing it is not changing anything. Its re-creating.
Something that the owner of the coin has every right to do.
Same thing I was arguing back then, however Luke pointed out that
malleability just refers to the possibility of modifying a transaction
after the fact.
I am not a fan of redefining dictionary words. I'll stick to the
universally excepted one, thanks.
Post by Christian Decker via bitcoin-dev
Nope, that is exactly the kind of dependency I was talking
about. Instead of nesting a construct like the current transactions
do, you rely on the order of tokens to imply that they belong
together.
if we
add new fields that a non-upgraded node doesn't know about and it
rejects transactions containing it, we'll have a hard-fork. It should
probably not reject transactions with unknown fields if the
transaction is included in a block.
This is addressed here;
https://github.com/bitcoin/bips/blob/master/bip-0134.mediawiki#future-extensibility
adiabat via bitcoin-dev
2016-09-21 22:45:55 UTC
Permalink
Hi-

One concern is that this doesn't seem compatible with Lightning as
currently written. Most relevant is that non-cooperative channel close
transactions in Lightning use OP_CHECKSEQUENCEVERIFY, which references the
sequence field of the txin; if the txin doesn't have a sequence number,
OP_CHECKSEQUENCEVERIFY can't work.

LockByBlock and LockByTime aren't described and there doesn't seem to be
code for them in the PR (186). If there's a way to make OP_CLTV and OP_CSV
work with this new format, please let us know, thanks!

-Tadge
Tom via bitcoin-dev
2016-09-22 08:47:03 UTC
Permalink
Post by adiabat via bitcoin-dev
Hi-
One concern is that this doesn't seem compatible with Lightning as
currently written. Most relevant is that non-cooperative channel close
transactions in Lightning use OP_CHECKSEQUENCEVERIFY, which references the
sequence field of the txin; if the txin doesn't have a sequence number,
OP_CHECKSEQUENCEVERIFY can't work.
LockByBlock and LockByTime aren't described and there doesn't seem to be
code for them in the PR (186). If there's a way to make OP_CLTV and OP_CSV
work with this new format, please let us know, thanks!
LockByBlock and LockByTime are still TODOs because I didn't have time to go
in-dept into how BIP68 does the encoding.
The intent is that these tags, while loading, will set the sequence integer in
the txin as the old version does. And while saving we do the reverse.

In other words; the lack of sequence number in the saved format doesn't affect
the in-memory format of the transaction. The in-memory version is the one that
script will operate on.

This means that there is no change in how CSV will work before and after on
any level other than serialisation.

Flexible Transactions is definitely meant to support the Lightning Network, so
any problems you find is something we should solve before it ships.

Thanks for your email!
Peter Todd via bitcoin-dev
2016-09-22 18:27:29 UTC
Permalink
Post by Tom via bitcoin-dev
Post by adiabat via bitcoin-dev
Hi-
One concern is that this doesn't seem compatible with Lightning as
currently written. Most relevant is that non-cooperative channel close
transactions in Lightning use OP_CHECKSEQUENCEVERIFY, which references the
sequence field of the txin; if the txin doesn't have a sequence number,
OP_CHECKSEQUENCEVERIFY can't work.
LockByBlock and LockByTime aren't described and there doesn't seem to be
code for them in the PR (186). If there's a way to make OP_CLTV and OP_CSV
work with this new format, please let us know, thanks!
LockByBlock and LockByTime are still TODOs because I didn't have time to go
in-dept into how BIP68 does the encoding.
The intent is that these tags, while loading, will set the sequence integer in
the txin as the old version does. And while saving we do the reverse.
In other words; the lack of sequence number in the saved format doesn't affect
the in-memory format of the transaction. The in-memory version is the one that
script will operate on.
This means that there is no change in how CSV will work before and after on
any level other than serialisation.
CSV uses per-input sequence numbers; you only have a per-tx equivalent.
--
https://petertodd.org 'peter'[:-1]@petertodd.org
Tom via bitcoin-dev
2016-09-22 18:37:29 UTC
Permalink
Post by Peter Todd via bitcoin-dev
CSV uses per-input sequence numbers; you only have a per-tx equivalent.
I think you misunderstand tagged systems at a very basic level. You think
that html can only use a bold tag <b> once in a document? Thats equivalent
to what you are saying.

Your comment is rather embarrassing, I have to point out. You may want to
read a bit more before you comment more.
Jonas Schnelli via bitcoin-dev
2016-09-22 19:59:12 UTC
Permalink
Hi Tom
Post by Tom via bitcoin-dev
I think you misunderstand tagged systems at a very basic level. You think
that html can only use a bold tag <b> once in a document? Thats equivalent
to what you are saying.
Would the "additional" segment contain the same amount of
nSequence-equivalent token as the number of inputs in the "inputs" segment?
What if you only want to add a per-input-token in the additional segment
for a certain input (assume last input)?
I guess the fundamental difference to html is the possible nesting.

However, I think that should be mentioned/specified in the BIP.

</jonas>
Tom via bitcoin-dev
2016-09-22 20:07:33 UTC
Permalink
On Thursday, 22 September 2016 21:59:12 CEST Jonas Schnelli via bitcoin-dev
Post by Jonas Schnelli via bitcoin-dev
Hi Tom
Post by Tom via bitcoin-dev
I think you misunderstand tagged systems at a very basic level. You
think that html can only use a bold tag <b> once in a document? Thats
equivalent to what you are saying.
Would the "additional" segment contain the same amount of
nSequence-equivalent token as the number of inputs in the "inputs" segment?
At this point I don't know what it should look like, I have not had time to
look deeply into BIP68. Is this what you would suggest it to look like?
I rather figured spending limitations would be assigned to an output, not
an input.
Post by Jonas Schnelli via bitcoin-dev
However, I think that should be mentioned/specified in the BIP.
It can be, and likely should be. This BIP doesn't pretend to be finished
yet.

I welcome any and all discussion about this, it only serves to make the end
result stronger!
Christian Decker via bitcoin-dev
2016-09-23 11:55:50 UTC
Permalink
Post by Tom via bitcoin-dev
Post by Peter Todd via bitcoin-dev
CSV uses per-input sequence numbers; you only have a per-tx equivalent.
I think you misunderstand tagged systems at a very basic level. You think
that html can only use a bold tag <b> once in a document? Thats equivalent
to what you are saying.
Your comment is rather embarrassing, I have to point out. You may want to
read a bit more before you comment more.
Not sure if the comparison to XML and HTML holds: the lack of closing
tags makes the meaning of individual tokens ambiguous, like I pointed
out before. The use of segments gives at most two levels of nesting,
so any relationship among tokens in the same segment has to rely on
their relative position, which could result in ambiguities, like
whether a tag refers to a single input or the transaction as a whole.

Cheers,
Christian
Tom via bitcoin-dev
2016-09-23 13:13:10 UTC
Permalink
On Friday, 23 September 2016 13:55:50 CEST Christian Decker via bitcoin-dev
Post by Christian Decker via bitcoin-dev
Not sure if the comparison to XML and HTML holds: the lack of closing
tags makes the meaning of individual tokens ambiguous, like I pointed
out before. The use of segments gives at most two levels of nesting,
so any relationship among tokens in the same segment has to rely on
their relative position, which could result in ambiguities, like
whether a tag refers to a single input or the transaction as a whole.
Practically all tagged formats make ordering a requirement, so indeed this
is relevant, and not unique.

For instance if you write;
<div> Some line </br>Another line</br>3rd line</div>
you can get a good idea of how ordering is relevant. You can reuse any item
many times.

Whenever there is a possible confusion the specification specifically
explains which order to use.

I'm not sure what you mean with the idea this;
Post by Christian Decker via bitcoin-dev
The use of segments gives at most two levels of nesting
It looks like you assume there is some opening and closing tags, since
otherwise there would be no nesting.
Such tags are not intended, nor documented.
Post by Christian Decker via bitcoin-dev
so any relationship among tokens in the same segment has to rely on
their relative position, which could result in ambiguities, like
whether a tag refers to a single input or the transaction as a whole.
I quoted parts of the spec in your previous email stating the same thing,
but I'll repeat here.
Any place that has any sort of possibility to be ambiguous is specified
specifically to have an order. This makes writing and parsing easier.

Since you wrote two emails now with the same issue, and I addressed it
twice, I would urge you to write out some examples which may be confusing
and if you find that the spec is indeed missing requirements then please
share it with us. I did this some time ago and it helps understanding the
ideas by having actual explicit examples. I am not aware of any sort of
ambiguities that the spec allows.

Cheers!
Loading...