[bitcoin-dev] {sign|verify}message replacement

[Karl Johan Alm via bitcoin-dev]

Hello,

I am considering writing a replacement for the message signing tools
that are currently broken for all but the legacy 1xx addresses. The
approach (suggested by Pieter Wuille) is to do a script based
approach. This does not seem to require a lot of effort for
implementing in Bitcoin Core*. Below is my proposal for this system:

A new structure SignatureProof is added, which is a simple scriptSig &
witnessProgram container that can be serialized. This is passed out
from/into the signer/verifier.

RPC commands:

sign <address> <message> [<prehashed>=false]

Generates a signature proof for <message> using the same method that
would be used to spend coins sent to <address>.**

verify <address> <message> <proof> [<prehashed>=false]

Deserializes and executes the proof using a custom signature checker
whose sighash is derived from <message>. Returns true if the check
succeeds, and false otherwise. The scriptPubKey is derived directly
from <address>.**

Feedback welcome.

-Kalle.

(*) Looks like you can simply use VerifyScript with a new signature
checker class. (h/t Nicolas Dorier)
(**) If <prehashed> is true, <message> is the sighash, otherwise
sighash=sha256d(message).

[Luke Dashjr via bitcoin-dev]

I don't see a need for a new RPC interface, just a new signature format.

Ideally, it should support not only just "proof I receive at this address",
but also "proof of funds" (as a separate feature) since this is a popular
misuse of the current message signing (which doesn't actually prove funds at
all). To do this, it needs to be capable of signing for multiple inputs.

Preferably, it should also avoid disclosing the public key for existing or
future UTXOs. But I don't think it's possible to avoid this without something
MAST-like first. Perhaps it can be a MAST upgrade later on, but the new
signature scheme should probably be designed with it in mind.

Luke


On Wednesday 14 March 2018 8:09:20 AM Karl Johan Alm via bitcoin-dev wrote:
> Hello,
>
> I am considering writing a replacement for the message signing tools
> that are currently broken for all but the legacy 1xx addresses. The
> approach (suggested by Pieter Wuille) is to do a script based
> approach. This does not seem to require a lot of effort for
> implementing in Bitcoin Core*. Below is my proposal for this system:
>
> A new structure SignatureProof is added, which is a simple scriptSig &
> witnessProgram container that can be serialized. This is passed out
> from/into the signer/verifier.
>
> RPC commands:
>
> sign <address> <message> [<prehashed>=false]
>
> Generates a signature proof for <message> using the same method that
> would be used to spend coins sent to <address>.**
>
> verify <address> <message> <proof> [<prehashed>=false]
>
> Deserializes and executes the proof using a custom signature checker
> whose sighash is derived from <message>. Returns true if the check
> succeeds, and false otherwise. The scriptPubKey is derived directly
> from <address>.**
>
> Feedback welcome.
>
> -Kalle.
>
> (*) Looks like you can simply use VerifyScript with a new signature
> checker class. (h/t Nicolas Dorier)
> (**) If <prehashed> is true, <message> is the sighash, otherwise
> sighash=sha256d(message).
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-***@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[Karl Johan Alm via bitcoin-dev]

On Wed, Mar 14, 2018 at 12:36 PM, Luke Dashjr <***@dashjr.org> wrote:
> Ideally, it should support not only just "proof I receive at this address",
> but also "proof of funds" (as a separate feature) since this is a popular
> misuse of the current message signing (which doesn't actually prove funds at
> all). To do this, it needs to be capable of signing for multiple inputs.

Re-reading this, I think what you mean is it should be possible to
create a proof for (a) specific UTXO(s), hence "inputs". That sounds
pretty useful, yeah!

So you could provide a mix of addresses and inputs (as txid:vout) and
it would generate a proof that signs the message for each input
(taking scriptPubKey from address or from the UTXO data directly on
the blockchain).

-Kalle.

[Luke Dashjr via bitcoin-dev]

On Thursday 15 March 2018 7:36:48 AM Karl Johan Alm wrote:
> On Wed, Mar 14, 2018 at 12:36 PM, Luke Dashjr <***@dashjr.org> wrote:
> > Ideally, it should support not only just "proof I receive at this
> > address", but also "proof of funds" (as a separate feature) since this
> > is a popular misuse of the current message signing (which doesn't
> > actually prove funds at all). To do this, it needs to be capable of
> > signing for multiple inputs.
>
> Re-reading this, I think what you mean is it should be possible to
> create a proof for (a) specific UTXO(s), hence "inputs". That sounds
> pretty useful, yeah!

Not necessarily specific UTXOs (that would contradict fungibility, as well as
be impossible for hot/cold wallet separation), but just to prove funds are
available. The current sign message cannot be used to prove present possession
of funds, only that you receive funds.

[Karl Johan Alm via bitcoin-dev]

On Thu, Mar 15, 2018 at 2:14 PM, Luke Dashjr <***@dashjr.org> wrote:
> Not necessarily specific UTXOs (that would contradict fungibility, as well as
> be impossible for hot/cold wallet separation), but just to prove funds are
> available. The current sign message cannot be used to prove present possession
> of funds, only that you receive funds.

By saying "not necessarily specific UTXOs", are you saying it may be
spent outputs? I'm a little confused I think.

On Thu, Mar 15, 2018 at 8:53 PM, Jim Posen <***@gmail.com> wrote:
> In this general signing-a-script context, I think a verifier might want to
> see the time conditions under which it may be spent. The proof container
> could include an optional nLockTime which defaults to 0 and nSequence which
> defaults to 0xFFFF...

Good point!

>> I think it would just use the default (SIGHASH_ALL?) for simplicity.
>> Is there a good reason to tweak it?
>
> I took another look and there should definitely be a byte appended to the
> end of the sig so that the encoding checks pass, but I think it might as
> well be a 0x00 byte since it's not actually a sighash flag.

I think the sighash flag affects the outcome of the actual
verification, but I could be mistaken.

-Kalle.

[Greg Sanders via bitcoin-dev]

Sorry if I missed the rationale earlier, but why not just do a transaction,
with a FORKID specifically for this? Then a node can have a mempool
acceptance test that returns true even if the signature is not valid as per
Bitcoin consensus, but only due to the FORKID?

This way basically any wallet can support this provided generic FORKID
support.

On Thu, Mar 15, 2018 at 8:38 PM, Karl Johan Alm via bitcoin-dev <
bitcoin-***@lists.linuxfoundation.org> wrote:

> On Thu, Mar 15, 2018 at 2:14 PM, Luke Dashjr <***@dashjr.org> wrote:
> > Not necessarily specific UTXOs (that would contradict fungibility, as
> well as
> > be impossible for hot/cold wallet separation), but just to prove funds
> are
> > available. The current sign message cannot be used to prove present
> possession
> > of funds, only that you receive funds.
>
> By saying "not necessarily specific UTXOs", are you saying it may be
> spent outputs? I'm a little confused I think.
>
> On Thu, Mar 15, 2018 at 8:53 PM, Jim Posen <***@gmail.com> wrote:
> > In this general signing-a-script context, I think a verifier might want
> to
> > see the time conditions under which it may be spent. The proof container
> > could include an optional nLockTime which defaults to 0 and nSequence
> which
> > defaults to 0xFFFF...
>
> Good point!
>
> >> I think it would just use the default (SIGHASH_ALL?) for simplicity.
> >> Is there a good reason to tweak it?
> >
> > I took another look and there should definitely be a byte appended to the
> > end of the sig so that the encoding checks pass, but I think it might as
> > well be a 0x00 byte since it's not actually a sighash flag.
>
> I think the sighash flag affects the outcome of the actual
> verification, but I could be mistaken.
>
> -Kalle.
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-***@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
>

[Karl Johan Alm via bitcoin-dev]

On Fri, Mar 16, 2018 at 1:59 AM, Greg Sanders <***@gmail.com> wrote:
> Sorry if I missed the rationale earlier, but why not just do a transaction,
> with a FORKID specifically for this? Then a node can have a mempool
> acceptance test that returns true even if the signature is not valid as per
> Bitcoin consensus, but only due to the FORKID?
>
> This way basically any wallet can support this provided generic FORKID
> support.

You'd basically have to provide an entire transaction rather than just
the signature, so there's some overhead. (Copy-pasting may become
unwieldy quicker.)

[Damian Williamson via bitcoin-dev]

That is very helpful Luke. I would not have been concerned if it was necessary to sign multiple times for multiple utxo's on different addresses but, since it is a feature it may as well be best usable. Signing for multiple inputs verifying that you have the priv key for each in your wallet is certainly usable for this popular misuse.


>Ideally, it should support not only just "proof I receive at this address",
but also "proof of funds" (as a separate feature) since this is a popular
misuse of the current message signing (which doesn't actually prove funds at
all). To do this, it needs to be capable of signing for multiple inputs.

________________________________
From: bitcoin-dev-***@lists.linuxfoundation.org <bitcoin-dev-***@lists.linuxfoundation.org> on behalf of Luke Dashjr via bitcoin-dev <bitcoin-***@lists.linuxfoundation.org>
Sent: Wednesday, 14 March 2018 11:36:47 PM
To: Karl Johan Alm; Bitcoin Protocol Discussion
Subject: Re: [bitcoin-dev] {sign|verify}message replacement

I don't see a need for a new RPC interface, just a new signature format.

Ideally, it should support not only just "proof I receive at this address",
but also "proof of funds" (as a separate feature) since this is a popular
misuse of the current message signing (which doesn't actually prove funds at
all). To do this, it needs to be capable of signing for multiple inputs.

Preferably, it should also avoid disclosing the public key for existing or
future UTXOs. But I don't think it's possible to avoid this without something
MAST-like first. Perhaps it can be a MAST upgrade later on, but the new
signature scheme should probably be designed with it in mind.

Luke


On Wednesday 14 March 2018 8:09:20 AM Karl Johan Alm via bitcoin-dev wrote:
> Hello,
>
> I am considering writing a replacement for the message signing tools
> that are currently broken for all but the legacy 1xx addresses. The
> approach (suggested by Pieter Wuille) is to do a script based
> approach. This does not seem to require a lot of effort for
> implementing in Bitcoin Core*. Below is my proposal for this system:
>
> A new structure SignatureProof is added, which is a simple scriptSig &
> witnessProgram container that can be serialized. This is passed out
> from/into the signer/verifier.
>
> RPC commands:
>
> sign <address> <message> [<prehashed>=false]
>
> Generates a signature proof for <message> using the same method that
> would be used to spend coins sent to <address>.**
>
> verify <address> <message> <proof> [<prehashed>=false]
>
> Deserializes and executes the proof using a custom signature checker
> whose sighash is derived from <message>. Returns true if the check
> succeeds, and false otherwise. The scriptPubKey is derived directly
> from <address>.**
>
> Feedback welcome.
>
> -Kalle.
>
> (*) Looks like you can simply use VerifyScript with a new signature
> checker class. (h/t Nicolas Dorier)
> (**) If <prehashed> is true, <message> is the sighash, otherwise
> sighash=sha256d(message).
> _______________________________________________
> bitcoin-dev mailing list
> bitcoin-***@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[Kalle Rosenbaum via bitcoin-dev]

Thank you.

I can't really see from your proposal if you had thought of this: A soft
fork can make old nodes accept invalid message signatures as valid. For
example, a "signer" can use a witness version unknown to the verifier to
fool the verifier. Witness version is detectable (just reject unknown
witness versions) but there may be more subtle changes. Segwit was not
"detectable" in that way, for example.

This is the reason why I withdrew BIP120. If you have thought about the
above, I'd be very interested.

/Kalle

Sent from my Sinclair ZX81

Den 14 mars 2018 16:10 skrev "Karl Johan Alm via bitcoin-dev" <
bitcoin-***@lists.linuxfoundation.org>:

Hello,

I am considering writing a replacement for the message signing tools
that are currently broken for all but the legacy 1xx addresses. The
approach (suggested by Pieter Wuille) is to do a script based
approach. This does not seem to require a lot of effort for
implementing in Bitcoin Core*. Below is my proposal for this system:

A new structure SignatureProof is added, which is a simple scriptSig &
witnessProgram container that can be serialized. This is passed out
from/into the signer/verifier.

RPC commands:

sign <address> <message> [<prehashed>=false]

Generates a signature proof for <message> using the same method that
would be used to spend coins sent to <address>.**

verify <address> <message> <proof> [<prehashed>=false]

Deserializes and executes the proof using a custom signature checker
whose sighash is derived from <message>. Returns true if the check
succeeds, and false otherwise. The scriptPubKey is derived directly
from <address>.**

Feedback welcome.

-Kalle.

(*) Looks like you can simply use VerifyScript with a new signature
checker class. (h/t Nicolas Dorier)
(**) If <prehashed> is true, <message> is the sighash, otherwise
sighash=sha256d(message).

[Anthony Towns via bitcoin-dev]

On 14 March 2018 5:46:55 AM GMT-04:00, Kalle Rosenbaum via bitcoin-dev <bitcoin-***@lists.linuxfoundation.org> wrote:
>Thank you.
>
>I can't really see from your proposal if you had thought of this: A
>soft
>fork can make old nodes accept invalid message signatures as valid. For
>example, a "signer" can use a witness version unknown to the verifier
>to
>fool the verifier. Witness version is detectable (just reject unknown
>witness versions) but there may be more subtle changes. Segwit was not
>"detectable" in that way, for example.
>
>This is the reason why I withdrew BIP120. If you have thought about the
>above, I'd be very interested.
>
>/Kalle
>
>Sent from my Sinclair ZX81
>
>Den 14 mars 2018 16:10 skrev "Karl Johan Alm via bitcoin-dev" <
>bitcoin-***@lists.linuxfoundation.org>:
>
>Hello,
>
>I am considering writing a replacement for the message signing tools
>that are currently broken for all but the legacy 1xx addresses. The
>approach (suggested by Pieter Wuille) is to do a script based
>approach. This does not seem to require a lot of effort for
>implementing in Bitcoin Core*. Below is my proposal for this system:
>
>A new structure SignatureProof is added, which is a simple scriptSig &
>witnessProgram container that can be serialized. This is passed out
>from/into the signer/verifier.
>
>RPC commands:
>
>sign <address> <message> [<prehashed>=false]
>
>Generates a signature proof for <message> using the same method that
>would be used to spend coins sent to <address>.**
>
>verify <address> <message> <proof> [<prehashed>=false]
>
>Deserializes and executes the proof using a custom signature checker
>whose sighash is derived from <message>. Returns true if the check
>succeeds, and false otherwise. The scriptPubKey is derived directly
>from <address>.**
>
>Feedback welcome.
>
>-Kalle.
>
>(*) Looks like you can simply use VerifyScript with a new signature
>checker class. (h/t Nicolas Dorier)
>(**) If <prehashed> is true, <message> is the sighash, otherwise
>sighash=sha256d(message).
>_______________________________________________
>bitcoin-dev mailing list
>bitcoin-***@lists.linuxfoundation.org
>https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

Wouldn't it be sufficient for old nodes to check for standardness of the spending script and report non-standard scripts as either invalid outright, or at least highly questionable? That should prevent confusion as long as soft forks are only making nonstandard behaviours invalid.

Cheers,
aj

--
Sent from my phone.

[Karl Johan Alm via bitcoin-dev]

On Wed, Mar 14, 2018 at 5:46 AM, Kalle Rosenbaum <***@rosenbaum.se> wrote:
> I can't really see from your proposal if you had thought of this: A soft
> fork can make old nodes accept invalid message signatures as valid. For
> example, a "signer" can use a witness version unknown to the verifier to
> fool the verifier. Witness version is detectable (just reject unknown
> witness versions) but there may be more subtle changes. Segwit was not
> "detectable" in that way, for example.
>
> This is the reason why I withdrew BIP120. If you have thought about the
> above, I'd be very interested.

I'm not sure I see the problem. The scriptPubKey is derived directly
from the address in all cases, which means the unknown witness version
would have to be committed to in the address itself.

So yeah, I can make a P2SH address with a witness version > 0 and a to
me unknown pubkey and then fool you into thinking I own it, but I
don't really see why you'd ultimately care. In other words, if I can
SPEND funds sent to that address today, I can prove that I can spend
today, which is the purpose of the tool, I think.

For the case where the witness version HAS been upgraded, the above
still applies, but I'm not sure it's a big issue. And it doesn't
really require an old node. I just need to set witness version >
current witness version and the problem applies to all nodes.

On Wed, Mar 14, 2018 at 8:36 AM, Luke Dashjr <***@dashjr.org> wrote:
> I don't see a need for a new RPC interface, just a new signature format.

All right.

> Ideally, it should support not only just "proof I receive at this address",
> but also "proof of funds" (as a separate feature) since this is a popular
> misuse of the current message signing (which doesn't actually prove funds at
> all). To do this, it needs to be capable of signing for multiple inputs.

I assume by inputs you mean addresses/keys. The address field could
optionally be an array. That'd be enough?

> Preferably, it should also avoid disclosing the public key for existing or
> future UTXOs. But I don't think it's possible to avoid this without something
> MAST-like first. Perhaps it can be a MAST upgrade later on, but the new
> signature scheme should probably be designed with it in mind.

I'd love to not have to reveal the public key, but I'm not sure how it
would be done, even with MAST.

On Wed, Mar 14, 2018 at 12:12 PM, Anthony Towns <***@erisian.com.au> wrote:
> Wouldn't it be sufficient for old nodes to check for standardness of the spending script and report non-standard scripts as either invalid outright, or at least highly questionable? That should prevent confusion as long as soft forks are only making nonstandard behaviours invalid.

That seems sensible to me. A warning would probably be useful, in case
the verifier is running old software.

-Kalle.

[Karl Johan Alm via bitcoin-dev]

On Thu, Mar 15, 2018 at 6:43 AM, Jim Posen <***@gmail.com> wrote:
> How are scripts with OP_CLTV and OP_CSV handled by verifiers? Do they always
> succeed? Or should an nLockTime and nSequence also be included in the proof
> in a way that can be parsed out and displayed to verifiers?

Good question.. Since you don't really have the input(s), I think it's
fine to always assume sufficient time/height on CLTV/CSV checks.

> I assume any signatures in the scriptSig/witness data would have no sighash
> type?

I think it would just use the default (SIGHASH_ALL?) for simplicity.
Is there a good reason to tweak it?

-Kalle.

[Jim Posen via bitcoin-dev]

>
> Good question.. Since you don't really have the input(s), I think it's
> fine to always assume sufficient time/height on CLTV/CSV checks.
>

In this general signing-a-script context, I think a verifier might want to
see the time conditions under which it may be spent. The proof container
could include an optional nLockTime which defaults to 0 and nSequence which
defaults to 0xFFFF...


> I think it would just use the default (SIGHASH_ALL?) for simplicity.
> Is there a good reason to tweak it?
>

I took another look and there should definitely be a byte appended to the
end of the sig so that the encoding checks pass, but I think it might as
well be a 0x00 byte since it's not actually a sighash flag.

[Pieter Wuille via bitcoin-dev]

Hello,

Thanks for starting a discussion about this idea.

A few comments inline:

On Wed, Mar 14, 2018 at 1:09 AM, Karl Johan Alm via bitcoin-dev <
bitcoin-***@lists.linuxfoundation.org> wrote:

> Hello,
>
> I am considering writing a replacement for the message signing tools
> that are currently broken for all but the legacy 1xx addresses. The
> approach (suggested by Pieter Wuille) is to do a script based
> approach. This does not seem to require a lot of effort for
> implementing in Bitcoin Core*. Below is my proposal for this system:
>
> A new structure SignatureProof is added, which is a simple scriptSig &
> witnessProgram container that can be serialized. This is passed out
> from/into the signer/verifier.
>

You need a bit more logic to deal with softforks and compatibility. The
question is which script validation flags you verify with:
* If you make them fixed, it means signatures can't evolve with new address
types being introduced that rely on new features.
* If you make it just consensus flags (following mainnet), it means that
people with old software will see future invalid signatures as always
valid; this is probably not acceptable.
* If you make it standardness flags, you will get future valid signatures
that fail to verify.

One solution is to include a version number in the signature, which
explicitly corresponds to a set of validation flags. When the version
number is something a verifier doesn't know, it can be reported as
inconclusive (it's relying on features you don't know about).

An solution is to verify twice; once with all consensus rules you know
about, and once with standardness rules. If they're both valid, the
signature is valid. If they're both invalid, the signature is invalid. If
they're different (consensus valid but standardness invalid), you report
the signature validation as inconclusive (it's relying on features you
don't know about). This approach works as long as new features only use
previous standardness-invalid scripts, but perhaps a version number is
still needed to indicate the standardness flags.

RPC commands:
>
> sign <address> <message> [<prehashed>=false]
>

Why not extend the existing signmessage/verifymessage RPC? For legacy
addresses it can fall back to the existing signature algorithm, while using
the script-based approach for all others.


>
> Generates a signature proof for <message> using the same method that
> would be used to spend coins sent to <address>.**
>
> verify <address> <message> <proof> [<prehashed>=false]
>
> Deserializes and executes the proof using a custom signature checker
> whose sighash is derived from <message>. Returns true if the check
> succeeds, and false otherwise. The scriptPubKey is derived directly
> from <address>.**
>
> Feedback welcome.
>
> -Kalle.
>
>
(**) If <prehashed> is true, <message> is the sighash, otherwise
> sighash=sha256d(message).
>

That's very dangerous I'm afraid. It could be used to trick someone into
signing off on an actual transaction, if you get them to sign a "random
looking" prehashed message. Even if you have a prehashed message, there is
no problem with treating it as hex input to a second hashing step, so I
think the prehashed option isn't needed. It's why the existing message
signing functionality always forcibly prefixes "Bitcoin signed message:",
to avoid signing something that unintentionally corresponds to a message
intended for another goal.

Cheers,

--
Pieter

[Karl Johan Alm via bitcoin-dev]

Pieter,

Thanks for the feedback. Comments below:

On Mon, Mar 26, 2018 at 5:53 PM, Pieter Wuille <***@gmail.com> wrote:
> One solution is to include a version number in the signature, which
> explicitly corresponds to a set of validation flags. When the version number
> is something a verifier doesn't know, it can be reported as inconclusive
> (it's relying on features you don't know about).
>
> An solution is to verify twice; once with all consensus rules you know
> about, and once with standardness rules. If they're both valid, the
> signature is valid. If they're both invalid, the signature is invalid. If
> they're different (consensus valid but standardness invalid), you report the
> signature validation as inconclusive (it's relying on features you don't
> know about). This approach works as long as new features only use previous
> standardness-invalid scripts, but perhaps a version number is still needed
> to indicate the standardness flags.

I think the double verify approach seems promising. I assume old nodes
consider new consensus rule enforcing transactions as non-standard but
valid. If this is always the case, it may be an idea to simply fail
verification with a message indicating the node is unable to verify
due to unknown consensus rules.

>> RPC commands:
>>
>> sign <address> <message> [<prehashed>=false]
>
> Why not extend the existing signmessage/verifymessage RPC? For legacy
> addresses it can fall back to the existing signature algorithm, while using
> the script-based approach for all others.

Yes, I initially thought it would be better to not use it as the
legacy behavior could be depended on god knows where, but I think
adding a legacy mode or simply doing the old way for 1xx is
sufficient.

>> (**) If <prehashed> is true, <message> is the sighash, otherwise
>> sighash=sha256d(message).
>
>
> That's very dangerous I'm afraid. It could be used to trick someone into
> signing off on an actual transaction, if you get them to sign a "random
> looking" prehashed message. Even if you have a prehashed message, there is
> no problem with treating it as hex input to a second hashing step, so I
> think the prehashed option isn't needed. It's why the existing message
> signing functionality always forcibly prefixes "Bitcoin signed message:", to
> avoid signing something that unintentionally corresponds to a message
> intended for another goal.

Eek.. good point...