Discussion:
proposal: extend WIF format for segwit
(too old to reply)
Thomas Voegtlin via bitcoin-dev
2017-09-15 08:55:37 UTC
Permalink
The Wallet Import Format (WIF) currently appends a 0x01 byte after the
raw private key, when that key needs to be used in conjunction with a
compressed public key. This allows wallets to associate a single Bitcoin
address to a WIF key.

It would be useful to extend the semantics of that byte, to signal for
segwit scripts, because these scripts result in different addresses.
That way, a WIF private key can still be associated to a single Bitcoin
address.

What WIF currently does is:

Nothing -> uncompressed pubkey
0x01 -> compressed pubkeys, non-segwit (can be used in P2PKH or P2SH)

We could extend it as follows:

0x02 -> segwit script embedded in P2SH (P2WPKH or P2WSH)
0x03 -> native segwit script (P2WKH or P2WSH)


Note 1: This is similar to my {x,y,z}{pub,prv} proposal for bip32
extended keys. (see other thread)

Note 2: It is probably not useful to use distinct bytes for P2WKH and
P2WSH, because the P2SH script is not known anyway. We did not do it for
non-segwit addresses, I guess we should keep it the way it is.

Note 3: we could also use a bech32 format for the private key, if it is
going to be used with a bech32 address. I am not sure if such a format
has been proposed already.

Note 4: my proposal will not result in a user visible change at the
beginning of the string, like we have for compressed/uncompressed. This
could be improved.
Pieter Wuille via bitcoin-dev
2017-09-17 02:29:41 UTC
Permalink
On Sep 15, 2017 01:56, "Thomas Voegtlin via bitcoin-dev" <
bitcoin-***@lists.linuxfoundation.org> wrote:

Note 3: we could also use a bech32 format for the private key, if it is
going to be used with a bech32 address. I am not sure if such a format
has been proposed already.


I've been working on an "extended bech32" format with 12 character checksum
rather than 6, for private keys and other things that need stronger
protection. It would guarantee correcting 4 errors, where normal bech32 can
only detect (but not correct) 4.

The rationale is that in the case of an address, if an error is detected,
you can ask the receiver for a corrected version. As that option doesn't
exist for private keys you want something stronger.

This has been a low-priority thing for me, though, and the computation work
to find a good checksum is significant.

Cheers,
--
Pieter
Thomas Voegtlin via bitcoin-dev
2017-09-17 08:10:17 UTC
Permalink
Post by Pieter Wuille via bitcoin-dev
This has been a low-priority thing for me, though, and the computation work
to find a good checksum is significant.
Thanks for the info. I guess this means that a bech32 format for private
keys is not going to happen soon. Even if such a format was available,
the issue would remain for segwit-in-p2sh addresses, which use base58.

The ambiguity of the WIF format is currently holding me from releasing a
segwit-capable version of Electrum. I believe it is not acceptable to
use the current WIF format with segwit scripts; that would just create
technological debt, forcing wallets to try all possible scripts. There
is a good reason why WIF adds a 0x01 byte for compressed pubkeys; it
makes it unambiguous.

I see only two options:
1. Disable private keys export in Electrum Segwit wallets, until a
common WIF extension has been agreed on.
2. Define my own WIF extension for Electrum, and go ahead with it.

Defining my own format does make sense for the xpub/xprv format, because
Electrum users need to share master public keys across Electrum wallets.
It makes much less sense for WIF, though, because WIF is mostly used to
import/sweep keys from other wallets.

I would love to know what other wallet developers are going to do,
especially Core. Are you going to export private keys used in segwit
scripts in the current WIF format?
AJ West via bitcoin-dev
2017-09-17 14:42:52 UTC
Permalink
Hi I have a small interjection about the point on error correction (excuse
me if it seems elementary). Isn't there an argument to be made where a
wallet software should never attempt to figure out the 'correct' address,
or in this case private key? I don't think it's crazy to suggest somebody
could import a slightly erroneous WIF, the software gracefully
error-corrects any problem, but then the user copies that error onward such
as in their backup processes like a paper wallet. I always hate to advocate
against a feature, I'm just worried too much error correcting removes the
burden of exactitude and attention of the user (eg. "I know I can have up
to 4 errors").

I'm pretty sure I read those arguments somewhere in a documentation or
issue tracker/forum post. Maybe I'm misunderstanding the bigger picture in
this particular case, but I was just reminded of that concept (even if it
only applies generally).

Thanks,
AJ West

On Sun, Sep 17, 2017 at 4:10 AM, Thomas Voegtlin via bitcoin-dev <
Post by Thomas Voegtlin via bitcoin-dev
Post by Pieter Wuille via bitcoin-dev
This has been a low-priority thing for me, though, and the computation
work
Post by Pieter Wuille via bitcoin-dev
to find a good checksum is significant.
Thanks for the info. I guess this means that a bech32 format for private
keys is not going to happen soon. Even if such a format was available,
the issue would remain for segwit-in-p2sh addresses, which use base58.
The ambiguity of the WIF format is currently holding me from releasing a
segwit-capable version of Electrum. I believe it is not acceptable to
use the current WIF format with segwit scripts; that would just create
technological debt, forcing wallets to try all possible scripts. There
is a good reason why WIF adds a 0x01 byte for compressed pubkeys; it
makes it unambiguous.
1. Disable private keys export in Electrum Segwit wallets, until a
common WIF extension has been agreed on.
2. Define my own WIF extension for Electrum, and go ahead with it.
Defining my own format does make sense for the xpub/xprv format, because
Electrum users need to share master public keys across Electrum wallets.
It makes much less sense for WIF, though, because WIF is mostly used to
import/sweep keys from other wallets.
I would love to know what other wallet developers are going to do,
especially Core. Are you going to export private keys used in segwit
scripts in the current WIF format?
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Mark Friedenbach via bitcoin-dev
2017-09-17 15:36:48 UTC
Permalink
Bech32 and WIF payload format are mostly orthogonal issues. You can design a new wallet import format now and later switch it to Bech32.
Hi I have a small interjection about the point on error correction (excuse me if it seems elementary). Isn't there an argument to be made where a wallet software should never attempt to figure out the 'correct' address, or in this case private key? I don't think it's crazy to suggest somebody could import a slightly erroneous WIF, the software gracefully error-corrects any problem, but then the user copies that error onward such as in their backup processes like a paper wallet. I always hate to advocate against a feature, I'm just worried too much error correcting removes the burden of exactitude and attention of the user (eg. "I know I can have up to 4 errors").
I'm pretty sure I read those arguments somewhere in a documentation or issue tracker/forum post. Maybe I'm misunderstanding the bigger picture in this particular case, but I was just reminded of that concept (even if it only applies generally).
Thanks,
AJ West
Post by Thomas Voegtlin via bitcoin-dev
Post by Pieter Wuille via bitcoin-dev
This has been a low-priority thing for me, though, and the computation work
to find a good checksum is significant.
Thanks for the info. I guess this means that a bech32 format for private
keys is not going to happen soon. Even if such a format was available,
the issue would remain for segwit-in-p2sh addresses, which use base58.
The ambiguity of the WIF format is currently holding me from releasing a
segwit-capable version of Electrum. I believe it is not acceptable to
use the current WIF format with segwit scripts; that would just create
technological debt, forcing wallets to try all possible scripts. There
is a good reason why WIF adds a 0x01 byte for compressed pubkeys; it
makes it unambiguous.
1. Disable private keys export in Electrum Segwit wallets, until a
common WIF extension has been agreed on.
2. Define my own WIF extension for Electrum, and go ahead with it.
Defining my own format does make sense for the xpub/xprv format, because
Electrum users need to share master public keys across Electrum wallets.
It makes much less sense for WIF, though, because WIF is mostly used to
import/sweep keys from other wallets.
I would love to know what other wallet developers are going to do,
especially Core. Are you going to export private keys used in segwit
scripts in the current WIF format?
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Karl Johan Alm via bitcoin-dev
2018-04-04 06:06:19 UTC
Permalink
I took the liberty of turning this into a BIP proposal -- the
formatted version can be seen here:
https://github.com/kallewoof/bips/blob/bip-typed-wif/bip-extended-privkey.mediawiki

<pre>
BIP: XXX
Layer: Applications
Title: Typed Private Keys
Author: Karl-Johan Alm <karljohan-***@garage.co.jp>
Comments-Summary: No comments yet.
Comments-URI: https://github.com/bitcoin/bips/wiki/Comments:BIP-XXX
Status: Draft
Type: Standards Track
Created: 2018-04-04
License: CC0-1.0
</pre>

== Abstract ==

An extension to the private key (WIF) format to specify what kind of
public key the private key corresponds to.

== Motivation ==

There are several types of public keys which can all be associated
with a given private key: P2PKH (legacy <code>1...</code> format),
P2SH-P2WPKH (SegWit public key inside P2SH), P2WPKH (bech32), etc.

While private keys have a 1-byte suffix indicating whether the
corresponding public key is compressed (<code>0x01</code>) or not
(<code>0x00</code>), there is no way of knowing what kind of public
keys were associated with the private key. As a result, when importing
a private key, the wallet has to assume all kinds, and keep track of
each possible alternative.

By extending the suffix, we can specify what kind of public key was
associated with the given private key.

== Specification ==

Currently, private keys are stored as a uint256 (private key data)
followed by a uint8 (compressed flag). The latter is extended to
specify the public key types:

{|class="wikitable" style="text-align: center;"
|-
!Value
!Type
!Compr
!Clarification
|-
|<code>0x00</code>||P2PKH_UNCOMPRESSED||No||Uncompressed legacy public
key. Unknown public key format
|-
|<code>0x01</code>||P2PKH_COMPRESSED||Yes||Compressed legacy public
key. Unknown public key format
|-
|<code>0x80</code>||P2PKH||Yes||Compressed legacy public key. Legacy
public key format (<code>1...</code>)
|-
|<code>0x81</code>||P2WPKH||Yes||Bech32 format (native Segwit)
|-
|<code>0x82</code>||P2WPKH_P2SH||Yes||Segwit nested in BIP16 P2SH
(<code>3...</code>)
|-
|<code>0x85</code>||P2SH|| &mdash; ||Non-Segwit BIP16 P2SH (<code>3...</code>)
|-
|<code>0x86</code>||P2WSH|| &mdash; ||Native Segwit P2SH
|-
|<code>0x87</code>||P2WSH_P2SH|| &mdash; ||Native Segwit P2SH nested
in BIP16 P2SH
|}

When a wallet imports a private key, it will have two outcomes:

* the key is using one of the legacy types, in which case all types
must be accounted for
* the key is using one of the extended types, in which case the wallet
need only track the specific corresponding public key

== Rationale ==

TODO

== Compatibility ==

This proposal is not backwards compatible, in that software that does
not recognize the new types will not understand the compressed flag.
It would be trivial to change this, by keeping the 'uncompressed'
state as it is (0) and changing 'compressed' to be 'anything not 0',
as opposed to 'the value 1'.

The proposal *is* backwards compatible in that new wallet software
will always understand the old WIF format, however. It will, as it
does today, assume that any kind of public key is possible, and will
have to track all of them, as it has to today.

== Acknowledgements ==

This BIP is based on the initial proposal by Thomas Voegtlin
<***@electrum.org> on the Bitcoin Dev mailing
list<ref>https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-September/015007.html</ref>
and the Electrum 3.0
implementation<ref>https://github.com/spesmilo/electrum/blob/82e88cb89df35288b80dfdbe071da74247351251/RELEASE-NOTES#L95-L108</ref>

== Reference implementation ==

There is a partial implementation which adds, but does not use, the
types described in this BIP here:
https://github.com/bitcoin/bitcoin/pull/12869

== References ==

<references/>

== Copyright ==

This document is licensed under the Creative Commons CC0 1.0 Universal license.

On Mon, Sep 18, 2017 at 12:36 AM, Mark Friedenbach via bitcoin-dev
Post by Mark Friedenbach via bitcoin-dev
Bech32 and WIF payload format are mostly orthogonal issues. You can design a
new wallet import format now and later switch it to Bech32.
On Sep 17, 2017, at 7:42 AM, AJ West via bitcoin-dev
Hi I have a small interjection about the point on error correction (excuse
me if it seems elementary). Isn't there an argument to be made where a
wallet software should never attempt to figure out the 'correct' address, or
in this case private key? I don't think it's crazy to suggest somebody could
import a slightly erroneous WIF, the software gracefully error-corrects any
problem, but then the user copies that error onward such as in their backup
processes like a paper wallet. I always hate to advocate against a feature,
I'm just worried too much error correcting removes the burden of exactitude
and attention of the user (eg. "I know I can have up to 4 errors").
I'm pretty sure I read those arguments somewhere in a documentation or issue
tracker/forum post. Maybe I'm misunderstanding the bigger picture in this
particular case, but I was just reminded of that concept (even if it only
applies generally).
Thanks,
AJ West
On Sun, Sep 17, 2017 at 4:10 AM, Thomas Voegtlin via bitcoin-dev
Post by Thomas Voegtlin via bitcoin-dev
Post by Pieter Wuille via bitcoin-dev
This has been a low-priority thing for me, though, and the computation work
to find a good checksum is significant.
Thanks for the info. I guess this means that a bech32 format for private
keys is not going to happen soon. Even if such a format was available,
the issue would remain for segwit-in-p2sh addresses, which use base58.
The ambiguity of the WIF format is currently holding me from releasing a
segwit-capable version of Electrum. I believe it is not acceptable to
use the current WIF format with segwit scripts; that would just create
technological debt, forcing wallets to try all possible scripts. There
is a good reason why WIF adds a 0x01 byte for compressed pubkeys; it
makes it unambiguous.
1. Disable private keys export in Electrum Segwit wallets, until a
common WIF extension has been agreed on.
2. Define my own WIF extension for Electrum, and go ahead with it.
Defining my own format does make sense for the xpub/xprv format, because
Electrum users need to share master public keys across Electrum wallets.
It makes much less sense for WIF, though, because WIF is mostly used to
import/sweep keys from other wallets.
I would love to know what other wallet developers are going to do,
especially Core. Are you going to export private keys used in segwit
scripts in the current WIF format?
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Karl-Johan Alm via bitcoin-dev
2018-04-10 02:54:08 UTC
Permalink
Hello,

I made slight modification to the BIP, dropping the 0x80 jump to 0x10:
https://github.com/kallewoof/bips/blob/bip-typed-wif/bip-extended-privkey.mediawiki

I will make the corresponding changes to the reference implementation shortly.

If there are no objections I would also like to request a BIP number.


On Wed, Apr 4, 2018 at 3:06 PM, Karl Johan Alm
Post by Karl Johan Alm via bitcoin-dev
I took the liberty of turning this into a BIP proposal -- the
https://github.com/kallewoof/bips/blob/bip-typed-wif/bip-extended-privkey.mediawiki
<pre>
BIP: XXX
Layer: Applications
Title: Typed Private Keys
Comments-Summary: No comments yet.
Comments-URI: https://github.com/bitcoin/bips/wiki/Comments:BIP-XXX
Status: Draft
Type: Standards Track
Created: 2018-04-04
License: CC0-1.0
</pre>
== Abstract ==
An extension to the private key (WIF) format to specify what kind of
public key the private key corresponds to.
== Motivation ==
There are several types of public keys which can all be associated
with a given private key: P2PKH (legacy <code>1...</code> format),
P2SH-P2WPKH (SegWit public key inside P2SH), P2WPKH (bech32), etc.
While private keys have a 1-byte suffix indicating whether the
corresponding public key is compressed (<code>0x01</code>) or not
(<code>0x00</code>), there is no way of knowing what kind of public
keys were associated with the private key. As a result, when importing
a private key, the wallet has to assume all kinds, and keep track of
each possible alternative.
By extending the suffix, we can specify what kind of public key was
associated with the given private key.
== Specification ==
Currently, private keys are stored as a uint256 (private key data)
followed by a uint8 (compressed flag). The latter is extended to
{|class="wikitable" style="text-align: center;"
|-
!Value
!Type
!Compr
!Clarification
|-
|<code>0x00</code>||P2PKH_UNCOMPRESSED||No||Uncompressed legacy public
key. Unknown public key format
|-
|<code>0x01</code>||P2PKH_COMPRESSED||Yes||Compressed legacy public
key. Unknown public key format
|-
|<code>0x80</code>||P2PKH||Yes||Compressed legacy public key. Legacy
public key format (<code>1...</code>)
|-
|<code>0x81</code>||P2WPKH||Yes||Bech32 format (native Segwit)
|-
|<code>0x82</code>||P2WPKH_P2SH||Yes||Segwit nested in BIP16 P2SH
(<code>3...</code>)
|-
|<code>0x85</code>||P2SH|| &mdash; ||Non-Segwit BIP16 P2SH (<code>3...</code>)
|-
|<code>0x86</code>||P2WSH|| &mdash; ||Native Segwit P2SH
|-
|<code>0x87</code>||P2WSH_P2SH|| &mdash; ||Native Segwit P2SH nested
in BIP16 P2SH
|}
* the key is using one of the legacy types, in which case all types
must be accounted for
* the key is using one of the extended types, in which case the wallet
need only track the specific corresponding public key
== Rationale ==
TODO
== Compatibility ==
This proposal is not backwards compatible, in that software that does
not recognize the new types will not understand the compressed flag.
It would be trivial to change this, by keeping the 'uncompressed'
state as it is (0) and changing 'compressed' to be 'anything not 0',
as opposed to 'the value 1'.
The proposal *is* backwards compatible in that new wallet software
will always understand the old WIF format, however. It will, as it
does today, assume that any kind of public key is possible, and will
have to track all of them, as it has to today.
== Acknowledgements ==
This BIP is based on the initial proposal by Thomas Voegtlin
list<ref>https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2017-September/015007.html</ref>
and the Electrum 3.0
implementation<ref>https://github.com/spesmilo/electrum/blob/82e88cb89df35288b80dfdbe071da74247351251/RELEASE-NOTES#L95-L108</ref>
== Reference implementation ==
There is a partial implementation which adds, but does not use, the
https://github.com/bitcoin/bitcoin/pull/12869
== References ==
<references/>
== Copyright ==
This document is licensed under the Creative Commons CC0 1.0 Universal license.
On Mon, Sep 18, 2017 at 12:36 AM, Mark Friedenbach via bitcoin-dev
Post by Mark Friedenbach via bitcoin-dev
Bech32 and WIF payload format are mostly orthogonal issues. You can design a
new wallet import format now and later switch it to Bech32.
On Sep 17, 2017, at 7:42 AM, AJ West via bitcoin-dev
Hi I have a small interjection about the point on error correction (excuse
me if it seems elementary). Isn't there an argument to be made where a
wallet software should never attempt to figure out the 'correct' address, or
in this case private key? I don't think it's crazy to suggest somebody could
import a slightly erroneous WIF, the software gracefully error-corrects any
problem, but then the user copies that error onward such as in their backup
processes like a paper wallet. I always hate to advocate against a feature,
I'm just worried too much error correcting removes the burden of exactitude
and attention of the user (eg. "I know I can have up to 4 errors").
I'm pretty sure I read those arguments somewhere in a documentation or issue
tracker/forum post. Maybe I'm misunderstanding the bigger picture in this
particular case, but I was just reminded of that concept (even if it only
applies generally).
Thanks,
AJ West
On Sun, Sep 17, 2017 at 4:10 AM, Thomas Voegtlin via bitcoin-dev
Post by Thomas Voegtlin via bitcoin-dev
Post by Pieter Wuille via bitcoin-dev
This has been a low-priority thing for me, though, and the computation work
to find a good checksum is significant.
Thanks for the info. I guess this means that a bech32 format for private
keys is not going to happen soon. Even if such a format was available,
the issue would remain for segwit-in-p2sh addresses, which use base58.
The ambiguity of the WIF format is currently holding me from releasing a
segwit-capable version of Electrum. I believe it is not acceptable to
use the current WIF format with segwit scripts; that would just create
technological debt, forcing wallets to try all possible scripts. There
is a good reason why WIF adds a 0x01 byte for compressed pubkeys; it
makes it unambiguous.
1. Disable private keys export in Electrum Segwit wallets, until a
common WIF extension has been agreed on.
2. Define my own WIF extension for Electrum, and go ahead with it.
Defining my own format does make sense for the xpub/xprv format, because
Electrum users need to share master public keys across Electrum wallets.
It makes much less sense for WIF, though, because WIF is mostly used to
import/sweep keys from other wallets.
I would love to know what other wallet developers are going to do,
especially Core. Are you going to export private keys used in segwit
scripts in the current WIF format?
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Loading...