Discussion:
[bitcoin-dev] Committed bloom filters for improved wallet performance and SPV security
bfd--- via bitcoin-dev
2016-05-09 08:26:06 UTC
Permalink
We introduce several concepts that rework the lightweight Bitcoin
client model in a manner which is secure, efficient and privacy
compatible.

Thea properties of BIP37 SPV [0] are unfortunately not as strong as
originally thought:

* The expected privacy of the probabilistic nature of bloom
filters does not exist [1][2], any user with a BIP37 SPV wallet
should be operating under no expectation of privacy.
Implementation flaws make this effect significantly worse, the
behavior meaning that no matter how high the false positive
rate (up to simply downloading the whole blocks verbatim) the
intent of the client connection is recoverable.

* Significant processing load is placed on nodes in the Bitcoin
network by lightweight clients, a single syncing wallet causes
(at the time of writing) 80GB of disk reads and a large amount
of CPU time to be consumed processing this data. This carries
significant denial of service risk [3], non-distinguishable
clients can repeatedly request taxing blocks causing
reprocessing on every request. Processed data is unique to every
client, and can not be cached or made more efficient while
staying within specification.

* Wallet clients can not have strong consistency or security
expectations, BIP37 merkle paths allow for a wallet to validate
that an output was spendable at some point in time but does not
prove that this output is not spent today.

* Nodes in the network can denial of service attack all BIP37 SPV
wallet clients by simply returning null filter results for
requests, the wallet has no way of discerning if it has been
lied to and may be made simply unaware that any payment has been
made to them. Many nodes can be queried in a probabilistic manor
but this increases the already heavy network load with little
benefit.



We propose a new concept which can work towards addressing these
shortcomings.


A Bloom Filter Digest is deterministically created of every block
encompassing the inputs and outputs of the containing transactions,
the filter parameters being tuned such that the filter is a small
portion of the size of the total block data. To determine if a block
has contents which may be interesting a second bloom filter of all
relevant key material is created. A binary comparison between the two
filters returns true if there is probably matching transactions, and
false if there is certainly no matching transactions. Any matched
blocks can be downloaded in full and processed for transactions which
may be relevant.

The BFD can be used verbatim in replacement of BIP37, where the filter
can be cached between clients without needing to be recomputed. It can
also be used by normal pruned nodes to do re-scans locally of their
wallet without needing to have the block data available to scan, or
without reading the entire block chain from disk.

-

For improved probabilistic security the bloom filters can be presented
to lightweight clients by semi-trusted oracles. A client wallet makes
an assumption that they trust a set, or subset of remote parties
(wallet vendors, services) which all all sign the BFD for each block.
The BFD can be downloaded from a single remote source, and the hash of
the filters compared against others in the trust set. Agreement is a
weak suggestion that the filter has not been tampered with, assuming
that these parties are not conspiring to defraud the client.

The oracles do not learn any additional information about the client
wallet, the client can download the block data from either nodes on
the network, HTTP services, NTTP, or any other out of band
communication method that provides the privacy desired by the client.

-

The security model of the oracle bloom filter can be vastly improved
by instead committing a hash of the BFD inside every block as a soft-
fork consensus rule change. After this, every node in the network would
build the filter and validate that the hash in the block is correct,
then make a conscious choice discard it for space savings or cache the
data to disk.

With a commitment to the filter it becomes impossible to lie to
lightweight clients by omission. Lightweight clients are provided with
a block header, merkle path, and the BFD. Altering the BFD invalidates
the merkle proof, it's validity is a strong indicator that the client
has an unadulterated picture of the UTXO condition without needing to
build one itself. A strong assurance that the hash of the BFD means
that the filters can be downloaded out of band along with the block
data at the leisure of the client, allowing for significantly greater
privacy and taking load away from the P2P Bitcoin network.

Committing the BFD is not a hard forking change, and does not require
alterations to mining software so long as the coinbase transaction
scriptSig is not included in the bloom filter.


[0] https://github.com/bitcoin/bips/blob/master/bip-0037.mediawiki
[1] https://eprint.iacr.org/2014/763.pdf
[2] https://jonasnick.github.io/blog/2015/02/12/privacy-in-bitcoinj/
[3] https://github.com/petertodd/bloom-io-attack
Gregory Maxwell via bitcoin-dev
2016-05-09 08:57:08 UTC
Permalink
On Mon, May 9, 2016 at 8:26 AM, bfd--- via bitcoin-dev
Post by bfd--- via bitcoin-dev
We introduce several concepts that rework the lightweight Bitcoin
client model in a manner which is secure, efficient and privacy
compatible.
[...]
Post by bfd--- via bitcoin-dev
A Bloom Filter Digest is deterministically created of every block
I think this is a fantastic idea.

Some napkin work shows that it has pretty good communications
bandwidth so long as you assume that the wallet has many keys (e.g.
more than the number of the outputs in the block)-- otherwise BIP37
uses less bandwidth, but you note its terrible privacy problems.

You should be aware that when the filter is transmitted but not
updated, as it is in these filtering applications, the bloom filter is
not the most communication efficient data structure.

The most efficient data structure is similar to a bloom filter, but
you use more bits and only one hash function. The result will be
mostly zero bits. Then you entropy code it using RLE+Rice coding or an
optimal binomial packer (e.g.
https://people.xiph.org/~greg/binomial_codec.c). This is about 45%
more space efficient than a bloom filter. ... it's just a PITA to
update, though that is inapplicable here. Entropy coding for this can
be quite fast, if many lookups are done the decompression could even
be faster than having to use two dozen hash functions for each lookup.

The intuition is that this kind of simple hash-bitmap is great, but
space inefficient if you don't have compression since most of the bits
are 0 you end up spending a bit to send less than a bit of
information. A bloom filter improve the situation by using the
multiple filters to increase the ones density to 50%, but the
increased collisions create overhead. This is important when its a
in-memory data-structure that you're updating often, but not here.

One thing to do with matching blocks is after finding the matches the
node could potentially consult some PIR to get the blocks it cares
about... thus preventing a leak of which blocks it was interested in,
but not taking PIR costs for the whole chain or requiring the
implementation of PIR tree search (which is theoretically simple but
in practice hard to implement).
Bob McElrath via bitcoin-dev
2016-05-11 20:06:48 UTC
Permalink
I like this idea, but let's run some numbers...
Post by bfd--- via bitcoin-dev
A Bloom Filter Digest is deterministically created of every block
Bloom filters completely obfuscate the required size of the filter for a desired
false-positive rate. But, an optimal filter is linear in the number of elements
it contains for fixed false-positive rate, and logarithmic in the false-positive
rate. (This comment applies to a RLL encoded Bloom filter Greg mentioned, but
that's not the only way) That is for N elements and false positive rate
\epsilon:

filter size = - N \log_2 \epsilon

Given that the data that would be put into this particular filter is *already*
hashed, it makes more sense and is faster to use a Cuckoo[1] filter, choosing a
fixed false-positive rate, given expected wallet sizes. For Bloom filters,
multiply the above formula by 1.44.

To prevent light clients from downloading more blocks than necessary, the
false-positive rate should be roughly less than 1/(block height). If we take
the false positive rate to be 1e-6 for today's block height ~ 410000, this is
about 20 bits per element. So for todays block's, this is a 30kb filter, for a
3% increase in block size, if blocks commit to the filter. Thus the required
size of the filter commitment is roughly:

filter size = N \log_2 H

where H is the block height. If bitcoin had these filters from the beginning, a
light client today would have to download about 12MB of data in filters. My
personal SPV wallet is using 31MB currently. It's not clear this is a bandwidth
win, though it's definitely a win for computing load on full nodes.


[1] https://www.cs.cmu.edu/~dga/papers/cuckoo-conext2014.pdf

--
Cheers, Bob McElrath

"For every complex problem, there is a solution that is simple, neat, and wrong."
-- H. L. Mencken
Bob McElrath via bitcoin-dev
2016-05-11 20:29:33 UTC
Permalink
Eerrrr....let me revise that last paragraph. That's 12 *GB* of filters at
today's block height (at fixed false-positive rate 1e-6. Compared to block
headers only which are about 33 MB today. So this proposal is not really
compatible with such a wallet being "light"...

Damn units...
Post by Bob McElrath via bitcoin-dev
I like this idea, but let's run some numbers...
Post by bfd--- via bitcoin-dev
A Bloom Filter Digest is deterministically created of every block
Bloom filters completely obfuscate the required size of the filter for a desired
false-positive rate. But, an optimal filter is linear in the number of elements
it contains for fixed false-positive rate, and logarithmic in the false-positive
rate. (This comment applies to a RLL encoded Bloom filter Greg mentioned, but
that's not the only way) That is for N elements and false positive rate
filter size = - N \log_2 \epsilon
Given that the data that would be put into this particular filter is *already*
hashed, it makes more sense and is faster to use a Cuckoo[1] filter, choosing a
fixed false-positive rate, given expected wallet sizes. For Bloom filters,
multiply the above formula by 1.44.
To prevent light clients from downloading more blocks than necessary, the
false-positive rate should be roughly less than 1/(block height). If we take
the false positive rate to be 1e-6 for today's block height ~ 410000, this is
about 20 bits per element. So for todays block's, this is a 30kb filter, for a
3% increase in block size, if blocks commit to the filter. Thus the required
filter size = N \log_2 H
where H is the block height. If bitcoin had these filters from the beginning, a
light client today would have to download about 12MB of data in filters. My
personal SPV wallet is using 31MB currently. It's not clear this is a bandwidth
win, though it's definitely a win for computing load on full nodes.
[1] https://www.cs.cmu.edu/~dga/papers/cuckoo-conext2014.pdf
--
Cheers, Bob McElrath
"For every complex problem, there is a solution that is simple, neat, and wrong."
-- H. L. Mencken
!DSPAM:5733934b206851108912031!
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
!DSPAM:5733934b206851108912031!
--
Cheers, Bob McElrath

"For every complex problem, there is a solution that is simple, neat, and wrong."
-- H. L. Mencken
Leo Wandersleb via bitcoin-dev
2016-07-28 21:07:29 UTC
Permalink
gmaxwell just made me aware of this mail thread [0]. Some days ago I had
independently and naively started implementing "something similar" [1].

My version totally ignored the commitment and signing part but I'm pretty sure
that 12GB is overkill. My code is currently broken and I have no time to work on
it much but I thought it might be helpful to chime in.

At this point in time the difference between 80GB and 3GB (as my current 1.5GB
of only outputs would suggest if I had covered the inputs) or even 12GB makes
the difference of being able to store it on a phone, vs. not being able to. 80GB
"compressed" to 3GB is not that bad at all. Unfortunately, with segWit this will
be worse, with the higher transaction count per MB.

Regards,

Leo

[0]
https://www.reddit.com/r/Bitcoin/comments/4v28jl/how_have_fungiblity_problems_affected_you_in/d5ux6aq
[1] https://github.com/Giszmo/TransactionFinder
Post by Bob McElrath via bitcoin-dev
Eerrrr....let me revise that last paragraph. That's 12 *GB* of filters at
today's block height (at fixed false-positive rate 1e-6. Compared to block
headers only which are about 33 MB today. So this proposal is not really
compatible with such a wallet being "light"...
Damn units...
Post by Bob McElrath via bitcoin-dev
I like this idea, but let's run some numbers...
Post by bfd--- via bitcoin-dev
A Bloom Filter Digest is deterministically created of every block
Bloom filters completely obfuscate the required size of the filter for a desired
false-positive rate. But, an optimal filter is linear in the number of elements
it contains for fixed false-positive rate, and logarithmic in the false-positive
rate. (This comment applies to a RLL encoded Bloom filter Greg mentioned, but
that's not the only way) That is for N elements and false positive rate
filter size = - N \log_2 \epsilon
Given that the data that would be put into this particular filter is *already*
hashed, it makes more sense and is faster to use a Cuckoo[1] filter, choosing a
fixed false-positive rate, given expected wallet sizes. For Bloom filters,
multiply the above formula by 1.44.
To prevent light clients from downloading more blocks than necessary, the
false-positive rate should be roughly less than 1/(block height). If we take
the false positive rate to be 1e-6 for today's block height ~ 410000, this is
about 20 bits per element. So for todays block's, this is a 30kb filter, for a
3% increase in block size, if blocks commit to the filter. Thus the required
filter size = N \log_2 H
where H is the block height. If bitcoin had these filters from the beginning, a
light client today would have to download about 12MB of data in filters. My
personal SPV wallet is using 31MB currently. It's not clear this is a bandwidth
win, though it's definitely a win for computing load on full nodes.
[1] https://www.cs.cmu.edu/~dga/papers/cuckoo-conext2014.pdf
--
Cheers, Bob McElrath
"For every complex problem, there is a solution that is simple, neat, and wrong."
-- H. L. Mencken
!DSPAM:5733934b206851108912031!
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
!DSPAM:5733934b206851108912031!
--
Cheers, Bob McElrath
"For every complex problem, there is a solution that is simple, neat, and wrong."
-- H. L. Mencken
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Erik Aronesty via bitcoin-dev
2017-01-06 22:07:36 UTC
Permalink
- N \log_2 \epsilon * 1.44

N = 41000 blocks
epsilon = 1/41000 (fp rate)

= 904689.8bits

~ 1 MB


On Thu, Jul 28, 2016 at 5:07 PM, Leo Wandersleb via bitcoin-dev <
Post by Leo Wandersleb via bitcoin-dev
gmaxwell just made me aware of this mail thread [0]. Some days ago I had
independently and naively started implementing "something similar" [1].
My version totally ignored the commitment and signing part but I'm pretty sure
that 12GB is overkill. My code is currently broken and I have no time to work on
it much but I thought it might be helpful to chime in.
At this point in time the difference between 80GB and 3GB (as my current 1.5GB
of only outputs would suggest if I had covered the inputs) or even 12GB makes
the difference of being able to store it on a phone, vs. not being able to. 80GB
"compressed" to 3GB is not that bad at all. Unfortunately, with segWit this will
be worse, with the higher transaction count per MB.
Regards,
Leo
[0]
https://www.reddit.com/r/Bitcoin/comments/4v28jl/how_
have_fungiblity_problems_affected_you_in/d5ux6aq
[1] https://github.com/Giszmo/TransactionFinder
Post by Bob McElrath via bitcoin-dev
Eerrrr....let me revise that last paragraph. That's 12 *GB* of filters
at
Post by Bob McElrath via bitcoin-dev
today's block height (at fixed false-positive rate 1e-6. Compared to
block
Post by Bob McElrath via bitcoin-dev
headers only which are about 33 MB today. So this proposal is not really
compatible with such a wallet being "light"...
Damn units...
Post by Bob McElrath via bitcoin-dev
I like this idea, but let's run some numbers...
Post by bfd--- via bitcoin-dev
A Bloom Filter Digest is deterministically created of every block
Bloom filters completely obfuscate the required size of the filter for
a desired
Post by Bob McElrath via bitcoin-dev
Post by Bob McElrath via bitcoin-dev
false-positive rate. But, an optimal filter is linear in the number of
elements
Post by Bob McElrath via bitcoin-dev
Post by Bob McElrath via bitcoin-dev
it contains for fixed false-positive rate, and logarithmic in the
false-positive
Post by Bob McElrath via bitcoin-dev
Post by Bob McElrath via bitcoin-dev
rate. (This comment applies to a RLL encoded Bloom filter Greg
mentioned, but
Post by Bob McElrath via bitcoin-dev
Post by Bob McElrath via bitcoin-dev
that's not the only way) That is for N elements and false positive rate
filter size = - N \log_2 \epsilon
Given that the data that would be put into this particular filter is
*already*
Post by Bob McElrath via bitcoin-dev
Post by Bob McElrath via bitcoin-dev
hashed, it makes more sense and is faster to use a Cuckoo[1] filter,
choosing a
Post by Bob McElrath via bitcoin-dev
Post by Bob McElrath via bitcoin-dev
fixed false-positive rate, given expected wallet sizes. For Bloom
filters,
Post by Bob McElrath via bitcoin-dev
Post by Bob McElrath via bitcoin-dev
multiply the above formula by 1.44.
To prevent light clients from downloading more blocks than necessary,
the
Post by Bob McElrath via bitcoin-dev
Post by Bob McElrath via bitcoin-dev
false-positive rate should be roughly less than 1/(block height). If
we take
Post by Bob McElrath via bitcoin-dev
Post by Bob McElrath via bitcoin-dev
the false positive rate to be 1e-6 for today's block height ~ 410000,
this is
Post by Bob McElrath via bitcoin-dev
Post by Bob McElrath via bitcoin-dev
about 20 bits per element. So for todays block's, this is a 30kb
filter, for a
Post by Bob McElrath via bitcoin-dev
Post by Bob McElrath via bitcoin-dev
3% increase in block size, if blocks commit to the filter. Thus the
required
Post by Bob McElrath via bitcoin-dev
Post by Bob McElrath via bitcoin-dev
filter size = N \log_2 H
where H is the block height. If bitcoin had these filters from the
beginning, a
Post by Bob McElrath via bitcoin-dev
Post by Bob McElrath via bitcoin-dev
light client today would have to download about 12MB of data in
filters. My
Post by Bob McElrath via bitcoin-dev
Post by Bob McElrath via bitcoin-dev
personal SPV wallet is using 31MB currently. It's not clear this is a
bandwidth
Post by Bob McElrath via bitcoin-dev
Post by Bob McElrath via bitcoin-dev
win, though it's definitely a win for computing load on full nodes.
[1] https://www.cs.cmu.edu/~dga/papers/cuckoo-conext2014.pdf
--
Cheers, Bob McElrath
"For every complex problem, there is a solution that is simple, neat,
and wrong."
Post by Bob McElrath via bitcoin-dev
Post by Bob McElrath via bitcoin-dev
-- H. L. Mencken
!DSPAM:5733934b206851108912031!
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
!DSPAM:5733934b206851108912031!
--
Cheers, Bob McElrath
"For every complex problem, there is a solution that is simple, neat,
and wrong."
Post by Bob McElrath via bitcoin-dev
-- H. L. Mencken
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
bfd--- via bitcoin-dev
2017-01-03 20:24:35 UTC
Permalink
I believe the filter can be more compact than this, but even if not an
order of magnitude saving of disk space is still significant.
Post by Bob McElrath via bitcoin-dev
Eerrrr....let me revise that last paragraph. That's 12 *GB* of filters at
today's block height (at fixed false-positive rate 1e-6. Compared to block
headers only which are about 33 MB today. So this proposal is not really
compatible with such a wallet being "light"...
Damn units...
Post by Bob McElrath via bitcoin-dev
I like this idea, but let's run some numbers...
Post by bfd--- via bitcoin-dev
A Bloom Filter Digest is deterministically created of every block
Bloom filters completely obfuscate the required size of the filter for a desired
false-positive rate. But, an optimal filter is linear in the number of elements
it contains for fixed false-positive rate, and logarithmic in the false-positive
rate. (This comment applies to a RLL encoded Bloom filter Greg mentioned, but
that's not the only way) That is for N elements and false positive rate
filter size = - N \log_2 \epsilon
Given that the data that would be put into this particular filter is *already*
hashed, it makes more sense and is faster to use a Cuckoo[1] filter, choosing a
fixed false-positive rate, given expected wallet sizes. For Bloom filters,
multiply the above formula by 1.44.
To prevent light clients from downloading more blocks than necessary, the
false-positive rate should be roughly less than 1/(block height). If we take
the false positive rate to be 1e-6 for today's block height ~ 410000, this is
about 20 bits per element. So for todays block's, this is a 30kb filter, for a
3% increase in block size, if blocks commit to the filter. Thus the required
filter size = N \log_2 H
where H is the block height. If bitcoin had these filters from the beginning, a
light client today would have to download about 12MB of data in filters. My
personal SPV wallet is using 31MB currently. It's not clear this is a bandwidth
win, though it's definitely a win for computing load on full nodes.
[1] https://www.cs.cmu.edu/~dga/papers/cuckoo-conext2014.pdf
--
Cheers, Bob McElrath
"For every complex problem, there is a solution that is simple, neat, and wrong."
-- H. L. Mencken
!DSPAM:5733934b206851108912031!
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
!DSPAM:5733934b206851108912031!
--
Cheers, Bob McElrath
"For every complex problem, there is a solution that is simple, neat, and wrong."
-- H. L. Mencken
Jonas Schnelli via bitcoin-dev
2017-01-01 21:01:23 UTC
Permalink
Hi
Post by bfd--- via bitcoin-dev
We introduce several concepts that rework the lightweight Bitcoin
client model in a manner which is secure, efficient and privacy
compatible.
The BFD can be used verbatim in replacement of BIP37, where the filter
can be cached between clients without needing to be recomputed. It can
also be used by normal pruned nodes to do re-scans locally of their
wallet without needing to have the block data available to scan, or
without reading the entire block chain from disk.
I started exploring the potential of BFD after this specification.

What would be the preferred/recommended way to handle 0-conf/mempool
filtering – if & once BDF would have been deployed (any type,
semi-trusted oracles or protocol-level/softfork)?

From the user-experience perspective, this is probably pretty important
(otherwise the experience will be that incoming funds can take serval
minutes to hours until they appear).
Using BIP37 bloom filters just for mempool filtering would obviously
result in the same unwanted privacy-setup.

</jonas>
bfd--- via bitcoin-dev
2017-01-03 20:18:59 UTC
Permalink
The concept combined with the weak blocks system where miners commit
to potential transaction inclusion with fractional difficulty blocks
is possible. I'm not personally convinced that unconfirmed transaction
display in a wallet is worth the privacy trade-off. The user has very
little to gain from this knowledge until the txn is in a block.
Post by Jonas Schnelli via bitcoin-dev
Hi
Post by bfd--- via bitcoin-dev
We introduce several concepts that rework the lightweight Bitcoin
client model in a manner which is secure, efficient and privacy
compatible.
The BFD can be used verbatim in replacement of BIP37, where the filter
can be cached between clients without needing to be recomputed. It can
also be used by normal pruned nodes to do re-scans locally of their
wallet without needing to have the block data available to scan, or
without reading the entire block chain from disk.
I started exploring the potential of BFD after this specification.
What would be the preferred/recommended way to handle 0-conf/mempool
filtering – if & once BDF would have been deployed (any type,
semi-trusted oracles or protocol-level/softfork)?
From the user-experience perspective, this is probably pretty important
(otherwise the experience will be that incoming funds can take serval
minutes to hours until they appear).
Using BIP37 bloom filters just for mempool filtering would obviously
result in the same unwanted privacy-setup.
</jonas>
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Aaron Voisine via bitcoin-dev
2017-01-03 22:18:21 UTC
Permalink
Unconfirmed transactions are incredibly important for real world use.
Merchants for instance are willing to accept credit card payments of
thousands of dollars and ship the goods despite the fact that the
transaction can be reversed up to 60 days later. There is a very large cost
to losing the ability to have instant transactions in many or even most
situations. This cost is typically well above the fraud risk.

It's important to recognize that bitcoin serves a wide variety of use cases
with different profiles for time sensitivity and fraud risk.

Aaron

On Tue, Jan 3, 2017 at 12:41 PM bfd--- via bitcoin-dev <
Post by bfd--- via bitcoin-dev
The concept combined with the weak blocks system where miners commit
to potential transaction inclusion with fractional difficulty blocks
is possible. I'm not personally convinced that unconfirmed transaction
display in a wallet is worth the privacy trade-off. The user has very
little to gain from this knowledge until the txn is in a block.
Post by Jonas Schnelli via bitcoin-dev
Hi
Post by bfd--- via bitcoin-dev
We introduce several concepts that rework the lightweight Bitcoin
client model in a manner which is secure, efficient and privacy
compatible.
The BFD can be used verbatim in replacement of BIP37, where the filter
can be cached between clients without needing to be recomputed. It can
also be used by normal pruned nodes to do re-scans locally of their
wallet without needing to have the block data available to scan, or
without reading the entire block chain from disk.
I started exploring the potential of BFD after this specification.
What would be the preferred/recommended way to handle 0-conf/mempool
filtering – if & once BDF would have been deployed (any type,
semi-trusted oracles or protocol-level/softfork)?
From the user-experience perspective, this is probably pretty important
(otherwise the experience will be that incoming funds can take serval
minutes to hours until they appear).
Using BIP37 bloom filters just for mempool filtering would obviously
result in the same unwanted privacy-setup.
</jonas>
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
adiabat via bitcoin-dev
2017-01-03 23:06:26 UTC
Permalink
Mempool transactions have their place, but "unconfirmed" and "SPV" don't
belong together. Only a full node can tell if a transaction may get
confirmed, or is nonsense. Unfortunately all the light / SPV wallets I
know of show mempool transactions, which makes it hard to go back... (e.g.
"why doesn't your software show 0-conf! your wallet is broken!", somewhat
akin to people complaining about RBF)

So, this is easy, just don't worry about mempool filtering. Why are light
clients looking at the mempool anyway? Maybe if there were some way to
provide SPV proofs of all inputs, but that's a bit of a mess for full nodes
to do.

Without mempool filtering, I think the committed bloom filters would be a
great improvement over the current bloom filter setup, especially for
lightning network use cases (with lightning, not finding out about a
transaction can make you lose money). I want to work on it and may be able
to at some point as it's somewhat related to lightning.

Also, if you're running a light client, and storing the filters the way you
store block headers, there's really no reason to go all the way back to
height 0. You can start grabbing headers at some point a while ago, before
your set of keys was generated. I think it'd be very worth it even with
GB-scale disk usage.

-Tadge


On Tue, Jan 3, 2017 at 5:18 PM, Aaron Voisine via bitcoin-dev <
Post by Aaron Voisine via bitcoin-dev
Unconfirmed transactions are incredibly important for real world use.
Merchants for instance are willing to accept credit card payments of
thousands of dollars and ship the goods despite the fact that the
transaction can be reversed up to 60 days later. There is a very large cost
to losing the ability to have instant transactions in many or even most
situations. This cost is typically well above the fraud risk.
It's important to recognize that bitcoin serves a wide variety of use
cases with different profiles for time sensitivity and fraud risk.
Aaron
Post by bfd--- via bitcoin-dev
The concept combined with the weak blocks system where miners commit
to potential transaction inclusion with fractional difficulty blocks
is possible. I'm not personally convinced that unconfirmed transaction
display in a wallet is worth the privacy trade-off. The user has very
little to gain from this knowledge until the txn is in a block.
Post by Jonas Schnelli via bitcoin-dev
Hi
Post by bfd--- via bitcoin-dev
We introduce several concepts that rework the lightweight Bitcoin
client model in a manner which is secure, efficient and privacy
compatible.
The BFD can be used verbatim in replacement of BIP37, where the filter
can be cached between clients without needing to be recomputed. It can
also be used by normal pruned nodes to do re-scans locally of their
wallet without needing to have the block data available to scan, or
without reading the entire block chain from disk.
I started exploring the potential of BFD after this specification.
What would be the preferred/recommended way to handle 0-conf/mempool
filtering – if & once BDF would have been deployed (any type,
semi-trusted oracles or protocol-level/softfork)?
From the user-experience perspective, this is probably pretty important
(otherwise the experience will be that incoming funds can take serval
minutes to hours until they appear).
Using BIP37 bloom filters just for mempool filtering would obviously
result in the same unwanted privacy-setup.
</jonas>
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Aaron Voisine via bitcoin-dev
2017-01-03 23:46:00 UTC
Permalink
If the sender doesn't control the receiver's network connection, then the
information the receiver gains by watching the mempool is if the
transaction has propagated across the bitcoin network. This is useful to
know in all kinds of situations.


Aaron Voisine
co-founder and CEO
breadwallet <http://breadwallet.com>
Post by adiabat via bitcoin-dev
Mempool transactions have their place, but "unconfirmed" and "SPV" don't
belong together. Only a full node can tell if a transaction may get
confirmed, or is nonsense. Unfortunately all the light / SPV wallets I
know of show mempool transactions, which makes it hard to go back... (e.g.
"why doesn't your software show 0-conf! your wallet is broken!", somewhat
akin to people complaining about RBF)
So, this is easy, just don't worry about mempool filtering. Why are light
clients looking at the mempool anyway? Maybe if there were some way to
provide SPV proofs of all inputs, but that's a bit of a mess for full nodes
to do.
Without mempool filtering, I think the committed bloom filters would be a
great improvement over the current bloom filter setup, especially for
lightning network use cases (with lightning, not finding out about a
transaction can make you lose money). I want to work on it and may be able
to at some point as it's somewhat related to lightning.
Also, if you're running a light client, and storing the filters the way
you store block headers, there's really no reason to go all the way back to
height 0. You can start grabbing headers at some point a while ago, before
your set of keys was generated. I think it'd be very worth it even with
GB-scale disk usage.
-Tadge
On Tue, Jan 3, 2017 at 5:18 PM, Aaron Voisine via bitcoin-dev <
Post by Aaron Voisine via bitcoin-dev
Unconfirmed transactions are incredibly important for real world use.
Merchants for instance are willing to accept credit card payments of
thousands of dollars and ship the goods despite the fact that the
transaction can be reversed up to 60 days later. There is a very large cost
to losing the ability to have instant transactions in many or even most
situations. This cost is typically well above the fraud risk.
It's important to recognize that bitcoin serves a wide variety of use
cases with different profiles for time sensitivity and fraud risk.
Aaron
On Tue, Jan 3, 2017 at 12:41 PM bfd--- via bitcoin-dev <
Post by bfd--- via bitcoin-dev
The concept combined with the weak blocks system where miners commit
to potential transaction inclusion with fractional difficulty blocks
is possible. I'm not personally convinced that unconfirmed transaction
display in a wallet is worth the privacy trade-off. The user has very
little to gain from this knowledge until the txn is in a block.
Post by Jonas Schnelli via bitcoin-dev
Hi
Post by bfd--- via bitcoin-dev
We introduce several concepts that rework the lightweight Bitcoin
client model in a manner which is secure, efficient and privacy
compatible.
The BFD can be used verbatim in replacement of BIP37, where the filter
can be cached between clients without needing to be recomputed. It can
also be used by normal pruned nodes to do re-scans locally of their
wallet without needing to have the block data available to scan, or
without reading the entire block chain from disk.
I started exploring the potential of BFD after this specification.
What would be the preferred/recommended way to handle 0-conf/mempool
filtering – if & once BDF would have been deployed (any type,
semi-trusted oracles or protocol-level/softfork)?
From the user-experience perspective, this is probably pretty important
(otherwise the experience will be that incoming funds can take serval
minutes to hours until they appear).
Using BIP37 bloom filters just for mempool filtering would obviously
result in the same unwanted privacy-setup.
</jonas>
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
bfd--- via bitcoin-dev
2017-01-04 00:10:14 UTC
Permalink
Unfortunately a non validating SPV wallet has absolutely no idea if
the information about an unconfirmed transaction they are seeing is
anything but properly formatted. They are connecting to an easily
manipulated, sybil attacked, and untrusted network and then asking
them for financial information. Seeing an unconfirmed transaction in a
wallet that's not also fully validating is at best meaningless.
Post by Aaron Voisine via bitcoin-dev
If the sender doesn't control the receiver's network connection, then
the information the receiver gains by watching the mempool is if the
transaction has propagated across the bitcoin network. This is useful
to know in all kinds of situations.
Aaron Voisine
co-founder and CEO
breadwallet [2]
Post by adiabat via bitcoin-dev
Mempool transactions have their place, but "unconfirmed" and "SPV"
don't belong together. Only a full node can tell if a transaction
may get confirmed, or is nonsense. Unfortunately all the light /
SPV wallets I know of show mempool transactions, which makes it hard
to go back... (e.g. "why doesn't your software show 0-conf! your
wallet is broken!", somewhat akin to people complaining about RBF)
So, this is easy, just don't worry about mempool filtering. Why are
light clients looking at the mempool anyway? Maybe if there were
some way to provide SPV proofs of all inputs, but that's a bit of a
mess for full nodes to do.
Without mempool filtering, I think the committed bloom filters would
be a great improvement over the current bloom filter setup,
especially for lightning network use cases (with lightning, not
finding out about a transaction can make you lose money). I want to
work on it and may be able to at some point as it's somewhat related
to lightning.
Also, if you're running a light client, and storing the filters the
way you store block headers, there's really no reason to go all the
way back to height 0. You can start grabbing headers at some point
a while ago, before your set of keys was generated. I think it'd be
very worth it even with GB-scale disk usage.
-Tadge
On Tue, Jan 3, 2017 at 5:18 PM, Aaron Voisine via bitcoin-dev
Unconfirmed transactions are incredibly important for real world
use. Merchants for instance are willing to accept credit card
payments of thousands of dollars and ship the goods despite the fact
that the transaction can be reversed up to 60 days later. There is a
very large cost to losing the ability to have instant transactions
in many or even most situations. This cost is typically well above
the fraud risk.
It's important to recognize that bitcoin serves a wide variety of
use cases with different profiles for time sensitivity and fraud
risk.
Aaron
On Tue, Jan 3, 2017 at 12:41 PM bfd--- via bitcoin-dev
The concept combined with the weak blocks system where miners commit
to potential transaction inclusion with fractional difficulty blocks
is possible. I'm not personally convinced that unconfirmed
transaction
display in a wallet is worth the privacy trade-off. The user has very
little to gain from this knowledge until the txn is in a block.
Post by Jonas Schnelli via bitcoin-dev
Hi
Post by bfd--- via bitcoin-dev
We introduce several concepts that rework the lightweight Bitcoin
client model in a manner which is secure, efficient and privacy
compatible.
The BFD can be used verbatim in replacement of BIP37, where the
filter
Post by Jonas Schnelli via bitcoin-dev
Post by bfd--- via bitcoin-dev
can be cached between clients without needing to be recomputed.
It can
Post by Jonas Schnelli via bitcoin-dev
Post by bfd--- via bitcoin-dev
also be used by normal pruned nodes to do re-scans locally of
their
Post by Jonas Schnelli via bitcoin-dev
Post by bfd--- via bitcoin-dev
wallet without needing to have the block data available to scan,
or
Post by Jonas Schnelli via bitcoin-dev
Post by bfd--- via bitcoin-dev
without reading the entire block chain from disk.
I started exploring the potential of BFD after this specification.
What would be the preferred/recommended way to handle
0-conf/mempool
Post by Jonas Schnelli via bitcoin-dev
filtering – if & once BDF would have been deployed (any type,
semi-trusted oracles or protocol-level/softfork)?
From the user-experience perspective, this is probably pretty
important
Post by Jonas Schnelli via bitcoin-dev
(otherwise the experience will be that incoming funds can take
serval
Post by Jonas Schnelli via bitcoin-dev
minutes to hours until they appear).
Using BIP37 bloom filters just for mempool filtering would
obviously
Post by Jonas Schnelli via bitcoin-dev
result in the same unwanted privacy-setup.
</jonas>
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev [1]
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev [1]
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev [1]
------
[1] https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
[2] http://breadwallet.com
Aaron Voisine via bitcoin-dev
2017-01-04 00:36:34 UTC
Permalink
Knowing that a transaction is property formatted and that it has been
broadcast to the gossip network is useful in many situations. You're only
thinking about whether you can know a transaction is valid and/or settled.
This is not the only possible useful information in actual real world use.
Any situation where credit card transactions are accepted today for
instance, it is useful to know that a transaction has been initiated, even
though it can be reversed at any time up to 60 days later.

Aaron Voisine
co-founder and CEO
breadwallet <http://breadwallet.com>
Post by bfd--- via bitcoin-dev
Unfortunately a non validating SPV wallet has absolutely no idea if
the information about an unconfirmed transaction they are seeing is
anything but properly formatted. They are connecting to an easily
manipulated, sybil attacked, and untrusted network and then asking
them for financial information. Seeing an unconfirmed transaction in a
wallet that's not also fully validating is at best meaningless.
Post by Aaron Voisine via bitcoin-dev
If the sender doesn't control the receiver's network connection, then
the information the receiver gains by watching the mempool is if the
transaction has propagated across the bitcoin network. This is useful
to know in all kinds of situations.
Aaron Voisine
co-founder and CEO
breadwallet [2]
Mempool transactions have their place, but "unconfirmed" and "SPV"
Post by adiabat via bitcoin-dev
don't belong together. Only a full node can tell if a transaction
may get confirmed, or is nonsense. Unfortunately all the light /
SPV wallets I know of show mempool transactions, which makes it hard
to go back... (e.g. "why doesn't your software show 0-conf! your
wallet is broken!", somewhat akin to people complaining about RBF)
So, this is easy, just don't worry about mempool filtering. Why are
light clients looking at the mempool anyway? Maybe if there were
some way to provide SPV proofs of all inputs, but that's a bit of a
mess for full nodes to do.
Without mempool filtering, I think the committed bloom filters would
be a great improvement over the current bloom filter setup,
especially for lightning network use cases (with lightning, not
finding out about a transaction can make you lose money). I want to
work on it and may be able to at some point as it's somewhat related
to lightning.
Also, if you're running a light client, and storing the filters the
way you store block headers, there's really no reason to go all the
way back to height 0. You can start grabbing headers at some point
a while ago, before your set of keys was generated. I think it'd be
very worth it even with GB-scale disk usage.
-Tadge
On Tue, Jan 3, 2017 at 5:18 PM, Aaron Voisine via bitcoin-dev
Unconfirmed transactions are incredibly important for real world
use. Merchants for instance are willing to accept credit card
payments of thousands of dollars and ship the goods despite the fact
that the transaction can be reversed up to 60 days later. There is a
very large cost to losing the ability to have instant transactions
in many or even most situations. This cost is typically well above
the fraud risk.
It's important to recognize that bitcoin serves a wide variety of
use cases with different profiles for time sensitivity and fraud
risk.
Aaron
On Tue, Jan 3, 2017 at 12:41 PM bfd--- via bitcoin-dev
The concept combined with the weak blocks system where miners commit
to potential transaction inclusion with fractional difficulty blocks
is possible. I'm not personally convinced that unconfirmed
transaction
display in a wallet is worth the privacy trade-off. The user has very
little to gain from this knowledge until the txn is in a block.
Hi
We introduce several concepts that rework the lightweight Bitcoin
client model in a manner which is secure, efficient and privacy
compatible.
The BFD can be used verbatim in replacement of BIP37, where the
Post by bfd--- via bitcoin-dev
filter
can be cached between clients without needing to be recomputed.
Post by bfd--- via bitcoin-dev
It can
also be used by normal pruned nodes to do re-scans locally of
Post by bfd--- via bitcoin-dev
their
wallet without needing to have the block data available to scan,
Post by bfd--- via bitcoin-dev
or
without reading the entire block chain from disk.
I started exploring the potential of BFD after this specification.
What would be the preferred/recommended way to handle
0-conf/mempool
filtering – if & once BDF would have been deployed (any type,
semi-trusted oracles or protocol-level/softfork)?
From the user-experience perspective, this is probably pretty
important
(otherwise the experience will be that incoming funds can take
serval
minutes to hours until they appear).
Using BIP37 bloom filters just for mempool filtering would
obviously
result in the same unwanted privacy-setup.
</jonas>
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev [1]
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev [1]
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev [1]
------
[1] https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
[2] http://breadwallet.com
Eric Voskuil via bitcoin-dev
2017-01-04 06:06:31 UTC
Permalink
Credit card reversals involve an escrow agent with control over the entire network and with a strong interest in preserving the network. A better analogy would be blind acceptance of any slip of paper under the assumption that it is sufficient currency. It may or may not be so, but you are on your own in either case.

e
Knowing that a transaction is property formatted and that it has been broadcast to the gossip network is useful in many situations. You're only thinking about whether you can know a transaction is valid and/or settled. This is not the only possible useful information in actual real world use. Any situation where credit card transactions are accepted today for instance, it is useful to know that a transaction has been initiated, even though it can be reversed at any time up to 60 days later.
Aaron Voisine
co-founder and CEO
breadwallet
Post by bfd--- via bitcoin-dev
Unfortunately a non validating SPV wallet has absolutely no idea if
the information about an unconfirmed transaction they are seeing is
anything but properly formatted. They are connecting to an easily
manipulated, sybil attacked, and untrusted network and then asking
them for financial information. Seeing an unconfirmed transaction in a
wallet that's not also fully validating is at best meaningless.
Post by Aaron Voisine via bitcoin-dev
If the sender doesn't control the receiver's network connection, then
the information the receiver gains by watching the mempool is if the
transaction has propagated across the bitcoin network. This is useful
to know in all kinds of situations.
Aaron Voisine
co-founder and CEO
breadwallet [2]
Post by adiabat via bitcoin-dev
Mempool transactions have their place, but "unconfirmed" and "SPV"
don't belong together. Only a full node can tell if a transaction
may get confirmed, or is nonsense. Unfortunately all the light /
SPV wallets I know of show mempool transactions, which makes it hard
to go back... (e.g. "why doesn't your software show 0-conf! your
wallet is broken!", somewhat akin to people complaining about RBF)
So, this is easy, just don't worry about mempool filtering. Why are
light clients looking at the mempool anyway? Maybe if there were
some way to provide SPV proofs of all inputs, but that's a bit of a
mess for full nodes to do.
Without mempool filtering, I think the committed bloom filters would
be a great improvement over the current bloom filter setup,
especially for lightning network use cases (with lightning, not
finding out about a transaction can make you lose money). I want to
work on it and may be able to at some point as it's somewhat related
to lightning.
Also, if you're running a light client, and storing the filters the
way you store block headers, there's really no reason to go all the
way back to height 0. You can start grabbing headers at some point
a while ago, before your set of keys was generated. I think it'd be
very worth it even with GB-scale disk usage.
-Tadge
On Tue, Jan 3, 2017 at 5:18 PM, Aaron Voisine via bitcoin-dev
Unconfirmed transactions are incredibly important for real world
use. Merchants for instance are willing to accept credit card
payments of thousands of dollars and ship the goods despite the fact
that the transaction can be reversed up to 60 days later. There is a
very large cost to losing the ability to have instant transactions
in many or even most situations. This cost is typically well above
the fraud risk.
It's important to recognize that bitcoin serves a wide variety of
use cases with different profiles for time sensitivity and fraud
risk.
Aaron
On Tue, Jan 3, 2017 at 12:41 PM bfd--- via bitcoin-dev
The concept combined with the weak blocks system where miners commit
to potential transaction inclusion with fractional difficulty blocks
is possible. I'm not personally convinced that unconfirmed
transaction
display in a wallet is worth the privacy trade-off. The user has very
little to gain from this knowledge until the txn is in a block.
Post by Jonas Schnelli via bitcoin-dev
Hi
Post by bfd--- via bitcoin-dev
We introduce several concepts that rework the lightweight Bitcoin
client model in a manner which is secure, efficient and privacy
compatible.
The BFD can be used verbatim in replacement of BIP37, where the
filter
Post by Jonas Schnelli via bitcoin-dev
Post by bfd--- via bitcoin-dev
can be cached between clients without needing to be recomputed.
It can
Post by Jonas Schnelli via bitcoin-dev
Post by bfd--- via bitcoin-dev
also be used by normal pruned nodes to do re-scans locally of
their
Post by Jonas Schnelli via bitcoin-dev
Post by bfd--- via bitcoin-dev
wallet without needing to have the block data available to scan,
or
Post by Jonas Schnelli via bitcoin-dev
Post by bfd--- via bitcoin-dev
without reading the entire block chain from disk.
I started exploring the potential of BFD after this specification.
What would be the preferred/recommended way to handle
0-conf/mempool
Post by Jonas Schnelli via bitcoin-dev
filtering – if & once BDF would have been deployed (any type,
semi-trusted oracles or protocol-level/softfork)?
From the user-experience perspective, this is probably pretty
important
Post by Jonas Schnelli via bitcoin-dev
(otherwise the experience will be that incoming funds can take
serval
Post by Jonas Schnelli via bitcoin-dev
minutes to hours until they appear).
Using BIP37 bloom filters just for mempool filtering would
obviously
Post by Jonas Schnelli via bitcoin-dev
result in the same unwanted privacy-setup.
</jonas>
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev [1]
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev [1]
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev [1]
------
[1] https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
[2] http://breadwallet.com
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Leo Wandersleb via bitcoin-dev
2017-01-04 16:13:41 UTC
Permalink
Post by adiabat via bitcoin-dev
Also, if you're running a light client, and storing the filters the way you
store block headers, there's really no reason to go all the way back to height
0. You can start grabbing headers at some point a while ago, before your set
of keys was generated. I think it'd be very worth it even with GB-scale disk
usage.
The really great benefit of having this index is that you could implement rather
efficient cold wallet spending once the wallet has the full index.

With Mycelium you can currently spend funds from a paper wallet or a BIP39
sentence but at the cost of sharing all addresses with our servers. Schildbach
would share addresses with random full nodes for hours or days to find funds of
a new private key with unknown creation date. With CBF it would still be a
matter of maybe minutes on a phone to identify relevant blocks and download
these but it would be very feasible to implement a private, cold storage
spending feature.

Also the index could be further partitioned by P2PKH, P2PK, P2SH, … This would
lead to a very minor privacy leak for a reasonable reduction in index size.
--
Leo Wandersleb
bfd--- via bitcoin-dev
2017-01-03 22:28:56 UTC
Permalink
The concept was not particularly targeted towards businesses, but
allowing for significantly improved wallet performance and reducing
privacy for lite clients. You would expect that a business has the
capacity to run a fully validating, fully storing node of their own.
If they’re not something is fundamentally broken with Bitcoin, or
their rationale of continuing to use it.
Post by Aaron Voisine via bitcoin-dev
Unconfirmed transactions are incredibly important for real world use.
Merchants for instance are willing to accept credit card payments of
thousands of dollars and ship the goods despite the fact that the
transaction can be reversed up to 60 days later. There is a very large
cost to losing the ability to have instant transactions in many or
even most situations. This cost is typically well above the fraud
risk.
It's important to recognize that bitcoin serves a wide variety of use
cases with different profiles for time sensitivity and fraud risk.
Aaron
On Tue, Jan 3, 2017 at 12:41 PM bfd--- via bitcoin-dev
Post by bfd--- via bitcoin-dev
The concept combined with the weak blocks system where miners commit
to potential transaction inclusion with fractional difficulty blocks
is possible. I'm not personally convinced that unconfirmed
transaction
display in a wallet is worth the privacy trade-off. The user has very
little to gain from this knowledge until the txn is in a block.
Post by Jonas Schnelli via bitcoin-dev
Hi
Post by bfd--- via bitcoin-dev
We introduce several concepts that rework the lightweight Bitcoin
client model in a manner which is secure, efficient and privacy
compatible.
The BFD can be used verbatim in replacement of BIP37, where the
filter
Post by Jonas Schnelli via bitcoin-dev
Post by bfd--- via bitcoin-dev
can be cached between clients without needing to be recomputed.
It can
Post by Jonas Schnelli via bitcoin-dev
Post by bfd--- via bitcoin-dev
also be used by normal pruned nodes to do re-scans locally of
their
Post by Jonas Schnelli via bitcoin-dev
Post by bfd--- via bitcoin-dev
wallet without needing to have the block data available to scan,
or
Post by Jonas Schnelli via bitcoin-dev
Post by bfd--- via bitcoin-dev
without reading the entire block chain from disk.
I started exploring the potential of BFD after this specification.
What would be the preferred/recommended way to handle
0-conf/mempool
Post by Jonas Schnelli via bitcoin-dev
filtering – if & once BDF would have been deployed (any type,
semi-trusted oracles or protocol-level/softfork)?
From the user-experience perspective, this is probably pretty
important
Post by Jonas Schnelli via bitcoin-dev
(otherwise the experience will be that incoming funds can take
serval
Post by Jonas Schnelli via bitcoin-dev
minutes to hours until they appear).
Using BIP37 bloom filters just for mempool filtering would
obviously
Post by Jonas Schnelli via bitcoin-dev
result in the same unwanted privacy-setup.
</jonas>
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Jonas Schnelli via bitcoin-dev
2017-01-04 07:47:10 UTC
Permalink
Hi
Post by Aaron Voisine via bitcoin-dev
Unconfirmed transactions are incredibly important for real world use.
Merchants for instance are willing to accept credit card payments of
thousands of dollars and ship the goods despite the fact that the
transaction can be reversed up to 60 days later. There is a very large
cost to losing the ability to have instant transactions in many or
even most situations. This cost is typically well above the fraud risk.
It's important to recognize that bitcoin serves a wide variety of use
cases with different profiles for time sensitivity and fraud risk.
I agree that unconfirmed transactions are incredibly important, but not
over SPV against random peers.

If you offer users/merchants a feature (SPV 0-conf against random
peers), that is fundamentally insecure, it will – sooner or later – lead
to some large scale fiasco, hurting Bitcoins reputation and trust from
merchants.

Merchants using and trusting 0-conf SPV transactions (retrieved from
random peers) is something we should **really eliminate** through
education and by offering different solution.

There are plenty, more sane options. If you can't run your own full-node
as a merchant (trivial), maybe co-use a wallet-service with centralized
verification (maybe use two of them), I guess Copay would be one of
those wallets (as an example). Use them in watch-only mode.

For end-users SPV software, I think it would be recommended to...
... disable unconfirmed transactions during SPV against random peers
... enable unconfirmed transactions when using SPV against a trusted
peer with preshared keys after BIP150
... if unconfirmed transactions are disabled, show how it can be enabled
(how to run a full-node [in a box, etc.])
... educate, inform users that a transaction with no confirmation can be
"stopped" or "redirected" any time, also inform about the risks during
low-conf phase (1-5).

I though see the point that it's nice to make use of the "incoming
funds..." feature in SPV wallets. But – for the sake of stability and
(risk-)scaling – we may want to recommend to scarify this feature and –
in the same turn – to use privacy-preserving BFD's.

</jonas>
Aaron Voisine via bitcoin-dev
2017-01-04 08:56:21 UTC
Permalink
It's easy enough to mark a transaction as "pending". People with bank
accounts are familiar with the concept.

Although the risk of accepting gossip information from multiple random
peers, in the case where the sender does not control the receivers network
is still minimal. Random node operators have no incentive to send fake
transactions, and would need to control all the nodes a client connects to,
and find a non-false-positive address belonging to the victims wallet.

It's not impossible, but it's non trivial, would only temporarily show a
pending transaction, and provide no benefit to the node operator. There are
much juicier targets for an attacker with the ability to sybil attack the
entire bitcoin p2p network.

Aaron
Post by Jonas Schnelli via bitcoin-dev
Hi
Post by Aaron Voisine via bitcoin-dev
Unconfirmed transactions are incredibly important for real world use.
Merchants for instance are willing to accept credit card payments of
thousands of dollars and ship the goods despite the fact that the
transaction can be reversed up to 60 days later. There is a very large
cost to losing the ability to have instant transactions in many or
even most situations. This cost is typically well above the fraud risk.
It's important to recognize that bitcoin serves a wide variety of use
cases with different profiles for time sensitivity and fraud risk.
I agree that unconfirmed transactions are incredibly important, but not
over SPV against random peers.
If you offer users/merchants a feature (SPV 0-conf against random
peers), that is fundamentally insecure, it will – sooner or later – lead
to some large scale fiasco, hurting Bitcoins reputation and trust from
merchants.
Merchants using and trusting 0-conf SPV transactions (retrieved from
random peers) is something we should **really eliminate** through
education and by offering different solution.
There are plenty, more sane options. If you can't run your own full-node
as a merchant (trivial), maybe co-use a wallet-service with centralized
verification (maybe use two of them), I guess Copay would be one of
those wallets (as an example). Use them in watch-only mode.
For end-users SPV software, I think it would be recommended to...
... disable unconfirmed transactions during SPV against random peers
... enable unconfirmed transactions when using SPV against a trusted
peer with preshared keys after BIP150
... if unconfirmed transactions are disabled, show how it can be enabled
(how to run a full-node [in a box, etc.])
... educate, inform users that a transaction with no confirmation can be
"stopped" or "redirected" any time, also inform about the risks during
low-conf phase (1-5).
I though see the point that it's nice to make use of the "incoming
funds..." feature in SPV wallets. But – for the sake of stability and
(risk-)scaling – we may want to recommend to scarify this feature and –
in the same turn – to use privacy-preserving BFD's.
</jonas>
Jorge Timón via bitcoin-dev
2017-01-04 10:13:02 UTC
Permalink
There were talks about implementing spv mode for bitcoin core without using
bloom filters. Less efficient because it downloads full blocks, but better
for privacy. Perhaps other spv implementations should consider doing the
same instead of committing the filters in the block?

Now I feel I was missing something. I guess you can download the whole
block you're interested in instead of only your txs and that gives you
privacy.
But how do you get to know which blocks are you interested in?

If the questions are too basic or offtopic for the thread, I'm happy
getting answers privately (but then maybe I get them more than once).


On 4 Jan 2017 09:57, "Aaron Voisine via bitcoin-dev" <
bitcoin-***@lists.linuxfoundation.org> wrote:

It's easy enough to mark a transaction as "pending". People with bank
accounts are familiar with the concept.

Although the risk of accepting gossip information from multiple random
peers, in the case where the sender does not control the receivers network
is still minimal. Random node operators have no incentive to send fake
transactions, and would need to control all the nodes a client connects to,
and find a non-false-positive address belonging to the victims wallet.

It's not impossible, but it's non trivial, would only temporarily show a
pending transaction, and provide no benefit to the node operator. There are
much juicier targets for an attacker with the ability to sybil attack the
entire bitcoin p2p network.

Aaron
Post by Jonas Schnelli via bitcoin-dev
Hi
Post by Aaron Voisine via bitcoin-dev
Unconfirmed transactions are incredibly important for real world use.
Merchants for instance are willing to accept credit card payments of
thousands of dollars and ship the goods despite the fact that the
transaction can be reversed up to 60 days later. There is a very large
cost to losing the ability to have instant transactions in many or
even most situations. This cost is typically well above the fraud risk.
It's important to recognize that bitcoin serves a wide variety of use
cases with different profiles for time sensitivity and fraud risk.
I agree that unconfirmed transactions are incredibly important, but not
over SPV against random peers.
If you offer users/merchants a feature (SPV 0-conf against random
peers), that is fundamentally insecure, it will – sooner or later – lead
to some large scale fiasco, hurting Bitcoins reputation and trust from
merchants.
Merchants using and trusting 0-conf SPV transactions (retrieved from
random peers) is something we should **really eliminate** through
education and by offering different solution.
There are plenty, more sane options. If you can't run your own full-node
as a merchant (trivial), maybe co-use a wallet-service with centralized
verification (maybe use two of them), I guess Copay would be one of
those wallets (as an example). Use them in watch-only mode.
For end-users SPV software, I think it would be recommended to...
... disable unconfirmed transactions during SPV against random peers
... enable unconfirmed transactions when using SPV against a trusted
peer with preshared keys after BIP150
... if unconfirmed transactions are disabled, show how it can be enabled
(how to run a full-node [in a box, etc.])
... educate, inform users that a transaction with no confirmation can be
"stopped" or "redirected" any time, also inform about the risks during
low-conf phase (1-5).
I though see the point that it's nice to make use of the "incoming
funds..." feature in SPV wallets. But – for the sake of stability and
(risk-)scaling – we may want to recommend to scarify this feature and –
in the same turn – to use privacy-preserving BFD's.
</jonas>
Adam Back via bitcoin-dev
2017-01-04 11:00:55 UTC
Permalink
I think this discussion started from the block bloom filter where
there is a bloom filter commitment in the block which can be
downloaded and is much smaller than the block. An SPV node based on
that model would download headers and bloom filters, verify the bloom
filter is committed to, and test locally if any addresses managed by
the wallet are in the filter (or false positives for being in it), and
then download blocks with hits. Apparently there are maybe 50% more
compact alternatives to bloom filters but people have been using bloom
filter as a short-hand for that. The block bloom filter does seem to
have higher overhead than the query model, but it offers much better
privacy. I think there was previous discussion about maybe doing
something with portions of blocks so you can know which half or
quarter of the block etc.

Adam


On 4 January 2017 at 10:13, Jorge Timón via bitcoin-dev
Post by Jorge Timón via bitcoin-dev
There were talks about implementing spv mode for bitcoin core without using
bloom filters. Less efficient because it downloads full blocks, but better
for privacy. Perhaps other spv implementations should consider doing the
same instead of committing the filters in the block?
Now I feel I was missing something. I guess you can download the whole block
you're interested in instead of only your txs and that gives you privacy.
But how do you get to know which blocks are you interested in?
If the questions are too basic or offtopic for the thread, I'm happy getting
answers privately (but then maybe I get them more than once).
On 4 Jan 2017 09:57, "Aaron Voisine via bitcoin-dev"
It's easy enough to mark a transaction as "pending". People with bank
accounts are familiar with the concept.
Although the risk of accepting gossip information from multiple random
peers, in the case where the sender does not control the receivers network
is still minimal. Random node operators have no incentive to send fake
transactions, and would need to control all the nodes a client connects to,
and find a non-false-positive address belonging to the victims wallet.
It's not impossible, but it's non trivial, would only temporarily show a
pending transaction, and provide no benefit to the node operator. There are
much juicier targets for an attacker with the ability to sybil attack the
entire bitcoin p2p network.
Aaron
Post by Jonas Schnelli via bitcoin-dev
Hi
Post by Aaron Voisine via bitcoin-dev
Unconfirmed transactions are incredibly important for real world use.
Merchants for instance are willing to accept credit card payments of
thousands of dollars and ship the goods despite the fact that the
transaction can be reversed up to 60 days later. There is a very large
cost to losing the ability to have instant transactions in many or
even most situations. This cost is typically well above the fraud risk.
It's important to recognize that bitcoin serves a wide variety of use
cases with different profiles for time sensitivity and fraud risk.
I agree that unconfirmed transactions are incredibly important, but not
over SPV against random peers.
If you offer users/merchants a feature (SPV 0-conf against random
peers), that is fundamentally insecure, it will – sooner or later – lead
to some large scale fiasco, hurting Bitcoins reputation and trust from
merchants.
Merchants using and trusting 0-conf SPV transactions (retrieved from
random peers) is something we should **really eliminate** through
education and by offering different solution.
There are plenty, more sane options. If you can't run your own full-node
as a merchant (trivial), maybe co-use a wallet-service with centralized
verification (maybe use two of them), I guess Copay would be one of
those wallets (as an example). Use them in watch-only mode.
For end-users SPV software, I think it would be recommended to...
... disable unconfirmed transactions during SPV against random peers
... enable unconfirmed transactions when using SPV against a trusted
peer with preshared keys after BIP150
... if unconfirmed transactions are disabled, show how it can be enabled
(how to run a full-node [in a box, etc.])
... educate, inform users that a transaction with no confirmation can be
"stopped" or "redirected" any time, also inform about the risks during
low-conf phase (1-5).
I though see the point that it's nice to make use of the "incoming
funds..." feature in SPV wallets. But – for the sake of stability and
(risk-)scaling – we may want to recommend to scarify this feature and –
in the same turn – to use privacy-preserving BFD's.
</jonas>
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Aaron Voisine via bitcoin-dev
2017-01-06 07:07:34 UTC
Permalink
Credit card transactions are simply an expample of a widely used payment
system that has frequent fraud and chargebacks. The argument I'm making is
that different people in different situations value speed and convenience
over a known fraud risk. Instant zero confirmation transactions are
extremely useful despite the risk in all kinds of real world situations.
You can't substitute your own value judgements for everyone in every
situation.

Aaron
With a credit card you have an institution worth billions of dollars
asserting that a payment has been made, with the option that it may be
retracted under special circumstances by the card issuer.
Unconfirmed Bitcoin transactions with a SPV client has you trusting
that the un-authenticated DNS seed lookup has not been tampered with,
the connection to the random node that you connect to has not been
tampered with, and that the seed nor the node are attempting to
manipulate you.
The two scenarios aren’t even remotely comparable.
Post by Aaron Voisine via bitcoin-dev
It's easy enough to mark a transaction as "pending". People with bank
accounts are familiar with the concept.
Although the risk of accepting gossip information from multiple random
peers, in the case where the sender does not control the receivers
network is still minimal. Random node operators have no incentive to
send fake transactions, and would need to control all the nodes a
client connects to, and find a non-false-positive address belonging to
the victims wallet.
It's not impossible, but it's non trivial, would only temporarily show
a pending transaction, and provide no benefit to the node operator.
There are much juicier targets for an attacker with the ability to
sybil attack the entire bitcoin p2p network.
Aaron
Post by Jonas Schnelli via bitcoin-dev
Hi
Post by Aaron Voisine via bitcoin-dev
Unconfirmed transactions are incredibly important for real world
use.
Post by Aaron Voisine via bitcoin-dev
Merchants for instance are willing to accept credit card payments
of
Post by Aaron Voisine via bitcoin-dev
thousands of dollars and ship the goods despite the fact that the
transaction can be reversed up to 60 days later. There is a very
large
Post by Aaron Voisine via bitcoin-dev
cost to losing the ability to have instant transactions in many or
even most situations. This cost is typically well above the fraud
risk.
Post by Aaron Voisine via bitcoin-dev
It's important to recognize that bitcoin serves a wide variety of
use
Post by Aaron Voisine via bitcoin-dev
cases with different profiles for time sensitivity and fraud risk.
I agree that unconfirmed transactions are incredibly important, but
not
over SPV against random peers.
If you offer users/merchants a feature (SPV 0-conf against random
peers), that is fundamentally insecure, it will – sooner or later
– lead
to some large scale fiasco, hurting Bitcoins reputation and trust
from
merchants.
Merchants using and trusting 0-conf SPV transactions (retrieved from
random peers) is something we should **really eliminate** through
education and by offering different solution.
There are plenty, more sane options. If you can't run your own
full-node
as a merchant (trivial), maybe co-use a wallet-service with
centralized
verification (maybe use two of them), I guess Copay would be one of
those wallets (as an example). Use them in watch-only mode.
For end-users SPV software, I think it would be recommended to...
... disable unconfirmed transactions during SPV against random peers
... enable unconfirmed transactions when using SPV against a trusted
peer with preshared keys after BIP150
... if unconfirmed transactions are disabled, show how it can be
enabled
(how to run a full-node [in a box, etc.])
... educate, inform users that a transaction with no confirmation
can be
"stopped" or "redirected" any time, also inform about the risks
during
low-conf phase (1-5).
I though see the point that it's nice to make use of the "incoming
funds..." feature in SPV wallets. But – for the sake of stability
and
(risk-)scaling – we may want to recommend to scarify this feature
and –
in the same turn – to use privacy-preserving BFD's.
</jonas>
Chris Priest via bitcoin-dev
2017-01-05 07:06:36 UTC
Permalink
On 1/3/17, Jonas Schnelli via bitcoin-dev
Post by Jonas Schnelli via bitcoin-dev
There are plenty, more sane options. If you can't run your own full-node
as a merchant (trivial), maybe co-use a wallet-service with centralized
verification (maybe use two of them), I guess Copay would be one of
those wallets (as an example). Use them in watch-only mode.
The best way is to connect to the mempool of each miner and check to
see if they have your txid in their mempool.

https://www.antpool.com/api/is_in_mempool?txid=334847bb...
https://www.f2pool.com/api/is_in_mempool?txid=334847bb...
https://bw.com/api/is_in_mempool?txid=334847bb...
https://bitfury.com/api/is_in_mempool?txid=334847bb...
https://btcc.com/api/is_in_mempool?txid=334847bb...

If each of these services return "True", and you know those services
so not engage in RBF, then you can assume with great confidence that
your transaction will be in the next block, or in a block very soon.
If any one of those services return "False", then you must assume that
it is possible that there is a double spend floating around, and that
you should wait to see if that tx gets confirmed. The problem is that
not every pool runs such a service to check the contents of their
mempool...

This is an example of mining centralization increasing the security of
zero confirm. If more people mined, this method will not work as well
because it would require you to call the API of hundreds of different
potential block creators.
Eric Voskuil via bitcoin-dev
2017-01-05 07:45:18 UTC
Permalink
Post by Chris Priest via bitcoin-dev
On 1/3/17, Jonas Schnelli via bitcoin-dev
Post by Jonas Schnelli via bitcoin-dev
There are plenty, more sane options. If you can't run your own full-node
as a merchant (trivial), maybe co-use a wallet-service with centralized
verification (maybe use two of them), I guess Copay would be one of
those wallets (as an example). Use them in watch-only mode.
The best way is to connect to the mempool of each miner and check to
see if they have your txid in their mempool.
https://www.antpool.com/api/is_in_mempool?txid=334847bb...
https://www.f2pool.com/api/is_in_mempool?txid=334847bb...
https://bw.com/api/is_in_mempool?txid=334847bb...
https://bitfury.com/api/is_in_mempool?txid=334847bb...
https://btcc.com/api/is_in_mempool?txid=334847bb...
If each of these services return "True", and you know those services
so not engage in RBF, then you can assume with great confidence that
your transaction will be in the next block, or in a block very soon.
If any one of those services return "False", then you must assume that
it is possible that there is a double spend floating around, and that
you should wait to see if that tx gets confirmed. The problem is that
not every pool runs such a service to check the contents of their
mempool...
This is an example of mining centralization increasing the security of
zero confirm.
A world connected up to a few web services to determine payment validity
is an example of a bitcoin security catastrophe.

e
Christian Decker via bitcoin-dev
2017-01-05 14:48:33 UTC
Permalink
Post by Eric Voskuil via bitcoin-dev
Post by Chris Priest via bitcoin-dev
The best way is to connect to the mempool of each miner and check to
see if they have your txid in their mempool.
https://www.antpool.com/api/is_in_mempool?txid=334847bb...
https://www.f2pool.com/api/is_in_mempool?txid=334847bb...
https://bw.com/api/is_in_mempool?txid=334847bb...
https://bitfury.com/api/is_in_mempool?txid=334847bb...
https://btcc.com/api/is_in_mempool?txid=334847bb...
If each of these services return "True", and you know those services
so not engage in RBF, then you can assume with great confidence that
your transaction will be in the next block, or in a block very soon.
If any one of those services return "False", then you must assume that
it is possible that there is a double spend floating around, and that
you should wait to see if that tx gets confirmed. The problem is that
not every pool runs such a service to check the contents of their
mempool...
This is an example of mining centralization increasing the security of
zero confirm.
A world connected up to a few web services to determine payment validity
is an example of a bitcoin security catastrophe.
e
And it's a great way to tell every miner who you are and what
transactions you are sending/receiving. An absolute privacy
nightmare...

-- cdecker
Chris Priest via bitcoin-dev
2017-01-06 20:15:46 UTC
Permalink
Its a method for determining the probability that a valid tx will be
mined in a block before that tx actually gets mined, which is useful
when accepting payments in situations when you can't wait for the full
confirmation. No one is saying all tx validation should be performed
by querying miners mempools, that's ridiculous. Obviously once the tx
gets it's first confirmation, you go back to determining validity the
way you always have. There is no "security catastrophe".

Even if you're running a full node, you can't know for certain that
any given tx will make it into a future block. You can't be certain
the future miner who finally does mine that tx will mine your TXID or
another TXID that spends the same inputs to another address (a double
spend). The only way to actually know for certain is to query every
single large hashpower mempool.
Post by Eric Voskuil via bitcoin-dev
Post by Chris Priest via bitcoin-dev
On 1/3/17, Jonas Schnelli via bitcoin-dev
Post by Jonas Schnelli via bitcoin-dev
There are plenty, more sane options. If you can't run your own full-node
as a merchant (trivial), maybe co-use a wallet-service with centralized
verification (maybe use two of them), I guess Copay would be one of
those wallets (as an example). Use them in watch-only mode.
The best way is to connect to the mempool of each miner and check to
see if they have your txid in their mempool.
https://www.antpool.com/api/is_in_mempool?txid=334847bb...
https://www.f2pool.com/api/is_in_mempool?txid=334847bb...
https://bw.com/api/is_in_mempool?txid=334847bb...
https://bitfury.com/api/is_in_mempool?txid=334847bb...
https://btcc.com/api/is_in_mempool?txid=334847bb...
If each of these services return "True", and you know those services
so not engage in RBF, then you can assume with great confidence that
your transaction will be in the next block, or in a block very soon.
If any one of those services return "False", then you must assume that
it is possible that there is a double spend floating around, and that
you should wait to see if that tx gets confirmed. The problem is that
not every pool runs such a service to check the contents of their
mempool...
This is an example of mining centralization increasing the security of
zero confirm.
A world connected up to a few web services to determine payment validity
is an example of a bitcoin security catastrophe.
e
James MacWhyte via bitcoin-dev
2017-01-06 21:35:58 UTC
Permalink
It's my opinion that the purpose of this list and bitcoin protocol
development in general is to build the base functionality that other
companies and individuals require to provide usability to the end-user. The
0-conf debate is a UX issue. If end users shouldn't rely on 0-conf, it is
up to wallet developers to hide 0-conf transactions or mark them
appropriately. Instead of using this list to debate what wallet designers
should or shouldn't do, we should just provide the tools and "let the
market sort it out". If wallet developers start getting inundated with
complaints that 0-conf transactions are causing confusion and loss, they
will find a solution. If the tools they require for the solution don't
exist, they will come to this list to request action.

Am I wrong?

On Fri, Jan 6, 2017 at 12:16 PM Chris Priest via bitcoin-dev <
Post by Chris Priest via bitcoin-dev
Its a method for determining the probability that a valid tx will be
mined in a block before that tx actually gets mined, which is useful
when accepting payments in situations when you can't wait for the full
confirmation. No one is saying all tx validation should be performed
by querying miners mempools, that's ridiculous. Obviously once the tx
gets it's first confirmation, you go back to determining validity the
way you always have. There is no "security catastrophe".
Even if you're running a full node, you can't know for certain that
any given tx will make it into a future block. You can't be certain
the future miner who finally does mine that tx will mine your TXID or
another TXID that spends the same inputs to another address (a double
spend). The only way to actually know for certain is to query every
single large hashpower mempool.
Post by Eric Voskuil via bitcoin-dev
Post by Chris Priest via bitcoin-dev
On 1/3/17, Jonas Schnelli via bitcoin-dev
Post by Jonas Schnelli via bitcoin-dev
There are plenty, more sane options. If you can't run your own
full-node
Post by Eric Voskuil via bitcoin-dev
Post by Chris Priest via bitcoin-dev
Post by Jonas Schnelli via bitcoin-dev
as a merchant (trivial), maybe co-use a wallet-service with centralized
verification (maybe use two of them), I guess Copay would be one of
those wallets (as an example). Use them in watch-only mode.
The best way is to connect to the mempool of each miner and check to
see if they have your txid in their mempool.
https://www.antpool.com/api/is_in_mempool?txid=334847bb...
https://www.f2pool.com/api/is_in_mempool?txid=334847bb...
https://bw.com/api/is_in_mempool?txid=334847bb...
https://bitfury.com/api/is_in_mempool?txid=334847bb...
https://btcc.com/api/is_in_mempool?txid=334847bb...
If each of these services return "True", and you know those services
so not engage in RBF, then you can assume with great confidence that
your transaction will be in the next block, or in a block very soon.
If any one of those services return "False", then you must assume that
it is possible that there is a double spend floating around, and that
you should wait to see if that tx gets confirmed. The problem is that
not every pool runs such a service to check the contents of their
mempool...
This is an example of mining centralization increasing the security of
zero confirm.
A world connected up to a few web services to determine payment validity
is an example of a bitcoin security catastrophe.
e
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Eric Voskuil via bitcoin-dev
2017-01-06 21:50:47 UTC
Permalink
It is a useful aspect of discussion at this level as it helps higher lever developers understand the actual tradeoffs. Clearly some do not. The market will eventually sort them out, but the discussion both gives developers the necessary information.

It also helps core development prioritize resources. I personally would not prioritize core work to facilitate zero conf. I would even spend time to discourage it, as others have done.

I think the cautions in this thread about doing privacy and system security damaging things (like checking mining pools for zero conf transactions) will prevent some wasted time, which benefits everyone.

e
It's my opinion that the purpose of this list and bitcoin protocol development in general is to build the base functionality that other companies and individuals require to provide usability to the end-user. The 0-conf debate is a UX issue. If end users shouldn't rely on 0-conf, it is up to wallet developers to hide 0-conf transactions or mark them appropriately. Instead of using this list to debate what wallet designers should or shouldn't do, we should just provide the tools and "let the market sort it out". If wallet developers start getting inundated with complaints that 0-conf transactions are causing confusion and loss, they will find a solution. If the tools they require for the solution don't exist, they will come to this list to request action.
Am I wrong?
Post by Chris Priest via bitcoin-dev
Its a method for determining the probability that a valid tx will be
mined in a block before that tx actually gets mined, which is useful
when accepting payments in situations when you can't wait for the full
confirmation. No one is saying all tx validation should be performed
by querying miners mempools, that's ridiculous. Obviously once the tx
gets it's first confirmation, you go back to determining validity the
way you always have. There is no "security catastrophe".
Even if you're running a full node, you can't know for certain that
any given tx will make it into a future block. You can't be certain
the future miner who finally does mine that tx will mine your TXID or
another TXID that spends the same inputs to another address (a double
spend). The only way to actually know for certain is to query every
single large hashpower mempool.
Post by Eric Voskuil via bitcoin-dev
Post by Chris Priest via bitcoin-dev
On 1/3/17, Jonas Schnelli via bitcoin-dev
Post by Jonas Schnelli via bitcoin-dev
There are plenty, more sane options. If you can't run your own full-node
as a merchant (trivial), maybe co-use a wallet-service with centralized
verification (maybe use two of them), I guess Copay would be one of
those wallets (as an example). Use them in watch-only mode.
The best way is to connect to the mempool of each miner and check to
see if they have your txid in their mempool.
https://www.antpool.com/api/is_in_mempool?txid=334847bb...
https://www.f2pool.com/api/is_in_mempool?txid=334847bb...
https://bw.com/api/is_in_mempool?txid=334847bb...
https://bitfury.com/api/is_in_mempool?txid=334847bb...
https://btcc.com/api/is_in_mempool?txid=334847bb...
If each of these services return "True", and you know those services
so not engage in RBF, then you can assume with great confidence that
your transaction will be in the next block, or in a block very soon.
If any one of those services return "False", then you must assume that
it is possible that there is a double spend floating around, and that
you should wait to see if that tx gets confirmed. The problem is that
not every pool runs such a service to check the contents of their
mempool...
This is an example of mining centralization increasing the security of
zero confirm.
A world connected up to a few web services to determine payment validity
is an example of a bitcoin security catastrophe.
e
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
bfd--- via bitcoin-dev
2017-01-06 02:04:22 UTC
Permalink
You might as well replace Bitcoin with a system where these parties
sign transactions and skip mining altogether, it would have the same
properties and be significantly more effient.
Post by Chris Priest via bitcoin-dev
On 1/3/17, Jonas Schnelli via bitcoin-dev
Post by Jonas Schnelli via bitcoin-dev
There are plenty, more sane options. If you can't run your own full-node
as a merchant (trivial), maybe co-use a wallet-service with
centralized
verification (maybe use two of them), I guess Copay would be one of
those wallets (as an example). Use them in watch-only mode.
The best way is to connect to the mempool of each miner and check to
see if they have your txid in their mempool.
https://www.antpool.com/api/is_in_mempool?txid=334847bb...
https://www.f2pool.com/api/is_in_mempool?txid=334847bb...
https://bw.com/api/is_in_mempool?txid=334847bb...
https://bitfury.com/api/is_in_mempool?txid=334847bb...
https://btcc.com/api/is_in_mempool?txid=334847bb...
If each of these services return "True", and you know those services
so not engage in RBF, then you can assume with great confidence that
your transaction will be in the next block, or in a block very soon.
If any one of those services return "False", then you must assume that
it is possible that there is a double spend floating around, and that
you should wait to see if that tx gets confirmed. The problem is that
not every pool runs such a service to check the contents of their
mempool...
This is an example of mining centralization increasing the security of
zero confirm. If more people mined, this method will not work as well
because it would require you to call the API of hundreds of different
potential block creators.
Tom Harding via bitcoin-dev
2017-03-15 22:36:09 UTC
Permalink
Agreed.

In contrast, BIP37 as used today is totally decentralized, and can me
made much more secure, private, and scalable -- without giving up the
utility of unconfirmed transactions.

Please don't read into this statement a belief that all the coffees
should go on the chain, or that the security or privacy of BIP37 compare
favorably to any other particular thing.

https://docs.google.com/presentation/d/13MzUo2iIH9JBW29TgtPMoaMXxeEdanWDfi6SlfO-LlA
Post by bfd--- via bitcoin-dev
You might as well replace Bitcoin with a system where these parties
sign transactions and skip mining altogether, it would have the same
properties and be significantly more effient.
Post by Chris Priest via bitcoin-dev
On 1/3/17, Jonas Schnelli via bitcoin-dev
Post by Jonas Schnelli via bitcoin-dev
There are plenty, more sane options. If you can't run your own full-node
as a merchant (trivial), maybe co-use a wallet-service with centralized
verification (maybe use two of them), I guess Copay would be one of
those wallets (as an example). Use them in watch-only mode.
The best way is to connect to the mempool of each miner and check to
see if they have your txid in their mempool.
https://www.antpool.com/api/is_in_mempool?txid=334847bb...
https://www.f2pool.com/api/is_in_mempool?txid=334847bb...
https://bw.com/api/is_in_mempool?txid=334847bb...
https://bitfury.com/api/is_in_mempool?txid=334847bb...
https://btcc.com/api/is_in_mempool?txid=334847bb...
If each of these services return "True", and you know those services
so not engage in RBF, then you can assume with great confidence that
your transaction will be in the next block, or in a block very soon.
If any one of those services return "False", then you must assume that
it is possible that there is a double spend floating around, and that
you should wait to see if that tx gets confirmed. The problem is that
not every pool runs such a service to check the contents of their
mempool...
This is an example of mining centralization increasing the security of
zero confirm. If more people mined, this method will not work as well
because it would require you to call the API of hundreds of different
potential block creators.
bfd--- via bitcoin-dev
2017-03-16 00:25:01 UTC
Permalink
Sorry, this is not the case.

Your slides gloss over the simple fact that compact fraud proofs in
Bitcoin aren't possible, and that the "SPV" implemented today bears
absolutely no resemblance in security properties to the version
described in the Bitcoin white paper. In the white paper SPV clients
have the same security as fully validating nodes, in the implementation
of BIP37 they have absolutely no security except the vague hope that
they are not being lied to, and that the chain with the most work they
are seeing is actually valid, both are very weak assumptions.

The suggested solution in no way precludes unconfirmed transactions from
being used with a commitment scheme, my comments and others are just
recognising that they are an almost useless indicator for people who
aren't validating.

During the validationless mining failure around the BIP66 activation
miners produced 6 invalid blocks in a chain, and many more invalid
blocks in isolated bursts for a period lasting several months. Due to
the instability of the network you are completely unreasonable to accept
anything except multiple confirmations, the true number for safety is
probably more like 60 or 120, not 6, or 3, or 1 as some Bitcoin
exchanges use today.


0000000000000000009cc829aa25b40b2cd4eb83dd498c12ad0d26d90c439d99.bin
bad-version(0x00000002)
0000000000000000032527aa796d3672e32e5f85a452d3a584a28fc7efbcd5d0.bin
bad-version(0x00000002)
000000000000000003ae1223f4926ec86100885cfe1484dc52fd67e042a19b12.bin
bad-version(0x00000002)
0000000000000000083cbdbb25c1607527c8f3fdb16f0d048c4439a73b501cb6.bin
bad-version(0x00000002)
00000000000000000954ed93eda1e79e8261137548fa9ccf4d516bb384a3660b.bin
bad-version(0x00000002)
00000000000000000afc9fbe7cfe8a6b50502d509ba626beb2e2d6c15d1d3ee3.bin
bad-version(0x00000002)
00000000000000000b6adf92bc192b3c21210f456ab21b5e46951665c74cfab2.bin
bad-version(0x00000002)
00000000000000000c9bb4a508fff34f5450d9c62ef2cb833e53909a4c549de5.bin
bad-version(0x00000002)
0000000000000000116322b5f25826787b01f7a70fb322837b68dff8216cefc4.bin
bad-version(0x00000002)
000000000000000012aac0664cd8b6cbc3ea485921a05f2c4340f928b0226d3c.bin
bad-version(0x00000002)

"SPV" like you're describing can exist, or validationless mining can
exist, both can not simultaneously.
Post by Tom Harding via bitcoin-dev
Agreed.
In contrast, BIP37 as used today is totally decentralized, and can me
made much more secure, private, and scalable -- without giving up the
utility of unconfirmed transactions.
Please don't read into this statement a belief that all the coffees
should go on the chain, or that the security or privacy of BIP37
compare favorably to any other particular thing.
https://docs.google.com/presentation/d/13MzUo2iIH9JBW29TgtPMoaMXxeEdanWDfi6SlfO-LlA
Post by bfd--- via bitcoin-dev
You might as well replace Bitcoin with a system where these parties
sign transactions and skip mining altogether, it would have the same
properties and be significantly more effient.
Post by Chris Priest via bitcoin-dev
On 1/3/17, Jonas Schnelli via bitcoin-dev
Post by Jonas Schnelli via bitcoin-dev
There are plenty, more sane options. If you can't run your own full-node
as a merchant (trivial), maybe co-use a wallet-service with
centralized
verification (maybe use two of them), I guess Copay would be one of
those wallets (as an example). Use them in watch-only mode.
The best way is to connect to the mempool of each miner and check to
see if they have your txid in their mempool.
https://www.antpool.com/api/is_in_mempool?txid=334847bb...
https://www.f2pool.com/api/is_in_mempool?txid=334847bb...
https://bw.com/api/is_in_mempool?txid=334847bb...
https://bitfury.com/api/is_in_mempool?txid=334847bb...
https://btcc.com/api/is_in_mempool?txid=334847bb...
If each of these services return "True", and you know those services
so not engage in RBF, then you can assume with great confidence that
your transaction will be in the next block, or in a block very soon.
If any one of those services return "False", then you must assume that
it is possible that there is a double spend floating around, and that
you should wait to see if that tx gets confirmed. The problem is that
not every pool runs such a service to check the contents of their
mempool...
This is an example of mining centralization increasing the security of
zero confirm. If more people mined, this method will not work as well
because it would require you to call the API of hundreds of different
potential block creators.
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Tom Harding via bitcoin-dev
2017-03-16 15:05:11 UTC
Permalink
compact fraud proofs in Bitcoin aren't possible
In the white paper SPV clients have the same security as fully
validating nodes
In addition to not existing, if compact fraud proofs did exist, trying
to ensure they are seen by SPV clients has the same problems as BIP37.
in the implementation of BIP37 they have absolutely no security except
the vague hope that they are not being lied to, and that the chain
with the most work they are seeing is actually valid, both are very
weak assumptions.
Since real money is involved, the near total absence of documented fraud
along these lines belies the strong language.
During the validationless mining failure around the BIP66 activation
miners produced 6 invalid blocks in a chain, and many more invalid
blocks in isolated bursts for a period lasting several months. Due to
the instability of the network you are completely unreasonable to
accept anything except multiple confirmations
This affected all users, not just SPV.
Chris Belcher via bitcoin-dev
2017-02-17 00:28:59 UTC
Permalink
I believe this proposal still suffers from one problem that bip37 did,
albiet by a much lesser extent. Combining the partial information from
the block downloads with the transaction subgraph information from the
blockchain can in some cases still reveal which addresses belong to the
wallet. Nonetheless this proposal still has many benefits and is well
worth working on.

==BIP37==

As a recap, probably the biggest and most problematic way that bip37 was
broken was by combining the partial wallet information from the bloom
filter with the transaction subgraph information from the blockchain

Suppose a wallet synchronizes it's history, if it spent a coin from its
address A, it must also also add the change address B to the bloom
filter, which is connected to A directly on transaction graph.

As an example, consider five typical transactions that consume one input
each and produce two outputs.
A, B, C, D, E refer to transactions. A1, A2, etc refer to addresses
within those transactions

-> C1
A1 -> B2 -> C2
-> B2 -> D1
-> D2 -> E1
-> E2

If a bip37 bloom filter matches addresses A1, B2, D2, E1 then it can be
seen that they form a "peel chain" [this terminology comes from
https://cseweb.ucsd.edu/~smeiklejohn/files/imc13.pdf]

-> X
A1 -> X -> X
-> B2 -> X
-> D2 -> E1
-> X

The same five transactions with non-matching addresses replaced by X.
The peel chain is visible, it's clear that B2, D2, E1 are change
addresses which belong to the same wallet as A1.

For a given false-positive rate fp and given length of peel chain C, the
odds of a false positive peel chain happening by chance is fp^C which
rapidly gets very small as the wallet makes more transactions (increases C).

If only one address was matched from the above group (for example B2)
then it likely to be a false positive by the fact that it doesn't make
any transactions to another address that also matches the bloom filter.
Another possibility is that the address is a payment output that the
wallet received but hasn't spent yet, but the wallet cant spend it
without adding the change address to the bloom filter and thus revealing
itself to the spy.

I believe the committed bloom filter proposal is vulnerability to this
same kind of attack because it still leaks information about which
addresses the wallet is interested in.

==Committed Bloom Filter Maths==

I'll try to analyze this now. I'll find the expectation value of the
number of transaction subgraphs in those blocks that appear just by
chance. If this expectation goes to zero, then the only transaction
subgraph left will be the real one that the wallet is actually
interested in. In that case it will be possible to spy on the wallet.

Assuming outputs have the same probability of being spent in each time
interval (i.e. they are spent in a Poisson process) This is
approximately true, see the graphs from
[https://letstalkbitcoin.com/blog/post/rise-of-the-zombie-bitcoins].
This means we can assign
a single probability P that an output is spent in each block.

Assume every transaction has one change address only and spending of
unconfirmed change doesn't happen (its more efficient to use RBF to add
additional outputs anyway)

Number of transactions per block = Q (about 1800 today)
Number of outputs per block = Z = 2*Q (approximately)
Length of peel chain = Number of transactions in wallet = C
Average time an output is unspent for = T (about 1 month, very roughly
estimating from the above blog post)
Probability an output being spent in any particular later block = P =
10minutes/T

Assume no false positive blocks
Say wallet downloaded two blocks and they are ordered by block height
The expected number of tx subgraphs between them, E(#G)
E(#G) = number of outputs created in block 1 that get spent in block 2
= Z*P

Say the wallet downloaded three blocks
Expected number of subgraphs going through them all
E(#G) = number of outputs created in block 1 get spent in block 2, that
create a change address which gets spent in block 3
= Z*P*P

Say the wallet downloaded C blocks
Expected number of tx subgraphs going through all the blocks by chance
E(#G) = Z*P^C
which gets small quickly as C goes up, because P < 1

Now drop the assumption about no false positive blocks.

Let the number of candidate blocks be D.
This is how many blocks the wallet scans, it's related to how far in the
past the wallet's keys was created. At one extreme wallet was created at
genesis block and so D = ~450000, at other extreme created now so D = 0.
Note that D = 0 must also imply C = 0

Expected number of false positive blocks downloaded = F = fp*D

In all these situations the blocks are sorted by block height

Suppose have C=2, F=1, and false one is in the middle.
I want to find E(#G|CF), the expected number of transaction subgraphs
that appear just by chance, given C and F.
E(#G|CF) = how many outputs which are created in block 1 get spent in
block 3
= Z*P

Same situation, but false one at the start instead of middle.
E(#G|CF) = how many outputs which are created in block 2 get spent in
block 3
= Z*P

Same situation but false one could be anywhere, result in the sum of the
probability for any false block position
E(#G|CF) = C(3, 1)*Z*P = 3*Z*P

where C() is the number of order-independent ways of choosing 1 element
out of a set of 3 elements, also known as the binomial coefficient

Now suppose C=3 and F=1
The same argument leads to
E(#G|CF) = C(4, 1)*Z*P^2 = 4*Z*P^2


Now suppose C=3 and F=2, with fp blocks at the end
E(#G|CF)
= how many outputs are created in block 1, are spent in block 2 and
change address spent in block 3
= Z*P^2

Same situation but fp blocks can be anywhere, add up all the possible
combinations of them within the rest
E(#G|CF) = C(5, 2)*Z*P^2 = 5*Z*P^2

With these same rules, its clear the general expression for any F and C
E(#G|CF) = C(F + C, F)*Z*P^(C - 1)

A more interesting value might be the time evolution of E(#G)
Let B be the blocks in the blockchain since the wallet creation date, as you
know it increases at an average rate of one every ten minutes

w = wallet transaction creation rate, expressed per-block
C = w * B
F = fp * B

J = average blocks between wallet transactions = 1440 (10 days)
w = 1/J

E(#G|B) = C((fp + w)*B, fp*B)*Z*P^(w*B - 1)

This goes to zero as B becomes big, although choosing very high values
of fp makes it go to zero slower.

This is only approximate maths, in actuality you cannot take the number
of false positive blocks to be fp*B, you have to sum over all blocks
weighted by probability. And outputs might not be spent in an exact
Poisson process so you cant just multiply by P each time. Plus if your
false positive rate is very high then some of your false positive blocks
will actually contain your real transactions, this analysis
double-counts them.

Using some reasonable values and plotting E(#G|B) against B can show how
quickly it drops and therefore leaves only the true transaction subgraph.

(note: in LibreOffice Math and Microsoft Excel the binomial coefficient
function is COMBIN)

==Notes==

*) The expected number of transaction subgraphs that happen by chance
goes to zero eventually as the blockchain steps ahead. Unless the fp
rate is very high (close to 1) and time between wallet transaction very
long, in which case the binomial coefficient term gets larger more
quicker than the exponential decay P^B term gets smaller.
*) fp rate doesn't help in most cases that much compared to the
exponential drop-off from time ticking ahead requiring more downloading
of blocks
*) its good for privacy if bitcoin outputs are spent more frequently so
P is higher, because that creates more transaction subgraphs in the
anonymity set.
*) its good for privacy if more outputs are made per block, although
still only linearly which is no match for the exponential reduction from
the P^B term.
*) its good for privacy to make less of your own transactions (increase
J and reduce w), for low-activity users the privacy of committed bloom
filters can be actually pretty good, for high-activity users who use
bitcoin's blockchain all the time it's not very good
*) For the reasonable values I tried for a once-a-month user with fp=1%,
their chance-transaction-subgraph-count drops below 1 in about eight months.
*) Because of the exponential nature, E(#G) goes from "billions of
billions" to "about 10" fairly quickly.

==Discussion of ways to mitigate this==

One way is to not use change outputs. This is unrealistic, doesn't match
people's behavior and money must be divisible.

A better way to mitigate this is to not leak the information that all
those blocks are interesting to the same wallet. Don't download all
blocks from the same archival node. If you download blocks from many
different nodes, it gives an incentive for surveillance startups to
create lots of sybil nodes they control and can then correlate together
block downloads with the wallet IP address. Many such startups are
already doing this today to try to detect the origin IP address of
broadcasted transactions.

Another solution could be to download a few blocks from different nodes
with new tor circuits used. This would delink the wallet IP address from
the downloads and would help a lot. This has the issue that tor is
slower (but still not as slow as downloading the entire blockchain)

Another way a wallet could be correlated with its block downloads is
timing correlations. At any one time only a certain number of peers
would be downloading blocks which narrows down which wallets are
downloading what. However even today Bitcoin Core downloads blocks in
parallel from many nodes so there's probably quite a large anonymity set
for lightweight wallets using committed bloom filters. Plus timing
correlation can be reduced simply by waiting longer. Wallets are not
sync'd from backup very often so it might be okay to wait.

Another way to improve privacy could be for the wallet to choose random
transaction subgraphs and download all the blocks related to them as well.

Wallet developers might choose to allow the user to configure their own
fp rate. This is probably not a good idea since the relationship between
fp rate and anonymity set is non-obvious. It might be better to ask the
user how often they expect to make transactions.

==Conclusion==

I think this committed bloom filter idea is very good and much better
than bip37, but for good privacy for when bitcoin is used often still
requires certain behavior namely downloading blocks
from many different peers with new tor circuits.

Note that I've been dealing with counting transaction subgraphs but
actually finding them from blocks might also be computationally
infeasible. Although a Bayesian approach worked very
well for similar transaction subgraph linking
[https://arxiv.org/pdf/1612.06747v3.pdf]

It would also be interesting to analyze what information a spy can get
if they are missing some blocks that the wallet downloaded.

For the long term, private and high-volume bitcoin use will be best
served by off-chain transactions. They will probably be a huge win just
because the large and public blockchain is such a non-private
way of doing things.
Post by bfd--- via bitcoin-dev
We introduce several concepts that rework the lightweight Bitcoin
client model in a manner which is secure, efficient and privacy
compatible.
Thea properties of BIP37 SPV [0] are unfortunately not as strong as
* The expected privacy of the probabilistic nature of bloom
filters does not exist [1][2], any user with a BIP37 SPV wallet
should be operating under no expectation of privacy.
Implementation flaws make this effect significantly worse, the
behavior meaning that no matter how high the false positive
rate (up to simply downloading the whole blocks verbatim) the
intent of the client connection is recoverable.
* Significant processing load is placed on nodes in the Bitcoin
network by lightweight clients, a single syncing wallet causes
(at the time of writing) 80GB of disk reads and a large amount
of CPU time to be consumed processing this data. This carries
significant denial of service risk [3], non-distinguishable
clients can repeatedly request taxing blocks causing
reprocessing on every request. Processed data is unique to every
client, and can not be cached or made more efficient while
staying within specification.
* Wallet clients can not have strong consistency or security
expectations, BIP37 merkle paths allow for a wallet to validate
that an output was spendable at some point in time but does not
prove that this output is not spent today.
* Nodes in the network can denial of service attack all BIP37 SPV
wallet clients by simply returning null filter results for
requests, the wallet has no way of discerning if it has been
lied to and may be made simply unaware that any payment has been
made to them. Many nodes can be queried in a probabilistic manor
but this increases the already heavy network load with little
benefit.
We propose a new concept which can work towards addressing these
shortcomings.
A Bloom Filter Digest is deterministically created of every block
encompassing the inputs and outputs of the containing transactions,
the filter parameters being tuned such that the filter is a small
portion of the size of the total block data. To determine if a block
has contents which may be interesting a second bloom filter of all
relevant key material is created. A binary comparison between the two
filters returns true if there is probably matching transactions, and
false if there is certainly no matching transactions. Any matched
blocks can be downloaded in full and processed for transactions which
may be relevant.
The BFD can be used verbatim in replacement of BIP37, where the filter
can be cached between clients without needing to be recomputed. It can
also be used by normal pruned nodes to do re-scans locally of their
wallet without needing to have the block data available to scan, or
without reading the entire block chain from disk.
-
For improved probabilistic security the bloom filters can be presented
to lightweight clients by semi-trusted oracles. A client wallet makes
an assumption that they trust a set, or subset of remote parties
(wallet vendors, services) which all all sign the BFD for each block.
The BFD can be downloaded from a single remote source, and the hash of
the filters compared against others in the trust set. Agreement is a
weak suggestion that the filter has not been tampered with, assuming
that these parties are not conspiring to defraud the client.
The oracles do not learn any additional information about the client
wallet, the client can download the block data from either nodes on
the network, HTTP services, NTTP, or any other out of band
communication method that provides the privacy desired by the client.
-
The security model of the oracle bloom filter can be vastly improved
by instead committing a hash of the BFD inside every block as a soft-
fork consensus rule change. After this, every node in the network would
build the filter and validate that the hash in the block is correct,
then make a conscious choice discard it for space savings or cache the
data to disk.
With a commitment to the filter it becomes impossible to lie to
lightweight clients by omission. Lightweight clients are provided with
a block header, merkle path, and the BFD. Altering the BFD invalidates
the merkle proof, it's validity is a strong indicator that the client
has an unadulterated picture of the UTXO condition without needing to
build one itself. A strong assurance that the hash of the BFD means
that the filters can be downloaded out of band along with the block
data at the leisure of the client, allowing for significantly greater
privacy and taking load away from the P2P Bitcoin network.
Committing the BFD is not a hard forking change, and does not require
alterations to mining software so long as the coinbase transaction
scriptSig is not included in the bloom filter.
[0] https://github.com/bitcoin/bips/blob/master/bip-0037.mediawiki
[1] https://eprint.iacr.org/2014/763.pdf
[2] https://jonasnick.github.io/blog/2015/02/12/privacy-in-bitcoinj/
[3] https://github.com/petertodd/bloom-io-attack
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
bfd--- via bitcoin-dev
2017-04-01 23:49:03 UTC
Permalink
Post by Chris Belcher via bitcoin-dev
I think this committed bloom filter idea is very good and much better
than bip37, but for good privacy for when bitcoin is used often still
requires certain behavior namely downloading blocks
from many different peers with new tor circuits.
Note that I've been dealing with counting transaction subgraphs but
actually finding them from blocks might also be computationally
infeasible. Although a Bayesian approach worked very
well for similar transaction subgraph linking
[https://arxiv.org/pdf/1612.06747v3.pdf]
It would also be interesting to analyze what information a spy can get
if they are missing some blocks that the wallet downloaded.
For the long term, private and high-volume bitcoin use will be best
served by off-chain transactions. They will probably be a huge win just
because the large and public blockchain is such a non-private
way of doing things.
Thank you for the analysis, this generally matches my views about the
properties offered by the system.

I've generally developed the opinion that BIP37 is effectively unused
by all but a very small number of wallets and services now, setting up
sinkhole nodes in the network to monitor `filterload` commands seems
to back that up.

Loading...