Discussion:
Libconsensus separated repository (was Bitcoin Core and hard forks)
(too old to reply)
Jorge Timón via bitcoin-dev
2015-07-23 14:30:06 UTC
Permalink
On Thu, Jul 23, 2015 at 2:49 AM, Eric Voskuil via bitcoin-dev
Only being partly serious - I strongly am in favor of a sufficiently
modularized codebase that swapping out consensus rules is fairly
straightforward and easy to test...
We (libbitcoin) have taken the time to publish and maintain bitcoind's
"libbitcoinconsensus" source files as an independent C++ library (with
Java and Python bindings).
https://en.bitcoin.it/wiki/Libbitcoin_Consensus
It can be easily verified against bitcoind sources and in builds of
libbitcoin-blockchain it can be swapped out for libbitcoin's native
consensus checks.
https://en.bitcoin.it/wiki/Libbitcoin_Blockchain#Consensus_Validation
So there is really no reason to consider the original client synonymous
with consensus. I initially argued for this library to be natively
isolated from bitcoind, but that didn't seem to be in the cards so we
did it independently.
I think there were some misunderstandings in our previous conversation
about this topic.
I completely agree with having a separated repository for libconsensus
(that's the whole point, alternative implementations can be
consensus-safe by using it, and in the event of a schism fork[1], they
can fork just that smaller project without having to relay on Bitcoin
Core [satoshi] at all).
But I thought you also wanted Bitcoin Core to use libconsensus instead
of just having a subtree/subrepository like it currently does with
libsecp256k1.
I'm not sure if that would ever be accepted, but in any case we're
certainly far away from that goal. Here are some things that need to
happen first:

1) Finish encapsulating consensus code so that it can be built without
any (we've done it only with script-related code so far). Here are
some related PRs (other people havee done other things that help with
this as well):

** MERGED or DELETED
*** MERGED Consensus: Decouple pow from chainparams #5812 [consensuspow]
*** DELETED MOVEONLY: Move constants and globals to consensus.h and
policy.o #5696 [consensus_policy0]
*** DELETED Refactor: Create CCoinsViewEfficient interface for
CCoinsViewCache #5747 [coins]
*** MERGED Chainparams: Refactor: Decouple IsSuperMajority from
Params() #5968 [params_consensus]
*** MERGED Remove redundant getter
CChainParams::SubsidyHalvingInterval() #5996 [params_subsidy]
*** MERGED Separate CValidationState from main #5669 [consensus]
*** DELETED Consensus: Refactor: Separate CheckFinalTx from
main::IsFinalTx #6063 [consensus_finaltx]
*** MERGED Consensus: Decouple ContextualCheckBlockHeader from
checkpoints #5975 [consensus_checkpoints]
*** MERGED Separate Consensus::CheckTxInputs and GetSpendHeight in
CheckInputs #6061 [consensus_inputs]
*** MERGED Bugfix: Don't check the genesis block header before
accepting it #6299 [5975-quick-fix]
** REVIEW Optimizations: Consensus: In AcceptToMemoryPool,
ConnectBlock, and CreateNewBlock #6445 [consensus-txinputs-0.12.99]
** REBASE MOVEONLY: Move most of consensus functions (pre-block) #6051
[consensus_moveonly]
** REBASE Consensus: Refactor: Turn CBlockIndex::GetMedianTimePast
into independent function #6009 [consensus_mediantime]
** DEPENDENT Consensus: Refactor: Consensus version of
CheckBlockHeader() #6035 [consensus_checkblockheader]
** DEPENDENT Consensus: Consensus version of pow functions [consensus_pow2]

2) Finish libconsensus's API: expose more things than VerifyScript, at
the very least, also expose VerifyTx, VerifyHeader and VerifyBlock.
Feedback from alternative implementations like libbitcoin is extremely
valuable here. Some related closed-for-now PRs:

** DEPENDENT API: Expose bitcoinconsensus_verify_header() in
libconsensus #5995 [consensus_header]
** DEPENDENT API: Expose bitcoinconsensus_verify_block() in
libconsensus #5946 [consensus_tip]
** REBASE Chainparams: Explicit Consensus::Params arg in consensus
functions #6024 [params_consensus2]

3) Move libconsensus to a separate repository as a
subtree/subrepository of Bitcoin Core.

Only after all that we can discuss whether Bitcoin Core itself should
include libconsensus' code or just use its API directly.
I hope that after all this, libbitcoin also reconsiders whether to
reimplement its own libconsensus or use the "official" one directly
instead.
In any case I agree with your stated need for this isolation (if not the
means) for the reasons you state. The community needs to move beyond a
largely singular and monolithic codebase that is holding that position
in part due to fear about consensus bug forks.
I completely agree. That's the goal of libconsensus (and an
alternative implementation like libbitcoin being able to use it
without sacrificing any of its current or future design differences
from Bitcoin Core would be a sign of success in this reward).
Unfortunately any changes that touch consensus code are risky and
therefore slow. And when consensus encapsulation changes conflict with
other changes (not because the other changes need to change consensus
but because consensus code is still coupled with policy and other
bitcoind-specific code), refactors are never prioritized. Ironically,
you need to encapsulate the consensus code to avoid such conflicts,
which would make all non-consensus changes far less risky (reducing
the consensus-critical review development bottleneck).

Unfortunately and ironically again, safer, small and incremental
changes are less interesting for reviewers.
For example, I've been trying to move consensus code to the consensus
folder for a long time. The correctness of a MOVEONLY change is
trivial to review for anyone who knows how to copy/paste in its
favorite editor and how to use git diff, but will I ever get answers
to my questions in [1]?
I know there's many people who really care about this, Cory Fields,
Wladimir and Pieter Wuille to name a few have reviewed many of this
changes (I've just got used to publicly whine about lack of review on
this front and policy encapsulation [very related fronts] as an
attempt to get some attention: not always, but begging for review
actually works some times).

Another unfortunate fact is that although a script-only libconsensus
allows you to avoid a big part of all possible consensus fork bugs,
there cannot be users of a finished libconsensus to ask things to util
a finished libconsensus actually exists. At the same time, the future
users (alternative implementations, since bitcoin core is already
"using libconsensus") are the most relevant people to listen when it
comes to the C API. That's why I beg you to comment on [2], even if
#5995 is currently closed. Your input on [1] would be very appreciated
as well (maybe you think it's better to expose verifyTx before
exposing verifyHeader, even if exposing verifyHeader is something that
could be done faster).
To make choice regarding consensus an actual choice (and thereby actual
consensus) the modularity you suggest is essential. One must be able to
take new developments without having to take consensus changes. The
option to fork the codebase is not reasonable for most people. At this
point there is no defensible reason for coupling consensus checks with
other features.
Would you agree that asking people to fork an independent libconsensus
project instead of having to fork the full Bitcoin-qt is much more
reasonable?
I mean, I agree with your points. If "the specification of the
consensus rules is an implementation", then that implementation
shouldn't be coupled with a bunch of policy and non-consensus
technical choices (storage, dependencies, p2p protocol...). But I
still think that "the specification of the consensus rules should be a
concrete implementation" rather than based purely on a natural
language like English.
I believe that's the only point where we fundamentally disagree, but
it shouldn't be a barrier in our common goal of taking "power" away
from Bitcoin Core development. If we're successful Bitcoin Core won't
have any privileged position with regards to, say, libbitcoin when it
comes to deciding consensus rules changes.
You see, people like Mike Hearn believe that "uncontroversial
acceptance by Bitcoin Core devs" is the same as "uncontroversial
acceptance by all users of the system" (for a libbitcoin developer
like you, obviously a superset of Bitcoin Core's users). He thinks
that Gavin proposal is only a schism consensus fork[3] because the
code is in github/bitcoinxt/bitcoinxt instead of
github/bitcoin/bitcoin, not because PeterTodd-the-user-of-the-system
(he doesn't care about him) opposes it.
But let's imagine a different situation:

1) libconsensus us finished and used by libbitcoin
2) Bitcoin Core was unanimously in favor of Gavin's 32 GB initial
proposal and the changes are applied to bitcoin/bitcoin and
bitcoin/libconsensus (or Bitcoin Core has a dictator like Mike
wants[4] and he accepts it, it doesn't really matter for this
example).

But let's also assume that X% of the users and 10% of the miners are
against that Schism hardfork, and they don't want to be forced to
change the rules by any influential group, mining, economic or user
majority.
Libbitcoin cannot be forced to accept the next, controversial version
of bitcoin/libconsensus, so you guys fork libbitcoin/libconsensus out
of the last ok version.
Centralized-bitcoin and old-bitcoin would become 2 separated
currencies and some people would likely lose money in the transition
from one currency to 2 of them, but the users of old-bitcoin have the
right of keeping the rules they signed up for and the only responsible
people for this likely-catastrophic schism would be the Bitcoin core
developers for trying to impose consensus changes into others against
their will. Trying to impose consensus changes against the will of
some users is wrong, and it is irrelevant if that happens in Bitcoin
Core or Bitcoin Tx (if it is uncontroversial, it's also irrelevant
where it gets implemented first).

I really believe bitcoin needs competitive alternative implementations
and I believe libconsensus is a tool to help that happen and reduce
the "gatekeeping" friction that there's (unfortunately) around Bitcoin
Core. I look forward to any potential collaboration on this front.
Even if you still want to maintain a reimplementation of libconsensus
(which I humbly think it's a mistake, but I don't think there's any
point on keep discussing that, since we know we disagree) we can
collaborate on the future common API of a complete libconsensus (with
verifyBlock and all). I really hope we can do that.

[1] https://github.com/bitcoin/bitcoin/pull/6051#issuecomment-120708121

[2] (kind of outdated, but the API is what matters here)
https://github.com/jtimon/bitcoin/commit/00b9b227afc8669a877984561329dde75d3d8942

[3] https://github.com/jtimon/bips/blob/bip-forks/bip-forks.org#schism1-hardforks

[4] http://lmgtfy.com/?q=mike+hearn+dictator&l=1
Milly Bitcoin via bitcoin-dev
2015-07-23 14:57:38 UTC
Permalink
Post by Jorge Timón via bitcoin-dev
[4] http://lmgtfy.com/?q=mike+hearn+dictator&l=1
_______________________________________________
You spend too much time on reddit. All this drama queen stuff is
getting ridiculous.


Russ
Jorge Timón via bitcoin-dev
2015-07-23 21:02:20 UTC
Permalink
On Thu, Jul 23, 2015 at 4:57 PM, Milly Bitcoin via bitcoin-dev
Post by Milly Bitcoin via bitcoin-dev
Post by Jorge Timón via bitcoin-dev
[4] http://lmgtfy.com/?q=mike+hearn+dictator&l=1
Mike has sincerely said that he would like "Bitcoin Core to have a
benevolent dictator like other free software projects", and I wanted
to make clear that I wasn't putting words in his mouth but it's
actually something very easy to find on the internet. But I now
realize that the search can be interpreted as me calling him dictator
or something of the sort. That wasn't my intention. In fact, Mike's
point of view on Bitcoin Core development wasn't even relevant for my
example so I shouldn't even have mentioned him in the first place. I
apologize for both mistakes, but please let's keep this thread focused
on libconsensus.
Post by Milly Bitcoin via bitcoin-dev
You spend too much time on reddit.
I actually don't spend much time on reddit: I don't particularly like
it. But I do spend some time in reddit so, I agree: I spend too much
time on reddit.
Milly Bitcoin via bitcoin-dev
2015-07-23 21:30:18 UTC
Permalink
Post by Jorge Timón via bitcoin-dev
Mike has sincerely said that he would like "Bitcoin Core to have a
benevolent dictator like other free software projects", and I wanted
to make clear that I wasn't putting words in his mouth
He is just pointing out reality. Decentralization is really just a
collection of centralized processes. If you ever want to change anything
someone, somewhere has to make a decision.

Satoshi was the first dictator and he set the consensus rules for his
version of Bitcoin and that is what people accepted. Those rules did not
appear by magic nor did they appear via some unexplained 'consensus'
mechanism. He made decisions and updated the code or accepted updates
made by others.

Russ
Eric Voskuil via bitcoin-dev
2015-07-28 06:40:42 UTC
Permalink
Post by Jorge Timón via bitcoin-dev
Only being partly serious - I strongly am in favor of a sufficiently
modularized codebase that swapping out consensus rules is fairly
straightforward and easy to test...
We (libbitcoin) have taken the time to publish and maintain bitcoind's
"libbitcoinconsensus" source files as an independent C++ library...
I think there were some misunderstandings in our previous conversation
about this topic.
I completely agree with having a separated repository for libconsensus
(that's the whole point, alternative implementations can be
consensus-safe by using it, and in the event of a schism fork[1], they
can fork just that smaller project without having to relay on Bitcoin
Core [satoshi] at all).
But I thought you also wanted Bitcoin Core to use libconsensus instead
of just having a subtree/subrepository like it currently does with
libsecp256k1.
libsecp256k1 has it's own repository, libbitcoinconsensus doesn't. A
separate repository was what I considered as a requirement for us to use it.
Post by Jorge Timón via bitcoin-dev
I'm not sure if that would ever be accepted, but in any case we're
certainly far away from that goal.
If it's not certain whether this would even be accepted, the commitment
to a community consensus library is too weak to take a strong dependency
on. But for us it's moot, as we have made the already accomplished that
goal.
Post by Jorge Timón via bitcoin-dev
Here are some things that need to
1) Finish encapsulating consensus code so that it can be built without
any (we've done it only with script-related code so far). Here are
some related PRs (other people have done other things that help with
...
Post by Jorge Timón via bitcoin-dev
2) Finish libconsensus's API: expose more things than VerifyScript, at
the very least, also expose VerifyTx, VerifyHeader and VerifyBlock.
Feedback from alternative implementations like libbitcoin is extremely
In our earlier discussion I believe you said that the library would not
be undergoing significant change or feature creep. If this is the very
least that's projected it would seem that constraint will not hold.
Post by Jorge Timón via bitcoin-dev
3) Move libconsensus to a separate repository as a
subtree/subrepository of Bitcoin Core.
Only after all that we can discuss whether Bitcoin Core itself should
include libconsensus' code or just use its API directly.
I don't think it's a question of whether it *should* use its own library
as it is published for others - this is a practically self-evident
conclusion.
Post by Jorge Timón via bitcoin-dev
I hope that after all this, libbitcoin also reconsiders whether to
reimplement its own libconsensus or use the "official" one directly
instead.
We use a fork of libsecp256k1 and would probably do the same with the
consensus library if it was cleanly isolated.
Post by Jorge Timón via bitcoin-dev
In any case I agree with your stated need for this isolation (if not the
means) for the reasons you state. The community needs to move beyond a
largely singular and monolithic codebase that is holding that position
in part due to fear about consensus bug forks.
I completely agree. That's the goal of libconsensus (and an
alternative implementation like libbitcoin being able to use it
without sacrificing any of its current or future design differences
from Bitcoin Core would be a sign of success in this reward).
It's a performance sacrifice, and then there's the OpenSSL dependency,
but these are both optional within our stack - so the application
developer has the option. So the only downside is that we are
maintaining the conditional compilation.
Post by Jorge Timón via bitcoin-dev
Unfortunately any changes that touch consensus code are risky and
therefore slow. And when consensus encapsulation changes conflict with
other changes (not because the other changes need to change consensus
but because consensus code is still coupled with policy and other
bitcoind-specific code), refactors are never prioritized. Ironically,
you need to encapsulate the consensus code to avoid such conflicts,
which would make all non-consensus changes far less risky (reducing
the consensus-critical review development bottleneck).
Unfortunately and ironically again, safer, small and incremental
changes are less interesting for reviewers.
For example, I've been trying to move consensus code to the consensus
folder for a long time. The correctness of a MOVEONLY change is
trivial to review for anyone who knows how to copy/paste in its
favorite editor and how to use git diff, but will I ever get answers
to my questions in [1]?
I think it's worthwhile work, especially if you are passionate about the
longer term objectives. I haven't been involved in these reviews as I
spend very little time with the satoshi client
Post by Jorge Timón via bitcoin-dev
I know there's many people who really care about this, Cory Fields,
Wladimir and Pieter Wuille to name a few have reviewed many of this
changes (I've just got used to publicly whine about lack of review on
this front and policy encapsulation [very related fronts] as an
attempt to get some attention: not always, but begging for review
actually works some times).
Well a cynic might observe that fear of consensus bugs is what keeps
people on the satoshi client, and therefore accelerating the development
of a clean and independent consensus library would be a very low priority.
Post by Jorge Timón via bitcoin-dev
Another unfortunate fact is that although a script-only libconsensus
allows you to avoid a big part of all possible consensus fork bugs,
there cannot be users of a finished libconsensus to ask things to until
a finished libconsensus actually exists.
Software is never finished, but this exists and we are using it.
Post by Jorge Timón via bitcoin-dev
At the same time, the future
users (alternative implementations, since bitcoin core is already
"using libconsensus")
It is using the same source files, but AFAICT not the library.
Post by Jorge Timón via bitcoin-dev
are the most relevant people to listen when it
comes to the C API. That's why I beg you to comment on [2], even if
#5995 is currently closed. Your input on [1] would be very appreciated
as well (maybe you think it's better to expose verifyTx before
exposing verifyHeader, even if exposing verifyHeader is something that
could be done faster).
I haven't looked at any of these commits, but I'll make some time to at
least give a cursory review.
Post by Jorge Timón via bitcoin-dev
To make choice regarding consensus an actual choice (and thereby actual
consensus) the modularity you suggest is essential. One must be able to
take new developments without having to take consensus changes. The
option to fork the codebase is not reasonable for most people. At this
point there is no defensible reason for coupling consensus checks with
other features.
Would you agree that asking people to fork an independent libconsensus
project instead of having to fork the full Bitcoin-qt is much more
reasonable?
Yes, of course. We've already done it. For each release of the satoshi
client since we made libbitcoin-consensus I've copied the sources. It's
pretty much automated and easy to visually verify that the sources
match. That would be quite a bit more difficult if there wasn't an
independent build.
Post by Jorge Timón via bitcoin-dev
I mean, I agree with your points. If "the specification of the
consensus rules is an implementation", then that implementation
shouldn't be coupled with a bunch of policy and non-consensus
technical choices (storage, dependencies, p2p protocol...). But I
still think that "the specification of the consensus rules should be a
concrete implementation" rather than based purely on a natural
language like English.
Useful specifications often have two reference implementations. It's the
idea that there can be only one legitimate implementation that's
problematic.
Post by Jorge Timón via bitcoin-dev
I believe that's the only point where we fundamentally disagree, but
it shouldn't be a barrier in our common goal of taking "power" away
from Bitcoin Core development. If we're successful Bitcoin Core won't
have any privileged position with regards to, say, libbitcoin when it
comes to deciding consensus rules changes.
I don't think we disagree on anything fundamental. My issues with the
library were (1) the lack of isolation, (2) the fact that the satoshi
client wouldn't actually use the library, and (3) backtracking to use
OpenSSL, which we had recently removed from libbitcoin.

..
Post by Jorge Timón via bitcoin-dev
1) libconsensus us finished and used by libbitcoin
2) Bitcoin Core was unanimously in favor of Gavin's 32 GB initial
proposal and the changes are applied to bitcoin/bitcoin and
bitcoin/libconsensus (or Bitcoin Core has a dictator like Mike
wants[4] and he accepts it, it doesn't really matter for this
example).
But let's also assume that X% of the users and 10% of the miners are
against that Schism hardfork, and they don't want to be forced to
change the rules by any influential group, mining, economic or user
majority.
Libbitcoin cannot be forced to accept the next, controversial version
of bitcoin/libconsensus, so you guys fork libbitcoin/libconsensus out
of the last ok version.
This is already done.
Post by Jorge Timón via bitcoin-dev
Centralized-bitcoin and old-bitcoin would become 2 separated
currencies and some people would likely lose money in the transition
from one currency to 2 of them, but the users of old-bitcoin have the
right of keeping the rules they signed up for and the only responsible
people for this likely-catastrophic schism would be the Bitcoin core
developers for trying to impose consensus changes into others against
their will. Trying to impose consensus changes against the will of
some users is wrong, and it is irrelevant if that happens in Bitcoin
Core or Bitcoin Tx (if it is uncontroversial, it's also irrelevant
where it gets implemented first).
I don't disagree, but you previously argued that *everyone* had to agree
on consensus. Above you are making the argument that people can choose
to disagree. Yes, this is important. Yet it's unrealistic for any
alternative consensus to overcome inertia unless there are widely
deployed independent implementations.
Post by Jorge Timón via bitcoin-dev
I really believe bitcoin needs competitive alternative implementations
and I believe libconsensus is a tool to help that happen and reduce
the "gatekeeping" friction that there's (unfortunately) around Bitcoin
Core. I look forward to any potential collaboration on this front.
Even if you still want to maintain a reimplementation of libconsensus
(which I humbly think it's a mistake, but I don't think there's any
point on keep discussing that, since we know we disagree) we can
collaborate on the future common API of a complete libconsensus (with
verifyBlock and all). I really hope we can do that.
I'm maintain our fork of the consensus sources until they are (1)
properly isolated from the satoshi client and (2) the satoshi client
uses the actual library.

I don't see it as important or even productive to try and wedge the
consensus library into the repository/build for bitcoind and Bitcoin-QT.
It's straightforward to maintain the consensus library independently.
Always willing to work with you on it, although we're all busy, and this
isn't my top priority presently.

e
Wladimir J. van der Laan via bitcoin-dev
2015-07-28 08:47:19 UTC
Permalink
Post by Eric Voskuil via bitcoin-dev
It's a performance sacrifice, and then there's the OpenSSL dependency,
but these are both optional within our stack - so the application
developer has the option. So the only downside is that we are
maintaining the conditional compilation.
Now that BIP66 became active, and only strict DER signatures are allowed, the OpenSSL dependency can be removed from consensus.

Pieter Wuille will do an announcement on this soon.

Wladimir
Jorge Timón via bitcoin-dev
2015-07-28 09:58:33 UTC
Permalink
Ok, I'm going to separate terms: current-libconsensus from theoretical
future-libconsensus (implementing ALL consensus rules).
Post by Eric Voskuil via bitcoin-dev
libsecp256k1 has it's own repository, libbitcoinconsensus doesn't. A
separate repository was what I considered as a requirement for us to use it.
We want to complete future-libconsensus (decouple all the consensus
rules from the rest of the bitcoin core code) first.
Then we can move future-libconsensus to a subrepository/subtree like
libsecp256k1 and I believe everybody wants this to eventually happen.
Separating current-libconsensus now may make completing
future-libconsensus harder.
Post by Eric Voskuil via bitcoin-dev
Post by Jorge Timón via bitcoin-dev
I'm not sure if that would ever be accepted, but in any case we're
certainly far away from that goal.
If it's not certain whether this would even be accepted, the commitment
to a community consensus library is too weak to take a strong dependency
on. But for us it's moot, as we have made the already accomplished that
goal.
What I mean is that once it is separated to a subtree, there's one more step:

Make Bitcoin Core use future-libconsensus' API instead of a subtree.
Decoupling future-libconsensus from Bitcoin Core is one thing, and
Decoupling Bitcoin Core from future-libconsensus is another thing: you
need to decouple Bitcoin Core from all future-libconsensus
implementation internals. For example, script/sign (part of Bitcoin
Core) depends on individual non-API-exposed classes in
current-libconsensus.
Moving from a subtree to a completely separated library is what I
don't know will ever happen, but I don't think this is "unfairly
advantageous" for Bitcoin Core or anything like that: other
implementations can also use future-libcosensus as a subtree instead
of the C API as well.

You have accomplished the goal of separating curren-libconsensus, not
future-libconsensus.
In fact, if you complete the equivalent of future-libconsensus in
libbitcoin and separate that, maybe that's a better place to start
drafting a full API.
Post by Eric Voskuil via bitcoin-dev
Post by Jorge Timón via bitcoin-dev
2) Finish libconsensus's API: expose more things than VerifyScript, at
the very least, also expose VerifyTx, VerifyHeader and VerifyBlock.
Feedback from alternative implementations like libbitcoin is extremely
In our earlier discussion I believe you said that the library would not
be undergoing significant change or feature creep. If this is the very
least that's projected it would seem that constraint will not hold.
future-libconsensus will not have significant changes *once it is
completed*. Currently future-libconsensus is spread around many places
inclusing src/main, so that obviously needs to change before it can be
separated to an independent repo.
Post by Eric Voskuil via bitcoin-dev
Post by Jorge Timón via bitcoin-dev
3) Move libconsensus to a separate repository as a
subtree/subrepository of Bitcoin Core.
Only after all that we can discuss whether Bitcoin Core itself should
include libconsensus' code or just use its API directly.
I don't think it's a question of whether it *should* use its own library
as it is published for others - this is a practically self-evident
conclusion.
Well, Bitcoin Core is "currently the only user of future-libconsensus"
since bitcoin core and future-libconsensus are currently mutually
coupled.
Bitcoin Core will always keep using future-libconsensus. The only
question is whether it will use it through the C API or as a
subtree/subrepository (both options are also available to other
implementations). I don't know if decoupling Bitcoin Core from
future-libconsensus' implementation details enough to be able to
directly use the API is worth it or if anyone will be interested in
doing so. For me this last step is not all that interesting: if we
have an independent repo with a full API that other implementations
can use, I don't really care about Bitcoin Core not going through the
API and using including all the code directly instead.
Post by Eric Voskuil via bitcoin-dev
Post by Jorge Timón via bitcoin-dev
I hope that after all this, libbitcoin also reconsiders whether to
reimplement its own libconsensus or use the "official" one directly
instead.
We use a fork of libsecp256k1 and would probably do the same with the
consensus library if it was cleanly isolated.
Great.
Post by Eric Voskuil via bitcoin-dev
Post by Jorge Timón via bitcoin-dev
In any case I agree with your stated need for this isolation (if not the
means) for the reasons you state. The community needs to move beyond a
largely singular and monolithic codebase that is holding that position
in part due to fear about consensus bug forks.
I completely agree. That's the goal of libconsensus (and an
alternative implementation like libbitcoin being able to use it
without sacrificing any of its current or future design differences
from Bitcoin Core would be a sign of success in this reward).
It's a performance sacrifice, and then there's the OpenSSL dependency,
but these are both optional within our stack - so the application
developer has the option. So the only downside is that we are
maintaining the conditional compilation.
As I told you before the reason why current-libconsensus is using
OpenSSL instead of libsecp256k1 is that the very authors of
libsecp256k1 warned that using libsecp256k1 for validation was
consensus risky. As Wladimir said, Pieter Wuille will make an
announcement about this soon.
In any case, as I told you in previous conversations, the plan is to
move from OpenSSL to libsecp256k1 for validation too (so libconsensus
wil drop the OpenSSL dependency and this is just a temporary concern).
Post by Eric Voskuil via bitcoin-dev
I think it's worthwhile work, especially if you are passionate about the
longer term objectives. I haven't been involved in these reviews as I
spend very little time with the satoshi client
Well, the questions about the API are just in english, no need to
deeply know Bitcoin Core's (satoshi client) internals.
But maybe we should have an independent mailing list for
consensus-only things. Not all future-libconsensus users will be
interested in Bitcoin Core-specific discussions, and making them
subscribe and filter seems like an unnecessary burden to
participation.
Post by Eric Voskuil via bitcoin-dev
Well a cynic might observe that fear of consensus bugs is what keeps
people on the satoshi client, and therefore accelerating the development
of a clean and independent consensus library would be a very low priority.
I think that's precisely what makes it a high priority in the eyes of
all the people working on it or reviewing related changes.
But, yes, I guess "evil-thinking", maybe that's what make it a low
priority for someone evil that wants Bitcoin Core's implementation
have more importance than it shold forever. I prefer not to evil-think
and just attribute it to having other priorities or just apathy about
it.
Post by Eric Voskuil via bitcoin-dev
Post by Jorge Timón via bitcoin-dev
Another unfortunate fact is that although a script-only libconsensus
allows you to avoid a big part of all possible consensus fork bugs,
there cannot be users of a finished libconsensus to ask things to until
a finished libconsensus actually exists.
Software is never finished, but this exists and we are using it.
By "finished" I mean a future-libconsensus that implements ALL
consensus rules. We don't have that yet.
Post by Eric Voskuil via bitcoin-dev
Post by Jorge Timón via bitcoin-dev
At the same time, the future
users (alternative implementations, since bitcoin core is already
"using libconsensus")
It is using the same source files, but AFAICT not the library.
Bitcoin Core is the ONLY "user" of future-libconsensus (which actually
only exists inside Bitcoin Core and it's not exposed).
Current-libconsensus is used by Bitcoin Core and also exposed as an
independent build (not a separated repository yet).

Once future-Bitcoin's API is completed and the code in a different
repo, how is Bitcoin Core using the API instead of the sources
directly of any importance to other implementations?
That's really the part that I cannot understand. It will be a problem
Bitcoin Core, but if other implementations want to have (and maybe
solve later) the same problem they can use a subtree too and start
coupling their code with implementations details from
future-libconsensus.
Why would they want to do that? Again, I have no idea. I don't
understand what the complain is here.
Post by Eric Voskuil via bitcoin-dev
Post by Jorge Timón via bitcoin-dev
are the most relevant people to listen when it
comes to the C API. That's why I beg you to comment on [2], even if
#5995 is currently closed. Your input on [1] would be very appreciated
as well (maybe you think it's better to expose verifyTx before
exposing verifyHeader, even if exposing verifyHeader is something that
could be done faster).
I haven't looked at any of these commits, but I'll make some time to at
least give a cursory review.
Great. I mean, I wasn't asking about reviewing the commits themselves
(which is also great if you do), but rather on answering the questions
I'm making there, ie: what to expose next (ie VerifyTx or
VerifyHeader)? would this be an acceptable way to expose VerifyHeader
? Which of he step-checks functions is worth exposing too (Bitcoin
Core is currently using some to prevent DoS attacks, for example)?
Post by Eric Voskuil via bitcoin-dev
Post by Jorge Timón via bitcoin-dev
Would you agree that asking people to fork an independent libconsensus
project instead of having to fork the full Bitcoin-qt is much more
reasonable?
Yes, of course. We've already done it. For each release of the satoshi
client since we made libbitcoin-consensus I've copied the sources. It's
pretty much automated and easy to visually verify that the sources
match. That would be quite a bit more difficult if there wasn't an
independent build.
Well, neither libbitcoincosnensus nor libbitcoin-consensus implements
all the consensus rules.
That's what makes them different from future-libconsensus.
But great, we're confirming more views that we share.
Post by Eric Voskuil via bitcoin-dev
Useful specifications often have two reference implementations. It's the
idea that there can be only one legitimate implementation that's
problematic.
Well, this is where I fear we will never agree. I think "Bitcoin is
different" in this reward and you disagree.
Maybe Pieter's explanation is more convincing to you:

Otherwise, I think I'll stop trying convincing you.
But one last try:

If there's 2 "specification implementations":

1) widely deployed but containing 1 bug
2) not deployed anywhere but more readable and without the bug.

When the bug is found, is it a consensus rule or not?
If it is, it turns out the second implementation wasn't an specification at all.
If it's not, nobody has been ever following consensus rules!!
Post by Eric Voskuil via bitcoin-dev
Post by Jorge Timón via bitcoin-dev
I believe that's the only point where we fundamentally disagree, but
it shouldn't be a barrier in our common goal of taking "power" away
from Bitcoin Core development. If we're successful Bitcoin Core won't
have any privileged position with regards to, say, libbitcoin when it
comes to deciding consensus rules changes.
I don't think we disagree on anything fundamental. My issues with the
library were (1) the lack of isolation, (2) the fact that the satoshi
client wouldn't actually use the library, and (3) backtracking to use
OpenSSL, which we had recently removed from libbitcoin.
1) Working on it
2) The Satohi client has been using all along and it will use it
forever (maybe not through the API, but I don't get what the problem
with that is).
3) There will be an announce about this soon.
Post by Eric Voskuil via bitcoin-dev
Post by Jorge Timón via bitcoin-dev
Libbitcoin cannot be forced to accept the next, controversial version
of bitcoin/libconsensus, so you guys fork libbitcoin/libconsensus out
of the last ok version.
This is already done.
Once future-libconsensus exists as a separated repository, I don't
think you want to preemptively fork it unless you're actually changing
something.
Post by Eric Voskuil via bitcoin-dev
I don't disagree, but you previously argued that *everyone* had to agree
on consensus. Above you are making the argument that people can choose
to disagree. Yes, this is important. Yet it's unrealistic for any
alternative consensus to overcome inertia unless there are widely
deployed independent implementations.
Ideally schism consensus-rules changes will never happen, since they
effectively divide the currency in 2 and force the users to chose
(apart from other complications).
But if someone is trying to impose a schism fork (from Bitcoin Core,
from the new future-lbiconsensus repo or from anywhere else) it
becomes trivial to protect your implementation against forced changes.
Ideally all consensus changes will be uncontroversial or not happen at all.
But I think about non-ideal cases too.
Post by Eric Voskuil via bitcoin-dev
I'm maintain our fork of the consensus sources until they are (1)
properly isolated from the satoshi client and (2) the satoshi client
uses the actual library.
1) Reasonable.
2) You mean use it through the API? Seriously, why do you care?
Post by Eric Voskuil via bitcoin-dev
I don't see it as important or even productive to try and wedge the
consensus library into the repository/build for bitcoind and Bitcoin-QT.
It's straightforward to maintain the consensus library independently.
To be clear, Bitcoin Core's using future-libconsensus through a
subtree instead of the API is not a preference or a goal: it is just
how things will be just after completing and separating
future-libconsensus. Making it use the API instead of the subtree will
be additional work. I'm not sure I will want/have time to do it and I
don't know of anyone planning to do that (which seems very reasonably
given that a separated future-libconsensus is a dependency for such a
change).
Post by Eric Voskuil via bitcoin-dev
Always willing to work with you on it, although we're all busy, and this
isn't my top priority presently.
Is it because "fear of consensus bugs is what keeps people on the
satoshi client" and you want to keep things this way?
Sorry, just joking about your previous "cynic" comment. See? Don't
attribute to malice what can be attributed to lack of time.
Eric Voskuil via bitcoin-dev
2015-07-29 20:38:39 UTC
Permalink
Post by Jorge Timón via bitcoin-dev
Post by Eric Voskuil via bitcoin-dev
I haven't looked at any of these commits, but I'll make some time to at
least give a cursory review.
Great. I mean, I wasn't asking about reviewing the commits themselves
(which is also great if you do), but rather on answering the questions
I'm making there, ie: what to expose next (ie VerifyTx or
VerifyHeader)?
Oh, I misunderstood your ask then. I don't have a preference on
prioritizing VerifyTx vs VerifyHeader.
Post by Jorge Timón via bitcoin-dev
would this be an acceptable way to expose
VerifyHeader?
I'm not sure how you mean to expose it, could you clarify?
Post by Jorge Timón via bitcoin-dev
Which of he step-checks functions is worth exposing too (Bitcoin
Core is currently using some to prevent DoS attacks, for example)?
I don't see any reason to expose checkpoints in this library. They are
trivial to implement and are not part of consensus.
Post by Jorge Timón via bitcoin-dev
Post by Eric Voskuil via bitcoin-dev
Post by Jorge Timón via bitcoin-dev
Would you agree that asking people to fork an independent libconsensus
project instead of having to fork the full Bitcoin-qt is much more
reasonable?
Yes, of course. We've already done it. For each release of the satoshi
client since we made libbitcoin-consensus I've copied the sources. It's
pretty much automated and easy to visually verify that the sources
match. That would be quite a bit more difficult if there wasn't an
independent build.
Well, neither libbitcoincosnensus nor libbitcoin-consensus implements
all the consensus rules.
That's what makes them different from future-libconsensus.
But great, we're confirming more views that we share.
Nothing can eliminate all consensus risk, not even a common full node
implementation.
Post by Jorge Timón via bitcoin-dev
Post by Eric Voskuil via bitcoin-dev
Useful specifications often have two reference implementations. It's the
idea that there can be only one legitimate implementation that's
problematic.
Well, this is where I fear we will never agree. I think "Bitcoin is
different" in this reward and you disagree.
http://youtu.be/PxW5D9xCIsc
Otherwise, I think I'll stop trying convincing you.
Maybe I wasn't sufficiently explicit. It is problematic. That is the
core issue we are dealing with. That doesn't mean I disagree with the
objectives of an independent community consensus library.

The premise of the "one true library" idea is that there is *no way* to
sufficiently test for consensus bugs in any software release. That of
course means that each release of the satoshi client poses a significant
risk to the network. This risk is presently greater than that posed by
other implementations simply because of adoption. That is the basis of
the red herring argument:

https://blog.conformal.com/the-bitcoin-consensus-red-herring

The bottom line is that nobody has control over this process. There are,
and will always be, a multitude of consensus implementations that intend
to target the same coin. Presently there are multiple versions of the
satoshi client, and this has produced forks, and will continue to do so.

Isolating the satoshi consensus checks to an independent library serves
not to eliminate that risk, but can reduce it somewhat. Importantly it
will allow various implementations to overcome a perception problem,
which will improve implementation diversity and developer participation.
Post by Jorge Timón via bitcoin-dev
Post by Eric Voskuil via bitcoin-dev
Post by Jorge Timón via bitcoin-dev
I believe that's the only point where we fundamentally disagree, but
it shouldn't be a barrier in our common goal of taking "power" away
from Bitcoin Core development. If we're successful Bitcoin Core won't
have any privileged position with regards to, say, libbitcoin when it
comes to deciding consensus rules changes.
I don't think we disagree on anything fundamental. My issues with the
library were (1) the lack of isolation, (2) the fact that the satoshi
client wouldn't actually use the library, and (3) backtracking to use
OpenSSL, which we had recently removed from libbitcoin.
1) Working on it
For the sake of clarity, this is now a non-issue for us.
Post by Jorge Timón via bitcoin-dev
2) The Satoshi client has been using all along and it will use it
forever (maybe not through the API, but I don't get what the problem
with that is).
Again, I consider this a requirement for us to link directly to it as a
library. If the sources are isolated into an independent repo, but the
satoshi client is embedding its own copies, one must continue to diff
the client sources against the library sources. We are doing this
already, so the benefit to having the independent repo is in no longer
having to do this.

There are also differences in the build system that can affect outcome.
Comparing those differences across repos can be more challenging. For
this reason I consider it important to your objective that the satoshi
client actually use the library - as I assume it will at some point.

If the satoshi client folks are to maintain a consensus library for the
community it's also important to show a commitment to its independence.
Dogfooding is of course a software engineering best practice. But there
is also the cynical perspective - the independent library in some ways
works against an advantage of the satoshi client.

I personally don't think the committers are parochial enough to let this
become an issue. We are all after something bigger. But if there was
push-back against using the library it would be a red flag. So until
that point passes I would just maintain our independent library, cloning
the sources from the satoshi client.
Post by Jorge Timón via bitcoin-dev
3) There will be an announce about this soon.
Yes, I've seen this as a temporary setback.

...
Post by Jorge Timón via bitcoin-dev
Post by Eric Voskuil via bitcoin-dev
Always willing to work with you on it, although we're all busy, and this
isn't my top priority presently.
Is it because "fear of consensus bugs is what keeps people on the
satoshi client" and you want to keep things this way?
No, I see it as less significant to the adoption of libbitcoin-server
than other issues we are working on, especially given the existence of
libbitcoin-consensus. I also trust you will make progress regardless.

e
Jorge Timón via bitcoin-dev
2015-07-29 21:46:49 UTC
Permalink
Post by Eric Voskuil via bitcoin-dev
Oh, I misunderstood your ask then. I don't have a preference on
prioritizing VerifyTx vs VerifyHeader.
Ok, let's assume we want to expose verifyHeader first (which I think
will be easier).
Post by Eric Voskuil via bitcoin-dev
Post by Jorge Timón via bitcoin-dev
would this be an acceptable way to expose
VerifyHeader?
I'm not sure how you mean to expose it, could you clarify?
In https://github.com/bitcoin/bitcoin/pull/5995 I had one (probably
stupid) proposal.
But it had so many preparations commits that I had to close it.
In the last commit
https://github.com/jtimon/bitcoin/commit/00b9b227afc8669a877984561329dde75d3d8942
you can see that I'm adding a new function in
script/bitcoinconsensus.cpp with the following declaration:

int bitcoinconsensus_verify_header(const unsigned char* blockHeader,
unsigned int blockHeaderLen,
const Consensus::Params& params, int64_t nTime, CBlockIndexBase*
pindexPrev, PrevIndexGetter indexGetter,
bitcoinconsensus_error* err)

The ugly parts that you may not like are the CBlockIndexBase struct
(or maybe it's not so unreasonable) and the function pointer
PrevIndexGetter.
To see their "ugliness" you can look at:

https://github.com/jtimon/bitcoin/commit/4528ec69617f1b6d6c8f0d73dc4091cded7c216c

The PrevIndexGetter function pointer that Bitcoin Core would use
internally would be:

const CBlockIndexBase* GetPrevIndex(const CBlockIndexBase* pindex)
{
return ((CBlockIndex*)pindex)->pprev;
}

with an ugly casting. But, well, I guess that's only ugly for Bitcoin
Core, not necessarily for other libconsensus users, which can define
their own function pointer, provided that it's of the form:

typedef const CBlockIndexBase* (*PrevIndexGetter)(const CBlockIndexBase*);

The struct that I think needs more refinement (and I just used what I
considered easier to implement at the time) is the CBlockIndexBase
struct itself:

+struct CBlockIndexBase
+{
+ //! pointer to the hash of the block, if any. Memory is owned by
this CBlockIndexBase
+ const uint256* phashBlock;
+ //! block header
+ int32_t nVersion;
+ uint256 hashMerkleRoot;
+ uint32_t nTime;
+ uint32_t nBits;
+ uint32_t nNonce;
+ //! height of the entry in the chain. The genesis block has height 0
+ int nHeight;
+};

I don't like phashBlock being a pointer instead of just a ref or even an object
Should that struct have a CBlockIndexBase* pprev; field (moving it
down from CBlockIndex)?
That's the kind of question where your feedback seems very important
from other-implementations developers (because you won't necessarily
take into account the difficulty of the refactors required in Bitcoin
Core to expose the right interface, and "libconsensus shouldn't care"
either, all we want is the best interface).
Post by Eric Voskuil via bitcoin-dev
Post by Jorge Timón via bitcoin-dev
Which of he step-checks functions is worth exposing too (Bitcoin
Core is currently using some to prevent DoS attacks, for example)?
I don't see any reason to expose checkpoints in this library. They are
trivial to implement and are not part of consensus.
Agreed, and I would say all of the checkpoint check separation has
been done already.
What I mean by step functions is...look at verfyHeader internals, for example:

https://github.com/jtimon/bitcoin/commit/11ede96f59f611ede596a1335e896b1fef4fb5b2

It internally calls Consensus::CheckBlockHeader (quite cheap with no
context required) and Consensus::ContextualCheckBlockHeader (not so
cheap).
Bitcoin Core never calls (yet) the full verifyHeader at once. It does
the cheap tests first and the expensive later. For example,

call CheckBlockHeader, then CheckBlock (also cheap), then
ContextualCheckBlockHeader and then ContextualCheckBlock.

The question is, will other implementations want access to these
not-full-but-cheap tests?
In other words, apart from exposing VerifyHeader that fully validates
all consensus rules for a header, do we also want to expose
CheckBlockHeader and ContextualCheckBlockHeader to give more
flexibility to libconsensus' users?

I think, yes, other implementations will want this for the same DoS
reasons that Bitcoin Core currently wants them. But it would be nice
to know what a second person thinks about this.
Post by Eric Voskuil via bitcoin-dev
Nothing can eliminate all consensus risk, not even a common full node
implementation.
In fact, one thing does: never changing the code again (but the cure
would be worse than the illness).
Agreed, any software changes in the consensus code can cause consensus
forks (and that's why you don't want to touch libconsensus that much
once it's separated).
Post by Eric Voskuil via bitcoin-dev
Post by Jorge Timón via bitcoin-dev
Post by Eric Voskuil via bitcoin-dev
Useful specifications often have two reference implementations. It's the
idea that there can be only one legitimate implementation that's
problematic.
Well, this is where I fear we will never agree. I think "Bitcoin is
different" in this reward and you disagree.
http://youtu.be/PxW5D9xCIsc
Otherwise, I think I'll stop trying convincing you.
Maybe I wasn't sufficiently explicit. It is problematic. That is the
core issue we are dealing with. That doesn't mean I disagree with the
objectives of an independent community consensus library.
The premise of the "one true library" idea is that there is *no way* to
sufficiently test for consensus bugs in any software release. That of
course means that each release of the satoshi client poses a significant
risk to the network. This risk is presently greater than that posed by
other implementations simply because of adoption. That is the basis of
Well, the "one true library" will be much better than the current "one
true full node".
The "one true library" would be the specification of the consensus
rules, but that doesn't mean you can't fork and modify it however you
want.
Post by Eric Voskuil via bitcoin-dev
The bottom line is that nobody has control over this process. There are,
and will always be, a multitude of consensus implementations that intend
to target the same coin. Presently there are multiple versions of the
satoshi client, and this has produced forks, and will continue to do so.
I get this point, even if the current satoshi client contains the
consensus rules specification (and many other things, obviously), that
doesn't mean is somehow protected from forking with itself if the
consensus code is changed in the wrong way accidentally. But the more
separated libconsensus and Bitcoin Core (satoshi client) are, the less
likely that changes in Bitcoin Core that weren't supposed to change
consensus rules actually do it by accident (like last time with the
migration out of bdb).
Post by Eric Voskuil via bitcoin-dev
Isolating the satoshi consensus checks to an independent library serves
not to eliminate that risk, but can reduce it somewhat. Importantly it
will allow various implementations to overcome a perception problem,
which will improve implementation diversity and developer participation.
I think alternative implementations using a full libconsensus can
increase their adoption a lot, since they become just as vulnerable to
consensus forks as Bitcoin Core (instead of more vulnerable like now).
Post by Eric Voskuil via bitcoin-dev
Post by Jorge Timón via bitcoin-dev
Post by Eric Voskuil via bitcoin-dev
Post by Jorge Timón via bitcoin-dev
I believe that's the only point where we fundamentally disagree, but
it shouldn't be a barrier in our common goal of taking "power" away
from Bitcoin Core development. If we're successful Bitcoin Core won't
have any privileged position with regards to, say, libbitcoin when it
comes to deciding consensus rules changes.
I don't think we disagree on anything fundamental. My issues with the
library were (1) the lack of isolation, (2) the fact that the satoshi
client wouldn't actually use the library, and (3) backtracking to use
OpenSSL, which we had recently removed from libbitcoin.
1) Working on it
For the sake of clarity, this is now a non-issue for us.
You mean libbitcoin's code is better organized than Bitcoin Core's?
I don't doubt it. Maybe we can create a full-libbitcoin-libconsensus
first and work on the API there.
Post by Eric Voskuil via bitcoin-dev
Post by Jorge Timón via bitcoin-dev
2) The Satoshi client has been using all along and it will use it
forever (maybe not through the API, but I don't get what the problem
with that is).
Again, I consider this a requirement for us to link directly to it as a
library. If the sources are isolated into an independent repo, but the
satoshi client is embedding its own copies, one must continue to diff
the client sources against the library sources. We are doing this
already, so the benefit to having the independent repo is in no longer
having to do this.
Oh, I see, you don't like that libsecp256k1 is currently a subtree of
Bitcoin Core either for the same reasons, right?
To not need to know when the changes in libconsensus are applied in
Bitcoin Core.
Mhmm, once libconsensus is complete, why would you care about it?
You just care about the libconsensus version (which doesn't have to
coincide with Bitcoin Core versions anymore).
Post by Eric Voskuil via bitcoin-dev
There are also differences in the build system that can affect outcome.
Comparing those differences across repos can be more challenging. For
this reason I consider it important to your objective that the satoshi
client actually use the library - as I assume it will at some point.
For the sake of clarity, please say "use the library's API". It's
going to use the library one way or another.
Post by Eric Voskuil via bitcoin-dev
If the satoshi client folks are to maintain a consensus library for the
community it's also important to show a commitment to its independence.
Dogfooding is of course a software engineering best practice. But there
is also the cynical perspective - the independent library in some ways
works against an advantage of the satoshi client.
I personally don't think the committers are parochial enough to let this
become an issue. We are all after something bigger. But if there was
push-back against using the library it would be a red flag. So until
that point passes I would just maintain our independent library, cloning
the sources from the satoshi client.
To be clear, I don't oppose to "dogfooding", it's just clear to me
that it will take even longer.
So what I don't understand is "once libbitcoin is complete and ready
for us to use, we will keep using our reimplementation of consensus
until Bitcoin Core uses the API as well. If Bitcoin core doesn't use
the API, we prefer not to use the library at all and keep having the
same consensus risk. We will do what we think it's worse for us until
Bitcoin Core uses the library through the API".
Post by Eric Voskuil via bitcoin-dev
Post by Jorge Timón via bitcoin-dev
3) There will be an announce about this soon.
Yes, I've seen this as a temporary setback.
And we will hopefully migrate the current libconsensus from openSSL to
libsecp256k1 soon. So we will be able to enjoy libsecp256k1's
performance improvements without risking consensus. One problem less.
Post by Eric Voskuil via bitcoin-dev
Post by Jorge Timón via bitcoin-dev
Post by Eric Voskuil via bitcoin-dev
Always willing to work with you on it, although we're all busy, and this
isn't my top priority presently.
Is it because "fear of consensus bugs is what keeps people on the
satoshi client" and you want to keep things this way?
No, I see it as less significant to the adoption of libbitcoin-server
than other issues we are working on, especially given the existence of
libbitcoin-consensus. I also trust you will make progress regardless.
This was just a joke because you said something similar earlier.
Don't take it seriously.
Jorge Timón via bitcoin-dev
2015-08-20 00:53:45 UTC
Permalink
Moving it here from the other thread.
No, as previously explained, once libconsensus is complete it can be
moved to a separate repository like libsecp256k1.
I don't see this happening any time soon, and I'm not sure why we should
wait for it.
Yes, unfortunately I don't see this happening any time soon either, at
least not with the amount of review I'm getting.
My initial hope was to complete libconsensus by 0.12 (one year should
be enough time, right?) but I was being too optimistic.
By "wait for it" I assume you mean waiting for libconsensus to be
complete before we separate it to a different repository.
The reason is just simplicity.
In our discussion leading up to libbitcoin building libbitcoin-consensus
we disagreed on whether intentional hard forks would (or even could)
happen. I think that issue is now settled. So my question remains how do
stakeholders (users/miners) maintain consensus when it's their
individual intent (the first objective of libconsensus), and diverge
when intended (which a direct dependency on libconsensus makes harder)?
IMO it's unreasonable to operate as if this won't happen, given that it has.
I believe the simplest option...
You might consider this as feedback from your customer base.
Mhmm, not sure I understand this point.
would be to fork the libconsensus
project and do the schism/controversial/contentious hardfork there.
But of course modifying libconsensus will be much easier than
modifying Bitcoin Core (if anything, because the amount of code is
much smaller).
That's a false dichotomy. We never would have considered forking Bitcoin
Core, and still wouldn't. Why would we set ourselves up for this
disruption, which would inevitably lead to us factoring the consensus
portions of libconsensus out of /bitcoin at the 11th hour?
We have to operate as if it can happen at any time. Otherwise we have
relinquished control of this vote and failed our users. Given that
operating assumption, it is much safer for us to have already done this
work (which we did). [It also provides a forcing function for us to
review in detail any consensus changes that get pushed out.]
Yes, you need to operate as if it can happen at any time. I now
understandbetter your position of having your own repository until a
complete libconsensus is separated.
In the meantime you will have to keep using your re-implementation of
the rest of the consensus rules (besides the script checks), but
fortunately the most risky and harder reimplementation is the part of
the script validation.
My question is why you would not embrace an independent consensus
repository? Your work to evolve it doesn't change.
Yes, I want a separated repository. I just wanted to start with a
separated folder first. Right now there's consensus code all over the
place, specially in main.cpp.
I think changing the order (separated repository first, moving code
from Bitcoin Core to libconsensus later) would increase the total
amount of work.

Here's another option that has recently crossed my mind:

1) Finish the libconsensus separation in an independent branch on top
of a given version, for example 0.11.
2) Separate a repository from that. Alternative implementations can
start using a full libconsensus
3) Rebase that branch on top of bitcoin/master and start to PR small
groups of commits. Once the whole branch has been merged, Bitcoin Core
can replace the consensus folder with the libconsensus subtree, so
that Bitcoin Core itself can start using a full libconsensus.

Ironically with this plan Bitcoin Core may not be the full node first
implementation to use a full libconsensus.
There will be some consensus fork bug risks during 3 (which at the
current speed I estimate it could easily take 3 or 4 years) and there
would be some redundant work (replicating every consensus change in
both Bitcoin Core and libconsensus).
On the bright side, we may be able to have a full libconsensus this
year (which was my goal after we exposed VerifyScript in the first
libconsensus).

Thoughts?
Tamas Blummer via bitcoin-dev
2015-08-20 07:14:11 UTC
Permalink
Jorge,

separating script engine into libconsensus was very helpful, since wrapped the piece of consensus
that would least likely to be captured exactly with an implementation from scratch. Thank you for your
effort there. Bits of Proof now uses its own or alternatively libconsensus for full validation.

I am sceptical however that a “full” consensus lib extracted from satoshi’s code is worth trying.
Not because it was impossible, but because the result would not be higher quality, if measured on agreement
with satoshi, than other re-implementations. It would actually be lower quality because of the antique tool set.

The rules outside script engine are simpler, therefore much easier to capture exactly. They are however
scattered around in the spaghetti and are often just a single if statement, also repeated elsewhere.

You would either have to very extensively refactor the code, that unlikely goes through as a PR, or
do what me and others did. Read satoshi code and rewrite the same. You have
a slight advantage of copy-paste small fragments, but I doubt the consensus relevant advantage of that.

Tamas Blummer
Post by Jorge Timón via bitcoin-dev
1) Finish the libconsensus separation in an independent branch on top
of a given version, for example 0.11.
2) Separate a repository from that. Alternative implementations can
start using a full libconsensus
3) Rebase that branch on top of bitcoin/master and start to PR small
groups of commits. Once the whole branch has been merged, Bitcoin Core
can replace the consensus folder with the libconsensus subtree, so
that Bitcoin Core itself can start using a full libconsensus.
Ironically with this plan Bitcoin Core may not be the full node first
implementation to use a full libconsensus.
There will be some consensus fork bug risks during 3 (which at the
current speed I estimate it could easily take 3 or 4 years) and there
would be some redundant work (replicating every consensus change in
both Bitcoin Core and libconsensus).
On the bright side, we may be able to have a full libconsensus this
year (which was my goal after we exposed VerifyScript in the first
libconsensus).
Thoughts?
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Jorge Timón via bitcoin-dev
2015-08-20 08:06:28 UTC
Permalink
Post by Tamas Blummer via bitcoin-dev
Jorge,
separating script engine into libconsensus was very helpful, since wrapped
the piece of consensus
that would least likely to be captured exactly with an implementation from
scratch. Thank you for your
effort there. Bits of Proof now uses its own or alternatively libconsensus
for full validation.
It wasn't just me: I didn't had the idea of creating a libconsensus
with a C API (thank Matt Corallo for that), I didn't removed all the
undesired dependencies or prepared the building part (thank Cory
Fields) and also thank at least Wladimir and Pieter who also
contributed in some ways I don't remember.
And of course also thank all the reviewers that made the PR merges possible.

I'm really happy to hear that libconsensus is being used, thank you
for your effort there too.
Post by Tamas Blummer via bitcoin-dev
I am sceptical however that a “full” consensus lib extracted from satoshi’s
code is worth trying.
Not because it was impossible, but because the result would not be higher
quality, if measured on agreement
with satoshi, than other re-implementations. It would actually be lower
quality because of the antique tool set.
I don't understand what you mean by "quality" in this context. One of
the goals is to have as little dependencies as possible (so "more
modern tool sets" may not be suitable for libconsensus). libsecp256k1
will keep on being a dependency (highly optimized C code) and that's
about it.
Ideally I would like to slowly move libconsensus from C++ to C too,
but it seems other people would prefer to move to C++11 instead.
Post by Tamas Blummer via bitcoin-dev
The rules outside script engine are simpler, therefore much easier to
capture exactly. They are however
scattered around in the spaghetti and are often just a single if statement,
also repeated elsewhere.
Yes, they are simpler and thus there's less risks of consensus fork
bugs, but it still exists.
It is true that the consensus code is currently spread all around
(specially in main.cpp), but completing libconsensus would solve that.
Lastly, since for consensus rules "the code is the specification", it
is unfortunate that the specification is coupled with a concrete
implementation (Bitcoin Core) and we should fix that.
Post by Tamas Blummer via bitcoin-dev
You would either have to very extensively refactor the code, that unlikely
goes through as a PR, or
do what me and others did. Read satoshi code and rewrite the same. You have
a slight advantage of copy-paste small fragments, but I doubt the consensus
relevant advantage of that.
But the goal is not reimplementing the consensus rules but rather
extract them from Bitcoin Core so that nobody needs to re-implement
them again.
It is not only exposing it but also separating it from Bitcoin Core so
that they can be changed without having to also change/take into
account non-consensus Bitcoin Core specific things.
A single PR would certainly be unacceptable, I was making many little
more acceptable ones (some of them already merged):

* [1/9] Consensus
** MERGED or DELETED
*** MERGED Consensus: Decouple pow from chainparams #5812 [consensuspow]
*** MERGED MOVEONLY: Move constants and globals to consensus.h #5696
[consensus_policy0]
*** DELETED Refactor: Create CCoinsViewEfficient interface for
CCoinsViewCache #5747 [coins]
*** MERGED Chainparams: Refactor: Decouple IsSuperMajority from
Params() #5968 [params_consensus]
*** MERGED Remove redundant getter
CChainParams::SubsidyHalvingInterval() #5996 [params_subsidy]
*** MERGED Separate CValidationState from main #5669 [consensus]
*** DELETED Consensus: Refactor: Separate CheckFinalTx from
main::IsFinalTx #6063 [consensus_finaltx]
*** MERGED Consensus: Decouple ContextualCheckBlockHeader from
checkpoints #5975 [consensus_checkpoints]
*** MERGED Separate Consensus::CheckTxInputs and GetSpendHeight in
CheckInputs #6061 [consensus_inputs]
*** MERGED Bugfix: Don't check the genesis block header before
accepting it #6299 [5975-quick-fix]
** REBASE Chainparams: Explicit Consensus::Params arg in consensus
functions #6024 [params_consensus2]
** REBASE Optimizations: Consensus: In AcceptToMemoryPool,
ConnectBlock, and CreateNewBlock #6445 [consensus-txinputs-0.12.99]
** REBASE MOVEONLY: Move most of consensus functions (pre-block) #6051
[consensus_moveonly]
** REBASE Consensus: Refactor: Turn CBlockIndex::GetMedianTimePast
into independent function #6009 [consensus_mediantime]
** DEPENDENT Consensus: Refactor: Consensus version of
CheckBlockHeader() #6035 [consensus_checkblockheader]
** DEPENDENT Consensus: Consensus version of pow functions [consensus_pow2]
** DEPENDENT API: Expose bitcoinconsensus_verify_header() in
libconsensus #5995 [consensus_header]
** DEPENDENT API: Expose bitcoinconsensus_verify_block() in
libconsensus #5946 [consensus_tip]
Tamas Blummer via bitcoin-dev
2015-08-20 08:35:13 UTC
Permalink
Every re-implementation, re-factoring even copy-paste introduces a risk of disagreement,
but also open the chance of doing the work better, in the sense of software engineering.
Post by Jorge Timón via bitcoin-dev
But the goal is not reimplementing the consensus rules but rather
extract them from Bitcoin Core so that nobody needs to re-implement
them again.
My goal is different. Compatibility with Bitcoin is important as I also want to deal with Bitcoins,
but it is also imperative to be able to create and serve other block chains with other rules and for those
I do not want to carry on the legacy of an antique tool set and a spaghetti style.

Bits of Proof uses scala (akka networking), java (api service), c++ (leveledb and now libconsensus)
and I am eager to integrate secp256k1 (c) as soon as part of consensus. The choices were
made because each piece appears best in what they do.

Tamas Blummer
Matt Corallo via bitcoin-dev
2015-08-20 17:44:28 UTC
Permalink
I dont think a libconsensus would have any kind of networking layer, nor
is C++ an antique tool set (hopefully libconsensus can avoid a boost
dependency, though thats not antique either). Ideally it would have a
simple API to give it blocks and a simple API for it to inform you of
what the current chain is. If you really want to get fancy maybe it has
pluggable block storage, too, but I dont see why you couldnt use this in
~any client?
Post by Tamas Blummer via bitcoin-dev
Every re-implementation, re-factoring even copy-paste introduces a risk of disagreement,
but also open the chance of doing the work better, in the sense of software engineering.
Post by Jorge Timón via bitcoin-dev
But the goal is not reimplementing the consensus rules but rather
extract them from Bitcoin Core so that nobody needs to re-implement
them again.
My goal is different. Compatibility with Bitcoin is important as I also want to deal with Bitcoins,
but it is also imperative to be able to create and serve other block chains with other rules and for those
I do not want to carry on the legacy of an antique tool set and a spaghetti style.
Bits of Proof uses scala (akka networking), java (api service), c++ (leveledb and now libconsensus)
and I am eager to integrate secp256k1 (c) as soon as part of consensus. The choices were
made because each piece appears best in what they do.
Tamas Blummer
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Tamas Blummer via bitcoin-dev
2015-08-20 21:26:19 UTC
Permalink
I know what you mean as I already have such a component with pluggable block store and networking.
While you are at it you could aim for isolation of bitcoin specific decisions and algos from generic block chain code.

The magnitude of refactoring you would have to do to get there from main.cpp and the rest of the hairball
is harder than a re-write from scratch, and the result will not be impressive, just hopefully working.
I think a slim API server was a lower hanging fruit in Core’s case.

BTW, support for refactoring is an example where you see if your tool set is modern.

Tamas Blummer
Post by Matt Corallo via bitcoin-dev
I dont think a libconsensus would have any kind of networking layer, nor
is C++ an antique tool set (hopefully libconsensus can avoid a boost
dependency, though thats not antique either). Ideally it would have a
simple API to give it blocks and a simple API for it to inform you of
what the current chain is. If you really want to get fancy maybe it has
pluggable block storage, too, but I dont see why you couldnt use this in
~any client?
Post by Tamas Blummer via bitcoin-dev
Every re-implementation, re-factoring even copy-paste introduces a risk of disagreement,
but also open the chance of doing the work better, in the sense of software engineering.
Post by Jorge Timón via bitcoin-dev
But the goal is not reimplementing the consensus rules but rather
extract them from Bitcoin Core so that nobody needs to re-implement
them again.
My goal is different. Compatibility with Bitcoin is important as I also want to deal with Bitcoins,
but it is also imperative to be able to create and serve other block chains with other rules and for those
I do not want to carry on the legacy of an antique tool set and a spaghetti style.
Bits of Proof uses scala (akka networking), java (api service), c++ (leveledb and now libconsensus)
and I am eager to integrate secp256k1 (c) as soon as part of consensus. The choices were
made because each piece appears best in what they do.
Tamas Blummer
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Matt Corallo via bitcoin-dev
2015-08-20 21:35:02 UTC
Permalink
Post by Tamas Blummer via bitcoin-dev
I know what you mean as I already have such a component with pluggable
block store and networking.
I'm not suggesting pluggable networking, I'm suggesting (and I think
everyone thinks the design should be) NO networking. The API is
ValidationResult libconsensus.HeyIFoundABlock(Block) and
ListOfBlocksToDownloadNext libconsensus.HeyIFoundAHeaderList(ListOfHeaders).
Post by Tamas Blummer via bitcoin-dev
While you are at it you could aim for isolation of bitcoin specific
decisions and algos from generic block chain code.
Are you suggesting to support altcoins? I dont think anyone cares about
supporting that.
Post by Tamas Blummer via bitcoin-dev
The magnitude of refactoring you would have to do to get there from
main.cpp and the rest of the hairball
is harder than a re-write from scratch,
I think you'd be very pleasantly surprised. It sounds like you havent
dug into Bitcoin Core validation code in years.
Post by Tamas Blummer via bitcoin-dev
and the result will not be
impressive, just hopefully working.
Hmm? The result would be an obviously correct consensus implementation
that everyone could use, instead of everyone going off and writing their
own and either being wrong, or never updating in the case of forks. Its
a huge deal to allow people to focus on making their libraries have good
APIs/Wallets/etc instead of focusing on making a working validation
engine (though maybe for that the p2p layer needs to also be in a library).
Post by Tamas Blummer via bitcoin-dev
I think a slim API server was a lower hanging fruit in Core’s case.
We have one, it just needs a few already obvious performance improvements.
Post by Tamas Blummer via bitcoin-dev
BTW, support for refactoring is an example where you see if your tool set is modern.
There are a number of good development tools for C++ that allow this....
Post by Tamas Blummer via bitcoin-dev
Tamas Blummer
On Aug 20, 2015, at 19:44, Matt Corallo via bitcoin-dev
I dont think a libconsensus would have any kind of networking layer, nor
is C++ an antique tool set (hopefully libconsensus can avoid a boost
dependency, though thats not antique either). Ideally it would have a
simple API to give it blocks and a simple API for it to inform you of
what the current chain is. If you really want to get fancy maybe it has
pluggable block storage, too, but I dont see why you couldnt use this in
~any client?
Post by Tamas Blummer via bitcoin-dev
Every re-implementation, re-factoring even copy-paste introduces a
risk of disagreement,
but also open the chance of doing the work better, in the sense of
software engineering.
Post by Jorge Timón via bitcoin-dev
But the goal is not reimplementing the consensus rules but rather
extract them from Bitcoin Core so that nobody needs to re-implement
them again.
My goal is different. Compatibility with Bitcoin is important as I
also want to deal with Bitcoins,
but it is also imperative to be able to create and serve other block
chains with other rules and for those
I do not want to carry on the legacy of an antique tool set and a spaghetti style.
Bits of Proof uses scala (akka networking), java (api service), c++
(leveledb and now libconsensus)
and I am eager to integrate secp256k1 (c) as soon as part of
consensus. The choices were
made because each piece appears best in what they do.
Tamas Blummer
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Tamas Blummer via bitcoin-dev
2015-08-21 06:46:26 UTC
Permalink
Thinking in Bitcoins only on the level of technology unnecessarily narrows your view.

OK, I hope to be pleasantly surprised.

Tamas Blummer
Post by Matt Corallo via bitcoin-dev
Post by Tamas Blummer via bitcoin-dev
I know what you mean as I already have such a component with pluggable
block store and networking.
I'm not suggesting pluggable networking, I'm suggesting (and I think
everyone thinks the design should be) NO networking. The API is
ValidationResult libconsensus.HeyIFoundABlock(Block) and
ListOfBlocksToDownloadNext libconsensus.HeyIFoundAHeaderList(ListOfHeaders).
Post by Tamas Blummer via bitcoin-dev
While you are at it you could aim for isolation of bitcoin specific
decisions and algos from generic block chain code.
Are you suggesting to support altcoins? I dont think anyone cares about
supporting that.
Post by Tamas Blummer via bitcoin-dev
The magnitude of refactoring you would have to do to get there from
main.cpp and the rest of the hairball
is harder than a re-write from scratch,
I think you'd be very pleasantly surprised. It sounds like you havent
dug into Bitcoin Core validation code in years.
Post by Tamas Blummer via bitcoin-dev
and the result will not be
impressive, just hopefully working.
Hmm? The result would be an obviously correct consensus implementation
that everyone could use, instead of everyone going off and writing their
own and either being wrong, or never updating in the case of forks. Its
a huge deal to allow people to focus on making their libraries have good
APIs/Wallets/etc instead of focusing on making a working validation
engine (though maybe for that the p2p layer needs to also be in a library).
Post by Tamas Blummer via bitcoin-dev
I think a slim API server was a lower hanging fruit in Core’s case.
We have one, it just needs a few already obvious performance improvements.
Post by Tamas Blummer via bitcoin-dev
BTW, support for refactoring is an example where you see if your tool set is modern.
There are a number of good development tools for C++ that allow this....
Post by Tamas Blummer via bitcoin-dev
Tamas Blummer
On Aug 20, 2015, at 19:44, Matt Corallo via bitcoin-dev
I dont think a libconsensus would have any kind of networking layer, nor
is C++ an antique tool set (hopefully libconsensus can avoid a boost
dependency, though thats not antique either). Ideally it would have a
simple API to give it blocks and a simple API for it to inform you of
what the current chain is. If you really want to get fancy maybe it has
pluggable block storage, too, but I dont see why you couldnt use this in
~any client?
Post by Tamas Blummer via bitcoin-dev
Every re-implementation, re-factoring even copy-paste introduces a
risk of disagreement,
but also open the chance of doing the work better, in the sense of
software engineering.
Post by Jorge Timón via bitcoin-dev
But the goal is not reimplementing the consensus rules but rather
extract them from Bitcoin Core so that nobody needs to re-implement
them again.
My goal is different. Compatibility with Bitcoin is important as I
also want to deal with Bitcoins,
but it is also imperative to be able to create and serve other block
chains with other rules and for those
I do not want to carry on the legacy of an antique tool set and a spaghetti style.
Bits of Proof uses scala (akka networking), java (api service), c++
(leveledb and now libconsensus)
and I am eager to integrate secp256k1 (c) as soon as part of
consensus. The choices were
made because each piece appears best in what they do.
Tamas Blummer
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Jorge Timón via bitcoin-dev
2015-08-21 19:46:33 UTC
Permalink
Post by Tamas Blummer via bitcoin-dev
Every re-implementation, re-factoring even copy-paste introduces a risk of disagreement,
but also open the chance of doing the work better, in the sense of software engineering.
But you don't want something better, you want something functionally identical.
You may want to watch sipa's explanation on why "the implementation is
the specification" and the reasons to separate libconsensus:

Post by Tamas Blummer via bitcoin-dev
Post by Jorge Timón via bitcoin-dev
But the goal is not reimplementing the consensus rules but rather
extract them from Bitcoin Core so that nobody needs to re-implement
them again.
My goal is different. Compatibility with Bitcoin is important as I also want to deal with Bitcoins,
but it is also imperative to be able to create and serve other block chains with other rules and for those
I do not want to carry on the legacy of an antique tool set and a spaghetti style.
Bits of Proof uses scala (akka networking), java (api service), c++ (leveledb and now libconsensus)
and I am eager to integrate secp256k1 (c) as soon as part of consensus. The choices were
made because each piece appears best in what they do.
Since you already depend on libconsensus for VerifyScript, wouldn't it
be nice that it also offered VerifyTx, VerifyHeader and VerifyBlock?
You would still have complete control over storage, concurrency,
networking, policy...
My plan is for the C API to interface with the external storage by
passing a function pointer to it.
Eric Lombrozo via bitcoin-dev
2015-08-21 20:07:15 UTC
Permalink
Unfortunately we have no way of rigorously proving functional equivalence
other than code review and unit testing. The simpler the consensus code
(and the more we can write it in a style that affords provability of
correctness) the easier it will be in the future to compare implementations.

Prior to swapping out implementations, we should at the least run it
through the gauntlet and perhaps run both implementations side-by-side.

All I/O should be treated abstractly in the API.

In C++ I really like using a nearly bare-bones signal template for most
async message handling, i.e.
https://github.com/ciphrex/mSIGNA/blob/master/deps/Signals/src/Signals.h

This greatly facilitates support for async bidirectional I/O, etc...with
minimal overhead.

But others might have other stylistic preferences.

- Eric

On Fri, Aug 21, 2015, 12:46 PM Jorge Timón <
Post by Tamas Blummer via bitcoin-dev
Post by Tamas Blummer via bitcoin-dev
Every re-implementation, re-factoring even copy-paste introduces a risk
of disagreement,
Post by Tamas Blummer via bitcoin-dev
but also open the chance of doing the work better, in the sense of
software engineering.
But you don't want something better, you want something functionally identical.
You may want to watch sipa's explanation on why "the implementation is
http://youtu.be/l3O4nh79CUU
Post by Tamas Blummer via bitcoin-dev
Post by Jorge Timón via bitcoin-dev
But the goal is not reimplementing the consensus rules but rather
extract them from Bitcoin Core so that nobody needs to re-implement
them again.
My goal is different. Compatibility with Bitcoin is important as I also
want to deal with Bitcoins,
Post by Tamas Blummer via bitcoin-dev
but it is also imperative to be able to create and serve other block
chains with other rules and for those
Post by Tamas Blummer via bitcoin-dev
I do not want to carry on the legacy of an antique tool set and a
spaghetti style.
Post by Tamas Blummer via bitcoin-dev
Bits of Proof uses scala (akka networking), java (api service), c++
(leveledb and now libconsensus)
Post by Tamas Blummer via bitcoin-dev
and I am eager to integrate secp256k1 (c) as soon as part of consensus.
The choices were
Post by Tamas Blummer via bitcoin-dev
made because each piece appears best in what they do.
Since you already depend on libconsensus for VerifyScript, wouldn't it
be nice that it also offered VerifyTx, VerifyHeader and VerifyBlock?
You would still have complete control over storage, concurrency,
networking, policy...
My plan is for the C API to interface with the external storage by
passing a function pointer to it.
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Tamas Blummer via bitcoin-dev
2015-08-22 11:04:44 UTC
Permalink
Post by Jorge Timón via bitcoin-dev
Post by Tamas Blummer via bitcoin-dev
Every re-implementation, re-factoring even copy-paste introduces a risk of disagreement,
but also open the chance of doing the work better, in the sense of software engineering.
But you don't want something better, you want something functionally identical.
You may want to watch sipa's explanation on why "the implementation is
http://youtu.be/l3O4nh79CUU http://youtu.be/l3O4nh79CUU
I do want something better, but not for the focus you have.

Not because what you produce was not high quality, but because quality is achieved at a very
high cost and is hard to uphold over generations of developer. You focus on a single use case
while there are many out there for distributed ledgers.

I think in an infrastructure for enterprise applications, building consensus on the ledger is a
cornerstone there, but is only a piece of the solution. I built several commercially successful
deployments where I delegated the consensus building to a border router, a Bitcoin Core,
then interfaced that trusted peer with my implementation that accepted Core’s decisions
in an SPV manner. One might think of this setup as wasteful and unsuitable for “small devices”
therefore an example of centralization people here try to avoid.

Enterprises have sufficient resources. Solving the business problem is valuable to them even at
magnitudes higher cost than a hobbyist would bear.

For mainstream adoption you need to get enterprises on board too, and that is what I care of.
Enterprises want code that is not only high quality, but is easy to maintain with a development
team with high attrition. One has to take whatever help is offered for that, and one is modern
languages and runtimes.

Bits of Proof’s own implementation of the scripts was not practically relevant in my commercially
successful deployments, because of the use of a border router, but it helped development,
enabling easier debug and precise error feedback esp. end even after Core had a reject message.

I integrated libconsensus only for the hope that is significantly fastens application side tx verification,
which it has turned out it does not, until secp265k1 is integrated.

I would likely use an other extended libconsensus too, but do not think there was a dependency on
that for enterprise development.

It would help there more to have a slim protocol server, no wallet, no rpc, no qt but a high
performance remoting API.
Post by Jorge Timón via bitcoin-dev
Since you already depend on libconsensus for VerifyScript, wouldn't it
be nice that it also offered VerifyTx, VerifyHeader and VerifyBlock?
You would still have complete control over storage, concurrency,
networking, policy...
My plan is for the C API to interface with the external storage by
passing a function pointer to it.
Storage and validation is non-trivially interconnected, but I now the separation can be done,
since I did it.

Excuse me, but function pointers is a pattern I used in the 80’s. I know that they are behind
the curtain of modern abstractions with similar use, I still prefer not to see them again.

Tamas Blummer
Eric Lombrozo via bitcoin-dev
2015-08-23 01:23:33 UTC
Permalink
I've been pushing for greater modularization since I first got into
bitcoin. I got quickly frustrated when I was only able to get through very
few things (i.e. moving core structure serialization classes to a separate
unit not called main). Working on Bitcoin has an added layer of frustration
that goes beyond most open source projects: even though we're clearly in
userland working at the application layer, a good layered protocol design
is still lacking. We have no standards process separate from what basically
amount to updates to one specific reference implementation. And we all need
to agree on any major change, since a blockchain that is easily forked in
contentious ways pretty much defeats its own purpose.

I went off to develop my own stack, where I could more easily avoid
politics and focus on engineering. But I now understand the politics are
inevitable. Bitcoin is inherently a cooperative project. Several people
have poured themselves passionately into the reference codebase, most of
whom did it (at least initially) purely as unpaid volunteers. There's a lot
of love that's gone into this. But it's become pretty clear that the
modularization is no longer merely a matter of good engineering - it is
essential to resolving serious political challenges.

Perhaps the most frustrating thing of all is watching people pushing for
relatively superficial yet highly controversial changes while we still lack
the proper infrastructure to handle these kinds of divergences of opinion
without either stagnating or becoming polarized.

I could continue working to reimplement an entire stack from scratch, as
several others have also done - but besides the serious effort duplication
this entails, it doesn't really seem like it will ultimately be a
convergent process. It's too easy to let ego and habit dictate one's
preferences rather than rational engineering considerations.

I know that some might feel I'm just preaching to the choir, but we should
probably take a step back from implementation hackery and try to specify
some core protocol layers, focusing on interfaces. Specifically, we need a
consensus layer that doesn't try to specify networking, storage, wallets,
UI, etc. Let different people improve upon these things independently in
their own implementations. What matters is that we all converge on a common
history and state. At the same time, let's open up more competition on all
these other things that are separate from the consensus layer.

If only we were to dedicate a fraction of the effort we've put into this
whole block size circus into what's actually important...and I blame myself
as well...

On Sat, Aug 22, 2015, 4:05 AM Tamas Blummer via bitcoin-dev <
Post by Tamas Blummer via bitcoin-dev
Every re-implementation, re-factoring even copy-paste introduces a risk of disagreement,
but also open the chance of doing the work better, in the sense of software engineering.
But you don't want something better, you want something functionally identical.
You may want to watch sipa's explanation on why "the implementation is
http://youtu.be/l3O4nh79CUU
I do want something better, but not for the focus you have.
Not because what you produce was not high quality, but because quality is
achieved at a very
high cost and is hard to uphold over generations of developer. You focus
on a single use case
while there are many out there for distributed ledgers.
I think in an infrastructure for enterprise applications, building
consensus on the ledger is a
cornerstone there, but is only a piece of the solution. I built several
commercially successful
deployments where I delegated the consensus building to a border router, a Bitcoin Core,
then interfaced that trusted peer with my implementation that accepted
Core’s decisions
in an SPV manner. One might think of this setup as wasteful and unsuitable
for “small devices”
therefore an example of centralization people here try to avoid.
Enterprises have sufficient resources. Solving the business problem is
valuable to them even at
magnitudes higher cost than a hobbyist would bear.
For mainstream adoption you need to get enterprises on board too, and
that is what I care of.
Enterprises want code that is not only high quality, but is easy to
maintain with a development
team with high attrition. One has to take whatever help is offered for
that, and one is modern
languages and runtimes.
Bits of Proof’s own implementation of the scripts was not practically
relevant in my commercially
successful deployments, because of the use of a border router, but it helped development,
enabling easier debug and precise error feedback esp. end even after Core
had a reject message.
I integrated libconsensus only for the hope that is significantly fastens
application side tx verification,
which it has turned out it does not, until secp265k1 is integrated.
I would likely use an other extended libconsensus too, but do not think
there was a dependency on
that for enterprise development.
It would help there more to have a slim protocol server, no wallet, no
rpc, no qt but a high
performance remoting API.
Since you already depend on libconsensus for VerifyScript, wouldn't it
be nice that it also offered VerifyTx, VerifyHeader and VerifyBlock?
You would still have complete control over storage, concurrency,
networking, policy...
My plan is for the C API to interface with the external storage by
passing a function pointer to it.
Storage and validation is non-trivially interconnected, but I now the
separation can be done,
since I did it.
Excuse me, but function pointers is a pattern I used in the 80’s. I know
that they are behind
the curtain of modern abstractions with similar use, I still prefer not to see them again.
Tamas Blummer
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Eric Lombrozo via bitcoin-dev
2015-08-23 02:19:26 UTC
Permalink
One thing it occurs to me (and I don't know if this has been suggested
before) we could do is separate the BIP process into at several distinct
areas:

1) Commit structure changes/consensus rule change proposals
- Consensus-building process (how are proposals debated, improved, vetted,
and selected)
- Update/deployment mechanisms for rule changes
- Specific hard fork proposals
- Specific soft fork proposals

2) Peer policies
- Seeding and discovery mechanisms
- Relay policies
- p2p message support

3) RPC

4) Everything else
Post by Eric Lombrozo via bitcoin-dev
I've been pushing for greater modularization since I first got into
bitcoin. I got quickly frustrated when I was only able to get through very
few things (i.e. moving core structure serialization classes to a separate
unit not called main). Working on Bitcoin has an added layer of frustration
that goes beyond most open source projects: even though we're clearly in
userland working at the application layer, a good layered protocol design
is still lacking. We have no standards process separate from what basically
amount to updates to one specific reference implementation. And we all need
to agree on any major change, since a blockchain that is easily forked in
contentious ways pretty much defeats its own purpose.
I went off to develop my own stack, where I could more easily avoid
politics and focus on engineering. But I now understand the politics are
inevitable. Bitcoin is inherently a cooperative project. Several people
have poured themselves passionately into the reference codebase, most of
whom did it (at least initially) purely as unpaid volunteers. There's a lot
of love that's gone into this. But it's become pretty clear that the
modularization is no longer merely a matter of good engineering - it is
essential to resolving serious political challenges.
Perhaps the most frustrating thing of all is watching people pushing for
relatively superficial yet highly controversial changes while we still lack
the proper infrastructure to handle these kinds of divergences of opinion
without either stagnating or becoming polarized.
I could continue working to reimplement an entire stack from scratch, as
several others have also done - but besides the serious effort duplication
this entails, it doesn't really seem like it will ultimately be a
convergent process. It's too easy to let ego and habit dictate one's
preferences rather than rational engineering considerations.
I know that some might feel I'm just preaching to the choir, but we should
probably take a step back from implementation hackery and try to specify
some core protocol layers, focusing on interfaces. Specifically, we need a
consensus layer that doesn't try to specify networking, storage, wallets,
UI, etc. Let different people improve upon these things independently in
their own implementations. What matters is that we all converge on a common
history and state. At the same time, let's open up more competition on all
these other things that are separate from the consensus layer.
If only we were to dedicate a fraction of the effort we've put into this
whole block size circus into what's actually important...and I blame myself
as well...
On Sat, Aug 22, 2015, 4:05 AM Tamas Blummer via bitcoin-dev <
Post by Tamas Blummer via bitcoin-dev
Every re-implementation, re-factoring even copy-paste introduces a risk of disagreement,
but also open the chance of doing the work better, in the sense of software engineering.
But you don't want something better, you want something functionally identical.
You may want to watch sipa's explanation on why "the implementation is
http://youtu.be/l3O4nh79CUU
I do want something better, but not for the focus you have.
Not because what you produce was not high quality, but because quality is
achieved at a very
high cost and is hard to uphold over generations of developer. You focus
on a single use case
while there are many out there for distributed ledgers.
I think in an infrastructure for enterprise applications, building
consensus on the ledger is a
cornerstone there, but is only a piece of the solution. I built several
commercially successful
deployments where I delegated the consensus building to a border router, a Bitcoin Core,
then interfaced that trusted peer with my implementation that accepted
Core’s decisions
in an SPV manner. One might think of this setup as wasteful and
unsuitable for “small devices”
therefore an example of centralization people here try to avoid.
Enterprises have sufficient resources. Solving the business problem is
valuable to them even at
magnitudes higher cost than a hobbyist would bear.
For mainstream adoption you need to get enterprises on board too, and
that is what I care of.
Enterprises want code that is not only high quality, but is easy to
maintain with a development
team with high attrition. One has to take whatever help is offered for
that, and one is modern
languages and runtimes.
Bits of Proof’s own implementation of the scripts was not practically
relevant in my commercially
successful deployments, because of the use of a border router, but it helped development,
enabling easier debug and precise error feedback esp. end even after Core
had a reject message.
I integrated libconsensus only for the hope that is significantly fastens
application side tx verification,
which it has turned out it does not, until secp265k1 is integrated.
I would likely use an other extended libconsensus too, but do not think
there was a dependency on
that for enterprise development.
It would help there more to have a slim protocol server, no wallet, no
rpc, no qt but a high
performance remoting API.
Since you already depend on libconsensus for VerifyScript, wouldn't it
be nice that it also offered VerifyTx, VerifyHeader and VerifyBlock?
You would still have complete control over storage, concurrency,
networking, policy...
My plan is for the C API to interface with the external storage by
passing a function pointer to it.
Storage and validation is non-trivially interconnected, but I now the
separation can be done,
since I did it.
Excuse me, but function pointers is a pattern I used in the 80’s. I know
that they are behind
the curtain of modern abstractions with similar use, I still prefer not
to see them again.
Tamas Blummer
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Tamas Blummer via bitcoin-dev
2015-08-23 06:42:17 UTC
Permalink
I see the huge amount of sweat and love that went into core and it actually hurts to see that most is expended in friction and lack of a vision for the software architecture.

To be concrete, this was my plan if dealing with the Core code base:

1) I'd consider the separation of networking and storage as suggested for a future extended libconsensus low priority, as their design should be (are) dominated by the need of the consensus logic only.

2) create an API to the consensus+networking+storage service that is not at the C++ language level but some scaleable cross-platform remoting, like eg. ZeroMQ.
This API should be minimal and simple, assuming that one fully trusts the node answering it. This API would unlock user land development by distinct teams with diverse technologies.

3) move the wallet, QT and RPC and other backward compatibility stuff (if e.g. there is some mining support) in-top of the new API and into distinct source code repositories.


Tamas Blummer
I've been pushing for greater modularization since I first got into bitcoin. I got quickly frustrated when I was only able to get through very few things (i.e. moving core structure serialization classes to a separate unit not called main). Working on Bitcoin has an added layer of frustration that goes beyond most open source projects: even though we're clearly in userland working at the application layer, a good layered protocol design is still lacking. We have no standards process separate from what basically amount to updates to one specific reference implementation. And we all need to agree on any major change, since a blockchain that is easily forked in contentious ways pretty much defeats its own purpose.
I went off to develop my own stack, where I could more easily avoid politics and focus on engineering. But I now understand the politics are inevitable. Bitcoin is inherently a cooperative project. Several people have poured themselves passionately into the reference codebase, most of whom did it (at least initially) purely as unpaid volunteers. There's a lot of love that's gone into this. But it's become pretty clear that the modularization is no longer merely a matter of good engineering - it is essential to resolving serious political challenges.
Perhaps the most frustrating thing of all is watching people pushing for relatively superficial yet highly controversial changes while we still lack the proper infrastructure to handle these kinds of divergences of opinion without either stagnating or becoming polarized.
I could continue working to reimplement an entire stack from scratch, as several others have also done - but besides the serious effort duplication this entails, it doesn't really seem like it will ultimately be a convergent process. It's too easy to let ego and habit dictate one's preferences rather than rational engineering considerations.
I know that some might feel I'm just preaching to the choir, but we should probably take a step back from implementation hackery and try to specify some core protocol layers, focusing on interfaces. Specifically, we need a consensus layer that doesn't try to specify networking, storage, wallets, UI, etc. Let different people improve upon these things independently in their own implementations. What matters is that we all converge on a common history and state. At the same time, let's open up more competition on all these other things that are separate from the consensus layer.
If only we were to dedicate a fraction of the effort we've put into this whole block size circus into what's actually important...and I blame myself as well...
Post by Jorge Timón via bitcoin-dev
Post by Tamas Blummer via bitcoin-dev
Every re-implementation, re-factoring even copy-paste introduces a risk of disagreement,
but also open the chance of doing the work better, in the sense of software engineering.
But you don't want something better, you want something functionally identical.
You may want to watch sipa's explanation on why "the implementation is
http://youtu.be/l3O4nh79CUU http://youtu.be/l3O4nh79CUU
I do want something better, but not for the focus you have.
Not because what you produce was not high quality, but because quality is achieved at a very
high cost and is hard to uphold over generations of developer. You focus on a single use case
while there are many out there for distributed ledgers.
I think in an infrastructure for enterprise applications, building consensus on the ledger is a
cornerstone there, but is only a piece of the solution. I built several commercially successful
deployments where I delegated the consensus building to a border router, a Bitcoin Core,
then interfaced that trusted peer with my implementation that accepted Core’s decisions
in an SPV manner. One might think of this setup as wasteful and unsuitable for “small devices”
therefore an example of centralization people here try to avoid.
Enterprises have sufficient resources. Solving the business problem is valuable to them even at
magnitudes higher cost than a hobbyist would bear.
For mainstream adoption you need to get enterprises on board too, and that is what I care of.
Enterprises want code that is not only high quality, but is easy to maintain with a development
team with high attrition. One has to take whatever help is offered for that, and one is modern
languages and runtimes.
Bits of Proof’s own implementation of the scripts was not practically relevant in my commercially
successful deployments, because of the use of a border router, but it helped development,
enabling easier debug and precise error feedback esp. end even after Core had a reject message.
I integrated libconsensus only for the hope that is significantly fastens application side tx verification,
which it has turned out it does not, until secp265k1 is integrated.
I would likely use an other extended libconsensus too, but do not think there was a dependency on
that for enterprise development.
It would help there more to have a slim protocol server, no wallet, no rpc, no qt but a high
performance remoting API.
Post by Jorge Timón via bitcoin-dev
Since you already depend on libconsensus for VerifyScript, wouldn't it
be nice that it also offered VerifyTx, VerifyHeader and VerifyBlock?
You would still have complete control over storage, concurrency,
networking, policy...
My plan is for the C API to interface with the external storage by
passing a function pointer to it.
Storage and validation is non-trivially interconnected, but I now the separation can be done,
since I did it.
Excuse me, but function pointers is a pattern I used in the 80’s. I know that they are behind
the curtain of modern abstractions with similar use, I still prefer not to see them again.
Tamas Blummer
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev <https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev>
Jorge Timón via bitcoin-dev
2015-08-29 23:30:39 UTC
Permalink
Post by Tamas Blummer via bitcoin-dev
I see the huge amount of sweat and love that went into core and it actually
hurts to see that most is expended in friction and lack of a vision for the
software architecture.
1) I'd consider the separation of networking and storage as suggested for a
future extended libconsensus low priority, as their design should be (are)
dominated by the need of the consensus logic only.
2) create an API to the consensus+networking+storage service that is not at
the C++ language level but some scaleable cross-platform remoting, like eg.
ZeroMQ.
This API should be minimal and simple, assuming that one fully trusts the
node answering it. This API would unlock user land development by distinct
teams with diverse technologies.
I plan to replicate the RPC API (or a subset of it) using ZMQ's
req/rep pattern, but #6103 comes first.
Post by Tamas Blummer via bitcoin-dev
3) move the wallet, QT and RPC and other backward compatibility stuff (if
e.g. there is some mining support) in-top of the new API and into distinct
source code repositories.
Well, the RPC is the API. For libconsensus, its C API is the API.
We've been talking about separating the wallet and qt to a different
repository for long, but modularization is a prerequisite.

Jorge Timón via bitcoin-dev
2015-08-29 23:25:22 UTC
Permalink
I completely agree and I share your frustration.
The importance of modularization is often disregarded but in my
opinion it has a deep positive impact in the long term: more people
are able to contribute with code and review (in the areas they know
better), the risks associated with each change become more clear
(there was a time when almost any change implied consensus risks),
more alternative code bases can be implemented on top of the basic
ones without fear of consensus bugs, etc.

When I first read some of the code in 2011, I concluded that almost
everything was in main.cpp (which I found ridiculous from a software
engineering perspective). When I started to contribute with code in
2014, main.cpp was still (and still is in my opinion) giant, but the
modularization had greatly improved thanks to changes like moving the
serialization code out of main (thank you very much for that). We
still have a lot of work ahead, but we've certainly advanced a lot.

Unfortunately we cannot force reviewers to pay more attention to
modularization PRs, many of them are usually more interested in
changes that add or remove functionality in the short term. This
problem gets exacerbated when modularization changes are required to
be done in small increments to make them more easily reviewable and
less disruptive to other open PRs, since it's harder for people to see
the big picture and the rationale for those small changes (that often
don't hcange functionality or performance at all).

I know we are not alone on this and people like Wladimir, Pieter, Cory
and Jonas Schnelli (at least, probably more people do) deeply care
about modularization, even if I subjectively and selfishly interpret
the lack of review on some of my PRs as a symptom of the opposite.
So I suggest that people who think this is a high priority join and
review each other's PRs on the subject.

Currently I focus on 3 modularization areas:

1) Chainparams: supporting multiple chains (ie multiple testchains is
all what Bitcoin cares about) is a great goal but there's still many
barriers to create a new testchain. I started this work with #3824,
but even after #6382 there are still more things to do.

2) Consensus: separating the consensus code, Matt Corallo had the idea
of also exposing it in libconsensus. I started with #3839, the latest
things I still have open are #6591 and #6445, please review.

3) Policy: separate node local policy code. Luke Dashjr started with
#5071, I started with #6335 (after several failed attempts), the next
little step blocking many other changes I have ready for way too long
is #6068 (#6424 also helps), please review.

I know Jonas Schnelli is focusing on the wallet. Cory Fields has
recently focused on checkpoints and chainparams.

Now that I know that you also care about modularization I will ask you
for review as well, I hope not to be too annoying like I've been with
Wladimir and Cory some times (and I usually am with some of my
coworkers at blockstream). Please do the same with me: point me to any
modularization PR you have opened.

Regarding your next post, I agree that an additional "Layer" field in
BIPs could be useful. Maybe you should start a BIP for that?
I've been pushing for greater modularization since I first got into bitcoin.
I got quickly frustrated when I was only able to get through very few things
(i.e. moving core structure serialization classes to a separate unit not
called main). Working on Bitcoin has an added layer of frustration that goes
beyond most open source projects: even though we're clearly in userland
working at the application layer, a good layered protocol design is still
lacking. We have no standards process separate from what basically amount to
updates to one specific reference implementation. And we all need to agree
on any major change, since a blockchain that is easily forked in contentious
ways pretty much defeats its own purpose.
I went off to develop my own stack, where I could more easily avoid politics
and focus on engineering. But I now understand the politics are inevitable.
Bitcoin is inherently a cooperative project. Several people have poured
themselves passionately into the reference codebase, most of whom did it (at
least initially) purely as unpaid volunteers. There's a lot of love that's
gone into this. But it's become pretty clear that the modularization is no
longer merely a matter of good engineering - it is essential to resolving
serious political challenges.
Perhaps the most frustrating thing of all is watching people pushing for
relatively superficial yet highly controversial changes while we still lack
the proper infrastructure to handle these kinds of divergences of opinion
without either stagnating or becoming polarized.
I could continue working to reimplement an entire stack from scratch, as
several others have also done - but besides the serious effort duplication
this entails, it doesn't really seem like it will ultimately be a convergent
process. It's too easy to let ego and habit dictate one's preferences rather
than rational engineering considerations.
I know that some might feel I'm just preaching to the choir, but we should
probably take a step back from implementation hackery and try to specify
some core protocol layers, focusing on interfaces. Specifically, we need a
consensus layer that doesn't try to specify networking, storage, wallets,
UI, etc. Let different people improve upon these things independently in
their own implementations. What matters is that we all converge on a common
history and state. At the same time, let's open up more competition on all
these other things that are separate from the consensus layer.
If only we were to dedicate a fraction of the effort we've put into this
whole block size circus into what's actually important...and I blame myself
as well...
On Sat, Aug 22, 2015, 4:05 AM Tamas Blummer via bitcoin-dev
Post by Tamas Blummer via bitcoin-dev
Every re-implementation, re-factoring even copy-paste introduces a risk of disagreement,
but also open the chance of doing the work better, in the sense of software engineering.
But you don't want something better, you want something functionally identical.
You may want to watch sipa's explanation on why "the implementation is
http://youtu.be/l3O4nh79CUU
I do want something better, but not for the focus you have.
Not because what you produce was not high quality, but because quality is
achieved at a very
high cost and is hard to uphold over generations of developer. You focus
on a single use case
while there are many out there for distributed ledgers.
I think in an infrastructure for enterprise applications, building
consensus on the ledger is a
cornerstone there, but is only a piece of the solution. I built several
commercially successful
deployments where I delegated the consensus building to a border router, a Bitcoin Core,
then interfaced that trusted peer with my implementation that accepted
Core’s decisions
in an SPV manner. One might think of this setup as wasteful and unsuitable
for “small devices”
therefore an example of centralization people here try to avoid.
Enterprises have sufficient resources. Solving the business problem is
valuable to them even at
magnitudes higher cost than a hobbyist would bear.
For mainstream adoption you need to get enterprises on board too, and
that is what I care of.
Enterprises want code that is not only high quality, but is easy to
maintain with a development
team with high attrition. One has to take whatever help is offered for
that, and one is modern
languages and runtimes.
Bits of Proof’s own implementation of the scripts was not practically
relevant in my commercially
successful deployments, because of the use of a border router, but it helped development,
enabling easier debug and precise error feedback esp. end even after Core
had a reject message.
I integrated libconsensus only for the hope that is significantly fastens
application side tx verification,
which it has turned out it does not, until secp265k1 is integrated.
I would likely use an other extended libconsensus too, but do not think
there was a dependency on
that for enterprise development.
It would help there more to have a slim protocol server, no wallet, no
rpc, no qt but a high
performance remoting API.
Since you already depend on libconsensus for VerifyScript, wouldn't it
be nice that it also offered VerifyTx, VerifyHeader and VerifyBlock?
You would still have complete control over storage, concurrency,
networking, policy...
My plan is for the C API to interface with the external storage by
passing a function pointer to it.
Storage and validation is non-trivially interconnected, but I now the
separation can be done,
since I did it.
Excuse me, but function pointers is a pattern I used in the 80’s. I know
that they are behind
the curtain of modern abstractions with similar use, I still prefer not to
see them again.
Tamas Blummer
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
Jorge Timón via bitcoin-dev
2015-08-29 22:08:24 UTC
Permalink
Post by Tamas Blummer via bitcoin-dev
Every re-implementation, re-factoring even copy-paste introduces a risk of disagreement,
but also open the chance of doing the work better, in the sense of software engineering.
But you don't want something better, you want something functionally identical.
You may want to watch sipa's explanation on why "the implementation is
http://youtu.be/l3O4nh79CUU
I do want something better, but not for the focus you have.
Not because what you produce was not high quality, but because quality is
achieved at a very
high cost and is hard to uphold over generations of developer. You focus on
a single use case
while there are many out there for distributed ledgers.
I think in an infrastructure for enterprise applications, building consensus
on the ledger is a
cornerstone there, but is only a piece of the solution. I built several
commercially successful
deployments where I delegated the consensus building to a border router, a Bitcoin Core,
then interfaced that trusted peer with my implementation that accepted
Core’s decisions
in an SPV manner. One might think of this setup as wasteful and unsuitable
for “small devices”
therefore an example of centralization people here try to avoid.
Enterprises have sufficient resources. Solving the business problem is
valuable to them even at
magnitudes higher cost than a hobbyist would bear.
For mainstream adoption you need to get enterprises on board too, and that
is what I care of.
Enterprises want code that is not only high quality, but is easy to maintain
with a development
team with high attrition. One has to take whatever help is offered for that,
and one is modern
languages and runtimes.
Bits of Proof’s own implementation of the scripts was not practically
relevant in my commercially
successful deployments, because of the use of a border router, but it helped development,
enabling easier debug and precise error feedback esp. end even after Core
had a reject message.
In fact I have been accused in the past (by at least Peter Todd) of
having "too many cases in mind" or "doing refactors that are good for
altchains".
That's why I'm very cautious about proposing changes that are not
strict improvments in maintainability to bitcoin itself.
But I actually have freicoin, sidechains and private chains (defined
in freimarkets, used in elements alpha as "block signing") in mind.
Some of the consensus changes I have in mind are support for multiple
assets or interest-bearing assets, for example.
But if you need to change the consensus rules you need to change the
code, there's no way around that.
It will be much simpler to only adapt libconsensus to other chains
than it is to adapt the whole Bitcoin Core code base.
Libconsensus can free you from the need of running "border routers"
(which you need to adapt if you depend on them and are supporting
chains with different rules).
When libconsensus has it's own independent repository, will I fork the
project to have a multi-consensus library supporting multiple
different chains (apart from bitcoin and its testchains)? Maybe, I'm
not sure it makes sense, maybe it's just simpler to maintain a
different project for each different chain (ie libfreicoinconsensus,
libbetaconsensus, etc).
Post by Tamas Blummer via bitcoin-dev
I integrated libconsensus only for the hope that is significantly fastens
application side tx verification,
which it has turned out it does not, until secp265k1 is integrated.
That is very sad to hear. The main reason to integrate libconsensus is
to avoid consensus fork bugs (or to not depend on the "border routers"
to avoid those bugs).
Post by Tamas Blummer via bitcoin-dev
I would likely use an other extended libconsensus too, but do not think
there was a dependency on
that for enterprise development.
It would help there more to have a slim protocol server, no wallet, no rpc,
no qt but a high
performance remoting API.
That's out of scope for libconsensus which will be stateless and whose
only API would be in C.
But the refactors in Bitcoin Core will hopefully make it easier to
support such a minimal node in it (you know you can "./configure
--disable-wallet --without-gui" already, right?, about RPC, that's the
remaining API!).
Post by Tamas Blummer via bitcoin-dev
Since you already depend on libconsensus for VerifyScript, wouldn't it
be nice that it also offered VerifyTx, VerifyHeader and VerifyBlock?
You would still have complete control over storage, concurrency,
networking, policy...
My plan is for the C API to interface with the external storage by
passing a function pointer to it.
Storage and validation is non-trivially interconnected, but I now the
separation can be done,
since I did it.
Excuse me, but function pointers is a pattern I used in the 80’s. I know
that they are behind
the curtain of modern abstractions with similar use, I still prefer not to see them again.
Yes, and the wheel it's an invention used in pre-historic times: that
doesn't make it less useful.
Do you have any other suggestion for interfacing with external storage
using a C API?
Wladimir J. van der Laan via bitcoin-dev
2015-07-28 08:43:14 UTC
Permalink
Post by Jorge Timón via bitcoin-dev
I think there were some misunderstandings in our previous conversation
about this topic.
I completely agree with having a separated repository for libconsensus
(that's the whole point, alternative implementations can be
consensus-safe by using it, and in the event of a schism fork[1], they
can fork just that smaller project without having to relay on Bitcoin
Core [satoshi] at all).
Indeed.
Post by Jorge Timón via bitcoin-dev
But I thought you also wanted Bitcoin Core to use libconsensus instead
of just having a subtree/subrepository like it currently does with
libsecp256k1.
I'm not sure if that would ever be accepted, but in any case we're
certainly far away from that goal. Here are some things that need to
I don't see any reason why Bitcoin Core would not use the consensus library. Eating our own dogfood and such.

Biggest risk, as I've said before, is that the refactoring loading to a (more complete) consensus library will result in code that is no longer bug-for-bug compatible with previous versions, thus defeating its entire purpose and introducing fork risk.

If that can be avoided - for example by going from here to there using pure code moves, as you're trying to do - I'm all for it.
Post by Jorge Timón via bitcoin-dev
2) Finish libconsensus's API: expose more things than VerifyScript, at
the very least, also expose VerifyTx, VerifyHeader and VerifyBlock.
Feedback from alternative implementations like libbitcoin is extremely
Agreed.
Post by Jorge Timón via bitcoin-dev
3) Move libconsensus to a separate repository as a
subtree/subrepository of Bitcoin Core.
If the rest is done, this is the easy part :)
Post by Jorge Timón via bitcoin-dev
Unfortunately and ironically again, safer, small and incremental
changes are less interesting for reviewers.
For example, I've been trying to move consensus code to the consensus
folder for a long time. The correctness of a MOVEONLY change is
trivial to review for anyone who knows how to copy/paste in its
favorite editor and how to use git diff, but will I ever get answers
to my questions in [1]?
Code review capacity is still our greatest bottleneck.
And I don't see any way out of that, unfortunately.
Post by Jorge Timón via bitcoin-dev
I know there's many people who really care about this, Cory Fields,
Wladimir and Pieter Wuille to name a few have reviewed many of this
changes (I've just got used to publicly whine about lack of review on
this front and policy encapsulation [very related fronts] as an
attempt to get some attention: not always, but begging for review
actually works some times).
I do really care about this.

Wladimir
Jorge Timón via bitcoin-dev
2015-07-28 10:09:22 UTC
Permalink
On Tue, Jul 28, 2015 at 10:43 AM, Wladimir J. van der Laan
Post by Wladimir J. van der Laan via bitcoin-dev
Post by Jorge Timón via bitcoin-dev
But I thought you also wanted Bitcoin Core to use libconsensus instead
of just having a subtree/subrepository like it currently does with
libsecp256k1.
I'm not sure if that would ever be accepted, but in any case we're
certainly far away from that goal. Here are some things that need to
I don't see any reason why Bitcoin Core would not use the consensus library. Eating our own dogfood and such.
As explained to Eric, it's not that I don't want Bitcoin Core to use
future-libconsensu through the API instead of a subtree: it's just
that that's more long-term and more work. And I don't see why other
implementations should really care about it.
Post by Wladimir J. van der Laan via bitcoin-dev
Biggest risk, as I've said before, is that the refactoring loading to a (more complete) consensus library will result in code that is no longer bug-for-bug compatible with previous versions, thus defeating its entire purpose and introducing fork risk.
If that can be avoided - for example by going from here to there using pure code moves, as you're trying to do - I'm all for it.
Well, pure movements will not be enough, parameters will have to
change, incompatible dependencies have to be removed (ie util.h which
contains globals), etc.
But yes, I think we can do it with only low-risk and easy-to-review commits.
Post by Wladimir J. van der Laan via bitcoin-dev
Post by Jorge Timón via bitcoin-dev
3) Move libconsensus to a separate repository as a
subtree/subrepository of Bitcoin Core.
If the rest is done, this is the easy part :)
And still, this doesn't require Bitcoin Core to use the API, a subtree
is enough at first.
This "easy step" doesn't guarantee that Bitcoin Core is using
future-libconsensus' API.
Post by Wladimir J. van der Laan via bitcoin-dev
Code review capacity is still our greatest bottleneck.
And I don't see any way out of that, unfortunately.
I really think these code separations help with this (ie there are
many more people in the world with enough knowledge to review the qt
or even policy parts than there's people able to review consensus
changes).
Post by Wladimir J. van der Laan via bitcoin-dev
I do really care about this.
I know and I said so.
Continue reading on narkive:
Loading...