[bitcoin-dev] Long-term vision for bitcoind (was libconsensus and bitcoin development process)

[Jorge Timón via bitcoin-dev]

On Fri, Sep 18, 2015 at 2:07 AM, Wladimir J. van der Laan via

My long-term vision of bitcoind is a P2P node with validation and blockchain store, with a couple of data sources that can be subscribed to or pulled from.

I agree with this long term vision.
Here's how I think it could happen:

1) Libconsensus is completed and moved to a subtree (which has libsecp
as an internal subtree)

2) Bitcoind becomes a subtree of bitcoin-wallet (which has
bitcoin-wallet and bitcoin-qt)

Without aggressively changing it for this purpose, libconsensus should
tend to become C, like libsecp, which is better for proving
correctness.
Hopefully at some point it won't take much to move to C.

Upper layers should move to C++11

Don't focus on the git subtrees, the basic architecture is bitcoin-qt
on top of bitcoin-wallet, bitcoin-wallet on top of bitcoind (and
friends like bitcoin-cli and bitcoin-tx), bitcoind on top of
libconsensus on top of libsecp256k1.

I believe this would maximize the number of people who can safely
contribute to the project.
I also believe this is the architecture most contributors have in mind
for the long term, but I may be wrong about it.

Criticisms to this plan?

[Eric Lombrozo via bitcoin-dev]

Here's what I propose as a long-term plan:

1) libconsensus
==========
We should probably start by defining an API for libconsensus. It should
support an abstract DB model, track chain state, provide query
mechanisms for blocks and transactions with optional pruning and
indexing, expose a subscription mechanism for events such as NEW_TIP,
REORG, etc, and contain a script interpreter.

We can develop the library in parallel with Bitcoin Core without too
much refactoring of Bitcoin Core itself...just moving pieces of Bitcoin
Core's consensus code into the new library, tracking code movements to
make merging easier. Yes, this is a bit ugly as it requires code
duplication...but it will temporarily avoid much of the downstream
pushback we're getting. The idea is that we can prove out the library
with some simple projects, then start removing the consensus stuff from
Bitcoin Core once we have greater acceptance of the library and better
documentation.


2) peer services
==========
We develop a peer services library that performs the tasks of peer
discovery and relay, with the ability to connect to appropriate peers
and queue messages. It uses libconsensus for all validation
functionality and as a datastore for the consensus state but maintains
its own database for peer history and statistics. I believe Cory has
been working on this already using libevent. I've already developed an
async library for this as well.


3) API/RPC
=======
We provide high level calls and pub/sub mechanisms. ZMQ has been
implemented and added already, but we could support other transports as
well.


4) Wallet
======
The wallet is split out into a separate process that connects to the
stack via the API/RPC layer.


- Eric

------ Original Message ------
From: "Jorge Timón" <bitcoin-***@lists.linuxfoundation.org>
To: "Wladimir J. van der Laan" <***@gmail.com>
Cc: "Bitcoin development mailing list"
<bitcoin-***@lists.linuxfoundation.org>
Sent: 9/22/2015 11:36:14 AM
Subject: [bitcoin-dev] Long-term vision for bitcoind (was libconsensus
and bitcoin development process)

Post by Jorge Timón via bitcoin-dev
On Fri, Sep 18, 2015 at 2:07 AM, Wladimir J. van der Laan via

My long-term vision of bitcoind is a P2P node with validation and
blockchain store, with a couple of data sources that can be subscribed
to or pulled from.

I agree with this long term vision.
1) Libconsensus is completed and moved to a subtree (which has libsecp
as an internal subtree)
2) Bitcoind becomes a subtree of bitcoin-wallet (which has
bitcoin-wallet and bitcoin-qt)
Without aggressively changing it for this purpose, libconsensus should
tend to become C, like libsecp, which is better for proving
correctness.
Hopefully at some point it won't take much to move to C.
Upper layers should move to C++11
Don't focus on the git subtrees, the basic architecture is bitcoin-qt
on top of bitcoin-wallet, bitcoin-wallet on top of bitcoind (and
friends like bitcoin-cli and bitcoin-tx), bitcoind on top of
libconsensus on top of libsecp256k1.
I believe this would maximize the number of people who can safely
contribute to the project.
I also believe this is the architecture most contributors have in mind
for the long term, but I may be wrong about it.
Criticisms to this plan?
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[Eric Lombrozo via bitcoin-dev]

I should also add that the mempool should exist in (2). This way the
peer services layer can manage all relay policy and mempool management.

------ Original Message ------
From: "Eric Lombrozo" <***@gmail.com>
To: "Jorge Timón" <***@jtimon.cc>; "Wladimir J. van der Laan"
<***@gmail.com>
Cc: "Bitcoin development mailing list"
<bitcoin-***@lists.linuxfoundation.org>
Sent: 9/22/2015 5:07:22 PM
Subject: Re: [bitcoin-dev] Long-term vision for bitcoind (was
libconsensus and bitcoin development process)

Post by Eric Lombrozo via bitcoin-dev
1) libconsensus
==========
We should probably start by defining an API for libconsensus. It should
support an abstract DB model, track chain state, provide query
mechanisms for blocks and transactions with optional pruning and
indexing, expose a subscription mechanism for events such as NEW_TIP,
REORG, etc, and contain a script interpreter.
We can develop the library in parallel with Bitcoin Core without too
much refactoring of Bitcoin Core itself...just moving pieces of Bitcoin
Core's consensus code into the new library, tracking code movements to
make merging easier. Yes, this is a bit ugly as it requires code
duplication...but it will temporarily avoid much of the downstream
pushback we're getting. The idea is that we can prove out the library
with some simple projects, then start removing the consensus stuff from
Bitcoin Core once we have greater acceptance of the library and better
documentation.
2) peer services
==========
We develop a peer services library that performs the tasks of peer
discovery and relay, with the ability to connect to appropriate peers
and queue messages. It uses libconsensus for all validation
functionality and as a datastore for the consensus state but maintains
its own database for peer history and statistics. I believe Cory has
been working on this already using libevent. I've already developed an
async library for this as well.
3) API/RPC
=======
We provide high level calls and pub/sub mechanisms. ZMQ has been
implemented and added already, but we could support other transports as
well.
4) Wallet
======
The wallet is split out into a separate process that connects to the
stack via the API/RPC layer.
- Eric
------ Original Message ------
Cc: "Bitcoin development mailing list"
Sent: 9/22/2015 11:36:14 AM
Subject: [bitcoin-dev] Long-term vision for bitcoind (was libconsensus
and bitcoin development process)

Post by Jorge Timón via bitcoin-dev
On Fri, Sep 18, 2015 at 2:07 AM, Wladimir J. van der Laan via

My long-term vision of bitcoind is a P2P node with validation and
blockchain store, with a couple of data sources that can be
subscribed to or pulled from.

I agree with this long term vision.
1) Libconsensus is completed and moved to a subtree (which has libsecp
as an internal subtree)
2) Bitcoind becomes a subtree of bitcoin-wallet (which has
bitcoin-wallet and bitcoin-qt)
Without aggressively changing it for this purpose, libconsensus should
tend to become C, like libsecp, which is better for proving
correctness.
Hopefully at some point it won't take much to move to C.
Upper layers should move to C++11
Don't focus on the git subtrees, the basic architecture is bitcoin-qt
on top of bitcoin-wallet, bitcoin-wallet on top of bitcoind (and
friends like bitcoin-cli and bitcoin-tx), bitcoind on top of
libconsensus on top of libsecp256k1.
I believe this would maximize the number of people who can safely
contribute to the project.
I also believe this is the architecture most contributors have in mind
for the long term, but I may be wrong about it.
Criticisms to this plan?
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev