[bitcoin-dev] Currency/exchange rate information API

[Luke Dashjr via bitcoin-dev]

Investigating what it would take to add fiat currency information to Bitcoin
Knots, I noticed Electrum currently has many implementations, one for each
exchange rate provider, due to lack of a common format for such data.

Therefore, I put together an initial draft of a BIP that could standardise
this so wallets (or other software) and exchange rate providers can simply
interoperate without a lot of overhead reimplementing the same thing many
ways.

One thing I am unsure about, is that currently this draft requires using XBT
(as BTC) for Bitcoin amounts. It would seem nicer to use satoshis, but those
don't really have a pseudo-ISO currency code to fit in nicely...

Current draft here:
https://github.com/luke-jr/bips/blob/bip-xchgrate/bip-xchgrate.mediawiki

Thoughts? Anything critical missing? Ways to make the interface better?

Luke

[アルム カールヨハン via bitcoin-dev]

I am surprised nothing like this exists already, so am all for the idea.

Maybe I am misunderstanding, but I'm not sure you want to have
thousand separators and other locale stuff in there. All currencies
including USD are often shown in Swedish using space or dot as
thousand separator and comma as decimal separator, e.g. "1.234,56 USD"
or "1 234,56 USD". I.e. this is something that the locale of the
user's environment defines, not something the server should have
opinions about. It is also not ideal to propose a given format based
on the user's locale, as some users have preferences for this (I
personally use US locale for numbers, and a US person who is visiting
Sweden wouldn't want this to suddenly change).

-Kalle.

On Sat, Mar 4, 2017 at 12:27 AM, Luke Dashjr via bitcoin-dev

Post by Luke Dashjr via bitcoin-dev
Investigating what it would take to add fiat currency information to Bitcoin
Knots, I noticed Electrum currently has many implementations, one for each
exchange rate provider, due to lack of a common format for such data.
Therefore, I put together an initial draft of a BIP that could standardise
this so wallets (or other software) and exchange rate providers can simply
interoperate without a lot of overhead reimplementing the same thing many
ways.
One thing I am unsure about, is that currently this draft requires using XBT
(as BTC) for Bitcoin amounts. It would seem nicer to use satoshis, but those
don't really have a pseudo-ISO currency code to fit in nicely...
https://github.com/luke-jr/bips/blob/bip-xchgrate/bip-xchgrate.mediawiki
Thoughts? Anything critical missing? Ways to make the interface better?
Luke
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[Tim Ruffing via bitcoin-dev]

I'm not sure if a BIP is the right thing in that case, given that the
provided functionality is not special to Bitcoin and can be used in
other contexts as well.

But ignoring this, the server should be authenticated at a
minimum. Otherwise manipulating exchange rates seems to be a nice
way for the attacker on the wire to make money...

Apart from that, my feeling is that it could be simplified. Is 
longpolling useful? And is the historical rate thing really necessary
for typical applications?

If yes, the client should be allowed to decide on which time scale the
data should be. (tick, min, hour, day, ...) That goes together with
clearly defining the type field (something like low, high, open, close,
but without flexibility). Think of a candle-stick chart basically.

Also, pushing may be more appropriate for "current" rates than polling.
Then no polling interval is necessary. On the other hand, this adds
complexity in other places, e.g., state.

Tim

Post by Luke Dashjr via bitcoin-dev
Investigating what it would take to add fiat currency information to
Bitcoin 
Knots, I noticed Electrum currently has many implementations, one for
each 
exchange rate provider, due to lack of a common format for such data.
Therefore, I put together an initial draft of a BIP that could
standardise 
this so wallets (or other software) and exchange rate providers can
simply 
interoperate without a lot of overhead reimplementing the same thing
many 
ways.
One thing I am unsure about, is that currently this draft requires
using XBT 
(as BTC) for Bitcoin amounts. It would seem nicer to use satoshis,
but those 
don't really have a pseudo-ISO currency code to fit in nicely...
    https://github.com/luke-jr/bips/blob/bip-xchgrate/bip-xchgrate.me
diawiki
Thoughts? Anything critical missing? Ways to make the interface better?
Luke
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[Luke Dashjr via bitcoin-dev]

Post by Tim Ruffing via bitcoin-dev
But ignoring this, the server should be authenticated at a
minimum. Otherwise manipulating exchange rates seems to be a nice
way for the attacker on the wire to make money...

HTTPS would be used for that. It's not something that needs to be at a higher
layer.

Post by Tim Ruffing via bitcoin-dev
Apart from that, my feeling is that it could be simplified. Is
longpolling useful?

I would think so, at least for Bitcoin since rates can change significantly in
a short period of time (or can they anymore? I haven't really watched lately.)

Post by Tim Ruffing via bitcoin-dev
And is the historical rate thing really necessary for typical applications?

When displaying historical transactions, it doesn't really make sense to use
the current market rate, but rather the market rate at the time the payment
was made. While wallets might simply cache it with the transaction, it would
be perhaps nicer if it could be automatically restored for seed-only
recoveries. In any case, if a service/wallet doesn't want to provide/use
historical information, it can simply not implement that part.

Post by Tim Ruffing via bitcoin-dev
If yes, the client should be allowed to decide on which time scale the
data should be. (tick, min, hour, day, ...) That goes together with
clearly defining the type field (something like low, high, open, close,
but without flexibility). Think of a candle-stick chart basically.

How is the current draft insufficient for this?

Post by Tim Ruffing via bitcoin-dev
Also, pushing may be more appropriate for "current" rates than polling.
Then no polling interval is necessary. On the other hand, this adds
complexity in other places, e.g., state.

Pushing is what longpolling does.

Luke

[Jonas Schnelli via bitcoin-dev]

I like the BIP. It can reduce workload during implementation on both sides of the API and it allows to show the user more data without implementing tons of proprietary APIs.
It’s not directly Bitcoin specific (example: BIP32 is also not Bitcoin specific), but I think a BIP is the right way for this.

Post by Luke Dashjr via bitcoin-dev

Post by Tim Ruffing via bitcoin-dev
Apart from that, my feeling is that it could be simplified. Is
longpolling useful?

I would think so, at least for Bitcoin since rates can change significantly in
a short period of time (or can they anymore? I haven't really watched lately.)

Long polling is a simple push concept that works on most type of network configurations (NAT, proxy, etc.).
The only concern I see here is that an public API will quickly fill up the maximum allowed httpd connections.
But it’s solvable.

Post by Luke Dashjr via bitcoin-dev

Post by Tim Ruffing via bitcoin-dev
And is the historical rate thing really necessary for typical applications?

When displaying historical transactions, it doesn't really make sense to use
the current market rate, but rather the market rate at the time the payment
was made. While wallets might simply cache it with the transaction, it would
be perhaps nicer if it could be automatically restored for seed-only
recoveries. In any case, if a service/wallet doesn't want to provide/use
historical information, it can simply not implement that part.

I’m also not sure how useful historical datapoint are. I don’t think the use case where someone wants to restore from a seed and get all exchange rates during the time of the payment is something users are looking for.
However, It’s optional.

Post by Luke Dashjr via bitcoin-dev

Post by Tim Ruffing via bitcoin-dev
If yes, the client should be allowed to decide on which time scale the
data should be. (tick, min, hour, day, ...) That goes together with
clearly defining the type field (something like low, high, open, close,
but without flexibility). Think of a candle-stick chart basically.

How is the current draft insufficient for this?

Post by Tim Ruffing via bitcoin-dev
Also, pushing may be more appropriate for "current" rates than polling.
Then no polling interval is necessary. On the other hand, this adds
complexity in other places, e.g., state.

Pushing is what longpolling does.

Agree with Luke. A „real“ push (though I’d say long-polling is the real push, AFAIK it’s also the technique behind Apple’s iOS push channel) would require a complex server setup that complicates many things like load-balancer, mem-caching, etc.


</jonas>

[Tim Ruffing via bitcoin-dev]

Post by Luke Dashjr via bitcoin-dev

Post by Tim Ruffing via bitcoin-dev
But ignoring this, the server should be authenticated at a
minimum. Otherwise manipulating exchange rates seems to be a nice
way for the attacker on the wire to make money...

HTTPS would be used for that. It's not something that needs to be at
a higher 
layer.

Sure, HTTPS is the way to go. But I think that should be required or at
least noted in the BIP, because people could miss easily, e.g., "I
don't need TLS, all the data is public anyway."

Post by Luke Dashjr via bitcoin-dev
When displaying historical transactions, it doesn't really make sense
to use 
the current market rate, but rather the market rate at the time the
payment 
was made. While wallets might simply cache it with the transaction,
it would 
be perhaps nicer if it could be automatically restored for seed-only 
recoveries. In any case, if a service/wallet doesn't want to
provide/use 
historical information, it can simply not implement that part.

Having the rate at the time of payment is indeed very useful, yes.
However that requires just a single value per payment, and there is no
query that tells the server "give me the value closest to timestamp t"
or similar.
Of course the client can download and keep a large part of history and
extract the information on its own but I can imagine that not every
clients wants to do that, and also the client does not know in advance
the bounds (from, to) that it must query.

Post by Luke Dashjr via bitcoin-dev

Post by Tim Ruffing via bitcoin-dev
If yes, the client should be allowed to decide on which time scale

the

Post by Tim Ruffing via bitcoin-dev
data should be. (tick, min, hour, day, ...) That goes together with
clearly defining the type field (something like low, high, open,

close,

Post by Tim Ruffing via bitcoin-dev
but without flexibility). Think of a candle-stick chart basically.

How is the current draft insufficient for this?

In the current draft the client or the server cannot specify
granularity. If the clients only wants one value per day but for an
entire year, then it has to perform many requests or download and
process a very large response.
Also, I think it's okay that the type field allows for arbitrary user-
defined values, but it should also have some precisely defined values
(e.g. the mentioned low/high/open/close/typical).
For example, it's not clear currently what "low" means for a timestamp
(as opposed to a time span). Is it the low of the entire day or the low
since the previous record or something different?

One has to be careful not to add too much complexity though. As soon as
one moves away from timestamps to something like hours or days, all
kind of issues with timezone, daylight saving time etc. appear. Maybe a
simple way to let the client ask "give me one value for every interval
of 3600 seconds" or similar.

Post by Luke Dashjr via bitcoin-dev
Pushing is what longpolling does.

That makes a lot of sense, yes.


Tim

[Tim Ruffing via bitcoin-dev]

Post by Tim Ruffing via bitcoin-dev

Post by Luke Dashjr via bitcoin-dev
Pushing is what longpolling does.

That makes a lot of sense, yes.

Forgot one thing:
For longpolling, maybe we would like to have the possibility to request
some periodic message from the server. Otherwise clients cannot
distinguish between the situations 1. "value is still in the requested
bounds (minrate, maxrate)" and 2. "connection has dropped". So the user
may take a wrong decision because he assumed that the value is still
in bounds holds but actually the server has died.

Tim

[Luke Dashjr via bitcoin-dev]

Post by Tim Ruffing via bitcoin-dev
For longpolling, maybe we would like to have the possibility to request
some periodic message from the server. Otherwise clients cannot
distinguish between the situations 1. "value is still in the requested
bounds (minrate, maxrate)" and 2. "connection has dropped". So the user
may take a wrong decision because he assumed that the value is still
in bounds holds but actually the server has died.

That's the job of TCP.
Do you think we need to explicitly specify a keepalive configuration?
Seems like that would vary based on client or network.

[Luke Dashjr via bitcoin-dev]

Post by Tim Ruffing via bitcoin-dev
Having the rate at the time of payment is indeed very useful, yes.
However that requires just a single value per payment, and there is no
query that tells the server "give me the value closest to timestamp t"
or similar.
Of course the client can download and keep a large part of history and
extract the information on its own but I can imagine that not every
clients wants to do that, and also the client does not know in advance
the bounds (from, to) that it must query.

It would be a privacy leak to request only the specific timestamps, but I
suppose many wallets lack even basic privacy to begin with.

To address the bounds issue, I have specified that when from/to don't have an
exact record, that the previous/next (respectively) is provided.

Hopefully this addresses both concerns?

Post by Tim Ruffing via bitcoin-dev
In the current draft the client or the server cannot specify
granularity. If the clients only wants one value per day but for an
entire year, then it has to perform many requests or download and
process a very large response.

That's what the "timedelta" field solves, no?
If you want one value per day, you'd put 86400.

Post by Tim Ruffing via bitcoin-dev
Also, I think it's okay that the type field allows for arbitrary user-
defined values, but it should also have some precisely defined values
(e.g. the mentioned low/high/open/close/typical).
For example, it's not clear currently what "low" means for a timestamp
(as opposed to a time span). Is it the low of the entire day or the low
since the previous record or something different?

Is it not sufficient for the server to specify this in the description of the
given currency-pair feed?

Luke

[Tim Ruffing via bitcoin-dev]

Post by Luke Dashjr via bitcoin-dev
It would be a privacy leak to request only the specific timestamps,
but I 
suppose many wallets lack even basic privacy to begin with.
To address the bounds issue, I have specified that when from/to don't
have an 
exact record, that the previous/next (respectively) is provided.
Hopefully this addresses both concerns?

Yes, that solves it. You're right with the privacy concern however.

Post by Luke Dashjr via bitcoin-dev

Post by Tim Ruffing via bitcoin-dev
In the current draft the client or the server cannot specify
granularity. If the clients only wants one value per day but for an
entire year, then it has to perform many requests or download and
process a very large response.

That's what the "timedelta" field solves, no?
If you want one value per day, you'd put 86400.

Oh sure, I had overlooked that.

Post by Luke Dashjr via bitcoin-dev

Post by Tim Ruffing via bitcoin-dev
Also, I think it's okay that the type field allows for arbitrary

user-

Post by Tim Ruffing via bitcoin-dev
defined values, but it should also have some precisely defined

values

Post by Tim Ruffing via bitcoin-dev
(e.g. the mentioned low/high/open/close/typical).
For example, it's not clear currently what "low" means for a

timestamp

Post by Tim Ruffing via bitcoin-dev
(as opposed to a time span). Is it the low of the entire day or the

low

Post by Tim Ruffing via bitcoin-dev
since the previous record or something different?

Is it not sufficient for the server to specify this in the
description of the 
given currency-pair feed?

That works but a standardized way of indicating that piece of
information to the client is useful. Then the client can display a
"connection status" to the user, e.g., an "possible out-of-date"
warning like the warning sign in the Qt GUI when Bitcoin Core is
catching up the network.


Tim

[Tim Ruffing via bitcoin-dev]

Post by Tim Ruffing via bitcoin-dev
That works but a standardized way of indicating that piece of
information to the client is useful. Then the client can display a
"connection status" to the user, e.g., an "possible out-of-date"
warning like the warning sign in the Qt GUI when Bitcoin Core is
catching up the network.

Wait, forget this reply, I mixed up the two issues of keepalive and
definition of low, high etc... -.-

1. Keepalive for longpolling:
As I said, this can be useful for an out-of-date warning. I don't know
if this is better solved with TCP keepalive or on the higher layer.

2. Definition of low, high:
My feeling is that there is nothing wrong with providing exact
definitions in the BIP, i.e.., giving up the flexibility does not too
hurt much. However all of this is a minor issue after all.

Tim

[Andreas Schildbach via bitcoin-dev]

Post by Tim Ruffing via bitcoin-dev
And is the historical rate thing really necessary
for typical applications?

Yes, it is. Basically all incoming transactions are historical, as your
wallet is likely not online when it happens.

[Marcel Jamin via bitcoin-dev]

Why are multiple results separated by a line-feed rather than using a JSON Array?
* Clients ought to cache historical data, and using a line-feed format allows them to simply append to a cache file.
* Parsing JSON typically requires the entire data parsed together as a single memory object. Using simple lines to separate results, however, allows parsing a single result at a time.
What if long descriptions require line and paragraph breaks?
* Clients should word-wrap long lines, and JSON escapes newlines as "\n" which can be used doubly ("\n\n") for paragraph breaks.

I'd file this under premature optimization at the cost of
interoperability. If you use JSON, then please use it properly.

I'd also say it's the job of the parser to offer a way of doing that,
e.g. in .NET you can easily achieve that with Newtonsoft's JSON
parser: http://stackoverflow.com/questions/20374083/deserialize-json-array-stream-one-item-at-a-time.

On 4 March 2017 at 09:27, Luke Dashjr via bitcoin-dev

Investigating what it would take to add fiat currency information to Bitcoin
Knots, I noticed Electrum currently has many implementations, one for each
exchange rate provider, due to lack of a common format for such data.
Therefore, I put together an initial draft of a BIP that could standardise
this so wallets (or other software) and exchange rate providers can simply
interoperate without a lot of overhead reimplementing the same thing many
ways.
One thing I am unsure about, is that currently this draft requires using XBT
(as BTC) for Bitcoin amounts. It would seem nicer to use satoshis, but those
don't really have a pseudo-ISO currency code to fit in nicely...
https://github.com/luke-jr/bips/blob/bip-xchgrate/bip-xchgrate.mediawiki
Thoughts? Anything critical missing? Ways to make the interface better?
Luke
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[Andrew LeCody via bitcoin-dev]

This formatting of JSON isn't unheard of though, it's typically called JSON
Streaming[1]. As long as exchanges implementing the API actually follow the
BIP and keep one JSON object per line, it shouldn't be a problem to decode.

1. https://en.wikipedia.org/wiki/JSON_Streaming

On Tue, Mar 7, 2017 at 8:07 AM Marcel Jamin via bitcoin-dev <

Why are multiple results separated by a line-feed rather than using a

JSON Array?

* Clients ought to cache historical data, and using a line-feed format

allows them to simply append to a cache file.

* Parsing JSON typically requires the entire data parsed together as a

single memory object. Using simple lines to separate results, however,
allows parsing a single result at a time.

What if long descriptions require line and paragraph breaks?
* Clients should word-wrap long lines, and JSON escapes newlines as "\n"

which can be used doubly ("\n\n") for paragraph breaks.
I'd file this under premature optimization at the cost of
interoperability. If you use JSON, then please use it properly.
I'd also say it's the job of the parser to offer a way of doing that,
e.g. in .NET you can easily achieve that with Newtonsoft's JSON
http://stackoverflow.com/questions/20374083/deserialize-json-array-stream-one-item-at-a-time
.
On 4 March 2017 at 09:27, Luke Dashjr via bitcoin-dev

Investigating what it would take to add fiat currency information to

Bitcoin

Knots, I noticed Electrum currently has many implementations, one for

each

exchange rate provider, due to lack of a common format for such data.
Therefore, I put together an initial draft of a BIP that could

standardise

this so wallets (or other software) and exchange rate providers can

simply

interoperate without a lot of overhead reimplementing the same thing many
ways.
One thing I am unsure about, is that currently this draft requires using

XBT

(as BTC) for Bitcoin amounts. It would seem nicer to use satoshis, but

those

don't really have a pseudo-ISO currency code to fit in nicely...

https://github.com/luke-jr/bips/blob/bip-xchgrate/bip-xchgrate.mediawiki

Thoughts? Anything critical missing? Ways to make the interface better?
Luke
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev