[bitcoin-dev] Memory leaks?

[Jonathan Toomim (Toomim Bros) via bitcoin-dev]

I just noticed that several of my running bitcoind processes were using around 3+ GB of RAM, even though the mempool itself seemed to be under control.

***@prime:~/bin$ ./bitcoin-cli getmempoolinfo
{
"size" : 1896,
"bytes" : 37341328
}

[total memory usage not shown -- I restarted bitcoind as soon as I noticed, and didn't copy it down from top]

37 MB mempool, >3 GB RAM usage. Normally, when there aren't a lot of unconfirmed txns floating around the network, memory usage is around 600 MB, so this is quite unusual.

After restarting the process and letting it run for a few minutes, I get:

PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
[###] [XXXX] 20 0 1402M 317M 49836 S 1.0 8.2 0:41.71 ./bitcoind -daemon

***@prime:~/bin$ ./bitcoin-cli getmempoolinfo
{
"size" : 1072,
"bytes" : 670000
}

0.67 MB mempool, 317 MB RAM usage. Much more reasonable.


Here's another node I'm running that has been online longer, before restarting:

PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
[###] [XXXX] 20 0 4961M 3540M 11080 S 2.8 45.3 8h20:11 bin/bitcoind -daemon

***@feather:~$ bin/bitcoin-cli getmempoolinfo
{
"size" : 3045,
"bytes" : 39656126
}

39 MB mempool, 3540 MB total memory usage. After restarting bitcoind, I see:

[XXXX]@feather:~$ bin/bitcoin-cli stop
Bitcoin server stopping
[XXXX]@feather:~$ bin/bitcoind -daemon
Bitcoin server starting
[XXXX]@feather:~$ sleep 10; bin/bitcoin-cli getmempoolinfo
{
"size" : 39,
"bytes" : 47037
}


PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
[###] [XXXX] 20 0 1640M 247M 67960 S 0.0 3.2 0:05.17 bin/bitcoind -daemon




Does anybody have any guesses where we might be leaking memory, or what is using the additional 2.4 GB? I've been using minrelaytxfee=0.00003 or similar on my nodes. Maybe there's a leak in the minrelaytxfee code path? Has anyone else seen something similar?

This issue appears to happen both with Bitcoin Core 0.10.1 and with Bitcoin XT 0.11B.

[odinn via bitcoin-dev]

You should compare this to having set up a node on a completely clean
computer. It would also help to know what operating system(s) you are
using for both the oldie and the freshie.

Also, dump your XT, is poo. Then try again, look at Core nodes on
your oldie and freshie. Watch them for a bit.

Cheers,

O

Post by Jonathan Toomim (Toomim Bros) via bitcoin-dev
I just noticed that several of my running bitcoind processes were
using around 3+ GB of RAM, even though the mempool itself seemed to
be under control.
"bytes" : 37341328 }
[total memory usage not shown -- I restarted bitcoind as soon as I
noticed, and didn't copy it down from top]
37 MB mempool, >3 GB RAM usage. Normally, when there aren't a lot
of unconfirmed txns floating around the network, memory usage is
around 600 MB, so this is quite unusual.
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+
Command [###] [XXXX] 20 0 1402M 317M 49836 S 1.0 8.2
0:41.71 ./bitcoind -daemon
"bytes" : 670000 }
0.67 MB mempool, 317 MB RAM usage. Much more reasonable.
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+
Command [###] [XXXX] 20 0 4961M 3540M 11080 S 2.8 45.3
8h20:11 bin/bitcoind -daemon
"bytes" : 39656126 }
39 MB mempool, 3540 MB total memory usage. After restarting
: 39, "bytes" : 47037 }
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+
Command [###] [XXXX] 20 0 1640M 247M 67960 S 0.0 3.2
0:05.17 bin/bitcoind -daemon
Does anybody have any guesses where we might be leaking memory, or
what is using the additional 2.4 GB? I've been using
minrelaytxfee=0.00003 or similar on my nodes. Maybe there's a leak
in the minrelaytxfee code path? Has anyone else seen something
similar?
This issue appears to happen both with Bitcoin Core 0.10.1 and with Bitcoin XT 0.11B.
_______________________________________________ bitcoin-dev mailing
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

- --
http://abis.io ~
"a protocol concept to enable decentralization
and expansion of a giving economy, and a new social good"
https://keybase.io/odinn

[Jonathan Toomim (Toomim Bros) via bitcoin-dev]

Signed PGP part
It would also help to know what operating system(s) you are
using for both the oldie and the freshie.

Linux feather 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u3 (2015-08-04) x86_64 GNU/Linux
Linux server 3.2.0-4-amd64 #1 SMP Debian 3.2.60-1+deb7u3 x86_64 GNU/Linux
Linux prime 3.2.0-4-amd64 #1 SMP Debian 3.2.63-2+deb7u2 x86_64 GNU/Linux

This excessive memory consumption was seen on 3 machines, all of which run Debian. All three machines run p2pool as well as bitcoind. Two run XT, one runs Core.

You should compare this to having set up a node on a completely clean
computer.

I can't afford to do that. All of the servers I have are being used for something. Also, I'm not sure what it is you're trying to test for with that suggestion. The numbers I'm reporting are for bitcoind's resident set, not for the whole server's memory usage. I don't see how other processes running on the same machine are relevant unless you are suggesting that RPC calls (e.g. getblocktemplate) might be somehow responsible.

Also, dump your XT, is poo.

Not relevant. I addressed this message to both the Core and XT lists because the issue appears to affect both forks. Let's keep blocksize and governance debates to their own threads, please.

Repeating request: Has anyone else seen something similar? Can you report your mempool size and total bitcoind resident set size for your running full nodes?

[Dave Scotese via bitcoin-dev]

{
"size" : 1085,
"bytes" : 16151768
}

It has been running about a day. I'll report tomorrow too. This is a
Windows 8.1 box.

16 million divided by 1085 transactions is almost 15Kb per transaction =
unlikely, right?

Signed PGP part
It would also help to know what operating system(s) you are
using for both the oldie and the freshie.
Linux feather 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u3
(2015-08-04) x86_64 GNU/Linux
Linux server 3.2.0-4-amd64 #1 SMP Debian 3.2.60-1+deb7u3 x86_64 GNU/Linux
Linux prime 3.2.0-4-amd64 #1 SMP Debian 3.2.63-2+deb7u2 x86_64 GNU/Linux
This excessive memory consumption was seen on 3 machines, all of which run
Debian. All three machines run p2pool as well as bitcoind. Two run XT, one
runs Core.
You should compare this to having set up a node on a completely clean
computer.
I can't afford to do that. All of the servers I have are being used for
something. Also, I'm not sure what it is you're trying to test for with
that suggestion. The numbers I'm reporting are for bitcoind's resident set,
not for the whole server's memory usage. I don't see how other processes
running on the same machine are relevant unless you are suggesting that RPC
calls (e.g. getblocktemplate) might be somehow responsible.
Also, dump your XT, is poo.
Not relevant. I addressed this message to both the Core and XT lists
because the issue appears to affect both forks. Let's keep blocksize and
governance debates to their own threads, please.
Repeating request: Has anyone else seen something similar? Can you report
your mempool size and total bitcoind resident set size for your running
full nodes?
--
You received this message because you are subscribed to the Google Groups
"bitcoin-xt" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

--
I like to provide some work at no charge to prove my value. Do you need a
techie?
I own Litmocracy <http://www.litmocracy.com> and Meme Racing
<http://www.memeracing.net> (in alpha).
I'm the webmaster for The Voluntaryist <http://www.voluntaryist.com> which
now accepts Bitcoin.
I also code for The Dollar Vigilante <http://dollarvigilante.com/>.
"He ought to find it more profitable to play by the rules" - Satoshi
Nakamoto

[Jonathan Toomim (Toomim Bros) via bitcoin-dev]

16 million divided by 1085 transactions is almost 15Kb per transaction = unlikely, right?

The recent spam was about 15 kB per transaction, so that part sounds right.

The anomalous thing that I saw was that the total bitcoind process usage was about 50-100x higher than I would have expected if the mempool was the main determinant of memory usage scaling. Can you tell me how much memory Task Manager is reporting your bitcoin process as using both today and tomorrow?

{
"size" : 1085,
"bytes" : 16151768
}
It has been running about a day. I'll report tomorrow too. This is a Windows 8.1 box.
16 million divided by 1085 transactions is almost 15Kb per transaction = unlikely, right?

[Dave Scotese via bitcoin-dev]

It was about 360MB (30 minutes ago?), but is now about 460MB. I'm sure it
won't keep going up that fast.
{
"size" : 3413,
"bytes" : 41892350
}

16 million divided by 1085 transactions is almost 15Kb per transaction = unlikely, right?
The recent spam was about 15 kB per transaction, so that part sounds right.
The anomalous thing that I saw was that the total bitcoind process usage
was about 50-100x higher than I would have expected if the mempool was the
main determinant of memory usage scaling. Can you tell me how much memory
Task Manager is reporting your bitcoin process as using both today and
tomorrow?
On Oct 13, 2015, at 4:52 PM, Dave Scotese via bitcoin-dev <
{
"size" : 1085,
"bytes" : 16151768
}
It has been running about a day. I'll report tomorrow too. This is a Windows 8.1 box.
16 million divided by 1085 transactions is almost 15Kb per transaction = unlikely, right?
--
You received this message because you are subscribed to the Google Groups
"bitcoin-xt" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

--
I like to provide some work at no charge to prove my value. Do you need a
techie?
I own Litmocracy <http://www.litmocracy.com> and Meme Racing
<http://www.memeracing.net> (in alpha).
I'm the webmaster for The Voluntaryist <http://www.voluntaryist.com> which
now accepts Bitcoin.
I also code for The Dollar Vigilante <http://dollarvigilante.com/>.
"He ought to find it more profitable to play by the rules" - Satoshi
Nakamoto

[odinn via bitcoin-dev]

You wanted advice... you got it

On Oct 13, 2015, at 3:49 PM, odinn

Signed PGP part It would also help to know what operating
system(s) you are using for both the oldie and the freshie.

Linux feather 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u3
(2015-08-04) x86_64 GNU/Linux Linux server 3.2.0-4-amd64 #1 SMP
Debian 3.2.60-1+deb7u3 x86_64 GNU/Linux Linux prime 3.2.0-4-amd64
#1 SMP Debian 3.2.63-2+deb7u2 x86_64 GNU/Linux
This excessive memory consumption was seen on 3 machines, all of
which run Debian. All three machines run p2pool as well as
bitcoind. Two run XT, one runs Core.

You should compare this to having set up a node on a completely
clean computer.

I can't afford to do that. All of the servers I have are being used
for something. Also, I'm not sure what it is you're trying to test
for with that suggestion. The numbers I'm reporting are for
bitcoind's resident set, not for the whole server's memory usage. I
don't see how other processes running on the same machine are
relevant unless you are suggesting that RPC calls (e.g.
getblocktemplate) might be somehow responsible.

Also, dump your XT, is poo.

Not relevant. I addressed this message to both the Core and XT
lists because the issue appears to affect both forks. Let's keep
blocksize and governance debates to their own threads, please.
Repeating request: Has anyone else seen something similar? Can you
report your mempool size and total bitcoind resident set size for
your running full nodes?

- --
http://abis.io ~
"a protocol concept to enable decentralization
and expansion of a giving economy, and a new social good"
https://keybase.io/odinn

[Pavel Janík via bitcoin-dev]

Post by Jonathan Toomim (Toomim Bros) via bitcoin-dev
I just noticed that several of my running bitcoind processes were using around 3+ GB of RAM, even though the mempool itself seemed to be under control.
{
"size" : 1896,
"bytes" : 37341328
}
[total memory usage not shown -- I restarted bitcoind as soon as I noticed, and didn't copy it down from top]

The newer versions contain much more accurate reporting of memorypool usage that reflect the real memory usage:

{
"size": 58536,
"bytes": 770044257,
"usage": 1652554832
}

See PR #6410 (Implement accurate memory accounting for mempool).
--
Pavel Janík

[Tom Zander via bitcoin-dev]

Post by Jonathan Toomim (Toomim Bros) via bitcoin-dev
Does anybody have any guesses where we might be leaking memory, or what is
using the additional 2.4 GB? I've been using minrelaytxfee=0.00003 or
similar on my nodes. Maybe there's a leak in the minrelaytxfee code path?
Has anyone else seen something similar?

I suggest running it in valgrind with --leak-check=full for 10 minutes.

valgrind --leak-check=full src/bitcoind 2>&1 | tee out

This at least will show you any memory leaks at exit.
Naturally, the leaks you observe may just be design issues where cache can
grow to much and when the cache is cleaned on shutdown you won't see it in the
valgrind output.

[Mike Hearn via bitcoin-dev]

Leaks are not the only explanation possible. Caches and fragmentation can
also give this sort of effect. Unfortunately the tools to debug this aren't
great. You could try a build with tcmalloc and use it to investigate heap
stats.

Odinn, trolling like a 3 year old will get you swiftly banned. Last warning.

Post by Tom Zander via bitcoin-dev

Post by Jonathan Toomim (Toomim Bros) via bitcoin-dev
Does anybody have any guesses where we might be leaking memory, or what

is

Post by Jonathan Toomim (Toomim Bros) via bitcoin-dev
using the additional 2.4 GB? I've been using minrelaytxfee=0.00003 or
similar on my nodes. Maybe there's a leak in the minrelaytxfee code path?
Has anyone else seen something similar?

I suggest running it in valgrind with --leak-check=full for 10 minutes.
valgrind --leak-check=full src/bitcoind 2>&1 | tee out
This at least will show you any memory leaks at exit.
Naturally, the leaks you observe may just be design issues where cache can
grow to much and when the cache is cleaned on shutdown you won't see it in the
valgrind output.
--
You received this message because you are subscribed to the Google Groups
"bitcoin-xt" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

[odinn via bitcoin-dev]

For the record, Mr. Hearn, you do not own this list. I submit to you
that you have very little to say on this matter at this stage and your
idle threats to "ban people" based on their preferences, suggestions,
or characterizations of your chosen software project are at best very
silly.

I am not and never was on bitcoin-***@googlegroups.com and thus have
clipped that from the reply-to.

Post by Mike Hearn via bitcoin-dev
Leaks are not the only explanation possible. Caches and
fragmentation can also give this sort of effect. Unfortunately the
tools to debug this aren't great. You could try a build with
tcmalloc and use it to investigate heap stats.
Odinn, trolling like a 3 year old will get you swiftly banned. Last

Post by Tom Zander via bitcoin-dev

Post by Jonathan Toomim (Toomim Bros) via bitcoin-dev
Does anybody have any guesses where we might be leaking memory, or what

is

Post by Jonathan Toomim (Toomim Bros) via bitcoin-dev
using the additional 2.4 GB? I've been using
minrelaytxfee=0.00003 or similar on my nodes. Maybe there's a
leak in the minrelaytxfee code path? Has anyone else seen
something similar?

I suggest running it in valgrind with --leak-check=full for 10 minutes.
valgrind --leak-check=full src/bitcoind 2>&1 | tee out
This at least will show you any memory leaks at exit. Naturally,
the leaks you observe may just be design issues where cache can
grow to much and when the cache is cleaned on shutdown you won't
see it in the valgrind output.
-- You received this message because you are subscribed to the
Google Groups "bitcoin-xt" group. To unsubscribe from this group
and stop receiving emails from it, send an email to
https://groups.google.com/d/optout.

_______________________________________________ bitcoin-dev mailing
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

- --
http://abis.io ~
"a protocol concept to enable decentralization
and expansion of a giving economy, and a new social good"
https://keybase.io/odinn

[Wladimir J. van der Laan via bitcoin-dev]

Post by Jonathan Toomim (Toomim Bros) via bitcoin-dev
I just noticed that several of my running bitcoind processes were using around 3+ GB of RAM, even though the mempool itself seemed to be under control.
{
"size" : 1896,
"bytes" : 37341328
}
[total memory usage not shown -- I restarted bitcoind as soon as I noticed, and didn't copy it down from top]

This is *most likely* the mempool, but is just not reported correctly.

In the meantime you can mitigate the mempool growth by setting `-mintxfee`, see
https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding

This will be the default in 0.11.1 and 0.10.3. 0.12.0 will have better mempool management, as well as more precise reporting (as paveljanik already mentions.)

Wladimir

[Jonathan Toomim via bitcoin-dev]

Post by Wladimir J. van der Laan via bitcoin-dev
This is *most likely* the mempool, but is just not reported correctly.

I did some testing with PR #6410's better mempool reporting. The improved reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is about 2.5x to 3x higher than the serialized transaction sizes ("bytes":). The excess memory usage that I'm seeing is on the order of 100x higher than the mempool "bytes": value. As such, I think it's unlikely that this is the mempool, or at least not normal/correct mempool behavior.

Post by Wladimir J. van der Laan via bitcoin-dev
In the meantime you can mitigate the mempool growth by setting `-mintxfee`, see
https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding

I have mintxfee and minrelaytxfee set to about 0.00003, which is high enough to exclude essentially all of the of the 14700-14800 byte flood transactions. My nodes' mempools only contain about one or two blocks' worth of transactions. So I don't think this is correct either.



Some additional notes on this issue:

1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a Core bitcoind process (commit d78a880) overnight with no mining connected to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes, and RSS climbed to 1180 MB. An hour after that with more getblocktemplates, and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.] getmempoolinfo is still showing "usage" around 25MB or less.

I'll do some more testing with this and see if I can make it repeatable, and record the results more carefully. Expect a follow-up from me in a day or two.

2. valgrind did not show anything super promising. It did report this:

==6880== LEAK SUMMARY:
==6880== definitely lost: 0 bytes in 0 blocks
==6880== indirectly lost: 0 bytes in 0 blocks
==6880== possibly lost: 288 bytes in 1 blocks
==6880== still reachable: 10,552 bytes in 39 blocks
==6880== suppressed: 0 bytes in 0 blocks
(Bitcoin Core commit d78a880)

and this:
==6778== LEAK SUMMARY:
==6778== definitely lost: 0 bytes in 0 blocks
==6778== indirectly lost: 0 bytes in 0 blocks
==6778== possibly lost: 320 bytes in 1 blocks
==6778== still reachable: 10,080 bytes in 32 blocks
==6778== suppressed: 0 bytes in 0 blocks
(Bitcoin XT commit fe446d)

I haven't found anything in there yet that I think would produce the multi-GB memory usage after running for a few days, but I could be missing it. Email me if you want the full log.

I did not try running getblocktemplate while valgrind was running. I'll have to try that. I also have not let valgrind run for more than an hour.



P.S.: Sorry for all the cross-post confusion and consequent flamewar fallout. While it's probably too late for this thread, I'll make sure to post in a manner that keeps the threads clearly separate in the future (e.g. different subject lines).

[Multipool Admin via bitcoin-dev]

My nodes are continuously running getblocktemplate and getinfo, and I also
suspected the issue is in either gbt or the rpc server.

The instance only takes a few hours to get up to that memory usage.
On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" <

Post by Wladimir J. van der Laan via bitcoin-dev
This is *most likely* the mempool, but is just not reported correctly.
I did some testing with PR #6410's better mempool reporting. The improved
reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is
about 2.5x to 3x higher than the serialized transaction sizes ("bytes":).
The excess memory usage that I'm seeing is on the order of 100x higher than
the mempool "bytes": value. As such, I think it's unlikely that this is the
mempool, or at least not normal/correct mempool behavior.
guessing his bitcoind has been running longer than any of mine. His server
definitely has more RAM. I don't know which email list he is subscribed to
(probably XT), so I'm sharing it with both lists to make sure you're all
aware of how big an issue this can be.
In the meantime you can mitigate the mempool growth by setting
`-mintxfee`, see
https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding
I have mintxfee and minrelaytxfee set to about 0.00003, which is high
enough to exclude essentially all of the of the 14700-14800 byte flood
transactions. My nodes' mempools only contain about one or two blocks'
worth of transactions. So I don't think this is correct either.
1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a
Core bitcoind process (commit d78a880) overnight with no mining connected
to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around
400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I
ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around
800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes,
and RSS climbed to 1180 MB. An hour after that with more getblocktemplates,
and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.]
getmempoolinfo is still showing "usage" around 25MB or less.
I'll do some more testing with this and see if I can make it repeatable,
and record the results more carefully. Expect a follow-up from me in a day
or two.
==6880== definitely lost: 0 bytes in 0 blocks
==6880== indirectly lost: 0 bytes in 0 blocks
==6880== possibly lost: 288 bytes in 1 blocks
==6880== still reachable: 10,552 bytes in 39 blocks
==6880== suppressed: 0 bytes in 0 blocks
(Bitcoin Core commit d78a880)
==6778== definitely lost: 0 bytes in 0 blocks
==6778== indirectly lost: 0 bytes in 0 blocks
==6778== possibly lost: 320 bytes in 1 blocks
==6778== still reachable: 10,080 bytes in 32 blocks
==6778== suppressed: 0 bytes in 0 blocks
(Bitcoin XT commit fe446d)
I haven't found anything in there yet that I think would produce the
multi-GB memory usage after running for a few days, but I could be missing
it. Email me if you want the full log.
I did not try running getblocktemplate while valgrind was running. I'll
have to try that. I also have not let valgrind run for more than an hour.
P.S.: Sorry for all the cross-post confusion and consequent flamewar
fallout. While it's probably too late for this thread, I'll make sure to
post in a manner that keeps the threads clearly separate in the future
(e.g. different subject lines).
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[Multipool Admin via bitcoin-dev]

I should also mention that this is definitely not an attack coming from
connected nodes. My node experiencing the issue is only connected to 3
other nodes, all of which I control (via connect=).

--Adam

Post by Multipool Admin via bitcoin-dev
My nodes are continuously running getblocktemplate and getinfo, and I also
suspected the issue is in either gbt or the rpc server.
The instance only takes a few hours to get up to that memory usage.
On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" <

Post by Wladimir J. van der Laan via bitcoin-dev
This is *most likely* the mempool, but is just not reported correctly.
I did some testing with PR #6410's better mempool reporting. The improved
reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is
about 2.5x to 3x higher than the serialized transaction sizes ("bytes":).
The excess memory usage that I'm seeing is on the order of 100x higher than
the mempool "bytes": value. As such, I think it's unlikely that this is the
mempool, or at least not normal/correct mempool behavior.
guessing his bitcoind has been running longer than any of mine. His server
definitely has more RAM. I don't know which email list he is subscribed to
(probably XT), so I'm sharing it with both lists to make sure you're all
aware of how big an issue this can be.
In the meantime you can mitigate the mempool growth by setting `-mintxfee`, see
https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding
I have mintxfee and minrelaytxfee set to about 0.00003, which is high
enough to exclude essentially all of the of the 14700-14800 byte flood
transactions. My nodes' mempools only contain about one or two blocks'
worth of transactions. So I don't think this is correct either.
1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a
Core bitcoind process (commit d78a880) overnight with no mining connected
to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around
400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I
ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around
800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes,
and RSS climbed to 1180 MB. An hour after that with more getblocktemplates,
and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.]
getmempoolinfo is still showing "usage" around 25MB or less.
I'll do some more testing with this and see if I can make it repeatable,
and record the results more carefully. Expect a follow-up from me in a day
or two.
==6880== definitely lost: 0 bytes in 0 blocks
==6880== indirectly lost: 0 bytes in 0 blocks
==6880== possibly lost: 288 bytes in 1 blocks
==6880== still reachable: 10,552 bytes in 39 blocks
==6880== suppressed: 0 bytes in 0 blocks
(Bitcoin Core commit d78a880)
==6778== definitely lost: 0 bytes in 0 blocks
==6778== indirectly lost: 0 bytes in 0 blocks
==6778== possibly lost: 320 bytes in 1 blocks
==6778== still reachable: 10,080 bytes in 32 blocks
==6778== suppressed: 0 bytes in 0 blocks
(Bitcoin XT commit fe446d)
I haven't found anything in there yet that I think would produce the
multi-GB memory usage after running for a few days, but I could be missing
it. Email me if you want the full log.
I did not try running getblocktemplate while valgrind was running. I'll
have to try that. I also have not let valgrind run for more than an hour.
P.S.: Sorry for all the cross-post confusion and consequent flamewar
fallout. While it's probably too late for this thread, I'll make sure to
post in a manner that keeps the threads clearly separate in the future
(e.g. different subject lines).
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[Mike Hearn via bitcoin-dev]

OK, then running under Valgrind whilst sending gbt RPCs would be the next
step.

Post by Multipool Admin via bitcoin-dev
My nodes are continuously running getblocktemplate and getinfo, and I also
suspected the issue is in either gbt or the rpc server.
The instance only takes a few hours to get up to that memory usage.
On Oct 18, 2015 8:59 AM, "Jonathan Toomim via bitcoin-dev" <

Post by Wladimir J. van der Laan via bitcoin-dev
This is *most likely* the mempool, but is just not reported correctly.
I did some testing with PR #6410's better mempool reporting. The improved
reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is
about 2.5x to 3x higher than the serialized transaction sizes ("bytes":).
The excess memory usage that I'm seeing is on the order of 100x higher than
the mempool "bytes": value. As such, I think it's unlikely that this is the
mempool, or at least not normal/correct mempool behavior.
guessing his bitcoind has been running longer than any of mine. His server
definitely has more RAM. I don't know which email list he is subscribed to
(probably XT), so I'm sharing it with both lists to make sure you're all
aware of how big an issue this can be.
In the meantime you can mitigate the mempool growth by setting `-mintxfee`, see
https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding
I have mintxfee and minrelaytxfee set to about 0.00003, which is high
enough to exclude essentially all of the of the 14700-14800 byte flood
transactions. My nodes' mempools only contain about one or two blocks'
worth of transactions. So I don't think this is correct either.
1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a
Core bitcoind process (commit d78a880) overnight with no mining connected
to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around
400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I
ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around
800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes,
and RSS climbed to 1180 MB. An hour after that with more getblocktemplates,
and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.]
getmempoolinfo is still showing "usage" around 25MB or less.
I'll do some more testing with this and see if I can make it repeatable,
and record the results more carefully. Expect a follow-up from me in a day
or two.
==6880== definitely lost: 0 bytes in 0 blocks
==6880== indirectly lost: 0 bytes in 0 blocks
==6880== possibly lost: 288 bytes in 1 blocks
==6880== still reachable: 10,552 bytes in 39 blocks
==6880== suppressed: 0 bytes in 0 blocks
(Bitcoin Core commit d78a880)
==6778== definitely lost: 0 bytes in 0 blocks
==6778== indirectly lost: 0 bytes in 0 blocks
==6778== possibly lost: 320 bytes in 1 blocks
==6778== still reachable: 10,080 bytes in 32 blocks
==6778== suppressed: 0 bytes in 0 blocks
(Bitcoin XT commit fe446d)
I haven't found anything in there yet that I think would produce the
multi-GB memory usage after running for a few days, but I could be missing
it. Email me if you want the full log.
I did not try running getblocktemplate while valgrind was running. I'll
have to try that. I also have not let valgrind run for more than an hour.
P.S.: Sorry for all the cross-post confusion and consequent flamewar
fallout. While it's probably too late for this thread, I'll make sure to
post in a manner that keeps the threads clearly separate in the future
(e.g. different subject lines).
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
--

You received this message because you are subscribed to the Google Groups
"bitcoin-xt" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.

[Jonathan Toomim via bitcoin-dev]

I did that Sunday twice. I'll report the results soon. Short version is that it looks like valgrind is just finding 200 kB to 600 kB of pblocktemplate, which is declared as a static pointer. Not exactly the multi-GB leak I'm looking for, but possibly related.

I've also got two bitcoind processes running on the same machine that I started at the same time, running on different ports, all with the same settings, but one of which is serving getblocktemplate every 5-6 seconds and the other is not, while logging RSS on both every 6 seconds. RSS for the non-serving node is now 734 MB, and for the serving node 1997 MB. Graphs coming soon.

OK, then running under Valgrind whilst sending gbt RPCs would be the next step.
My nodes are continuously running getblocktemplate and getinfo, and I also suspected the issue is in either gbt or the rpc server.
The instance only takes a few hours to get up to that memory usage.

Post by Wladimir J. van der Laan via bitcoin-dev
This is *most likely* the mempool, but is just not reported correctly.

I did some testing with PR #6410's better mempool reporting. The improved reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is about 2.5x to 3x higher than the serialized transaction sizes ("bytes":). The excess memory usage that I'm seeing is on the order of 100x higher than the mempool "bytes": value. As such, I think it's unlikely that this is the mempool, or at least not normal/correct mempool behavior.

Post by Wladimir J. van der Laan via bitcoin-dev
In the meantime you can mitigate the mempool growth by setting `-mintxfee`, see
https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding

I have mintxfee and minrelaytxfee set to about 0.00003, which is high enough to exclude essentially all of the of the 14700-14800 byte flood transactions. My nodes' mempools only contain about one or two blocks' worth of transactions. So I don't think this is correct either.
1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a Core bitcoind process (commit d78a880) overnight with no mining connected to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes, and RSS climbed to 1180 MB. An hour after that with more getblocktemplates, and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.] getmempoolinfo is still showing "usage" around 25MB or less.
I'll do some more testing with this and see if I can make it repeatable, and record the results more carefully. Expect a follow-up from me in a day or two.
==6880== definitely lost: 0 bytes in 0 blocks
==6880== indirectly lost: 0 bytes in 0 blocks
==6880== possibly lost: 288 bytes in 1 blocks
==6880== still reachable: 10,552 bytes in 39 blocks
==6880== suppressed: 0 bytes in 0 blocks
(Bitcoin Core commit d78a880)
==6778== definitely lost: 0 bytes in 0 blocks
==6778== indirectly lost: 0 bytes in 0 blocks
==6778== possibly lost: 320 bytes in 1 blocks
==6778== still reachable: 10,080 bytes in 32 blocks
==6778== suppressed: 0 bytes in 0 blocks
(Bitcoin XT commit fe446d)
I haven't found anything in there yet that I think would produce the multi-GB memory usage after running for a few days, but I could be missing it. Email me if you want the full log.
I did not try running getblocktemplate while valgrind was running. I'll have to try that. I also have not let valgrind run for more than an hour.
P.S.: Sorry for all the cross-post confusion and consequent flamewar fallout. While it's probably too late for this thread, I'll make sure to post in a manner that keeps the threads clearly separate in the future (e.g. different subject lines).
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
--
You received this message because you are subscribed to the Google Groups "bitcoin-xt" group.
For more options, visit https://groups.google.com/d/optout.

[Jonathan Toomim via bitcoin-dev]

More notes:

1. I ran a side-by-side comparison with two bitcoind processes (Core, same recent git commit as before) on the same computer with the same settings running on different ports. With both processes, I logged RSS (via /proc/$pid/status) every 6 seconds. With one of those processes, I also ran bitcoin-cli getblocktemplate > /dev/null every 6 seconds. I let that run for about 30 hours. A graph and links to the CSVs of raw data are below. Results seem pretty clear: the getblocktemplate RPC is implicated in this issue.


http://toom.im/files/memlog8518.csv
http://toom.im/files/memlog-nogbt-8503.csv
Loading Image...


2. I ran valgrind twice, for about 6 hours each, on bitcoind while hitting it with getblocktemplate every 6 hours. Full valgrind output can be found at these two URLs:

http://toom.im/files/valgrind-gbt-1.log
http://toom.im/files/valgrind-gbt-2.log

The summary:

==4064== LEAK SUMMARY:
==4064== definitely lost: 0 bytes in 0 blocks
==4064== indirectly lost: 0 bytes in 0 blocks
==4064== possibly lost: 288 bytes in 1 blocks
==4064== still reachable: 527,594 bytes in 4,367 blocks
==4064== suppressed: 0 bytes in 0 blocks
The main components of that still reachable section seem to just be one output of CreateNewBlock that's cached in case another getblocktemplate request is received before any new transactions come in:

==4064== 98,304 bytes in 1 blocks are still reachable in loss record 39 of 40
==4064== at 0x4C29180: operator new(unsigned long) (vg_replace_malloc.c:324)
==4064== by 0x28EAA1: __gnu_cxx::new_allocator<CTransaction>::allocate(unsigned long, void const*) (new_allocator.h:104)
==4064== by 0x27EE44: __gnu_cxx::__alloc_traits<std::allocator<CTransaction> >::allocate(std::allocator<CTransaction>&, unsigned long) (alloc_traits.h:182)
==4064== by 0x26DFB0: std::_Vector_base<CTransaction, std::allocator<CTransaction> >::_M_allocate(unsigned long) (stl_vector.h:170)
==4064== by 0x2D5BDE: std::vector<CTransaction, std::allocator<CTransaction> >::_M_insert_aux(__gnu_cxx::__normal_iterator<CTransaction*, std::vector<CTransaction, std::allocator<CTransaction> > >, CTransaction const&) (vector.tcc:353)
==4064== by 0x2D3FF8: std::vector<CTransaction, std::allocator<CTransaction> >::push_back(CTransaction const&) (stl_vector.h:925)
==4064== by 0x2D113E: CreateNewBlock(CScript const&) (miner.cpp:298)
==4064== by 0x442D78: getblocktemplate(UniValue const&, bool) (rpcmining.cpp:513)
==4064== by 0x390CEB: CRPCTable::execute(std::string const&, UniValue const&) const (rpcserver.cpp:526)
==4064== by 0x41C5AB: HTTPReq_JSONRPC(HTTPRequest*, std::string const&) (httprpc.cpp:125)
==4064== by 0x3559BD: boost::detail::function::void_function_invoker2<bool (*)(HTTPRequest*, std::string const&), void, HTTPRequest*, std::string const&>::invoke(boost::detail::function::function_buffer&, HTTPRequest*, std::string const&) (function_template.hpp:112)
==4064== by 0x422520: boost::function2<void, HTTPRequest*, std::string const&>::operator()(HTTPRequest*, std::string const&) const (function_template.hpp:767)

There are a few other similar loss records (mostly referring to pblock or pblocktemplate in CreateNewBlock(...), but I see nothing that can explain the multi-GB memory consumption.

3. One user on the bitcointalk p2pool thread (https://bitcointalk.org/index.php?topic=18313.msg12733791#msg12733791) claimed that he had this memory usage issue on Linux, but not on Mac OS X, under a GBT workload in both situations. If this is true, that would suggest this might be a fragmentation issue due to poor memory allocation. The other likely hypothesis is bloated caches. Looking into those two possibilities will be my next steps.

Post by Jonathan Toomim via bitcoin-dev
I did that Sunday twice. I'll report the results soon. Short version is that it looks like valgrind is just finding 200 kB to 600 kB of pblocktemplate, which is declared as a static pointer. Not exactly the multi-GB leak I'm looking for, but possibly related.
I've also got two bitcoind processes running on the same machine that I started at the same time, running on different ports, all with the same settings, but one of which is serving getblocktemplate every 5-6 seconds and the other is not, while logging RSS on both every 6 seconds. RSS for the non-serving node is now 734 MB, and for the serving node 1997 MB. Graphs coming soon.

OK, then running under Valgrind whilst sending gbt RPCs would be the next step.
My nodes are continuously running getblocktemplate and getinfo, and I also suspected the issue is in either gbt or the rpc server.
The instance only takes a few hours to get up to that memory usage.

Post by Wladimir J. van der Laan via bitcoin-dev
This is *most likely* the mempool, but is just not reported correctly.

I did some testing with PR #6410's better mempool reporting. The improved reporting suggests that actual in-memory usage ("usage":) by CTxMemPool is about 2.5x to 3x higher than the serialized transaction sizes ("bytes":). The excess memory usage that I'm seeing is on the order of 100x higher than the mempool "bytes": value. As such, I think it's unlikely that this is the mempool, or at least not normal/correct mempool behavior.

Post by Wladimir J. van der Laan via bitcoin-dev
In the meantime you can mitigate the mempool growth by setting `-mintxfee`, see
https://github.com/bitcoin/bitcoin/blob/v0.11.0/doc/release-notes.md#transaction-flooding

I have mintxfee and minrelaytxfee set to about 0.00003, which is high enough to exclude essentially all of the of the 14700-14800 byte flood transactions. My nodes' mempools only contain about one or two blocks' worth of transactions. So I don't think this is correct either.
1. I think it's related to CreateNewBlock() and getblocktemplate. I ran a Core bitcoind process (commit d78a880) overnight with no mining connected to it, and (IIRC -- my memory is fuzzy) when I woke up it was using around 400 MB of RSS and the mempool was at around "bytes":10MB, "usage": 25MB. I ran ./bitcoin-cli getblocktemplate once, and IIRC the RSS shot up to around 800 MB. I then ran getblocktemplate every 5 seconds for about 30 minutes, and RSS climbed to 1180 MB. An hour after that with more getblocktemplates, and now RSS is at 1350 MB. [Edit: 1490 MB about 30 minutes later.] getmempoolinfo is still showing "usage" around 25MB or less.
I'll do some more testing with this and see if I can make it repeatable, and record the results more carefully. Expect a follow-up from me in a day or two.
==6880== definitely lost: 0 bytes in 0 blocks
==6880== indirectly lost: 0 bytes in 0 blocks
==6880== possibly lost: 288 bytes in 1 blocks
==6880== still reachable: 10,552 bytes in 39 blocks
==6880== suppressed: 0 bytes in 0 blocks
(Bitcoin Core commit d78a880)
==6778== definitely lost: 0 bytes in 0 blocks
==6778== indirectly lost: 0 bytes in 0 blocks
==6778== possibly lost: 320 bytes in 1 blocks
==6778== still reachable: 10,080 bytes in 32 blocks
==6778== suppressed: 0 bytes in 0 blocks
(Bitcoin XT commit fe446d)
I haven't found anything in there yet that I think would produce the multi-GB memory usage after running for a few days, but I could be missing it. Email me if you want the full log.
I did not try running getblocktemplate while valgrind was running. I'll have to try that. I also have not let valgrind run for more than an hour.
P.S.: Sorry for all the cross-post confusion and consequent flamewar fallout. While it's probably too late for this thread, I'll make sure to post in a manner that keeps the threads clearly separate in the future (e.g. different subject lines).
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
--
You received this message because you are subscribed to the Google Groups "bitcoin-xt" group.
For more options, visit https://groups.google.com/d/optout.

[Tom Zander via bitcoin-dev]

Post by Jonathan Toomim via bitcoin-dev
claimed that he had this memory usage issue on Linux, but not on Mac OS X,
under a GBT workload in both situations. If this is true, that would
suggest this might be a fragmentation issue due to poor memory allocation.

Please make sure you measure your memory usage correctly on Linux, it is
notoriously easy to get misleading info from tools like top.

I use this one on Linux.

$cat ~/bin/showmemusage
#!/bin/sh
if test -z "$1"; then
echo "need a pid"
exit
fi

mem=`echo 0 $(cat /proc/$1/smaps | grep Pss | awk '{print $2}' | \
sed 's#^#+#' ) | bc`
echo "$mem KB"

[Jonathan Toomim via bitcoin-dev]

The method I was using was essentially

grep VmRSS /proc/$pid/status

Comparing these two methods, I get

Your method (PSS):
2408313
My method (RSS):
VmRSS: 2410396 kB

Post by Tom Zander via bitcoin-dev
Please make sure you measure your memory usage correctly on Linux, it is
notoriously easy to get misleading info from tools like top.
I use this one on Linux.
$cat ~/bin/showmemusage
#!/bin/sh
if test -z "$1"; then
echo "need a pid"
exit
fi
mem=`echo 0 $(cat /proc/$1/smaps | grep Pss | awk '{print $2}' | \
sed 's#^#+#' ) | bc`
echo "$mem KB"

[Multipool Admin via bitcoin-dev]

This is a real issue. The bitcoind process is getting killed every few
days when it reaches around 55gb of usage on my server.
On Oct 21, 2015 12:29 AM, "Tom Zander via bitcoin-dev" <

Post by Jonathan Toomim via bitcoin-dev

Post by Jonathan Toomim via bitcoin-dev
claimed that he had this memory usage issue on Linux, but not on Mac OS

X,

Post by Jonathan Toomim via bitcoin-dev
under a GBT workload in both situations. If this is true, that would
suggest this might be a fragmentation issue due to poor memory

allocation.
Please make sure you measure your memory usage correctly on Linux, it is
notoriously easy to get misleading info from tools like top.
I use this one on Linux.
$cat ~/bin/showmemusage
#!/bin/sh
if test -z "$1"; then
echo "need a pid"
exit
fi
mem=`echo 0 $(cat /proc/$1/smaps | grep Pss | awk '{print $2}' | \
sed 's#^#+#' ) | bc`
echo "$mem KB"
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[Jonathan Toomim via bitcoin-dev]

You may want to add a cron job to restart bitcoind every day or two as a damage control mechanism until we figure this out.

This is a real issue. The bitcoind process is getting killed every few days when it reaches around 55gb of usage on my server.

Post by Jonathan Toomim via bitcoin-dev
claimed that he had this memory usage issue on Linux, but not on Mac OS X,
under a GBT workload in both situations. If this is true, that would
suggest this might be a fragmentation issue due to poor memory allocation.

Please make sure you measure your memory usage correctly on Linux, it is
notoriously easy to get misleading info from tools like top.
I use this one on Linux.
$cat ~/bin/showmemusage
#!/bin/sh
if test -z "$1"; then
echo "need a pid"
exit
fi
mem=`echo 0 $(cat /proc/$1/smaps | grep Pss | awk '{print $2}' | \
sed 's#^#+#' ) | bc`
echo "$mem KB"
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev
--
You received this message because you are subscribed to the Google Groups "bitcoin-xt" group.
For more options, visit https://groups.google.com/d/optout.

[Btc Drak via bitcoin-dev]

I think this thread has gotten to the stage where it should be moved
to an issue on Github and not continue to CC the bitcoin-dev list (or
any other list tbh).

On Thu, Oct 22, 2015 at 5:27 PM, Jonathan Toomim via bitcoin-dev

Post by Jonathan Toomim via bitcoin-dev
You may want to add a cron job to restart bitcoind every day or two as a
damage control mechanism until we figure this out.
This is a real issue. The bitcoind process is getting killed every few days
when it reaches around 55gb of usage on my server.
On Oct 21, 2015 12:29 AM, "Tom Zander via bitcoin-dev"

Post by Tom Zander via bitcoin-dev

Post by Jonathan Toomim via bitcoin-dev
claimed that he had this memory usage issue on Linux, but not on Mac OS X,
under a GBT workload in both situations. If this is true, that would
suggest this might be a fragmentation issue due to poor memory allocation.

Please make sure you measure your memory usage correctly on Linux, it is
notoriously easy to get misleading info from tools like top.
I use this one on Linux.
$cat ~/bin/showmemusage
#!/bin/sh
if test -z "$1"; then
echo "need a pid"
exit
fi
mem=`echo 0 $(cat /proc/$1/smaps | grep Pss | awk '{print $2}' | \
sed 's#^#+#' ) | bc`
echo "$mem KB"
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

--
You received this message because you are subscribed to the Google Groups
"bitcoin-xt" group.
To unsubscribe from this group and stop receiving emails from it, send an
For more options, visit https://groups.google.com/d/optout.
_______________________________________________
bitcoin-dev mailing list
https://lists.linuxfoundation.org/mailman/listinfo/bitcoin-dev

[Rusty Russell via bitcoin-dev]

Post by Btc Drak via bitcoin-dev
I think this thread has gotten to the stage where it should be moved
to an issue on Github and not continue to CC the bitcoin-dev list (or
any other list tbh).

Agreed. I couldn't see an issue, so I've opened one. Let's track this
there, please.

https://github.com/bitcoin/bitcoin/issues/6876

Cheers,
Rusty.

[Ross Bennetts via bitcoin-dev]

Post by Jonathan Toomim via bitcoin-dev
3. One user on the bitcointalk p2pool thread (
https://bitcointalk.org/index.php?topic=18313.msg12733791#msg12733791)
claimed that he had this memory usage issue on Linux, but not on Mac OS X,
under a GBT workload in both situations.

Thanks for the thorough investigation.
FYI, I believe I've experienced this issue on both Linux & Mac OSX.
I've moved my P2Pool/bitcoind node (rossbennetts.com:9332) from Ubuntu to
OSX since my P2Pool instability began and they both seem to become flaky
after a few hours of uptime (since ~v0.10.something). I've tried both core
and xt.
Two other bitcoin nodes, one connected to electrum-server and one
standalone, seem fine.

//long term lurker, first time poster...
Ross...

[Adam McKenna via bitcoin-dev]

I can do better than that:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND


10625 ******** 20 0 41.310g 0.035t 51172 S 1.0 57.6 47:11.17
bitcoind

Any idea what's going on here?

$ cat /proc/10625/status|grep -i VM
VmPeak: 43647420 kB
VmSize: 43319932 kB
VmLck: 60 kB
VmPin: 0 kB
VmHWM: 38286252 kB
VmRSS: 38014300 kB
VmData: 43103812 kB
VmStk: 236 kB
VmExe: 8844 kB
VmLib: 194656 kB
VmPTE: 79932 kB
VmSwap: 0 kB


On Tuesday, October 13, 2015 at 2:56:15 PM UTC-7, Jonathan Toomim (Toomim

Post by Jonathan Toomim (Toomim Bros) via bitcoin-dev
I just noticed that several of my running bitcoind processes were using
around 3+ GB of RAM, even though the mempool itself seemed to be under
control.
{
"size" : 1896,
"bytes" : 37341328
}
[total memory usage not shown -- I restarted bitcoind as soon as I
noticed, and didn't copy it down from top]
37 MB mempool, >3 GB RAM usage. Normally, when there aren't a lot of
unconfirmed txns floating around the network, memory usage is around 600
MB, so this is quite unusual.
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
[###] [XXXX] 20 0 1402M 317M 49836 S 1.0 8.2 0:41.71 ./bitcoind -daemon
{
"size" : 1072,
"bytes" : 670000
}
0.67 MB mempool, 317 MB RAM usage. Much more reasonable.
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
[###] [XXXX] 20 0 4961M 3540M 11080 S 2.8 45.3 8h20:11
bin/bitcoind -daemon
{
"size" : 3045,
"bytes" : 39656126
}
Bitcoin server stopping
Bitcoin server starting
{
"size" : 39,
"bytes" : 47037
}
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
[###] [XXXX] 20 0 1640M 247M 67960 S 0.0 3.2 0:05.17
bin/bitcoind -daemon
Does anybody have any guesses where we might be leaking memory, or what is
using the additional 2.4 GB? I've been using minrelaytxfee=0.00003 or
similar on my nodes. Maybe there's a leak in the minrelaytxfee code path?
Has anyone else seen something similar?
This issue appears to happen both with Bitcoin Core 0.10.1 and with Bitcoin XT 0.11B.