[Networkit] graph.indexEdges() dies on large graph
maximilian.vogel at student.kit.edu
Fri May 20 11:26:05 CEST 2016
I tried to estimate the size of the graph object. It takes about 2GB
(including edge ids), probably more.
On 19.05.2016 22:21, Stephen North wrote:
> I get various errors.
Which errors besides a stopping kernel?
> That’s about 60 bytes of memory per edge.
An edge is stored in the adjacency array of both nodes to enable fast
iterations over the neighbors of a node. Hence, an edge (including its
id) takes up 32 byte (not counting in any overhead of C++'s vector).
Also, do you have the full 4GB available? What about the OS's and other
processes usage of memory?
> Any thoughts about that? Would one expect to see an error when we hit the
> wall with memory, or only random errors? (ipython3 just says the kernel stopped
> and it’s going to restart it automatically.)
You're raising an interesting point. When I'm out of memory, the kernel
stops or the ipython shell just crashes. I guess it comes down to what
happens internally. If the C++ runtime just "crashes", I don't know
(yet) what can be done. If a bad allocation exception is thrown, this
could and should be passed to Python already. (At least I vaguely recall
receiving such an error in an IPython terminal.)
> I can try a bigger VM, so how big?
At least 8GB. It also depends on how many and which experiments you want
to do as you need some memory for them as well.
Please keep us updated if the errors persist with more memory.
More information about the NetworKit