[Networkit] Cython memory management
kolja.esders at student.kit.edu
Sun Apr 5 21:24:03 CEST 2015
Hi fellow developers,
as I am currently writing a link prediction extension for NetworKit I
discovered some weird behaviour regarding Cython memory management where
there is way more memory allocated than necessary.
In general this is no news since Cython copies objects to make them
available on the Python level. See Chris' stackoverflow post
But there are 2 curious things in my case:
- On the C++ level a specific method (MissingLinksFinder.findAll
of mine returns a vector that has a total size of ~124 MiB. Cython in
return allocates a total of 1.1 GiB of memory. Even regarding
vector-specific over-allocation this is way beyond what's expected.***
- Using std::move in the python code
is done for Graph etc.) yields no improvement.
You can have a look at a test that I added
demonstrate the C++ memory usage and compare that to the memory consumed
during execution of the following Python snippet:
from networkit import *
cd ~/some_folders/NetworKit-Esders # Edit this line accordingly
testGraph = readGraph("input/caidaRouterLevel.graph", Format.METIS)
missingLinks = linkprediction.MissingLinksFinder(trainingGraph).findAll(2)
# Around 1.1 GiB
commonNeighborsIndex = linkprediction.CommonNeighborsIndex(trainingGraph)
predictions = commonNeighborsIndex.runOnParallel(missingLinks) # Around 1.7
What are your thoughts on this? Do you encounter the same problem?
Any recommendations regarding my interface are also appreciated.
Let me know if you need more information.
** *Same is true for LinkPredictor::runOnParallel
be ~186 MiB and actually is ~1.7 GiB)
PS: I am using the newest Cython version.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the NetworKit