[Networkit] Distributed graph analysis in NetworKit

Ivan Raikov ivan.g.raikov at gmail.com
Mon Jul 24 20:53:01 CEST 2017


Hi Christian,

    Thanks for your suggestions. I did indeed look at some solutions
for distributed graph processing, but I was strongly biased towards
MPI so I wanted something that is easy to deploy on an MPI-based HPC
cluster and extensible via Python. GraphLab seems to have been bought
out and unless I am mistaken, the only available open-source
components are the basic graph datastructures, but not the analysis
routines. But as I already wrote on in my response in Henning, I will
try running NetworKit on a large-memory cluster node and see if that
might work. Thanks,

-Ivan

On Sat, Jul 22, 2017 at 2:45 AM, Christian Staudt
<christian.staudt at kit.edu> wrote:
> Hi Ivan,
> get more RAM. If that doesn’t help: Have you considered the many existing solutions for distributed graph processing? We had a paper on this a while ago [1], and surely there have been developments since then.
>
> @all: Should there ever be plans to write a new distributed graph computing backend, my hope is that there will be a careful discussion about whether adding that to NetworKit is a case of scope creep. [2][3]
>
> Best regards
> Chris
>
>
>
> [1]: https://www.researchgate.net/publication/289117288_An_Empirical_Comparison_of_Big_Graph_Frameworks_in_the_Context_of_Network_Analysis
> [2]: https://www.youtube.com/watch?v=kmJ59yyYza4
> [3]: https://www.pmi.org/learning/library/top-five-causes-scope-creep-6675
>
>> On 22. Jul 2017, at 07:32, Henning Meyerhenke <henning.meyerhenke at kit.edu> wrote:
>>
>> Dear Ivan,
>>
>> Thank you for raising this issue. This aspect has been, of course, in my
>> mind for some time. So far, there is no funding for this (nor did we
>> apply). Thus, do not expect a distributed backend any time soon.
>>
>> If there are, however, numerous serious requests in this regard, we
>> would have very good arguments towards funding agencies...
>>
>> On the other hand, for us there is rarely the need to go beyond shared
>> memory. Machines with 1TB RAM are fairly cheap these days and you can
>> fit a lot into 1TB (approx. 40G edges). This is sufficient for many
>> applications -- but certainly not for all.
>>
>> If you want to discuss your application in more detail, this could be
>> helpful for assessment. If you don't want to do that on the public
>> mailing list, feel free to send to my personal email address.
>>
>> Best,
>> Henning
>>
>>
>>
>> Am 21.07.17 um 20:51 schrieb Ivan Raikov:
>>> Hello,
>>>
>>>   I have recently come across NetworKit and rather like its interface
>>> and features. However, some of the graphs I would like to analyze are
>>> too big to fit in the memory of a single workstation, and so I wonder
>>> if there are any plans or interest in providing functionality for
>>> graph analysis in distributed-memory compute environments, such as
>>> MPI-based clusters. Thanks and best regards,
>>>
>>> -Ivan Raikov
>>>
>>> _______________________________________________
>>> NetworKit mailing list
>>> NetworKit at ira.uni-karlsruhe.de
>>> https://lists.ira.uni-karlsruhe.de/mailman/listinfo/networkit
>>>
>>
>> --
>>
>> ==========================================================
>> Karlsruhe Institute of Technology (KIT)
>> Institute of Theoretical Informatics (ITI)
>>
>> Prof. Dr. Henning Meyerhenke
>> Theoret. Informatics / Parallel Computing
>>
>> Phone: +49-721-608-41876
>> Web: http://parco.iti.kit.edu/henningm/
>>
>> KIT - The Research University in the Helmholtz Association
>> ==========================================================
>>
>> _______________________________________________
>> NetworKit mailing list
>> NetworKit at ira.uni-karlsruhe.de
>> https://lists.ira.uni-karlsruhe.de/mailman/listinfo/networkit
>
>
> _______________________________________________
> NetworKit mailing list
> NetworKit at ira.uni-karlsruhe.de
> https://lists.ira.uni-karlsruhe.de/mailman/listinfo/networkit



More information about the NetworKit mailing list