Dear Maurycy and Hans,
My name is Lorenzo Tartarini, neuroscience PhD student at UNIMORE.
I am extremely interested in the topic and willing to help developing the C++ code to achieve this feature.
I am currently working on a fairly big network (921 million synapses) built with the SONATA framework and trained with STDP. For some reason I need to chunk the nest.GetConnections(source=some_nodes) query even on an HPC with 360 GB of RAM (my saved pre,post,weight .h5 file is less than 1 GB and the edges.h5 file is 18 GB). I am currently working with a single MPI process using all 48 available cores ("local_num_threads": 48)
Building the network takes approximately 20 minutes, running thee simulation is pretty quick but then saving the simulation takes almost 10 hours, chunking the connectivity with block of almost 4 millions connections. I tried parallelizing this operation using the python multiprocessing package but received the error: python: /home/nest-simulator/nestkernel/vp_manager.h:207: void nest::VPManager::assert_thread_parallel() const: Assertion omp_get_num_threads() == static_cast< int >( n_threads_ )' failed. which, if I understood correctly, is due to OpenMP threads forking protection of NEST.
I tried the suggested pickle based solution found here: https://nest-simulator.readthedocs.io/en/v3.3/auto_examples/store_restore_ne... without a decent speed-up since the bottleneck stands in the nest connections query.
How can we get started on this? Do you have some pointers?
Best,
Lorenzo