Hans and Robin,

Thanks for the tip! 🙂

But I believe I didn't explained correctly myself.

In the code you suggest, I understand the local connections are normalized using the global amount of weights. That is, the sum of all the weights in the network will be one. This is not what I am interested in. I just want to normalize the sum of the presynaptic weights to a neuron (target). That is, I am interested in the same idea you publish on your weight normalization documentation page https://nest-simulator.readthedocs.io/en/v3.3/guides/weight_normalization.html

I guessed that the example code in this documentation page would work, because it is OK for me to receive only the targets on the MPI process executing the command (as you say in the documentation) because I want to normalize for every target neuron. And it works!, but only if I do not use MPI.

I am sorry if I am missing some point.

Best,

Xavier


From: Hans Ekkehard Plesser <hans.ekkehard.plesser@nmbu.no>
Sent: Thursday, March 16, 2023 3:44 PM
To: NEST User Mailing List <users@nest-simulator.org>
Subject: [NEST Users] Re: MPI error when using GetConnections
 

 

Hi Robin,

 

Thanks for pitching in—I'm not an expert in MPI4py myself ;).

 

local_data = nest.GetConnections(...)

global_data = COMM_WORLD.allgather(local_data)

 

`global_data` will now be a list of local_data's collected from all the nodes. I suppose `allgather` might be a bit overkill, because all data is gathered on all nodes. There's also `COMM_WORLD.gather` which collects the data on just 1 node.

 

This will not quite work and is not as efficient as possible in this case. GetConnections returns a synapse collection which can only be used to access local neurons. One needs to communicate the actual synapse properties of interest between processes. Furthermore, in this case we only need the total weight of the synapses, so we can first compute that locally and then reduce:

 

       conn = nest.GetConnections(target=neuron, synapse_model='static_synapse')

       w_syns = np.array( conn.weight )
       w_total = COMM_WORLD.allreduce(np.abs(w_syns).sum())
       if w_total > 0:

                w_normed = w_syns / w_total

                  ...

 

If you normalize several times during a longer simulation, you should call GetConnections() only the first time you need it, since it can be slow. As long as you do not add or remove connections, the conn object will remain valid.

 

I'd be much interested in how these ideas work :).

 

BTW, we will soon open abstract submission for this year's NEST Conference (https://nest-simulator.org/conference).

You could show your work there and discuss with the NEST community!

 

Best,

Hans Ekkehard

 

-- 

 

Prof. Dr. Hans Ekkehard Plesser

Head, Department of Data Science

 

Faculty of Science and Technology

Norwegian University of Life Sciences

PO Box 5003, 1432 Aas, Norway

 

Phone +47 6723 1560

Email hans.ekkehard.plesser@nmbu.no

Home http://arken.nmbu.no/~plesser

 

 

 

From: Robin Gilbert De Schepper <robingilbert.deschepper@unipv.it>
Date: Thursday, 16 March 2023 at 14:30
To: NEST User Mailing List <users@nest-simulator.org>
Subject: [NEST Users] Re: MPI error when using GetConnections

I can lend some insights to the `mpi4py` side of things. You can `pip install mpi4py`, but if your machine contains multiple MPI implementations, make sure to prepend the correct MPI wrappers to the command, e.g.:

 

CC=mpicc pip install mpi4py

 

Afterwards you can use:

 

import nest

from mpi4py.MPI import COMM_WORLD

 

...

 

local_data = nest.GetConnections(...)

global_data = COMM_WORLD.allgather(local_data)

 

`global_data` will now be a list of local_data's collected from all the nodes. I suppose `allgather` might be a bit overkill, because all data is gathered on all nodes. There's also `COMM_WORLD.gather` which collects the data on just 1 node.

 

On Thu, 16 Mar 2023 at 14:19, Xavier Otazu Porter <xotazu@cvc.uab.cat> wrote:

 

Hans,

 

 

> We clearly need to fix the bug. Could you create a minimal reproducer and then create an issue https://github.com/nest/nest-simulator/issues?

 

Yes I will.

 

> As a work-around for global normalization of weights, you could try to use MPI4Py to compute the sum of the local weights.

 

I should learn MPI4Py. I am not an expert in MPI (although I am fluent with threads programming). In addition I am clueless about what pyNEST functions to use in order to gather all the pre-synaptic connections of a neuron (different to GetConnections() ).

 

 

> If your network sizes do not go into the millions, maybe systems with large numbers of threads could help, e.g., JUSUF which has computed notes with 128 cores. You can apply for access through FENIX (https://fenix-ri.eu). The Human Brain Project provides relatively easy-to-apply for resources there.

 

The problem is that my architecture sometimes (many times, in fact) needs more than 128Gb RAM (just the RAM size of the nodes the cluster I am working on). Hence, I need to work with MPI to distribute my architecture through several nodes.

 

Now I am working with 1 mpi process (hence, just one node) and 32 threads. But it implies to work with "small" architectures.

 

 

> You could also issue a feature request issue on NEST for support for global normalization.

 

I will!

 

Thanks a lot in advance!

 

Xavier

 

 

 

 

 

 

Best regards,

Hans Ekkehard

 

-- 

 

Prof. Dr. Hans Ekkehard Plesser

Head, Department of Data Science

 

Faculty of Science and Technology

Norwegian University of Life Sciences

PO Box 5003, 1432 Aas, Norway

 

Phone +47 6723 1560

Email hans.ekkehard.plesser@nmbu.no

Home http://arken.nmbu.no/~plesser

 

 

 

From: Xavier Otazu <xotazu@cvc.uab.cat>
Date: Thursday, 16 March 2023 at 11:57
To: users@nest-simulator.org <users@nest-simulator.org>
Subject: [NEST Users] Re: MPI error when using GetConnections

[Du mottar ikke ofte e-post fra xotazu@cvc.uab.cat. Finn ut hvorfor dette er viktig p? https://aka.ms/LearnAboutSenderIdentification ]

Hi Hans!

Commenting out the DumpLayerConnections() lines, it also crashes.

In addition, the problem disappears when I reduce the size of the number of neurons or the connection shape (but it is not practical for me).

Hence, if it is only the normalization of the local MPI process, I understand the only way to normalize the overall sum of weights of a particular neuron is to not using MPI? It is impractical for me because I use big networks (several tenths of thousands).

Cheers,

Xavier
_______________________________________________
NEST Users mailing list -- users@nest-simulator.org
To unsubscribe send an email to users-leave@nest-simulator.org


Computer Vision Center
CONFIDENTIALITY WARNING

_______________________________________________
NEST Users mailing list -- users@nest-simulator.org
To unsubscribe send an email to users-leave@nest-simulator.org


 

--

Robin De Schepper, MSc (they/them)

Department of Brain and Behavioral Sciences

Unit of Neurophysiology

University of Pavia, Italy

Via Forlanini 6, 27100 Pavia - Italy

Tel: (+39) 038298-7607

http://www-5.unipv.it/dangelo/


Computer Vision Center
CONFIDENTIALITY WARNING