Hello all, I am trying to use as much parallelism as possible on my computing grid. In this sense, I combine MPI+OpenMP. MPI processes are spawned fine, but OpenMP threads always amount to 1, no matter that I set local_num_threads correctly (https://nest-simulator.readthedocs.io/en/latest/hpc/slurm_script.html#set-l…) as well as all SLURM variables. I am sure I have compiled nest with OpenMP too. I use slightly modified izhikevich neurons with stdp synapses. Can the model specifics be the cause of underutilizing available threads? Or is there something else in the system that could be the reason for that? Best regards, -- Stefan Dvoretskii _______________________________________________ NEST Users mailing list --users(a)nest-simulator.org To unsubscribe send an email tousers-leave(a)nest-simulator.org

Dear Sandra, I have done all of what you describe, and still only one OpenMP thread is being used on each node. (I set all above listed variables to 16 threads) For reproducibility: I attach my SLURM batch script and the simulation script. I use NEST 3.4 compiled with openmp and mpi, and Intel(R) MPI Library for Linux* OS, Version 2019 Update 12 Build 20210429 (id: e380127cb). Best regards, On Sun, 6 Aug 2023 at 12:02, Sandra Diaz &lt;s.diaz(a)fz-juelich.de&gt; wrote: Dear Stefan, Have you tried setting the OMP variable in your batch script? Something like: export OMP_NUM_THREADS=<No. of threads> You should do this before your srun call to the nest script and in combination to setting the local_num_threads, as well as the right value for -c (or --cpus-per-task). All the best, Sandra On 05.08.23 13:05, Stefan Dvoretskii wrote: -- Dr. Sandra Diaz Pier Simulation & Data Lab Neuroscience Jülich Supercomputing Centre Institute for Advanced Simulation Forschungszentrum Jülich Office: +49 2461 61-8913 --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Stefan Müller Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Dr. Ir. Pieter Jansens, Prof. Dr. Astrid Lambrecht, Prof. Dr. Frauke Melchior --------------------------------------------------------------------------------------------- --------------------------------------------------------------------------------------------- -- Stefan Dvoretskii _______________________________________________ NEST Users mailing list --users(a)nest-simulator.org To unsubscribe send an email tousers-leave(a)nest-simulator.org

Hi Stefan, I'm also having some unexplainable performance issue with OpenMP on our cluster, and have yet to find a solution, but perhaps the behavior I've observed would provide you some ideas. I run my Nest scripts on a single node (so no MPI) but have set the number of threads using `local_num_threads` in my script and also the `#SBATCH --ntasks=28`. Additionally, I've set the `MKL_THREADING_LAYER` environment variable per [GitHub issue #2573]( https://github.com/nest/nest-simulator/issues/2573). However, when monitoring the node via `htop`, I only sometimes see utilization close to 28x cores. The core utilization seems to be random from each SBATCH runs (sometimes 10x core, sometimes 6x cores) even when on an idle node of my cluster, thus the issue seems to be independent of how subscribed a node is. The GH issue #2573 also points to [GH issue #2401]( https://github.com/nest/nest-simulator/pull/2401) and [this documentation on threading]( https://nest-simulator.readthedocs.io/en/stable/hpc/threading.html#table-of… ). _______________________________________________ NEST Users mailing list -- users(a)nest-simulator.org To unsubscribe send an email to users-leave(a)nest-simulator.org