Hi Allen,

 

It would also be interesting if you could take a look at how memory consumption grows while the simulation is running (using top or similar).

 

Are you recording spike or membrane potential data to memory during the simulation? With you very long-running simulations, that could flood memory over time. A quick back-of-the envelope calculation:

 

648 neurons x 1 membrane potential per ms x 300000 ms x 24B / values (V_m, sender, time) = 4.3 GB 

 

If you do record data to memory during the entire simulation, try running without recording. If problem still occur, data may be collecting elsewhere and we would need to check further. Have you tried a comparable simulation without MUSIC?

 

Best,

Hans Ekkehard

 

 

 

-- 

 

Prof. Dr. Hans Ekkehard Plesser

Head, Department of Data Science

 

Faculty of Science and Technology

Norwegian University of Life Sciences

PO Box 5003, 1432 Aas, Norway

 

Phone +47 6723 1560

Email hans.ekkehard.plesser@nmbu.no

Home http://arken.nmbu.no/~plesser

 

 

On 24/11/2020, 10:10, "Trensch, Guido" <g.trensch@fz-juelich.de> wrote:

 

Dear Allen,

 

a “bad_alloc” exception does not indicate a disk space problem, but rather refers to a memory shortage. Please try to increase the base memory of your VM.

 

Best regards,

Guido

 

 

 

From: Allen Rabayda [mailto:arabayda@masonlive.gmu.edu]
Sent: Monday, 23 November 2020 17:24
To: NEST User Mailing List <users@nest-simulator.org>
Subject: [NEST Users] error std::bad_alloc: virtualbox

 

Dear Nest Community,

 

  Has anyone encountered a "bad_alloc" error like the one below and if so, any recommendations?   It appears to be a  VM memory issue but only using 21% of harddrive space (ref: below).

 

  My simulation successfully completes for 200,000 ms but errors out at 98% complete for 230,000 ms, 75% for 300,000 ms and 56% for 400,000 ms.

 

  I'm running on NEST 2.18.0 VirtualBox lubuntu 18.04 (ref: image of settings below).

 

  Thank you for any suggestions.

 

Best Regards,

--Allen

**********************************************

**** Error Message *****

 

>> # SIMULATION
>> nest.Simulate(300000)

Nov 21 17:10:28 NodeManager::prepare_nodes [Info]:
    Preparing 684 nodes for simulation.

Nov 21 17:10:28 MUSICManager::enter_runtime [Info]:
    Entering MUSIC runtime with tick = 1 ms

Nov 21 17:10:28 SimulationManager::start_updating_ [Info]:
    Number of local nodes: 684
    Simulation time (ms): 300000
    Number of OpenMP threads: 2
    Number of MPI processes: 1

 75 %: network time: 223698.0 ms, realtime factor: 0.6277Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "/home/nest/work/nest-install/lib/python3.6/site-packages/nest/ll_api.py", line 246, in stack_checker_func
    return f(*args, **kwargs)
  File "/home/nest/work/nest-install/lib/python3.6/site-packages/nest/lib/hl_api_simulation.py", line 66, in Simulate
    sr('ms Simulate')
  File "/home/nest/work/nest-install/lib/python3.6/site-packages/nest/ll_api.py", line 132, in catching_sli_run
    raise exceptionCls(commandname, message)
nest.ll_api.std::bad_alloc: ('std::bad_alloc in Simulate_d: C++ exception: std::bad_alloc', 'std::bad_alloc', <SLILiteral: Simulate_d>, ': C++ exception: std::bad_alloc')
********************************************


**** Folder Space on VirtualBox after Error ****

nest@nestvm:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            5.2G     0  5.2G   0% /dev
tmpfs           1.1G  1.1M  1.1G   1% /run
/dev/sda1        99G   20G   76G  21% /
tmpfs           5.2G     0  5.2G   0% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           5.2G     0  5.2G   0% /sys/fs/cgroup
SharedNest2     917G  447G  470G  49% /media/sf_SharedNest2
tmpfs           1.1G   16K  1.1G   1% /run/user/1000
/dev/sr0         74M   74M     0 100% /media/nest/VBox_GAs_6.0.10

***********************************


**** VirtualBox Settings *******

 

 



------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Volker Rieke
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------