Hi Hans,

Thanks for the tip! I ended up doing something slightly different, since in my case the simulation can go from regular speed to infinitely slow pretty fast (so that the last 10ms Run call before freezing never finishes).

In case it's useful to someone:
Below is the corresponding code,
Cheers!

```
class HangingSimulationError(Exception):
    pass

def nest_run(sim_time):
    import nest, time
    t = time.time()
    nest.Run(sim_time)
    return time.time() - t

def simulate_in_steps(simulation_time):
    """Simulate in steps and catch hanging simulations."""
    import nest
    step = 10 # (virtual time)
    ministep = nest.GetKernelStatus('resolution')
    N_break = 3
    split_t = 2 # (real time) (s) Run next chunk in mini steps if previous took longer
    break_t = 1 # (real time) (s) break if `N_break` consecutive mini-steps takes longer
    total_sim_time = 0
    kernel_time = nest.GetKernelStatus('time')
    with nest.RunManager():
        while total_sim_time < simulation_time:
            next_step_t = min(step, simulation_time - total_sim_time)
            step_real_t = nest_run(next_step_t)
            total_sim_time += next_step_t
            if step_real_t > split_t:
                # Run next {step}ms in ministeps
                # Break if 3 consecutive ministeps last too long
                count = 0
                print(f"Running next {step}ms in ministeps")
                for _ in range(int(step/ministep)):
                    if simulation_time == total_sim_time:
                        break
                    next_ministep_t = min(ministep, simulation_time - total_sim_time)
                    t = nest_run(next_ministep_t)
                    total_sim_time += next_ministep_t
                    if t > break_t:
                        count += 1
                    else:
                        count = 0
                    if count >= N_break:
                        raise HangingSimulationError(
                            f"Simulation froze at {nest.GetKernelStatus('time')}ms"
                        )
    assert nest.GetKernelStatus('time') - kernel_time == simulation_time
```

From: Hans Ekkehard Plesser <hans.ekkehard.plesser@nmbu.no>
Sent: Thursday, May 6, 2021 9:49 AM
To: NEST User Mailing List <users@nest-simulator.org>
Subject: [NEST Users] Re: Stop hanging simulations
 

 

Hi Tom,

 

As a DIY workaround, you can use the RunManager context to simulate in small steps and break if it gets too slow. I haven't tested the code, just sketching from memory. Instead of calling nest.Simulate(1000), use

 

with nest.RunManager():

    for _ in range(100):

        t = time.time()

        nest.Run(10)

        if time.time() - t > 5:

            break

 

The logic is as follows: You split the 1000 ms into 100 times 10 ms. This is fast with Run() within the RunManager(). You then use Python's time to see how long it takes to simulate 10 ms and break if it takes too long, here a 5 s limit. You can then use GetKernelStatus to get the current time in the simulation.

 

It would be interesting to add this as a kernel feature. Let me know if it works!

 

Best,

Hans Ekkehard

 

 

--

 

Prof. Dr. Hans Ekkehard Plesser

Head, Department of Data Science

 

Faculty of Science and Technology

Norwegian University of Life Sciences

PO Box 5003, 1432 Aas, Norway

 

Phone +47 6723 1560

Email hans.ekkehard.plesser@nmbu.no

Home http://arken.nmbu.no/~plesser

 

 

 

On 06/05/2021, 16:08, "TOM BUGNON" <bugnon@wisc.edu> wrote:

 

Hi all,

Under some circumstances simulations can slow down up to the point where the nest.Simulate() does not advance anymore and stays stuck at a given virtual time, with a "realtime factor" of 0. I suppose this can happen for instance when a network falls into a regime of runaway excitation in which a massive number of spikes are being exchanged.

I'm looking for a way to stop the simulation in such a case, (say when the realtime factor goes below a set threshold, or when the output files are not updated for a certain duration), ideally in such a way that the program can continue running rather than crashing. If anyone has a suggestion about how to go around this issue I'd be happy to hear it.
Thanks in advance! Best, Tom