Intermittent Vulkan crash - latest build, Windows 11, Nvidia GTX1080

[timp3289]

Running this file with latest master causes intermittent crashes on my machine under the debugger.
It throws VK_ERROR_DEVICE_LOST

demod.scopesession.txt

This is with Windows 11, MSYS2, gpu is a GTX 1080
It is possible the GPU is dying, but I haven't noticed this elsewhere.
It does not crash when running FurMark for a while - if you know a more appropriate stress test I am happy to run it and see.

The GPU info printed on startup is below:

Initializing Vulkan
    VK_KHR_get_physical_device_properties2: supported
    VK_EXT_debug_utils: supported
    Loader/API support available for Vulkan 1.4
    Vulkan 1.2 support available, requesting it
    Initializing glfw 3.4.0 Win32 WGL Null EGL OSMesa MinGW-w64 DLL
    GLFW required extensions:
        VK_KHR_surface
        VK_KHR_win32_surface
    Physical devices:
        Device 0: NVIDIA GeForce GTX 1080
            API version:            0x00403118 (0.1.3.280)
            Driver version:         0x8c178000 (560.94.0.0)
            Vendor ID:              10de
            Device ID:              1b80
            Device type:            Discrete GPU
            int64:                  yes
            int16:                  yes (allowed in SSBOs)
            int8:                   yes (allowed in SSBOs)
            atomic int64 in SSBO:   yes
            atomic int64 in shared: yes
            Max image dim 2D:       32768
            Max storage buf range:  4095 MB
            Max mem objects:        4294967295
            Max compute shared mem: 48 KB
            Max compute grp count:  2147483647 x 65535 x 65535
            Max compute invocs:     1536
            Max compute grp size:   1536 x 1024 x 64
            Memory types:
                Type 0
                    Heap index: 1
                Type 1
                    Heap index: 1
                Type 2
                    Heap index: 1
                Type 3
                    Heap index: 1
                Type 4
                    Heap index: 1
                Type 5
                    Heap index: 1
                Type 6
                    Heap index: 1
                Type 7
                    Heap index: 0
                    Device local
                Type 8
                    Heap index: 1
                    Host visible
                    Host coherent
                Type 9
                    Heap index: 1
                    Host visible
                    Host coherent
                    Host cached
                Type 10
                    Heap index: 2
                    Device local
                    Host visible
                    Host coherent
            Memory heaps:
                Heap 0
                    Size: 7 GB
                    Device local
                Heap 1
                    Size: 15 GB
                Heap 2
                    Size: 214 MB
                    Device local
        Selected device 0
            Queue families (5 total)
                Queue type 0
                    Queue count:          16
                    Timestamp valid bits: 64
                    Graphics
                    Compute
                    Transfer
                    Sparse binding
                Queue type 1
                    Queue count:          2
                    Timestamp valid bits: 64
                    Transfer
                    Sparse binding
                Queue type 2
                    Queue count:          8
                    Timestamp valid bits: 64
                    Compute
                    Transfer
                    Sparse binding
                Queue type 3
                    Queue count:          1
                    Timestamp valid bits: 32
                    Transfer
                    Sparse binding
                    Video decode
                Queue type 4
                    Queue count:          2
                    Timestamp valid bits: 32
                    Transfer
                    Sparse binding
                    Video encode
            Driver: vk::DriverId::eNvidiaProprietary
            Enabling 64-bit float support
            Enabling 64-bit integer support
            Enabling 16-bit integer support
            Enabling 32-bit atomic float support for SSBOs and shared memory
            Enabling 16-bit integer support for SSBOs
            Enabling 8-bit integer support
            Enabling 64-bit atomic int support for SSBOs
            Enabling 64-bit atomic int support for shared memory
            Enabling 8-bit integer support for SSBOs
            Device has VK_KHR_push_descriptor, requesting it
            Device has VK_KHR_shader_atomic_int64, requesting it
            Device has VK_KHR_shader_non_semantic_info, requesting it
            Device has VK_EXT_memory_budget, requesting it
            Device has VK_EXT_shader_atomic_float, requesting it
            Using heap 1, type 9 for pinned host memory
            Using heap 0, type 7 for card-local memory
            Sorted queues:
                Family=1 Index=1 Flags=0000000c
                Family=1 Index=0 Flags=0000000c
                Family=4 Index=1 Flags=0000004c
                Family=4 Index=0 Flags=0000004c
                Family=3 Index=0 Flags=0000002c
                Family=2 Index=7 Flags=0000000e
                Family=2 Index=6 Flags=0000000e
                Family=2 Index=5 Flags=0000000e
                Family=2 Index=4 Flags=0000000e
                Family=2 Index=3 Flags=0000000e
                Family=2 Index=2 Flags=0000000e
                Family=2 Index=1 Flags=0000000e
                Family=2 Index=0 Flags=0000000e
                Family=0 Index=14 Flags=0000000f
                Family=0 Index=15 Flags=0000000f
                Family=0 Index=0 Flags=0000000f
                Family=0 Index=13 Flags=0000000f
                Family=0 Index=12 Flags=0000000f
                Family=0 Index=11 Flags=0000000f
                Family=0 Index=10 Flags=0000000f
                Family=0 Index=9 Flags=0000000f
                Family=0 Index=8 Flags=0000000f
                Family=0 Index=7 Flags=0000000f
                Family=0 Index=6 Flags=0000000f
                Family=0 Index=5 Flags=0000000f
                Family=0 Index=4 Flags=0000000f
                Family=0 Index=3 Flags=0000000f
                Family=0 Index=2 Flags=0000000f
                Family=0 Index=1 Flags=0000000f

[azonenberg]

Device lost generally means a shader hanging and timing out, but there can be other causes too like a GPU driver crash.

Do you have any specific details as to what you were doing when the fault happened? Have you tried running under Vulkan validation layers to see if they report any problems?

[timp3289]

Sometimes I was interacting with the UI, doing things like changing the record length or adding filters
Sometimes it was in the background and I was using another app.

I have not tried with the validation layers, how do I do that?

[timp3289]

Vulkan log with gpu validation.txt
segfault with gpu validation.txt
segfault with cpu validation.txt

[timp3289]

Anecdotally, I have noticed an inconsistent hang (usually not a crash) which seems to happen when changing the record length (of FFTs and other things). This could be related if there is a timeout involved.
One of the ways this crash triggers (not always) is when changing the record length of the sine generator.