lost channels due to old states; now cln won't start anymore

[marimes]

The node was working fine until a reboot, which likely swapped in an outdated database, possibly due to an incorrect setup of my new HA cluster. As a result, I just lost a large number of channels due to FCs. Now my node won't come up again, and I'm looking for help in getting it fixed.

Here is the log from the start to the backtrace:

2025-06-05T17:34:57.941Z INFO    lightningd: v25.02.2
2025-06-05T17:34:58.586Z INFO    plugin-wss-proxy: Killing plugin: disabled itself: No module named 'websockets'
2025-06-05T17:34:58.587Z UNUSUAL plugin-bookkeeper: topic 'utxo_deposit' is not a known notification topic
2025-06-05T17:34:58.587Z UNUSUAL plugin-bookkeeper: topic 'utxo_spend' is not a known notification topic
2025-06-05T17:35:06.130Z INFO    plugin-bcli: bitcoin-cli initialized and connected to bitcoind.
2025-06-05T17:35:41.126Z UNUSUAL xxxx-chan#1931: gossipd lost track of announced channel: re-announcing!

[... about 40 of these lines]

2025-06-05T17:35:41.204Z INFO    plugin-clnrest: REST server running at http://127.0.0.1:3100
2025-06-05T17:35:41.205Z INFO    plugin-cln-grpc: Connecting to \"lightning-rpc\" and serving grpc on 127.0.0.1:9736
2025-06-05T17:36:01.228Z INFO    xxxx-chan#1670: Restarting onchaind (ONCHAIN): closed in block 899909
2025-06-05T17:36:01.228Z UNUSUAL xxxx-chan#1670: Peer permanent failure in ONCHAIN: Funding transaction spent: onchain txid xxxx (reason=unknown)
2025-06-05T17:36:01.228Z **BROKEN** xxxx-chan#1670: Cannot broadcast our commitment tx: they have a future one
2025-06-05T17:36:01.228Z INFO    xxxx-chan#1670: State changed from ONCHAIN to FUNDING_SPEND_SEEN

[... several pages of the above messages]

2025-06-05T17:36:02.338Z INFO    lightningd: --------------------------------------------------
2025-06-05T17:36:02.338Z INFO    lightningd: Server started with public key xxxx, alias DisasterNode (color #000000) and lightningd v25.02.2
2025-06-05T17:36:17.347Z **BROKEN** xxxx-chan#1704: Cannot broadcast our commitment tx: they have a future one
2025-06-05T17:36:17.621Z **BROKEN** xxxx-onchaind-chan#1670: ERROR: Unknown commitment #884423, recovering our funds!
2025-06-05T17:36:17.621Z INFO    xxxx-chan#1670: State changed from FUNDING_SPEND_SEEN to ONCHAIN
2025-06-05T17:36:18.082Z **BROKEN** xxxx-onchaind-chan#903: ERROR: Unknown commitment #116906, recovering our funds!

[... about an additional page of the above messages]

2025-06-05T17:36:58.545Z UNUSUAL xxx-chan#1673: Deferring incoming commit until we sync
2025-06-05T17:37:06.007Z UNUSUAL xxx-chan#1215: Peer permanent failure in ONCHAIN: Funding transaction spent: onchain txid xxx (reason=unknown)


[... and finally]

2025-06-05T17:37:40.984Z INFO    xxx-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 884842x3699x0
2025-06-05T17:38:04.009Z **BROKEN** lightningd: FATAL SIGNAL 11 (version v25.02.2)
2025-06-05T17:38:04.009Z **BROKEN** lightningd: backtrace: common/daemon.c:41 (send_backtrace) 0x5620587a37ef
2025-06-05T17:38:04.009Z **BROKEN** lightningd: backtrace: common/daemon.c:78 (crashdump) 0x5620587a382f
2025-06-05T17:38:04.009Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f9b9f47404f
2025-06-05T17:38:04.009Z **BROKEN** lightningd: backtrace: ccan/ccan/htable/htable.c:179 (htable_first_) 0x562058856984
2025-06-05T17:38:04.009Z **BROKEN** lightningd: backtrace: lightningd/onchain_control.c:48 (replay_tx_hash_first) 0x562058752db9
2025-06-05T17:38:04.009Z **BROKEN** lightningd: backtrace: lightningd/onchain_control.c:403 (replay_block) 0x562058752db9
2025-06-05T17:38:04.009Z **BROKEN** lightningd: backtrace: lightningd/bitcoind.c:512 (getrawblockbyheight_callback) 0x5620587212b9
2025-06-05T17:38:04.009Z **BROKEN** lightningd: backtrace: lightningd/plugin.c:661 (plugin_response_handle) 0x56205877024f
2025-06-05T17:38:04.009Z **BROKEN** lightningd: backtrace: lightningd/plugin.c:773 (plugin_read_json_one) 0x5620587739d8
2025-06-05T17:38:04.009Z **BROKEN** lightningd: backtrace: lightningd/plugin.c:824 (plugin_read_json) 0x562058773c74
2025-06-05T17:38:04.009Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:60 (next_plan) 0x562058856eb6
2025-06-05T17:38:04.009Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:422 (do_plan) 0x562058857341
2025-06-05T17:38:04.010Z **BROKEN** lightningd: backtrace: ccan/ccan/io/io.c:439 (io_ready) 0x5620588573fa
2025-06-05T17:38:04.010Z **BROKEN** lightningd: backtrace: ccan/ccan/io/poll.c:455 (io_loop) 0x562058858cb5
2025-06-05T17:38:04.010Z **BROKEN** lightningd: backtrace: lightningd/io_loop_with_timers.c:22 (io_loop_with_timers) 0x562058745a34
2025-06-05T17:38:04.010Z **BROKEN** lightningd: backtrace: lightningd/lightningd.c:1490 (main) 0x56205874b2d5
2025-06-05T17:38:04.010Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f9b9f45f249
2025-06-05T17:38:04.010Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x7f9b9f45f304
2025-06-05T17:38:04.010Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0x56205871eea0
2025-06-05T17:38:04.010Z **BROKEN** lightningd: backtrace: (null):0 ((null)) 0xffffffffffffffff