workloads+smite-scenarios: patch CLN to sync with new blocks faster

[NishantBansal2003]

ref: #143

I tried two approaches for CLN:

• Patch the polling interval from 30s (currently 2s) to 50ms
• Add a new syncblocks RPC that triggers an immediate chain sync, and tie it to bitcoind’s -blocknotify hook

In both cases, I ran campaign for 2 days and didn’t observe any target-side issues, so both approaches appear to be stable. To determine which approach performs better, I compared their coverage using the smite evaluation script. I ran five 1h trials for each approach, then replayed the generated corpus from each trial in local mode while measuring the CPU overhead for both CLN and bitcoind. Here are the resulting plots for reference (Let me know if running longer campaign trials would provide a better comparison?):

Target: cln

Median Coverage Over Time

Distribution Comparisons

Final Edge Coverage	Area Under Curve (Speed)

---

CPU Overhead (median)

With the 50ms polling interval approach:

INFO  [smite_scenarios::targets::bitcoind] Starting bitcoind...
INFO  [smite_scenarios::targets::bitcoind] Waiting for bitcoind to be ready...
INFO  [smite_scenarios::targets::bitcoind] bitcoind is ready
INFO  [smite_scenarios::targets::cln] Starting lightningd...
INFO  [smite_scenarios::targets::cln] Waiting for lightningd to be ready and synced...
INFO  [smite_scenarios::targets::cln] CLN identity pubkey: 021821158262e148a50c40b8e4eed6ad2f3a8a5dae475ba5c8269e8881a664fff6, blockheight: 101
INFO  [smite_scenarios::targets::cln] lightningd synced (blockheight=101)
INFO  [smite_scenarios::targets::cln] Both daemons are running, ready to fuzz
DEBUG [smite_scenarios::scenarios] Handshake complete, received target init
INFO  [smite::scenarios] Scenario initialized! Executing input...
INFO  [smite::runners] Reading input from "/input.bin"
DEBUG [smite_scenarios::scenarios::ir] [9.559µs] Executing IR program (28 instructions, 204 input bytes)
DEBUG [smite_scenarios::executor] [59.887µs] SendOpenChannel: 349 bytes
DEBUG [smite_scenarios::executor] [73.212µs] RecvAcceptChannel: waiting
DEBUG [smite_scenarios::executor] [17.164578ms] RecvAcceptChannel: received
DEBUG [smite_scenarios::executor] [23.921416ms] SendFundingCreated: 132 bytes
DEBUG [smite_scenarios::executor] [23.93913ms] RecvFundingSigned: waiting
DEBUG [smite_scenarios::executor] [41.149847ms] RecvFundingSigned: received
DEBUG [smite_scenarios::executor] [41.309537ms] BroadcastTransaction: txid=a982a06080d7d843d62b56a265ea1201d7f7ac47c7fd32cea854b4e1304906a8
DEBUG [smite_scenarios::executor] [73.766602ms] MineBlocks: mined 8 block(s)
DEBUG [smite_scenarios::executor] [73.781275ms] SendChannelReady: 77 bytes
DEBUG [smite_scenarios::executor] [84.409964ms] RecvChannelReady: waiting
DEBUG [smite_scenarios::executor] skipping gossip message type 258
DEBUG [smite_scenarios::executor] [166.947336ms] RecvChannelReady: received
DEBUG [smite_scenarios::scenarios::ir] [166.954511ms] Program executed successfully
DEBUG [smite_scenarios::scenarios::ir] [167.001796ms] Target responded with pong
INFO  [smite::scenarios] Test case ran successfully!
DEBUG [smite_scenarios::targets::cln] lightningd: requesting graceful shutdown via lightning-cli stop
DEBUG [smite_scenarios::targets::cln] lightningd: waiting for process to exit
DEBUG [smite::process] bitcoind: dropping running process, attempting shutdown
DEBUG [smite::process] bitcoind: sending SIGTERM to process group 7
DEBUG [smite::process] bitcoind: exited with exit status: 0

With the -blocknotify hook approach:

INFO  [smite_scenarios::targets::bitcoind] Starting bitcoind...
INFO  [smite_scenarios::targets::bitcoind] Waiting for bitcoind to be ready...
INFO  [smite_scenarios::targets::bitcoind] bitcoind is ready
INFO  [smite_scenarios::targets::cln] Starting lightningd...
INFO  [smite_scenarios::targets::cln] Waiting for lightningd to be ready and synced...
INFO  [smite_scenarios::targets::cln] CLN identity pubkey: 031b4bda84b978bcc29a6dc8c5a0fdbf66e6bbe4d4a29cda105c99d42510291bcc, blockheight: 101
INFO  [smite_scenarios::targets::cln] lightningd synced (blockheight=101)
INFO  [smite_scenarios::targets::cln] Both daemons are running, ready to fuzz
DEBUG [smite_scenarios::scenarios] Handshake complete, received target init
INFO  [smite::scenarios] Scenario initialized! Executing input...
INFO  [smite::runners] Reading input from "/input.bin"
DEBUG [smite_scenarios::scenarios::ir] [7.524µs] Executing IR program (28 instructions, 204 input bytes)
DEBUG [smite_scenarios::executor] [61.873µs] SendOpenChannel: 349 bytes
DEBUG [smite_scenarios::executor] [72.363µs] RecvAcceptChannel: waiting
DEBUG [smite_scenarios::executor] [17.631703ms] RecvAcceptChannel: received
DEBUG [smite_scenarios::executor] [23.81154ms] SendFundingCreated: 132 bytes
DEBUG [smite_scenarios::executor] [23.828741ms] RecvFundingSigned: waiting
DEBUG [smite_scenarios::executor] [42.806291ms] RecvFundingSigned: received
DEBUG [smite_scenarios::executor] [43.058768ms] BroadcastTransaction: txid=5f1961dff6b10103072fdd66d72c71a7a2ef0f4f7a0acef1c95871c6ea136cb0
DEBUG [smite_scenarios::executor] [72.396396ms] MineBlocks: mined 8 block(s)
DEBUG [smite_scenarios::executor] [72.413794ms] SendChannelReady: 77 bytes
DEBUG [smite_scenarios::executor] [81.454544ms] RecvChannelReady: waiting
DEBUG [smite_scenarios::executor] skipping gossip message type 258
DEBUG [smite_scenarios::executor] [119.480716ms] RecvChannelReady: received
DEBUG [smite_scenarios::scenarios::ir] [119.486254ms] Program executed successfully
DEBUG [smite_scenarios::scenarios::ir] [119.546017ms] Target responded with pong
INFO  [smite::scenarios] Test case ran successfully!
DEBUG [smite_scenarios::targets::cln] lightningd: requesting graceful shutdown via lightning-cli stop
DEBUG [smite_scenarios::targets::cln] lightningd: waiting for process to exit
DEBUG [smite::process] bitcoind: dropping running process, attempting shutdown
DEBUG [smite::process] bitcoind: sending SIGTERM to process group 6
DEBUG [smite::process] bitcoind: exited with exit status: 0

[Chand-ra]

ref: #143

I tried two approaches for CLN:

• Patch the polling interval from 30s (currently 2s) to 50ms
• Add a new syncblocks RPC that triggers an immediate chain sync, and tie it to bitcoind’s -blocknotify hook

In both cases, I ran campaign for 2 days and didn’t observe any target-side issues, so both approaches appear to be stable. To determine which approach performs better, I compared their coverage using the smite evaluation script. I ran five 1h trials for each approach, then replayed the generated corpus from each trial in local mode while measuring the CPU overhead for both CLN and bitcoind. Here are the resulting plots for reference (Let me know if running longer campaign trials would provide a better comparison?)

That is difficult to answer with just the plots. While the plots provide a qualitative way of comparing the efficacy of two approaches, the statistical tests performed in the evaluation script are supposed to be the ground-truth oracles to determine answers to questions of this nature.

Could you provide the summary statistics from the evaluation report? It should be a markdown table similar to this:

Target	Duration (h)	n (Baseline)	n (Exp.)	Median Cov. (Baseline)	Median Cov. (Exp.)	Adj. p-value (Cov.)	Â12 (Cov.)	Median AUC (Baseline)	Median AUC (Exp.)	Adj. p-value (AUC)	Â12 (AUC)	Union Cov. (Baseline)	Union Cov. (Exp.)	Execs/s (Baseline)	Execs/s (Exp.)
ldk	14.6636	13	13	13190	14199	0.00766076	0.810651	190769	200973	0.0402395	0.739645	13544	15269	363.63	157.57

[NishantBansal2003]

Target	Duration (h)	n (Baseline)	n (Exp.)	Median Cov. (Baseline)	Median Cov. (Exp.)	Adj. p-value (Cov.)	Â12 (Cov.)	Median AUC (Baseline)	Median AUC (Exp.)	Adj. p-value (AUC)	Â12 (AUC)	Union Cov. (Baseline)	Union Cov. (Exp.)	Execs/s (Baseline)	Execs/s (Exp.)
cln	1	5	5	13063	13428	0.00793651	1	13003.3	13370.2	0.00793651	1	13302	13684	59.39	64.08

[Chand-ra]

Target Duration (h) n (Baseline) n (Exp.) Median Cov. (Baseline) Median Cov. (Exp.) Adj. p-value (Cov.) Â12 (Cov.) Median AUC (Baseline) Median AUC (Exp.) Adj. p-value (AUC) Â12 (AUC) Union Cov. (Baseline) Union Cov. (Exp.) Execs/s (Baseline) Execs/s (Exp.)
cln 1 5 5 13063 13428 0.00793651 1 13003.3 13370.2 0.00793651 1 13302 13684 59.39 64.08

The experiment is definitely statistically under-powered (needs longer and more trials). But solely from the results here, the experimental configuration (which I assume is the blocknotify approach) is better than the baseline (assumably the polling approach) 100% of the time, with very little chances (~0.8%) of any of the trials being a false positive. True for both final coverage and AUC values.

Bottom line, this is the strongest possible result we can observe using our framework!

The execs/sec also seem to gain a significant improvement, and the CPU overhead is significantly lower. Given all of this, I think it's reasonable to assume that the blocknotify approach will be atleast as performant as the polling approach.

[Chand-ra]

The code looks fine to me, but I think it would be better to store patches in a separate directory like fuzzamoto does instead of alongside Docker files. Maybe in a workloads/<target>/patches/ directory?