Skip to content

fuzz: improve CI iteration strategy, add corpus minimization and summary reporting #4497

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
joostjager wants to merge 2 commits into
base: main
Choose a base branch
from
+75 −7
Open

fuzz: improve CI iteration strategy, add corpus minimization and summary reporting #4497

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d86297e
ci: split fuzz sanity check into separate parallel job
joostjager Mar 20, 2026
32f182a
fuzz: improve iteration scaling, add minimization and summary table
joostjager Mar 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 18 additions & 6 deletions .github/workflows/build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,21 @@ jobs:
- name: Simulate docs.rs build
run: ci/check-docsrs.sh

fuzz_sanity:
runs-on: self-hosted
env:
TOOLCHAIN: 1.75
steps:
- name: Checkout source code
uses: actions/checkout@v4
- name: Install Rust ${{ env.TOOLCHAIN }} toolchain
run: |
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --profile=minimal --default-toolchain ${{ env.TOOLCHAIN }}
- name: Sanity check fuzz targets on Rust ${{ env.TOOLCHAIN }}
run: |
cd fuzz
RUSTFLAGS="--cfg=fuzzing --cfg=secp256k1_fuzz --cfg=hashes_fuzz" cargo test --quiet --color always --lib --bins -j8

fuzz:
runs-on: self-hosted
env:
Expand Down Expand Up @@ -238,13 +253,10 @@ jobs:
key: fuzz-corpus-refs/heads/main-${{ github.sha }}
restore-keys: |
fuzz-corpus-refs/heads/main-
- name: Sanity check fuzz targets on Rust ${{ env.TOOLCHAIN }}
run: |
cd fuzz
RUSTFLAGS="--cfg=fuzzing --cfg=secp256k1_fuzz --cfg=hashes_fuzz" cargo test --verbose --color always --lib --bins -j8
cargo clean
- name: Run fuzzers
run: cd fuzz && ./ci-fuzz.sh && cd ..
env:
FUZZ_MINIMIZE: ${{ contains(github.event.pull_request.labels.*.name, 'fuzz-minimize') }}
Copy link
Collaborator

@TheBlueMatt TheBlueMatt Mar 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why bother if we do this on main every time anyway?

Copy link
Contributor Author

@joostjager joostjager Mar 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added it just for testing. Or maybe if a fuzz-affecting refactor PR is open for a long time and has accumulated a large corpus.

Can also remove the label trigger. Do you prefer that?

- name: Upload honggfuzz corpus
uses: actions/upload-artifact@v4
with:
Expand Down Expand Up @@ -308,7 +320,7 @@ jobs:
TOR_PROXY="127.0.0.1:9050" RUSTFLAGS="--cfg=tor" cargo test --verbose --color always -p lightning-net-tokio

notify-failure:
needs: [build-workspace, build-features, build-bindings, build-nostd, build-cfg-flags, build-sync, fuzz, linting, rustfmt, check_release, check_docs, benchmark, ext-test, tor-connect, coverage]
needs: [build-workspace, build-features, build-bindings, build-nostd, build-cfg-flags, build-sync, fuzz_sanity, fuzz, linting, rustfmt, check_release, check_docs, benchmark, ext-test, tor-connect, coverage]
if: failure() && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
permissions:
Expand Down
58 changes: 57 additions & 1 deletion fuzz/ci-fuzz.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,20 +30,76 @@ sed -i 's/lto = true//' Cargo.toml
export HFUZZ_BUILD_ARGS="--features honggfuzz_fuzz"

cargo --color always hfuzz build -j8

SUMMARY=""

for TARGET in src/bin/*.rs; do
FILENAME=$(basename $TARGET)
FILE="${FILENAME%.*}"
HFUZZ_RUN_ARGS="--exit_upon_crash -v -n8 --run_time 30"
CORPUS_DIR="hfuzz_workspace/$FILE/input"
CORPUS_COUNT=$(find "$CORPUS_DIR" -type f 2>/dev/null | wc -l)
# Run 8x the corpus size plus a baseline, ensuring full corpus replay
# with room for new mutations. The 10-minute hard cap (--run_time 600)
# prevents slow-per-iteration targets from running too long.
ITERATIONS=$((CORPUS_COUNT * 8 + 1000))
HFUZZ_RUN_ARGS="--exit_upon_crash -q -n8 -t 3 -N $ITERATIONS --run_time 600"
if [ "$FILE" = "chanmon_consistency_target" -o "$FILE" = "fs_store_target" ]; then
HFUZZ_RUN_ARGS="$HFUZZ_RUN_ARGS -F 64"
fi
export HFUZZ_RUN_ARGS
FUZZ_START=$(date +%s)
cargo --color always hfuzz run $FILE
FUZZ_END=$(date +%s)
FUZZ_TIME=$((FUZZ_END - FUZZ_START))
FUZZ_CORPUS_COUNT=$(find "$CORPUS_DIR" -type f 2>/dev/null | wc -l)
if [ -f hfuzz_workspace/$FILE/HONGGFUZZ.REPORT.TXT ]; then
cat hfuzz_workspace/$FILE/HONGGFUZZ.REPORT.TXT
for CASE in hfuzz_workspace/$FILE/SIG*; do
cat $CASE | xxd -p
done
exit 1
fi
if [ "$GITHUB_REF" = "refs/heads/main" ] || [ "$FUZZ_MINIMIZE" = "true" ]; then
HFUZZ_RUN_ARGS="-M -q -n8 -t 3"
export HFUZZ_RUN_ARGS
MIN_START=$(date +%s)
cargo --color always hfuzz run $FILE
MIN_END=$(date +%s)
MIN_TIME=$((MIN_END - MIN_START))
MIN_CORPUS_COUNT=$(find "$CORPUS_DIR" -type f 2>/dev/null | wc -l)
Comment on lines +62 to +69
Copy link
Collaborator

@ldk-claude-review-bot ldk-claude-review-bot Mar 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The crash check (lines 55-61) runs only after the fuzz step, but not after the minimization step here. Minimization still executes every corpus input through the target binary. If a crash occurs during minimization (e.g., a newly-mutated input from the fuzz step that was added to the corpus but didn't crash during fuzzing due to timing, or a non-deterministic crash), the HONGGFUZZ.REPORT.TXT would be written but never checked.

Consider adding a crash check after the minimization run as well:

Suggested change
if [ "$GITHUB_REF" = "refs/heads/main" ] || [ "$FUZZ_MINIMIZE" = "true" ]; then
HFUZZ_RUN_ARGS="-M -q -n8 -t 3"
export HFUZZ_RUN_ARGS
MIN_START=$(date +%s)
cargo --color always hfuzz run $FILE
MIN_END=$(date +%s)
MIN_TIME=$((MIN_END - MIN_START))
MIN_CORPUS_COUNT=$(find "$CORPUS_DIR" -type f 2>/dev/null | wc -l)
HFUZZ_RUN_ARGS="-M -q -n8 -t 3"
export HFUZZ_RUN_ARGS
MIN_START=$(date +%s)
cargo --color always hfuzz run $FILE
MIN_END=$(date +%s)
MIN_TIME=$((MIN_END - MIN_START))
MIN_CORPUS_COUNT=$(find "$CORPUS_DIR" -type f 2>/dev/null | wc -l)
if [ -f hfuzz_workspace/$FILE/HONGGFUZZ.REPORT.TXT ]; then
cat hfuzz_workspace/$FILE/HONGGFUZZ.REPORT.TXT
for CASE in hfuzz_workspace/$FILE/SIG*; do
cat $CASE | xxd -p
done
exit 1
fi

SUMMARY="${SUMMARY}${FILE}|${ITERATIONS}|${CORPUS_COUNT}|${FUZZ_CORPUS_COUNT}|${FUZZ_TIME}|${MIN_CORPUS_COUNT}|${MIN_TIME}\n"
else
SUMMARY="${SUMMARY}${FILE}|${ITERATIONS}|${CORPUS_COUNT}|${FUZZ_CORPUS_COUNT}|${FUZZ_TIME}|-|-\n"
fi
done

fmt_time() {
local secs=$1
local m=$((secs / 60))
local s=$((secs % 60))
if [ "$m" -gt 0 ]; then
printf "%dm %ds" "$m" "$s"
else
printf "%ds" "$s"
fi
}

# Print summary table
set +x
echo ""
echo "==== Fuzz Summary ===="
HDR="%-40s %7s %7s %-15s %9s %-15s %9s\n"
FMT="%-40s %7s %7s %6s %-9s %9s %6s %-9s %9s\n"
printf "$HDR" "Target" "Iters" "Corpus" " Fuzzed" "Fuzz time" " Minimized" "Min. time"
printf "$HDR" "------" "-----" "------" "---------------" "---------" "---------------" "---------"
echo -e "$SUMMARY" | while IFS='|' read -r name iters orig fuzzed ftime minimized mtime; do
[ -z "$name" ] && continue
fuzz_delta=$((fuzzed - orig))
if [ "$minimized" = "-" ]; then
printf "$FMT" "$name" "$iters" "$orig" "$fuzzed" "(+$fuzz_delta)" "$(fmt_time "$ftime")" "-" "" "-"
else
min_delta=$((minimized - fuzzed))
printf "$FMT" "$name" "$iters" "$orig" "$fuzzed" "(+$fuzz_delta)" "$(fmt_time "$ftime")" "$minimized" "($min_delta)" "$(fmt_time "$mtime")"
fi
done
echo "======================"
Loading