Improve the run_once script to better handle Ruby options and harness selection

So we don't need to deal with the `-r` ruby argument directly for
selecting harnesses.

Author: rafaelfranca

.github/workflows/test.yml

@@ -141,8 +141,8 @@ jobs:
         with:
           ruby-version: head
 
-      - name: Test run_once.sh
-        run: ./run_once.sh --yjit-stats benchmarks/railsbench/benchmark.rb
+      - name: Test run_once.rb
+        run: ./run_once.rb -- --yjit-stats benchmarks/railsbench/benchmark.rb
 
   all-tests:
     runs-on: ubuntu-latest

README.md

@@ -93,14 +93,14 @@ To run one or more specific benchmarks and record the data:
 
 ### Running a single benchmark
 
-This is the easiest way to run a single benchmark.
-It requires no setup at all and assumes nothing about the Ruby you are benchmarking.
-It's also convenient for profiling, debugging, etc, especially since all benchmarked code runs in that process.
+The easiest way to run a single benchmark once is using the `run_once.rb` script:
 
 ```
-ruby benchmarks/some_benchmark.rb
+./run_once.rb benchmarks/some_benchmark.rb
 ```
 
+This automatically sets the environment to run the benchmark once (no warmup iterations).
+
 ### Benchmark organization
 
 Benchmarks can be organized in three ways:
@@ -225,23 +225,29 @@ You can find several test harnesses in the `harness/` directory:
 * `chain` - a harness to chain multiple harnesses together
 * `mplr` - a harness for multiple iterations with time limits
 
-To use a specific harness, run a benchmark script directly with `-r` to require the harness:
+To use a specific harness, use the `run_once.rb` script:
 
 ```
-# Use default harness (loads harness/default.rb automatically)
-ruby benchmarks/railsbench/benchmark.rb
+# Use default harness
+./run_once.rb benchmarks/railsbench/benchmark.rb
 
 # Use the 'once' harness
-ruby -r./harness/once benchmarks/railsbench/benchmark.rb
+./run_once.rb --harness=once benchmarks/railsbench/benchmark.rb
 
 # Use the 'perf' harness
-ruby -r./harness/perf benchmarks/railsbench/benchmark.rb
+./run_once.rb --harness=perf benchmarks/railsbench/benchmark.rb
 
 # Use the 'stackprof' harness
-ruby -r./harness/stackprof benchmarks/railsbench/benchmark.rb
+./run_once.rb --harness=stackprof benchmarks/railsbench/benchmark.rb
 
 # Use the 'vernier' harness
-ruby -r./harness/vernier benchmarks/railsbench/benchmark.rb
+./run_once.rb --harness=vernier benchmarks/railsbench/benchmark.rb
+
+# Pass Ruby options like --yjit (use -- separator)
+./run_once.rb -- --yjit benchmarks/railsbench/benchmark.rb
+
+# Combine harness option with Ruby options
+./run_once.rb --harness=default -- --yjit-stats benchmarks/railsbench/benchmark.rb
 ```
 
 When using `run_benchmarks.rb`, you can specify a harness with the `--harness` option:
@@ -272,12 +278,11 @@ You can also use `--warmup`, `--bench`, or `--once` to set these environment var
 ./run_benchmarks.rb railsbench --once
 ```
 
-There is also a handy script for running benchmarks just once using
-`WARMUP_ITRS=0 MIN_BENCH_ITRS=1 MIN_BENCH_TIME=0`, for example
+You can also use the `run_once.rb` script to run benchmarks just once, for example
 with the `--yjit-stats` command-line option:
 
 ```
-./run_once.sh --yjit-stats benchmarks/railsbench/benchmark.rb
+./run_once.rb -- --yjit-stats benchmarks/railsbench/benchmark.rb
 ```
 
 ### Using perf
@@ -287,10 +292,10 @@ If `PERF` environment variable is present, it starts the perf subcommand after w
 
 ```sh
 # Use `perf record` for both warmup and benchmark
-perf record ruby --yjit-perf=map -r./harness/perf benchmarks/railsbench/benchmark.rb
+perf record ./run_once.rb --harness=perf -- --yjit-perf=map benchmarks/railsbench/benchmark.rb
 
 # Use `perf record` only for benchmark
-PERF=record ruby --yjit-perf=map -r./harness/perf benchmarks/railsbench/benchmark.rb
+PERF=record ./run_once.rb --harness=perf -- --yjit-perf=map benchmarks/railsbench/benchmark.rb
 ```
 
 This is the only harness that uses `run_benchmark`'s argument, `num_itrs_hint`.

harness/perf.rb

@@ -3,7 +3,7 @@
 # This is a relatively minimal harness meant for use with Linux perf(1).
 # Example usage:
 #
-#    $ PERF='record -e cycles' ruby -r./harness/perf benchmarks/fib.rb
+#    $ PERF='record -e cycles' ./run_once.rb --harness=perf benchmarks/fib.rb
 #
 # When recording with perf(1), make sure the benchmark runs long enough; you
 # can tweak the MIN_BENCH_ITRS environment variable to lengthen the run. A race

harness/stackprof.rb

@@ -3,8 +3,8 @@
 # Profile the benchmark (ignoring initialization code) with stackprof.
 # Customize stackprof options with an env var of STACKPROF_OPTS='key:value,...'.
 # Usage:
-# STACKPROF_OPTS='mode:object' MIN_BENCH_TIME=0 MIN_BENCH_ITRS=1 ruby -v -r./harness/stackprof benchmarks/.../benchmark.rb
-# STACKPROF_OPTS='mode:cpu,interval:10' MIN_BENCH_TIME=1 MIN_BENCH_ITRS=10 ruby -v -r./harness/stackprof benchmarks/.../benchmark.rb
+# STACKPROF_OPTS='mode:object' ./run_once.rb --harness=stackprof benchmarks/.../benchmark.rb
+# STACKPROF_OPTS='mode:cpu,interval:10' MIN_BENCH_ITRS=10 ./run_once.rb --harness=stackprof benchmarks/.../benchmark.rb
 
 require_relative "../lib/harness"
 require_relative "../lib/harness/extra"

harness/vernier.rb

@@ -3,8 +3,8 @@
 # Profile the benchmark (ignoring initialization code) using vernier and display the profile.
 # Set NO_VIERWER=1 to disable automatically opening the profile in a browser.
 # Usage:
-# MIN_BENCH_TIME=1 MIN_BENCH_ITRS=1 ruby -v -r./harness/vernier benchmarks/...
-# NO_VIEWER=1 MIN_BENCH_TIME=1 MIN_BENCH_ITRS=1 ruby -v -r./harness/vernier benchmarks/...
+# ./run_once.rb --harness=vernier benchmarks/...
+# NO_VIEWER=1 ./run_once.rb --harness=vernier benchmarks/...
 
 require_relative "../lib/harness"
 require_relative "../lib/harness/extra"

lib/harness/loader.rb

@@ -1,9 +1,9 @@
-# Use harness/default.rb by default. You can change it with -r option.
-# The -r option should be specified BEFORE requiring loader.
+# Use harness/default.rb by default. You can change it with the --harness option in run_once.rb
+# or with -r option when calling Ruby directly.
 # Examples:
-#   ruby benchmarks/railsbench/benchmark.rb                      # uses harness/default.rb
-#   ruby -r./harness/once benchmarks/railsbench/benchmark.rb     # uses harness/once.rb
-#   ruby -r./harness/ractor benchmarks/railsbench/benchmark.rb   # uses harness/ractor.rb
+#   ./run_once.rb benchmarks/railsbench/benchmark.rb                  # uses harness/default.rb
+#   ./run_once.rb --harness=once benchmarks/railsbench/benchmark.rb   # uses harness/once.rb
+#   ./run_once.rb --harness=ractor benchmarks/railsbench/benchmark.rb # uses harness/ractor.rb
 
 # Only load the default harness if no other harness has defined run_benchmark
 unless defined?(run_benchmark)

run_once.rb

@@ -0,0 +1,90 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+# Script to run a single benchmark once
+# Provides a clean interface for running benchmarks with different harnesses.
+# Examples:
+#   ./run_once.rb benchmarks/railsbench/benchmark.rb
+#   ./run_once.rb --harness=once benchmarks/fib.rb
+#   ./run_once.rb --harness=stackprof benchmarks/fib.rb
+#   ./run_once.rb -- --yjit-stats benchmarks/railsbench/benchmark.rb
+#   ./run_once.rb --harness=default -- --yjit benchmarks/fib.rb
+
+require 'optparse'
+require 'shellwords'
+
+# Parse options
+harness = nil
+ruby_args = []
+benchmark_file = nil
+
+parser = OptionParser.new do |opts|
+  opts.banner = "Usage: #{$0} [options] [--] [ruby-options] BENCHMARK_FILE"
+
+  opts.on("--harness=HARNESS", "Harness to use (default: default, options: once, bips, perf, ractor, stackprof, vernier, warmup, stats, continuous, chain, mplr)") do |h|
+    harness = h
+  end
+
+  opts.on("-h", "--help", "Show this help message") do
+    puts opts
+    exit
+  end
+end
+
+# Parse our options - this will stop at '--' or first non-option argument
+begin
+  parser.parse!
+rescue OptionParser::InvalidOption => e
+  puts "Error: #{e.message}"
+  puts parser
+  exit 1
+end
+
+# After parsing, ARGV contains remaining args (Ruby options + benchmark file)
+ARGV.each do |arg|
+  if arg.end_with?('.rb') && !benchmark_file
+    benchmark_file = arg
+  else
+    ruby_args << arg
+  end
+end
+
+if !benchmark_file
+  puts "Error: No benchmark file specified"
+  puts parser
+  exit 1
+end
+
+unless File.exist?(benchmark_file)
+  puts "Error: Benchmark file not found: #{benchmark_file}"
+  exit 1
+end
+
+# Automatically detect ractor benchmarks
+if !harness && benchmark_file.include?('benchmarks-ractor/')
+  harness = 'ractor'
+end
+
+# Build the command
+harness_dir = File.expand_path('harness', __dir__)
+harness_args = if harness && harness != 'default'
+  harness_path = File.join(harness_dir, harness)
+  unless File.exist?("#{harness_path}.rb")
+    puts "Error: Harness not found: #{harness}"
+    puts "Available harnesses: #{Dir.glob("#{harness_dir}/*.rb").map { |f| File.basename(f, '.rb') }.join(', ')}"
+    exit 1
+  end
+  ['-r', harness_path]
+else
+  []
+end
+
+# Set environment for running once
+ENV['WARMUP_ITRS'] = '0'
+ENV['MIN_BENCH_ITRS'] = '1'
+ENV['MIN_BENCH_TIME'] = '0'
+
+# Build and execute the command
+cmd = ['ruby', *ruby_args, *harness_args, benchmark_file]
+
+exec(*cmd)