Research Reproducibility for Small Engineering Projects

A small engineering experiment can be persuasive without being repeatable. A graph looks clean, a demo runs once, and a teammate remembers which command produced the result. Six weeks later, the data has moved, a dependency has updated, and nobody can recreate the number in the slide. The team has not necessarily done bad research; it has failed to preserve the conditions under which the claim was made.

ACM's artifact review and badging policy treats code, datasets, scripts, and generated outputs as artifacts that can be inspected. It distinguishes independently reproduced results from other forms of evidence and explicitly allows reasonable experimental tolerance rather than demanding identical numbers in every setting. A small team does not need a formal badge to benefit from the idea: make the path from question to result inspectable by someone who did not run the original experiment.

Write the claim before collecting more results

Start an experiment note with the exact question, the comparison, the measurement, and the acceptance rule. “Model B is faster” is incomplete. “On the same test set and hardware, median latency falls below a stated threshold without reducing accuracy beyond a stated tolerance” can be tested. Record what would count as a failed outcome. This protects against moving the target after seeing the data.

Separate observations from interpretations. A lower runtime on one machine is an observation. “This architecture will save every customer time” is a much larger claim. Keep notes on sample size, exclusions, and environmental limits beside the chart. If a parameter was chosen after exploring the test set, say so. Honest boundaries make useful findings more credible, not less.

Freeze the inputs that define the experiment

Record the source code commit, dependency versions, input data version, configuration, hardware, and random seeds when they matter. Do not rely on a file named final-data.csv as a version marker. Use a checksum or an immutable dataset snapshot and document how the data was prepared. If the source data cannot be shared for privacy or licensing reasons, provide a schema, synthetic sample, and instructions for authorized access. Explain what an independent reader can and cannot verify.

Dependencies change too. A lockfile is useful for application packages, while the workflow itself may depend on external actions or images. GitHub's workflow dependency lockfile guidance describes how pinned action commits make workflow code reviewable and reproducible at the dependency level. It also notes that such a lockfile does not control every build input. A repeatable run therefore needs more than one pin: data, environment, parameters, and code all contribute.

Make one command produce the reported table

A practical artifact has a short start path: set up the environment, obtain or generate the allowed inputs, run the experiment, and produce the plots or tables. Put commands in scripts rather than a collection of notebook cells that depend on hidden execution order. A notebook can still be a valuable explanation, but a fresh run should work from an empty state. Store raw results separately from the plotting code so a reviewer can inspect both the measurement and the presentation.

Test the path on a clean machine or isolated environment. Have someone other than the author follow the README without private verbal hints. Record any missing steps. If the run is too expensive to repeat fully, provide a small smoke run and a precise account of the full compute requirements. This lets a reviewer distinguish “cannot run here” from “the instructions are broken.”

1. Define the question and acceptance rule before the run.
2. Freeze code, data, dependencies, parameters, and environment.
3. Run a scripted pipeline from a clean state.
4. Save raw outputs with checksums and run metadata.
5. Generate tables and figures from those outputs.
6. Ask a second person to repeat a smoke run and note differences.

Handle randomness and tolerance explicitly

Many systems will not yield identical values on every rerun. Random seeds help identify a configuration, but GPU kernels, distributed scheduling, and data-order effects can still vary. Run enough repetitions for the claim being made, report spread rather than only the best number, and define an acceptable tolerance before comparing. ACM's policy notes that exact reproduction is not always expected; agreement must be judged in the context of the experiment and its claims.

Keep failures and negative runs where they affect interpretation. If three configurations failed and one succeeded, the reader needs to know how the winner was selected. Avoid cherry-picking the cleanest chart. A compact experiment log with run IDs, parameters, status, and output paths can preserve the search process without overwhelming the main article.

Protect sensitive data while keeping the method inspectable

Reproducibility does not justify publishing private datasets or credentials. Document access rules, redact personal fields, and use synthetic fixtures for public smoke tests. Keep secrets outside the artifact. If an internal reviewer has authorized access to a protected dataset, provide a controlled way to run the same pipeline and compare output summaries. State clearly when a public user can validate only the code path, not the original data-based result.

Consider an illustrative two-person team evaluating a search-ranking change. One engineer reports a 9% improvement on a private query set. The teammate cannot reconstruct it because the query set was silently edited and the metric script ran from an unsaved notebook cell. The team freezes a dataset snapshot, commits the metric script, records the model version and seeds, then reruns the comparison. The revised result is smaller but defensible. This hypothetical example shows why reproducibility is useful even when it changes a celebrated number.

Package the artifact for the next person

Include a README with purpose, prerequisites, exact commands, expected runtime, data availability, outputs, and known limitations. Link the reported figure to the run ID and raw data that produced it. ACM SIGIR's artifact-badging guidance highlights the value of documented, structured artifacts that other researchers can reuse. For a small team, the same habit reduces the cost of onboarding and prevents a result from depending on one person's memory.

Revisit an artifact when the project changes. A dependency patch or new dataset may produce a different answer; that is a new run, not a reason to overwrite the old evidence. Version the outputs and explain the change. The aim is not to freeze all future development. It is to let someone answer what was run, why it mattered, and whether the result still supports the claim.

Practical checklist

• State the hypothesis and failure rule before running.
• Record code, data, parameters, seeds, and environment.
• Use a clean scripted path to create reported outputs.
• Save raw results and link each figure to a run ID.
• Report variation, failed runs, and reasonable tolerance.
• Keep private data and secrets out of public artifacts.
• Test the instructions with a second person.
• Version new results instead of overwriting old evidence.