Good test authoring requires careful analytical thinking about what differences exist between two test cases and whether the differences between them are intentional or unintentional.
Write better, clearer tests. But also, take the time to document (using the jsPerf.com “Description” field and/or code comments) exactly what the intent of your test is, even to the nuanced detail. Call out the intentional differences, which will help others and your future self to better identify unintentional differences that could be skewing the test results.
Instead of trying to narrow in on a tiny snippet of your real code and benchmarking just that piece out of context, tests and benchmarks are better when they include a larger (while still relevant) context. Those tests also tend to run slower, which means any differences you spot are more relevant in context.