Flawed Benchmark Update Shows Pricey Intel Core i9-9900K Much Closer To AMD Ryzen 7 2700X
Intel and its hired benchmarking gun, Principled Technologies, have come under fire for the benchmarking methodology used to validate the Core i9-9900K as the "world's fastest gaming processor." At the heart of the issue is the latter's decision to enable Game Mode on AMD's Ryzen 7 2700X, resulting in a performance disparity upwards of 50 percent compared to the Core i9-9900K. Principled Technologies has since retested the Ryzen 7 2700X in Creator Mode, and while the new data still has the Core i9-9900K coming out ahead, the disparity isn't nearly as dramatic.
As you might already know, the Core i9-9900K is Intel's flagship CPU in its recently announced 9th generation Core processor family, and it's first 8-core/16-thread mainstream desktop part. It launched alongside the Core i7-9700K (8-core/8-thread) and Core i5-9600K (6-core/6-thread), all three of which are built on and benefit from Intel's 14nm++ manufacturing process.
During the launch event, Intel touted the Core i9-9900K as the fastest gaming processor on the planet. To back up its claim, Intel commissioned Principled Technologies, a third-party "marketing, learning, and testing services" company, to test the Core i9-9900K in a bunch of gaming benchmarks (19 in all) and compare the results to several other processors. Those results were made public on the same day as the 9th generation Core launch event.
Quick side note—performance reviews of the Core i9-9900K are still under embargo until October 19. That means reviews sites like us and others are unable to compare any benchmarks we might already have, with those that Principled Technologies obtained. It also leaves customers with a single official source of benchmarking data to decide whether to pre-order the part. So, it's not an ideal situation.
That said, it's the testing methodology Principled Technologies used that has come under scrutiny. Steve Burke at Gamers Nexus scored an exclusive interview with the testing lab's co-founder Bill Catchings, and in the 46-minute video posted online, Burke does a good job of covering some of the questionable testing decisions, including the type of cooling and specific settings.
What seems clear from the interview and Principled Technologies' response to the situation is that there was no ill intent. It doesn't appear the company was trying to 'game' the results to favor Intel. Testing CPUs and running gaming benchmarks isn't something it does all the time. And to the company's credit, after being informed of how enabling Game Mode on a mainstream Ryzen processor can (and in some cases, does) hurt gaming performance, it went back and retested all Ryzen chips with Creator Mode.
"Specific to AMD CPUs, we started the testing on Game Mode for AMD Ryzen Threadripper processors. Those results did indeed show Game Mode overall yielded the best gaming performance on Threadripper. For consistency, we then used Game Mode on all of the AMD processors. We have not added results from our testing of the AMD Ryzen 7 2700X in its default mode (i.e., Creator Mode) as well. That mode overall yielded the best gaming performance on the 2700X. We apologize for not testing both modes in the initial report," Catchings said in a follow-up statment (PDF).
Click to Enlarge (Source: Principled Technologies)
Switching to Creator Mode didn't always make much of a difference, but there were definitely cases where it did. More often than not, that was the case, actually. For example, in Gears of War 4, the performance advantage of the Core i9-9900K dropped from 46.4 percent to 24.9 percent when switching from Game Mode to Creator Mode on the AMD part. And in Ashes of the Singularity, the Core i9-9900K went from being 57.2 percent faster to 17.7 percent.
Intel embraced the new results, issuing an updated statement of its own to underscore its original point, which is that the Core i9-9900K is the fastest gaming processor.
"Given the feedback from the tech community, we are pleased that Principled Technologies ran additional tests. They've now published these results along with even more detail on the configurations used and the rationale. The results continue to show that the 9th Gen Intel Core i9-9900K is the world's best gaming processor. We are thankful to Principled Technologies' time and transparency throughout the process. We always appreciate feedback from the tech community and are looking forward to comprehensive third party reviews coming out on October 19," Intel said.
So there you have it, the Core i9-9900K was still shown to be faster in games than the Ryzen 7 2700X, just not to the same extent as the original testing methodology indicated. Whether Intel should have promoted the original set of benchmarks is a deeper discussion.
In any event, actual reviews are coming, and it won't be long before potential buyers have a much wider set of data and analysis to base their decisions on.
