New Video Shows Cortex-A15 Decimating Tegra 3; Be Wary of The Kool-Aid
Texas Instruments has posted a new video at YouTube that claims to show a dual-core OMAP5-based Cortex-A15 at 800MHz beating the snot out of an unspecified "commercially available" 1.3GHz quad-core device. Given that the only quad-core Tegra 3 tablet available is Asus Transformer Prime, it's not exactly hard to put two and two together. The Transformer Prime is extremely well-regarded, but there've already been rumors that the chip doesn't pack enough horsepower; Asus' highest-end TF700T, with its 1900x1200 screen, reportedly relies on a Qualcomm solution due to Tegra 3's limited memory bandwidth.
At first glance, the TI video seems to confirm that.
Both devices use Ice Cream Sandwich and are loading pages from EEMBC's BrowsingBench benchmark. TI claims that the systems are also downloading a video and playing back an MP3. The results show TI's 800MHz dual-core Cortex-A15 completing its page loads in 95s seconds, while the Tegra 3 device takes 201 seconds to finish the same workload. Huge victory for TI and OMAP5, right?
Not so fast. The reason we're so suspicious of these results is the size of the performance gap. Forget the dual-vs-quad argument -- clock for clock, the Cortex-A15 is expected to be 40% faster than the A9. That should put a theoretical 800MHz A15 just behind a 1.3GHz A9 in a best-case scenario. Instead, TI claims their chip is more than 2x as fast. A gap that large strongly implies that other factors are at work, with network configuration the prime suspect.
The Transformer Prime's WiFi reception and transmission speeds aren't all that great, with the first run of products disproportionately affected. TI's test, with its simultaneous video download, plays to that particular weakness. As part of an overall review, it's completely fair to measure and compare the Prime's WiFi performance with other shipping products -- but the OMAP5 TI demonstrated is months away from that point.
Next up, there's BrowsingBench itself. The test is designed to allow reviewers to simulate a wide range of network conditions, from high-latency/low bandwidth connections to "Sitting on the WiFi unit." BrowsingBench has a default mode, which it uses to calculate general performance—but that mode isn't being used here. Instead, all we see are page load times. There's no proof that the two systems are equally configured or even connecting to the same router.
Finally there's the question of when, exactly, it's fair to start publishing comparisons. It's always been a given that Tegra 3's quad-core A9's would face stiff competition from manufacturers like Qualcomm, but Qualcomm's Snapdragon S4 is set to ship in the first half of the year. OMAP5, in contrast, isn't expected until 2013. Even if TI's performance figures are valid, the video compares the performance of a system you can buy today with that of a system you won't be able to buy for at least 9 months. That's dubious, at best.
Nvidia is no stranger to benchmark shenanigans; the company has been caught with its hand in the cookie jar more times than we can remember -- but that doesn't make obfuscation and questionable result publishing a good idea. It's possible that TI's comparison is fully on the level, but comparing against hardware without a firm ship date is never fair -- and there's good reason to think the results weren't measured fairly.
At first glance, the TI video seems to confirm that.
Both devices use Ice Cream Sandwich and are loading pages from EEMBC's BrowsingBench benchmark. TI claims that the systems are also downloading a video and playing back an MP3. The results show TI's 800MHz dual-core Cortex-A15 completing its page loads in 95s seconds, while the Tegra 3 device takes 201 seconds to finish the same workload. Huge victory for TI and OMAP5, right?
Not so fast. The reason we're so suspicious of these results is the size of the performance gap. Forget the dual-vs-quad argument -- clock for clock, the Cortex-A15 is expected to be 40% faster than the A9. That should put a theoretical 800MHz A15 just behind a 1.3GHz A9 in a best-case scenario. Instead, TI claims their chip is more than 2x as fast. A gap that large strongly implies that other factors are at work, with network configuration the prime suspect.
The Transformer Prime's WiFi reception and transmission speeds aren't all that great, with the first run of products disproportionately affected. TI's test, with its simultaneous video download, plays to that particular weakness. As part of an overall review, it's completely fair to measure and compare the Prime's WiFi performance with other shipping products -- but the OMAP5 TI demonstrated is months away from that point.
Next up, there's BrowsingBench itself. The test is designed to allow reviewers to simulate a wide range of network conditions, from high-latency/low bandwidth connections to "Sitting on the WiFi unit." BrowsingBench has a default mode, which it uses to calculate general performance—but that mode isn't being used here. Instead, all we see are page load times. There's no proof that the two systems are equally configured or even connecting to the same router.
Finally there's the question of when, exactly, it's fair to start publishing comparisons. It's always been a given that Tegra 3's quad-core A9's would face stiff competition from manufacturers like Qualcomm, but Qualcomm's Snapdragon S4 is set to ship in the first half of the year. OMAP5, in contrast, isn't expected until 2013. Even if TI's performance figures are valid, the video compares the performance of a system you can buy today with that of a system you won't be able to buy for at least 9 months. That's dubious, at best.
Nvidia is no stranger to benchmark shenanigans; the company has been caught with its hand in the cookie jar more times than we can remember -- but that doesn't make obfuscation and questionable result publishing a good idea. It's possible that TI's comparison is fully on the level, but comparing against hardware without a firm ship date is never fair -- and there's good reason to think the results weren't measured fairly.
