NVIDIA ICAT Unveiled: In-Game Image Quality Analysis Made Easy
NVIDIA's Spatial Upscaling Takes on FSR And New ICAT Evaluation Tools
There's been a lot of discussion about image quality and super-sampled graphics in 3D games over the last few months. Ray tracing is quite expensive from a computational standpoint, so frame rates typically suffer when it's enabled. Upscaling technologies seek to claw some of that performance back, by rendering at a lower resolution and scaling up to the display's native pixel density, using high quality algorithms and other innovative techniques. NVIDIA's Deep Learning Super Sampling (DLSS) version 2, which uses machine learning to upscale games, was the only game in town for a while, but that has started to change.
More recently rival AMD introduced FidelityFX Super Resolution (FSR) in an effort to do something similar without the need for AI assistance, allowing upscaling on any graphics card with enough shader performance to handle the processing. Intel has also rattled its own sword about Xe Super Sampling (XeSS), which is also GPU agnostic, but also has modes that will use some Intel-only hardware features for better upscaling. Unlike DLSS and some parts of XeSS, FSR is interesting, because it works on any graphics architecture from AMD itself to NVIDIA and Intel using spatial data from the objects in the frame. In essence, FSR is a post-processing shader that games can call prior to drawing the HUD to upscale a 3D world to any resolution. The added benefit of wider compatibility in FSR is nice. The drawback, like other techniques, is that FSR still takes developer intervention and support within the game.
NVIDIA's Own Spatial Scaling
If our Two and a Half Geeks episode on FSR taught graphics vendors anything, it's that there's a desire for technology like FSR which runs on any graphics processor, but goes a step further with a global switch. That's exactly what NVIDIA has done starting with today's GeForce Game Ready driver. NVIDIA has had its own take on spatial upscaling for a little bit, available as a plugin for popular game engines, but now it can be force-enabled across all games. The switch is found in both the GeForce Experience app and the NVIDIA GeForce driver's control panel.
NIS options in the NVIDIA Control Panel
Like FSR, NVIDIA's spatial image scaler, which is aptly named NVIDIA Image Scaling is hardware agnostic, meaning it will run on AMD and NVIDIA graphics cards alike. The downside for Radeon owners is that games still have to be built with NIS in mind, since AMD's drivers don't have a global switch. A plug-in is available for game engines like Unreal and Unity and it will work on any DX12 Ultimate capable GPU, including those from its competitors. GeForce owners will be able to apply this upscaling to any game, however, thanks to this global switch.
What's a little confusing about the GeForce Experience control panel (above) is that it asks you to pick a resolution. This is what GFE will automatically optimize games to use, but not the only resolution which can be upscaled. Using the on-screen overlay indicator, we we found that we could select any resolution and it would be upscaled. As long as we didn't pick our display's native 1440p resolution, the NIS indicator was green, and our monitor always reported an output resolution of 1440p.
To try out NVIDIA Image Scaling, just update to the latest Game Ready driver 496.70 (or newer), which should be on NVIDIA's website by now. NVIDIA's control panel didn't really make it obvious, but the documentation states that once enabled, we needed to select the lower resolution in game so the driver will upscale to the display's native resolution. A handy overlay can be toggled, which shows NIS in blue when upscaling is not in use and in green when the image is being upscaled.
We took it for a spin in Marvel's Guardians of the Galaxy on our testbed system which has a Ryzen 7 5800X with 32 GB of DDR4-3200 and a GeForce RTX 3070 graphics card. NIS can be enabled with and without DLSS, which can really reduce the render resolution. We mostly stuck to the highest quality settings for each, so as to put as much strain on the graphics card as possible at different settings. Then for fun we tested with NIS set to 720p and DLSS set to Performance. That gives an input resolution of just 640x360, and it's a wonder the game doesn't look like something from the PS2 era as a result.
Overall, NIS adds a nice little bump to performance since it renders at a lower target resolution and then upscales the image. We'll talk about image quality in a little bit because the company also released a handy tool to help gamers see the difference between all these different technologies. Unfortunately, the bottom rung of the image resolution ladder isn't really any faster than our DLSS + NIS combo, so it looks like we've bumped up against a frame rate ceiling in this particular game. We'll get to how each of these look in just a moment.
DLSS 2.3 Update Boost Image Quality
The good news is that NVIDIA hasn't stood still with its temporal AI-assisted upscaling option, DLSS. The SDK for NVIDIA's machine learning algorithm has been released to developers, and some of the early results appear to be quite promising. The latest update to the AI model looks to eliminate temporal artifacts that are well-documented across games that used older versions of the technology. The AI now looks for trails and attempts to cull them from the processed image. These fixes are already available in a few games, so gamers can see the updated algorithm in person for themselves already.
The DLSS 2.3 update for Cyberpunk 2077 fixes trails off of moving objects like car mirrors
We got to see a couple of examples in motion in recent updates to Cyberpunk 2077 and Doom Eternal, and the results were quite convincing. Cars in Cyberpunk 2077 would frequently show trail artifacts off the vehicles used to traverse the streets of Night City, and the most recent update includes DLSS 2.3 so it's something we can all see in person. The Doom Eternal example that NVIDIA demonstrated corrects some issues with particle effects. Burning embers that float through the air were reduced to grainy trails of light in earlier versions, but gained back definition in a convincing way. Don't take NVIDIA's word for it, though; these are both already available for anybody with a GeForce RTX 2060 or higher to see.
NVIDIA ICAT: Image Quality Analysis Made Easy
NVIDIA's ICAT, or Image Comparison Analysis Tool, makes it easy to compare both still images and recorded video in side-by-side comparisons. The company is releasing ICAT today alongside its new upscaling features and it's free for download. Because the tool works with captured media, it's just as capable of processing video from AMD or Intel graphics as it is NVIDIA's own. The only caveat is that global NIS can't be captured by NVIDIA's Shadow Play, so a separate PC with a hardware capture card is required. We used a second PC with a Ryzen 5 5600G and an Avermedia LiveGamer 4K PCIe card to grab video from the testbed PC that we used for performance comparisons above.
Zooming in on video side-by-side really makes aliasing artifacts stick out
ICAT is very simple to use and has a lot of power under the hood. The first step is to bring in whatever captured videos you have. Those video clips don't have to start at the same place, since the timeline allows shifting video by clicking and dragging left and right, with two little frame-by-frame arrows on the far left. This allowed us to perfectly sync up to four video clips of the Guardians of the Galaxy benchmark in just a couple of minutes. The importance of that task cannot be understated, as it took literal hours to do that for a handful of games in our PlayStation 5 M.2 upgrade video when we compared load times. NVIDIA's tool is slick and easy for this task compared to Final Cut Pro X or basically any other non-linear video package.
Once everything was lined up, we could easily shift all four videos at once by grabbing one in the preview window and dragging around. The same is true for zooming; all four clips are kept in perfect sync not just in time, but in zoom level and pan location. ICAT is also capable of adjusting the playback speed, as well. All of this functionality literally could not possibly be any easier to do -- even with this first revision, NVIDIA's thought things out well.
We wanted to export some video ourselves to demonstrate how powerful it is, but ICAT is not capable of rendering and exporting a video of its own for uploading to YouTube. It's not even capable of remembering what it was doing the last time it was launched, as there is no project save ability, either. Both of those are features we asked NVIDIA to consider adding in the future, and we really hope that happens. Video exports would make demonstrating image comparisons very easy for you, our audience, down the road. We could have just captured video of the system running ICAT but even our beefy testbed was not able to keep up with four simultaneous video streams in real-time and then you're dealing with video being re-encoded twice over so some of the effect is lost.
One might ask why we're so interested in ICAT when we ourselves aren't totally sold on the idea of pixel peeping at high zoom levels and putting frames under a microscope, and that's a valid question. The reason is simple, though: NVIDIA has given us the tools to examine up close with side-by-side comparisons in motion, not just in still images. After all, we experience games in motion, not one frame at a time (unless you happen to be playing GLQuake on an S3 ViRGE in 1998, that is) and ICAT lets us experience that without putting our noses to the monitor. It's a far better method for comparing image quality than watching individual videos one at a time and trying to remember all the differences. In fact, some issues like flicker or other "dot crawl" effects only show up in video, even if individual still images look good.
Not only is ICAT able to do four-way comparisons of different videos synced in time, but it can also do side-by-side comparisons complete with a slider bar down the middle. This allows us to put up to two videos on the screen and drag the divider back and forth to see how image quality changes between settings. There's no zoom this time around, and we can only see two videos at once, but when we load four videos to the timeline we can select which one appears on the left and right in real time, so flipping between the different captures is simple.
Four-way face-off with DLSS and NIS in various combinations. Upscaling looks well-defined with 50% sharpening
As for the actual image quality between NIS, DLSS, and native rendering, we're pretty impressed with the combination of DLSS and NIS together. NVIDIA provides a sharpening filter with NIS which can look a little aliased under a microscope, but that goes away in motion at full speed and 100% zooming. There's a little bit of loss in texture detail which goes along with the reduced resolution. We found that if we dialed the sharpening slider down to around 25%, some of those aliasing artifacts went away and the drop became much less noticeable. The fact is there are dozens of combinations of DLSS, NIS, and sharpness to fool around with; we're confident that most gamers can find a combination that works for them.
However, for some, this is going to feel like dirty console talk. Image quality comparisons between the Xbox One and PlayStation often talked about these same things: input resolution, smoothing filters, and upscaling. Some members of the #PCMR will definitely look down their noses at NVIDIA's new features much like they have DLSS and FidelityFX Super Resolution, but that's fine. At least on the PC we get a choice as to whether we turn these things on and off, and now there's a powerful new tool by which we can judge image quality and make better, informed decisions.
Conclusions: Image Upscaling Tools Are Not Created Equal
NVIDIA dropped a big pile of image quality tools today. Whether it's NVIDIA Image Scaling, DLSS 2.3, or a combination of the two, it's never been easier to wring extra performance out of our games. The most important part of it all is that NVIDIA makes NIS available directly in the driver, something that no other upscaling technique has done to date. This is likely welcome relief from GeForce owners who haven't been able to upgrade their graphics cards in quite a while due to shortages and incredibly inflated scalpers pricing the cards out of most gamers' financial means.However, the big bomb for us is ICAT. NVIDIA's image quality analysis tool made great strides in image comparison by allowing us to use moving pictures, rather than still images. Throw in all the extra capabilities like video sync, zooming, and panning, and we've got a killer tool on our hands. Right now it's pretty basic without any video export or project files to save our work. We're sure we're not the only ones who would like to see those features added, as it would make creating YouTube videos with image quality comparisons dead simple. For now, you can head over to NVIDIA's site and download ICAT and try it out for yourself.
The introduction of ICAT also raises some interesting questions. In side-by-side comparisons, its much easier to see the image quality differences between the various DLSS and FSR modes currently available. Often, even with both methods set to similar modes, using similar input/output resolutions, there are distinct image quality differences, which have to be considered when doing direct performance comparisons. Two modes may have similar names, but if they offer different output image quality, and one is clearly superior, is it fair to directly compare their performance? We've only had our hands on ICAT for a short time, but you can bet these are things we'll have to discuss in future graphics coverage. If you have opinions on the subject, we'd love to hear from you in the comment below.
