Over the past few years, there’s been a lot of new technology that has had a profound impact on gaming. With DLSS Super Resolution, NVIDIA paved the way for other upscaling technologies like PlayStation‘s own PSSR and QSSR, and with Frame Generation, it has made it easier for gamers to get a taste of a high-framerate experience, but only in terms of perceived smoothness, as it’s the base framerate that determines the real smoothness of a game. This is still today the number one issue with Frame Generation tech, alongside image quality issues caused by generated frames interpolated with rendered ones.
PlayStation is well aware of the input delay issues only mitigated by technology like NVIDIA Reflex, and it seems like the company is seeking to solve them with a new frame prediction tech detailed in a recent patent.
“Frame generation super sampling may include generating an image frame embedding from a first image frame in an image stream and predicting a synthetic second image frame in the image stream using the image frame embedding of the first image in the image stream. The synthetic second image is displayed frame after the first image frame in the image stream,” reads the summary of the patent filed by Sony on October 15, spotted by Patentlyze.
On the surface, this may not sound much different from how tech like NVIDIA DLSS Frame Generation works, but it’s actually quite different. While Frame Generation“inserts a synthetic frame generated by a machine learning model between the last GPU-rendered frame and the one before that prior to display,”PlayStation‘s patented tech predicts the next frame based on the player’s expected input, using the same principle that is used for rollback netcode used in modern fighting games like Street Fighter 6, however at a whole different level in terms of complexity.
“The user input here may be made contemporaneously (e.g., within 15 ms) with the generation of the input last image or shortly before (e.g. greater than 15 ms display but less than two seconds before) the last image frame is generated. The idea being that the user input is tied to generation of the image frame; e.g., when a player in a game presses the jump button, the next few frames will be jump animation frames which can be learned by the encoder,” the patent explains.
This tech would essentially be an Algorithmic Frame Generation that could potentially slash input delay if implemented right. “Algorithmic Frame Generation uses an algorithm instead of machine learning and neural networks to generate a synthetic frame. According to aspects of the present disclosure, the Algorithmic frame generation may use either motion vectors or a real previous frame with the real last frame.”
Ultimately, this frame prediction tech patented by PlayStation would use an encoder that reads a rendered frame, optional motion vectors, and controller input, and then a decoder that outputs a generated frame that would be shown before the next real frame arrives, covering multiple possible options by generating multiple future frames. To mitigate the risk of an incorrect prediction, the system also has a non-AI fallback.
Interestingly enough, the patent doesn’t detail any additional hardware requirements beyond what’s already in current gaming systems, so this could potentially be deployed on current-generation systems. However, it is unlikely we will see frame prediction tech anytime soon, given the complexity of the system and how it would lead to a terrible experience by failing repeatedly in games like fast-paced action games or competitive shooters.
Nevertheless, this new PlayStation tech sounds incredibly interesting as much as other tech the company is working on, including one that could rewrite enemy behavior on the fly, and has the potential to bring incredibly smooth gaming experiences at 4K resolution if implemented correctly. As with the rollback netcode, which shares so many similarities with it, there will likely be growing pains, but they could definitely be worth it.
About the author: Francesco De Meo has been covering video games and technology since 2012, starting his career at small outlets like Gamersyndrome and GeekSnack.
After joining Wccftech gaming section in 2015, he quickly expanded his video gaming coverage with in-depth reporting, interviews with iconic industry figures such as Grasshopper Manufacture founder and No More Heroes creator Goichi “Suda51” Suda, Resident Evil series creator Shinji Mikami, Team NINJA’s president and Nioh series director Fumihiko Yasuda, and Silent Hill creator Keiichiro Toyama, reviews and on-the-ground coverage of major industry events such as Gamescom and E3.
When he’s not reporting or reviewing, Francesco can be found playing the genres he loves most, spending time with his six cats, reading, writing music, playing guitar and drumming for his progressive rock band.
Follow Wccftech on Google to get more of our news coverage in your feeds.
Deal of the Day
