OpenAI’s Sora is a true Milestone in Generative AI. It is a beautiful example of the state-of-the-art. It can create interactive 3D environments and videos on the spot.
Oddly enough, one of the breakthroughs that preceded it is a model architecture for artificial intelligence models named diffusion transformer, which arrived on the scene for AI research years before.
The diffusion transformer seems ready to revolutionize the field of GenAI by allowing GenAI models to grow above the limits of what was previously feasible. It also powers the newest image generator from AI startup Stability AI, called Stable Diffusion 3.0.
In June 2022, NYU computer science professor Saining Xie started the research that resulted in the diffusion transformer. Diffusion and the transformer are two machine-learning ideas that Xie combined with William Peebles. During Peebles’ internship at Meta’s AI research lab, he was his student and is currently co-lead Sora at OpenAI to build the diffusion transformer.
The majority of contemporary AI-powered media generators, such as OpenAI’s DALL-E 3, produce photos, videos, voice, music, 3D models, artwork, and more through a process known as diffusion.
Although it may not seem like the most obvious concept, noise is added gradually until a piece of media, like an image, is no longer recognizable. It is done repeatedly using a data set of distorted media fusion models trained on this. It gradually learns how to remove the noise and gets closer to a specific outcome piece of information (like a new image) step by step.
A U-Net is the standard “backbone,” or sort of engine, of diffusion models. The U-Net backbone gains proficiency in estimating the noise that needs to be eliminated. However, U-Nets are intricate, containing modules specifically made to slow down the diffusion pipeline significantly.
Transformers may replace U-Nets while also increasing performance and efficiency.
For sophisticated reasoning tasks, transformers are the preferred architecture. This architecture powers models such as GPT-4, Gemini, and ChatGPT.
Transformers have many distinctive qualities, but their “attention mechanism” sets them apart. Transformers evaluate the significance of all of the other input (any other distortion in an image) for every element of input information (in case of diffusion or picture noise) and use the results to produce the result (an indicator of the image distortion).
Transformers are not only more parallelizable than other model architectures, but they are also more straightforward thanks to the attention mechanism. To put it another way, training ever-larger transformer models is possible and can be performed with noticeable but reasonable increases in processing power.
In an email interview, Xie compared what transformers do to an engine change regarding how they aid in the diffusion process. The development of transformers represents a significant improvement in efficacy and scalability. It is especially true for models like Sora, which demonstrate the transformative power of transformers when used at scale by utilizing extensive model parameters and training on massive amounts of video data.
Since diffusion transformers are a concept introduced previously, why did it take so long for projects like Sora and Stable Diffusion to use them? According to Xie, the significance of possessing a flexible backbone model was only recently realized.
He claimed that the Sora team went beyond what was necessary to demonstrate how much more can be accomplished on a large scale with this strategy. They have stated that transformers will now be used for diffusion models instead of U-Nets.
According to Xie, diffusion transformers ought to be an easy replacement for current diffusion models, regardless of whether the models produce images, videos, audio, or other types of information. As it currently occurs, diffusion transformer training may result in specific errors and performance damage, but Xie thinks these issues can be resolved in the long run.
According to him, the critical lesson is elementary. Abandon U-Nets in favour of transformers, which are quicker, more effective, and more scalable. The integration of the realms of content creation and interpretation inside the diffusion transformer framework is something that interests me. These are currently like two distinct realms, one for creation and the other for understanding. These elements are being merged in the future, and standardizing the underlying architectures is necessary to make this integration possible. Transformers are an excellent choice for this.
We’re in for a wild ride whether Sora and Stable Diffusion 3.0 offer any indication of what to anticipate from diffusion transformers.
