Prompt as a new element of the film language, or How can Sora change the future of digital cinema?

Rakhmankulov Bogdan Marselevich ORCID: 0009-0008-0147-7878 Postgraduate student, Department of Advertising and Public Relations, St. Petersburg State University of Industrial Technologies and Design 191186, Russia, Saint Petersburg, Bolshaya Morskaya str., 18 bogdan.rakhmankulov@gmail.com Other publications by this author

Introduction

Cinema has come a long way from the first "moving pictures" to modern blockbusters with their complex special effects and deep narratives. The film language is at the heart of this art. From the point of view of semiotics, any language is traditionally understood as "a naturally occurring and developing system of sign units clothed in sound form, capable of expressing the totality of human concepts and thoughts, intended primarily for communication purposes" ^{[1, p. 652]}. The language of cinema is no exception. K. E. Razlogov notes that cinema is "a sound-visual reflection of the world, structured in a special way for the purpose of communication, that is, it is a certain sign system, an audiovisual "language", and its peculiarity lies in the fact that it "uses in its manifestations the signs of reality itself, the so-called cultural codes" ^{[2, p. 16]}. Key elements of the film language, such as frame and mise-en-scene, close-up and editing, camera movement, color and light, noise and music, have been developed and improved over many decades. They form a complex system through which cinematographers can influence the emotions of the viewer, convey certain ideas laid down by the author and create artistic worlds.

With the advent of artificial intelligence (AI) and its integration into the process of making films, a new era in cinematography is beginning. AI opens up opportunities for filmmakers that could only be dreamed of before, from automating routine tasks to generating unique content. For example, neural networks have long established themselves as assistants in creating digital special effects and improving graphics. According to visual effects specialist Tim Webber, winner of the Academy Award for Best Visual Effects ("Gravity"), artificial intelligence technologies are embedded in many film production tools, but they are used exclusively to "achieve creative results more quickly" ^[3]. In other words, artificial intelligence and neural networks do not yet replace the human creator, but only help him achieve creative ideas. The question of whether the moment will come when AI will be able to fully control creative processes remains open and causes lively discussions among philosophers, art historians and cultural scientists. I. G. Shestakova concludes that so far "in the tandem of man and computer, the contribution of man is the creative component, the contribution of the computer is the calculation," however, given the enormous With the speed of technology development, we are coming to the conclusion that the need for a human mind for any essential tasks is disappearing ^{[4, p. 58]}. Modern trends show that art created with the help of neural networks opens up new horizons in artistic practice, making technology not just a tool, but also a full-fledged participant in the creative process. This is confirmed by the research of other authors, such as L. Z. Manovich ^[5], M. A. Stepanov ^[6], T. E. Fadeeva ^[7], A. A. Druzhinina ^[8], who note the role of generative art artists in the development of cultural studies and the promotion of interdisciplinary cooperation. But this article reveals another direction of using artificial intelligence – a contribution to the development of the modern socio-cultural environment through the expansion of the possibilities of the film language. AI is becoming a catalyst for the creation of new forms of visual narrative, enriching the cultural landscape and providing new tools for exploring the diversity of human experience. This not only allows us to rethink traditional approaches to filmmaking, but also opens the door to an inclusive and multilateral dialogue in the cultural sphere, offering new perspectives for understanding socio-cultural processes in the digital age. The scientific novelty of this study lies not only in analyzing the potential for full automation of the film creation process, where artificial intelligence moves from the role of an assistant to the role of a full-fledged creator, but also in identifying the practical application of these technologies. This opens up unlimited possibilities for experimenting with narrative, visual style, and the realization of almost any ideas that were previously limited by the technical and financial capabilities of production. Thus, the practical application of the research results can affect areas ranging from the development of new forms of cinematography education to the creation of films with unlimited visual and narrative potential, accessible to a wide range of cinematographers without significant investments in the production process. It also suggests a new stage in the development of the virtual reality industry, where scenarios and visualizations can be created in response to user interaction. In addition, an important aspect of practical application is the development of an ethical and legal framework for the use of AI in film production, which requires a deep understanding of the interaction between humans and artificial intelligence in the creative process.

Superhuman creativity

The modern creative industry demonstrates the growing adoption of generative neural networks, sparking discussions about the nature of creativity, traditionally perceived as an exclusively human quality. An increasing discussion of non-human creativity in the industry suggests that a transformation of ideas about it should be expected. Previously, creativity was an exclusively individual human characteristic, but now it can be considered as a collective process in which a person and a machine act in symbiosis ^[9]. The historical context of such an interaction is not new in itself. There has always been a question about the role of the machine in human life, but now it has evolved into the opposite – the question of the role of man in the world of technological dominance ^{[4, p. 46]}.

Today we are on the verge of fundamental changes in the field of cinema. In addition to the already sensational generative neural networks capable of creating film posters, making storyboards and writing scripts, a new tool for the future of digital cinema has appeared on the market – a neural network from OpenAI capable of generating video based on text prompts, Sora. Technology offers the prospect of democratizing the filmmaking process, opening up opportunities for small production teams to produce content that can compete with the products of large studios, thereby expanding the boundaries of traditional filmmaking and providing new tools for creative expression. This is already being confirmed in practice – director and producer Tyler Perry planned to expand his own film studio worth $ 800 million, but after the release of Sora announced the suspension of work indefinitely for economic expediency: "I no longer need to travel to locations. If I want to be in snowy Colorado, I'll write it in the text. If I need a scene on the moon, it will also be text, and artificial intelligence will generate everything I need. I can just sit in the office and do everything from a computer, it's shocking. I am concerned that many jobs will be lost in the near future."^[10] With opportunities come new challenges.: both applied (job cuts and the replacement of specialists with artificial intelligence) and deeper issues of authorship, originality, and ethical dilemmas surrounding the use of machine learning in creative processes.

The controversy surrounding generative neural networks

Disputes over the use of artificial intelligence in art and culture have been going on for many years, but they usually remained within the professional circle. With the improvement of generative AI models, these discussions have multiplied exponentially. One of the latest high-profile scandals among filmmakers has been the use of ChatGPT in screenwriting. A chatbot capable of generating entire script blocks based on simple user prompts led to a strike by the Screenwriters Guild of the USA. Filmmakers are becoming increasingly vulnerable to artificial intelligence, as automated production is a way for studios to reduce costs and increase profits. The effectiveness of artificial intelligence in creating storylines and writing scripts in a short time is a significant advantage compared to more expensive traditional methods. But that's still not possible for now: ChatGPT cannot completely replace the human creativity and depth of thought needed to create stories that touch on emotions and feelings on a deep level. Although it can generate texts based on preset parameters, artificial intelligence is still limited in its ability to understand the nuances of human experience and cultural contexts, which is crucial for creating meaningful cinematic work. Discussions about the use of AI in cinema and other fields of art emphasize the need for a balance between technological innovation and preserving human uniqueness in the creative process. This is where the very collective creativity that M. A. Stepanov wrote about is important: "The focus of modern media art is not so much scientific and technological capabilities and their exploitation, but rather attention to our existence as a multitude of societies and beings, human and non–human, and the possibility of having a future" ^{[11, p. 93]}. The collaboration between human creativity and the computing power of artificial intelligence opens up new horizons for innovation and experimentation. This interaction allows us to combine the depth of human understanding of emotions, cultural contexts, and story arcs with the efficiency and speed of AI in processing and generating content. But along with these opportunities, new questions arise. Could it be that we are witnessing the rapid destruction of traditional cinema? Isn't it possible that human creativity will be overshadowed by the efficiency and simplicity of machines? Specialized specialists may find themselves out of work only because they will be replaced by artificial intelligence, and generative cinema itself may undermine the tradition of internal dialogue between creators, which has enriched cinematic art throughout time ^[12].

Another important issue of using generative neural networks (for working with both text and audiovisual content) is the stylization of the generated material. So, it is possible to "train" artificial intelligence in the style of specific authors, which will allow generating new works imitating the characteristic features of their work. The potential for abuse of this opportunity is obvious and raises important questions about the ethical and legal aspects of copyright and intellectual property. There is a need to develop clear regulatory frameworks that could regulate the use of such technologies, protecting the rights of authors and preventing the misuse of their work. It is extremely important to maintain transparency regarding the resources that are used to train the neural network, but so far neither Sora, ChatGPT, nor other generative models provide this ^{[13, pp. 11-14]}. Nevertheless, the balance between innovation and intellectual property protection is becoming a key challenge in the era of digitalization of art.

A new element of the film language is a text hint

According to the developers of Sora, the neural network "can create realistic and fantasy scenes based on text prompts" ^[14]. It is assumed that in the case of its full integration into film production, text prompts (prompta) can become a new element of the film language. This, in turn, has the potential to radically transform the film industry, providing directors and screenwriters with unique tools to realize creative concepts.

Promptings are text queries that are transformed by artificial intelligence into detailed visual images and scenes. With their help, directors and screenwriters can experiment with plots, characters, and sets, overcoming previous constraints related to production budgets and technical capabilities. As a result, the realization of fantasy worlds or the accurate reconstruction of historical periods will become much easier and more accessible.

Figure 1. A frame from the generated video for the text query "Historical footage of California during the gold rush" (English Historical footage of California during the gold rush)

As already noted, generating content based on a text query is an established practice in the creative industries. Specialists actively use generative models capable of converting texts into images (Midjourney, DALL-E, Stable Diffusion) and work in an interactive mode in natural language, answering questions and even creating artistic texts (ChatGPT). However, Sora is not just a creative tool. NVIDIA senior Researcher Jim Fan characterizes the new neural network as a "simulator of real and fantasy worlds" ^[15], offering a deep understanding and modeling of the world that goes beyond the usual content generation. In response to the request "Photorealistic close-up video of two pirate ships fighting each other while they sail inside a cup of coffee," Sora generates a 15-second video clip with high detail and 3D graphics elements. This neural network automatically simulates the movement of liquids and realistically determines the trajectories of objects, adapts the size of frames and plans to achieve optimal visual perception, including applying the tilt-shift effect to create the illusion of miniaturization of the scene.

Figure 2. A frame from the generated video for the text query "Photorealistic close-up video of two pirate ships fighting each other while they sail inside a cup of coffee" (English Photorealistic closeup video of two pirate ships battling each other as they sail inside a cup of coffee)

In addition, Sora will allow you to visualize ideas and concepts that would be difficult or even impossible to implement using traditional methods. This opens the door to new visual styles and narrative techniques, expanding the boundaries of cinematic language and offering viewers a unique visual experience. However, this also raises the question of the author's role in filmmaking. In a world where a machine can create visual content based on text queries, determining authorship and creative contribution becomes more difficult. This calls for a new understanding of creative processes and the relationship between Man and machine in the context of cinema.

Will the future of cinema be generated?

Potential discussions around the new Sora neural network will also address these issues. Can or should artificial intelligence create full-fledged films? Until recently, this question wasn't even raised, but then generative models appeared that convert text into video. Early versions of such models could generate visual content lasting several seconds, which is how the first AI films began to appear. Of course, so far this can only be attributed to experimental cinema. In such works, we see a lot of artifacts, inconsistent frames, differences in style, and many other digital defects. Rather, it is not a movie, but video art, often with a non-linear plot and without a clear narrative. So, in early 2023, the author of this article generated a series of one-second episodes using the Chinese generative model ModelScope under the general name "Blink of an Eye". All episodes symbolize different aspects of life, death, and rebirth and offer viewers an allegorical and visual experience, allowing them to interpret each second in their own way. The visual component of "Blink of an Eye" is characterized by the use of different color schemes and compositions to convey the emotions and ideas contained in each episode ^[16]. This approach to creating content using AI at an early stage not only demonstrates the technical capabilities of modern generative models, but also opens up new perspectives for exploring deep philosophical and aesthetic themes through cinema.

Figure 3. Footage of all episodes of "Blink of an Eye"

Another example is Paul Trillo's short film "Thank You for Not Responding", generated in the more advanced generative neural network Runway Gen–2 ^[17]. Unlike ModelScope's work, there are already much fewer artifacts and inaccuracies characteristic of neural networks, but it is still impossible to call the frames photorealistic and highly detailed. Nevertheless, such a generated movie has a special cinematic aesthetic, characteristic of the early work of neural networks to generate static images (Midjourney,
DALL-E and others). This is exactly what makes the painting unique. Today, Midjourney and DALL-E have stepped forward, their generation is already quite photorealistic, and there are fewer machine errors ("bugs") in the content. This means that neural networks for generating audiovisual content will also reach this level very soon. However, Sora, which we've already talked about a lot, has come very close to truly cinematic quality. The developers from OpenAI note that the generated video content has problems with rendering the movement of people's and animals' legs, and the model can still confuse spatial details (for example, right and left) and is difficult to accurately describe events that occur over time. But all this is only at the beta testing stage. In the context of the rapid development of technology, it can be expected that all shortcomings will be overcome in the foreseeable future. Given the pace of progress in artificial intelligence, the evolution of generative models promises to achieve a level of photorealism and cinematic detail that will make them indistinguishable from real video content. This opens up new horizons for experimentation in cinema and offers prospects for even closer interaction between AI technologies and the creative process.

Figure 4. A frame from the movie "Thank you for not answering"

Conclusion

The integration of Sora technology and similar artificial intelligence-based tools into film production opens up new horizons for the creative industry, transforming not only the methods of content creation, but also the very idea of the film language. Industrial, as its possible future element, provides a unique opportunity to experiment with narratives and visualization, democratizing the creative process and making it accessible to a wide range of creators. This promises to revolutionize the industry, authorship, and content production, bringing to the fore the creative symbiosis of Man and machine. Back in the mid-2010s, T. Elsesser and M. Hagener noted that digitalization would lead to the fact that "in the next decade, the theory of cinema will reinvent itself, despite the fact that cinema – as we knew it during the first 100 years of its history – continues to exist in the form of "inside-out", as the host of a new "parasite"" ^[18]. With the advent of new generative technologies, the need to "reinvent oneself" becomes even more urgent for the industry. Indeed, along with opportunities, questions arise about the future role of humans in creativity, the need for a balance between innovation and intellectual property protection, as well as the ethical and legal aspects of using AI in art. Major crises can unfold in the interaction of key elements of filmmaking, such as the script, cinematography, director's perspective, and acting, emphasizing the importance of synergy between the human and the nonhuman. The future of cinema will be defined at the intersection of technological innovation and philosophical reflections on the nature of creativity and the cultural value of art, emphasizing the importance of collaboration and mutual understanding between humans and artificial intelligence to overcome the challenges and take advantage of the opportunities provided by this new frontier.