Preface
As I gain more and more exposure to the AI field, some of my previously vague understandings are gradually becoming clearer, and my needs have also slowly transitioned from the simplest chat-based usage to text generation (text-to-image, text-to-speech, text-to-video, text-to-code, etc.).
If we further subdivide AI needs based on this, we can more precisely divide AI needs into the following levels (the division of these levels is based on different requirements for ”accuracy”, ”subjectivity”, ”execution requirements” in application scenarios, and the user's ”trust” in AI):
- Level 1: Information Query and Vague Answers
The AI need at this level is to quickly provide vague information or reference suggestions, similar to a search engine alternative. It is mainly applied in scenarios such as daily Q&A and encyclopedia queries, where users only need a roughly feasible answer rather than strictly accurate content, such as travel recommendations or recipe suggestions. These types of needs usually do not require strict verification unless they involve health or sensitive areas.
- Level 2: Open-ended Content Generation (High Subjectivity, Low Accuracy)
At this level, AI is mainly used to ”generate content preferred by users based on text provided by users, to satisfy subjective needs rather than precise standards.” Typical applications include image generation and text creation, where the output only needs to meet the user's aesthetic or emotional preferences, without considering the objective accuracy of the content, such as generating artistic images, videos, or writing stories based on text.
- Level 3: Executable Task Generation (Basic Accuracy Requirements)
The AI need at this level is to generate content with executability and basic functionality for direct use by machines or humans, such as code generation or preliminary scripts for data analysis. The generated content needs to pass basic tests and achieve certain functional requirements, but does not pursue absolute precision. Therefore, tasks like code generation need to meet certain running standards but do not have to be completely error-free.
- Level 4: High-Precision Generation and Decision Support (High Accuracy, Strict Logic)
This level requires AI to generate strictly accurate and logically rigorous content, especially suitable for rigorous fields such as medicine and law. Application scenarios include medical report generation and legal clause drafting. The generated content must achieve high precision and logical consistency, and usually requires further human verification to ensure there are no logical errors or ambiguities.
- Level 5: Complex, Multi-step Task Execution (Automated, Multimodal Tasks)
This level involves AI executing multi-step, multimodal tasks while maintaining logical consistency throughout the process. Applications such as intelligent customer service, virtual assistants, and autonomous driving require AI to integrate data inputs from various modalities, such as voice, images, and text, and respond precisely based on context. This requires AI to maintain high precision and consistency across continuous steps.
- Level 6: Real-time, Dynamic Decision-making and Adaptive Tasks
At this level, AI needs to have the ability to adapt to environmental changes in real-time and perform complex unstructured tasks. Typical applications include real-time route adjustment in autonomous driving or intelligent security monitoring. AI not only needs to dynamically perceive the environment but also find a balance between high precision and rapid response to ensure the reliability and safety of decisions.
- Level 7: Fully Autonomous Multimodal Collaboration and Creative Tasks
The highest level of AI need is the ability to autonomously perform multimodal collaboration and creative tasks, such as generating complex design schemes and scientific experiment suggestions. These tasks require AI to possess a high degree of creativity and self-evaluation capabilities, acting as a “collaborator.” While meeting strict logical requirements, AI must also balance innovation, such as automated scientific research experiment design or the generation of new drug molecular combinations.
This layering of AI usage from ”simple to complex” and ”needs from vague to precise” perfectly aligns with the requirements of different tasks in practical applications: it helps us understand how AI functions and output requirements change across different scenarios.
Note 1: Levels 4 to 7 belong to advanced applications of AI, which ordinary people generally do not encounter. Therefore, Levels 1 to 3 are the scenarios we encounter most frequently.
Note 2: Starting from Level 2, an important concept is involved: Multimodal AI.
Multimodal AI
FBI warning: This part of the content is relatively dry, but it is essential for the structure of the article. Friends who are not interested in these technical details can skip directly to the practical section later.
What is Multimodal AI?
Multimodal AI refers to the ability of AI systems to understand, process, and generate multiple types of data, such as text, images, audio, and video, and to establish connections between different data modalities. For example, a multimodal AI can generate images based on text descriptions (text-to-image) or generate text descriptions for scenes in a video (video-to-text). This capability makes AI more flexible in handling complex and diverse tasks, allowing it to more naturally simulate human multi-sensory information processing.
The Importance and Implementation Difficulties of Multimodality
The importance of multimodal AI lies in its ability to understand information provided by humans more comprehensively. Humans often rely on multiple modalities when communicating, such as language, facial expressions, body movements, and sounds, conveying information through multiple channels. Multimodal AI can integrate these different data sources to form a deeper understanding of information, which is particularly important in the following aspects:
Enhanced Understanding Capability: By integrating information from different modalities, AI can understand context more accurately. For example, in autonomous driving, AI systems need to simultaneously process vision (road images), sound (alarms or sounds of surrounding vehicles), and radar sensor data to make safer decisions.
Improved Generation Capability: Multimodal AI can generate content of one modality from another, such as generating images based on text or generating descriptions based on images. This capability is widely used in creative content generation, assisted writing, design, and other fields, helping people realize creative ideas more easily.
Achieving More Natural Human-Computer Interaction: Multimodal AI brings human-computer interaction closer to communication between humans. For example, in virtual assistants, by combining voice, image, and text understanding, AI can understand user needs more comprehensively and provide more practical answers.
The difficulty in implementing multimodal AI lies in the fact that different data modalities are inherently very different and processed differently. The features and structures of image data and text data are completely different, making it a major challenge to have AI learn to establish connections between them. In addition, how to effectively fuse information from different modalities to avoid information loss and bias is also an important topic in multimodal AI research.
Typical Applications of Multimodal AI
Multimodal AI applications are enriching the ways of human-computer interaction, making AI more flexible and intelligent in handling diverse tasks, and bringing brand-new possibilities to industries such as artistic creativity, film and television production, and content creation. Common multimodal AI applications include the following scenarios.
Text-to-Image Generation: Multimodal AI generates corresponding images based on text descriptions input by users. For example, if a user inputs “beach scene at sunset,” the AI generates an image that matches the description. Representative models include DALL-E, Stable Diffusion, and Midjourney, which are widely used in creative design, digital art, and the advertising industry.
Text-to-Video: In this application, AI generates video clips based on text descriptions. For example, by inputting “sunrise in the forest,” the AI can generate a video of the rising sun gradually illuminating the forest. This application is currently in its early stages but holds huge potential in fields such as entertainment, advertising, and education. Imagen Video and Meta's Make-A-Video are leading research models in this field.
Text-to-Speech: This application generates natural language speech output based on text descriptions. Typical applications include virtual assistants (such as Siri, Alexa, etc.) and text-to-speech assistive technologies. When a user inputs a piece of text, the AI generates the corresponding speech, such as using News Anchor to produce broadcasts. This application is common in scenarios like customer service, smart devices, and assistive tools.
Text-to-Music: Multimodal AI generates context-appropriate music based on text descriptions or emotional prompts. Users can input “warm and relaxing background music,” and the AI will generate a corresponding melody. For example, Google's MusicLM generates music clips that match user descriptions by learning the association between audio and text, making it suitable for short videos, podcast background music, and creative music production.
Image-to-Text Description (Image Captioning): AI generates text descriptions or tags based on image content to help users understand the image. For example, on social media platforms, it automatically generates descriptive text for images; in image search, it makes it easier for users to find relevant images through text descriptions. Representative models include OpenAI's CLIP and Google's Imagen.
Image-to-Audio: AI generates corresponding audio effects based on images, such as generating the sound of a thunderstorm when seeing a thunderstorm scene. This technology is highly useful in augmented reality (AR) and virtual reality (VR), making virtual scenes more immersive, and can also be applied to movie special effects and multimedia content production.
Video-to-Text Description (Video Captioning): AI analyzes video content and generates summary descriptions or frame-by-frame text annotations, facilitating content search and understanding. For example, generating automatic subtitles in news and documentaries, or providing detailed audio descriptions for accessible videos. Related models include Meta's VideoCLIP, among others.
Audio-to-Text：This application serves as the foundation for speech recognition systems, where AI converts audio content into text—such as transcribing speeches and meetings in real time. It is widely used in scenarios such as voice assistants, video captioning, and phone customer service, with typical systems including Google's Speech-to-Text and Apple's Siri.
Audio-to-Image：AI generates corresponding images or visual effects based on audio content—such as creating dynamic imagery from music or generating scene graphics from audio cues. This technology is commonly used in music video production and audio visualization, enhancing the expressiveness and artistic impact of sound.
Comprehensive Multimodal Generation：AI combines information from different modalities (such as text, images, and audio) to generate new content or provide compound services. For instance, autonomous driving AI systems generate decisions after processing multimodal data like visual inputs, radar, and sound; virtual assistants combine user-entered text and contextual images to deliver customized responses. These applications are particularly suited for complex scenarios, such as smart security and intelligent customer service.
Note: Due to the vast number of multimodal application domains, this article only focuses on the areas I am most interested in: 'Text-to-Image,' 'Text-to-Video,' 'Text-to-Speech,' and 'Text-to-Music.
Text-to-Image
In the field of multimodal AI text-to-image generation, AI generates corresponding images based on user-provided text descriptions, commonly applied in creative work, design, advertising, and entertainment. In recent years, this field has developed rapidly, giving rise to many representative models such as OpenAI's DALL-E, Stability AI's Stable Diffusion, and Midjourney. A brief overview of this field is as follows:
Primary Models and Technologies
• DALL-E: The DALL-E series of models developed by OpenAI is one of the pioneering projects in this field. DALL-E uses the Transformer model in deep learning, trained on a large amount of image-text pair data, enabling it to generate semantically consistent images based on input descriptions. DALL-E 2 and DALL-E 3 have made significant progress in image generation quality, detail expression, and diversity, with DALL-E 3 further optimizing the understanding and fine-grained control of user text descriptions.
• Stable Diffusion: Stable Diffusion is an open-source system based on diffusion models that generates images using a diffusion process. It can work under various input conditions, such as images, text, etc., and provides users with strong customization capabilities when generating images. The open-source nature of Stable Diffusion has made it widely popular in the creative community, allowing users to customize models and even fine-tune them to generate images of specific styles or themes.
• Midjourney: Midjourney is an AI platform focusing on image generation, primarily providing services through online communities and Discord servers. Midjourney is renowned for its artistic quality and unique style, making it suitable for creative and artistic expression scenarios. The generated images often possess a unique aesthetic effect, making them highly popular among designers and art creators.
Both DALL-E and Stable Diffusion can be called through some mainstream local Large Language Model UIs (such as LobeChat):
For Stable Diffusion, due to its open-source nature, it also supports various local deployment methods.
On the other hand, Midjourney can only be called by default through Discord (Discord is an instant messaging and VoIP social platform that allows communication via voice calls, video calls, text messages, and media). Users must join its Discord community to generate images, and there is no official API or local deployment option. Of course, the power of netizens is infinite: there are open-source projects that implement calls to Midjourney through other UIs via proxies, which I will discuss later.
Technical Principles
• Transformer Model: Models like DALL-E rely on Transformer models to generate images by encoding the semantic relationships between text and images. The Transformer model utilizes a self-attention mechanism to capture contextual information in text descriptions and then maps it to the image space, thereby generating the corresponding images.
• Diffusion Models: Stable Diffusion uses the principles of diffusion models, which step-by-step convert random noise into clear images. By continuously reducing noise, the model gradually draws images in the picture that match the text description. This model performs outstandingly in generating clear, detail-rich images, and the generation process has good stability and flexibility.
Note: The technical details of Midjourney have not been publicly disclosed, so its specific architecture remains unclear. However, judging from its generation effects and some known characteristics, Midjourney may not adopt traditional Transformer models (like DALL-E) or standard diffusion models (like Stable Diffusion). Its generation technology might be based on some adaptive generative model, belonging to an innovative multimodal generation approach that integrates diffusion models, style transfer, and other technologies to achieve unique artistic effects.
Application Scenarios
• Advertising and Marketing: Text-to-image generation shines brightly in the field of creative advertising. Marketers can use these models to quickly generate images that align with the brand's tone, thereby enhancing visual effects and reducing production costs.
• Games and Film & Television: In industries like gaming, film, and television that require a large amount of visual assets, AI-generated images can help designers quickly generate scene concept art or character designs, improving efficiency and exploring multiple visual styles.
• Social Media and Self-Media Creation: Many self-media creators use generation tools like DALL-E and Midjourney to produce unique visual content, enhancing the appeal of social media content.
Related Technology: Image-to-Text
Image-to-text, also known as image captioning, is a technology where AI automatically analyzes the content of an image and generates a text description. It is widely used in scenarios such as image search, automated tagging, and assisting the visually impaired. Representative models include OpenAI's CLIP, Google's Imagen Captioning, and Microsoft's Azure Computer Vision. Image-to-text technology typically utilizes deep neural networks to extract image features, and then generates natural language descriptions that match the image content through a text generation module, making it particularly suitable for scenarios requiring rapid understanding and indexing of images.
Image-to-text (image captioning) and text-to-image (image generation) are closely related in the field of multimodal AI, mainly reflected in the following aspects:
• Shared Multimodal Representation Space: Many advanced models (such as CLIP, DALL-E) represent images and text in the same multimodal space, thereby establishing a semantic correlation between images and text. This correlation enables the model not only to generate image descriptions (image-to-text) but also to generate semantically consistent images based on text. This common multimodal representation is the foundation of both tasks.
• Mutual Support of Training Data: Image-to-text and text-to-image models are typically trained using image data with text labels. Such data helps the model learn the correspondence between images and text, that is, understanding the image content while understanding the text description. This data interoperability allows progress made by the model in one task (such as more accurate image descriptions) to improve performance in the other task (such as generating images that better match the descriptions).
• Complementarity of Application Scenarios: In practical applications, image-to-text and text-to-image are often used in combination. For example, in the e-commerce field, image-to-text can automatically generate descriptions for products, while text-to-image can generate rendering effects or design samples based on product descriptions. This complementarity not only improves content production efficiency but also allows the two models to cooperate with each other, enriching the user experience.
• Bidirectional Enhancement of Understanding and Generation Capabilities: Image-to-text allows AI to better understand image content, while text-to-image teaches AI to conceptualize images from descriptions. The combination of the two helps enhance the model's bidirectional visual-language understanding and generation capabilities, enabling the model to not only generate descriptions through image recognition but also generate visual content based on descriptions, achieving more natural multimodal interaction.
Through these connections, image-to-text and text-to-image form a bidirectional complementary relationship in technology and application, jointly driving the development of multimodal AI.
Text-to-Video
In the ”text-to-video” field of multimodal AI, AI generates short videos that meet requirements based on text descriptions provided by users. Although this technology is still in its developmental stage, it has already shown great potential and received widespread attention in fields such as entertainment, advertising, and education. The main representative models include Meta's Make-A-Video and Google's Imagen Video. Below is a detailed introduction to this field.
Primary Models and Technologies
• Make-A-Video: Make-A-Video, launched by Meta, is currently one of the more advanced text-to-video models. This model has a clear advantage in generating dynamic visual content, capable of creating video clips with smooth transitions and fine details based on input text descriptions. Make-A-Video makes the generated video content more natural by using pre-training based on image-video pair datasets, combined with generative adversarial networks (GANs) and diffusion models.
• Imagen Video: Imagen Video, launched by Google, is a multimodal AI system that generates videos based on text descriptions. Imagen Video adopts a diffusion model architecture similar to Stable Diffusion and combines text encoding technology to generate videos with higher resolution and richer layers. The model can generate multiple scene elements and dynamic changes, making it suitable for generating short video content.
• Luma: Luma Labs focuses on high-quality 3D content generation, utilizing text descriptions to generate realistic 3D videos. This model is particularly suitable for scene generation requiring fine visual details, and is widely used in virtual reality, augmented reality, and film and television production fields to provide highly realistic visual effects.
• Runway: The Gen-2 model launched by Runway supports text-based multimodal video generation, capable of generating short videos from simple text descriptions. Runway excels in generating creative content and short videos, providing high-quality special effects and creative tools, and is widely used in creative content production, advertising, and short video platforms.
• Pika Labs: Pika Labs provides services for rapid video generation, capable of generating content-rich video clips based on short text descriptions. The Pika model has clear advantages in video generation speed and effects, making it suitable for scenarios requiring rapid generation of video content, such as social media creation and e-commerce advertising.
• Kling (Klein): The Kling model is a Chinese multimodal generative model, focusing on text-to-video functionality within the Chinese context. This model can generate realistic video content based on Chinese descriptions, suitable for scenarios such as domestic short video creation, advertising, and educational content generation, particularly fitting the creative needs of Chinese users.
These models each have their own focus in video generation performance, ranging from realistic 3D videos to rapidly generated creative short videos, all of which can bring innovative solutions to multimodal content creation.
Currently, models in the text-to-video field differ significantly in how they are used: some models provide open APIs (or are not public but have been analyzed and successfully simulated by third parties), allowing integration into a unified interface via third-party tools or proxy services for invocation, thereby supporting use in local or customized interfaces. However, more models do not provide public API interfaces and are limited to use on officially provided platforms or dedicated applications. This difference in design causes various models to differ in flexibility and integration: models supporting APIs are suitable for flexible integration in multiple scenarios, while models limited to dedicated interfaces rely on official platforms and are typically suitable for standardized applications.
Of course, the power of netizens remains infinite, which I will discuss later.
Technical Principles
Diffusion Models: Diffusion models are gradually becoming the mainstream choice in the field of video generation, performing particularly well in generating high-quality, coherent video content. The basic principle of diffusion models is to encode video content by gradually adding noise, and then reverse the process to denoise, thereby generating content that matches the description frame by frame. For example,Runway's Gen-2 model,Imagen Video和Pika Labsboth adopt diffusion models, generating high-quality video clips by gradually reducing noise. Due to their step-by-step generation method, diffusion models can outperform traditional generative adversarial networks in maintaining details and stability.
Generative Adversarial Networks (GANs): GANs were once one of the primary technologies for image and video generation, utilizing adversarial training between a generator and a discriminator to generate realistic video content.Make-A-Videomodels may introduce GAN structures into their frameworks, making each generated frame of video both coherent and highly visually realistic. However, the challenge of GANs lies in their tendency to produce unstable effects when generating high-resolution videos, especially since generating long sequences of continuous frames is complex, leading them to be gradually surpassed by diffusion models in the field of pure video generation.
Autoregressive Models: Some models generate video content frame-by-frame or clip-by-clip through autoregressive generation. Autoregressive models rely on information from the previous frame or several previous clips to generate video content sequentially, making them particularly suitable for short videos and specific-themed video generation.Lumamodels may introduce an autoregressive-like generation method in 3D scene generation, giving the video content high consistency and clarity in 3D rendering details.
Multimodal Feature EmbeddingMany video generation models, includingKling(Klein), use multimodal feature embedding technology to map features such as emotion, style, and scene from text descriptions into video features. This method allows the model to generate content that meets cultural and linguistic needs based on the Chinese context, improving the alignment between video generation and text descriptions. Kling focuses on the needs of Chinese users, enhancing the emotional expression and thematic consistency of generated content by optimizing the mapping from Chinese text to video.
Application Scenarios
• Advertising and Marketing: Text-to-video has significant applications in the advertising field, capable of rapidly generating short video clips that match brand tone, reducing advertising production costs, and providing rich visual content creativity.
• Social Media and Content Creation: We-media and short video creators can use these tools to generate short video assets, such as simple scene transitions or background effects, increasing the appeal and diversity of their content.
• Education and Training: In the field of education, text-to-video can help explain abstract concepts, generating explanatory dynamic content that is easy to understand. For example, animations of science experiment demonstrations or physical processes can be generated based on text descriptions, helping students better understand complex concepts.
Related Technology: Video-to-Text
Video-to-text, also known as video description generation, is a technology that automatically analyzes video content and generates text descriptions through AI. It is widely used in scenarios such as video search, automated subtitle generation, and content moderation. Representative models include Google's VideoBERT, Microsoft's Azure Video Indexer, and Meta's VideoCLIP. Video-to-text technology typically utilizes deep neural networks to extract temporal and visual features of videos, and then generates natural language descriptions matching the video content through a text generation module, making it particularly suitable for scenarios requiring rapid video understanding and indexing. Future development directions include multi-level plot descriptions, cross-lingual subtitle generation, and higher semantic accuracy.
Video-to-text (video description generation) and text-to-video (text-driven video generation) have a close correlation in multimodal AI, mainly reflected in the following aspects:
Shared Multimodal Understanding and Generation Capabilities: Both technologies require the model to understand the deep connection between video and text. Video-to-text requires converting visual and temporal information into linguistic descriptions, while text-to-video requires generating visual and temporal information that matches the content based on text descriptions. This bidirectional conversion relies on the multimodal model's general understanding of video and text, enabling the model to both parse content and generate semantically appropriate expressions.
Bidirectional Support of Datasets: Video-to-text and text-to-video typically use similar datasets, especially video datasets with text descriptions. These datasets contain paired relationships between videos and their text descriptions, enabling the model to master the mapping relationship from text to video while learning to generate descriptions, thereby promoting the mutual development of both tasks.
Mutual Enhancement of Application Scenarios: In many application scenarios, video-to-text and text-to-video can be used in combination. For example, in content creation, users can quickly obtain initial footage through text-to-video, and then use the automatic descriptions generated by video-to-text for content checking and refinement. In addition, in fields such as education and news, text-to-video can generate visual content, while video-to-text can generate subtitles and descriptions to facilitate content dissemination and understanding.
Complementarity of Cross-Domain Applications: Video-to-text can help improve the accuracy of text-to-video. For example, when generating videos with complex plots, the descriptions generated by the video-to-text module can be used to automatically proofread or optimize the generated video, ensuring consistency between the text and the video content.
Through these connections, video-to-text and text-to-video promote each other, forming a complementary relationship in technology and application, enabling multimodal AI to more comprehensively support complex video-text interaction needs.
Text-to-Speech (TTS)
In the ”text-to-speech” field of multimodal AI, AI generates natural and fluent speech based on text input by the user. This technology has been widely applied in fields such as voice assistants, customer service systems, and education, capable of generating speech content with different timbres, speaking rates, and emotions according to needs. Representative models include Google's Tacotron, OpenAI's Whisper, and Microsoft's Azure TTS. The following is a detailed introduction to this field, along with a brief introduction to “speech-to-text” technology.
Primary Models and Technologies
• Tacotron: Tacotron, introduced by Google, is an early end-to-end text-to-speech model. Tacotron can convert text into natural speech waveforms and can even synthesize specific tones and intonations. Tacotron 2 further optimized sound quality and stability, laying the foundation for subsequent TTS models.
• Azure Text-to-Speech (TTS): Microsoft's Azure TTS provides flexible text-to-speech services, supporting synthetic speech output in multiple languages, accents, and emotions. Users can adjust the voice style to suit the needs of different industries, such as intelligent customer service and educational broadcasting scenarios.
Note: TTS is the most commonly used form of text-to-speech, and it can be called directly within local LLM UIs.
Technical Principles
• Sequence-to-Sequence (Seq2Seq) Models: Tacotron and other text-to-speech models are typically based on the Seq2Seq model architecture, encoding text into speech features and then decoding them into audio waveforms. Such models utilize deep neural networks to learn the correspondence between text and speech.
• Waveform Generation Technology: Tacotron 2, Whisper, and Azure TTS use advanced waveform generators (such as WaveNet or WaveGlow) to produce more natural and clear speech. These waveform generators use high-frequency sampling and filtering to make speech synthesis closer to human voices.
Application Scenarios
• Virtual Assistants and Smart Devices: Text-to-speech is widely used in virtual assistants, such as Alexa and Siri. AI can generate natural speech based on user commands, enabling smart devices with interactive capabilities.
• Telephone Customer Service and Service Robots: Many enterprises use AI voice in customer service scenarios, improving service efficiency by generating natural speech. The generated speech can simulate human accents and intonations, improving customer experience.
• Education and Media: Text-to-speech is used in the education field for automated broadcasting and explanations, suitable for language learning, e-book reading, and course audio production. The media field also widely uses speech generation technology to automatically generate news broadcasts or audio content.
Related Technology: Speech-to-Text (ASR)
Speech-to-text is a technology that transcribes speech content in audio into text, widely used in scenarios such as voice assistants, subtitle generation, and telephone customer service. Representative models include Google Speech-to-Text, Whisper, and Microsoft's Azure ASR. ASR technology typically uses deep neural networks to analyze and recognize audio features, accurately transcribing speech content into text, making it particularly suitable for voice input scenarios that require written records. Future development directions include support for multiple languages and improved adaptability to noisy environments.
Note: Whisper is a model released by OpenAI, focusing on the ”speech-to-text” (ASR) task. Trained on a large dataset of diverse languages and contexts, it possesses high speech recognition capabilities, making it especially suitable for scenarios requiring accurate semantic expression, such as language learning or broadcasting. It is important to note that Whisper does not generate speech; rather, it is a core technology for speech recognition.
In many voice interaction scenarios, Whisper and TTS are often used in combination: Whisper is responsible for speech recognition, converting the user's voice into text, the AI model understands the user's needs based on the text content, and then the TTS model converts the response content into voice output. This workflow achieves a complete closed loop from voice to text and back to voice, suitable for applications such as voice assistants and customer service systems, taking LobeChat's ”Voice Service” as an example:
Speech-to-text (ASR) and text-to-speech (TTS) are highly correlated in technology and application, together forming a ”two-way speech-text conversion system”:
• Building a complete closed loop of voice interaction: ASR and TTS complement each other, providing a complete interactive experience for applications such as voice assistants, customer service systems, and smart devices. The user inputs voice, the system converts it to text via ASR, understands the intent, and then synthesizes a natural voice reply via TTS, thereby achieving full-duplex voice communication. Smart assistants like Siri and Alexa rely on this interactive closed loop, allowing users to complete operations without text input.
• Sharing speech and text features and training data: ASR and TTS technologies typically use similar speech-text paired data for training. In multimodal AI models, the mapping relationship between speech and text is key. Through the same or similar datasets, ASR can generate accurate text from speech, while TTS uses this text to generate appropriate speech. The use of bidirectional training data improves the accuracy and consistency of the models.
• Improving the naturalness and intelligence of human-computer interaction: The combination of ASR and TTS makes the human-computer interaction experience more natural and coherent. For example, in customer service bots, a user's voice request can be accurately transcribed and understood, and then a natural voice reply is provided through TTS, making the interaction closer to human conversation. In the future, with the development of multimodal AI models, the integration of ASR and TTS will allow devices to perceive and understand users' emotions and intentions more intelligently.
• Promoting cross-lingual and multimodal translation and transcription: In cross-lingual applications, ASR and TTS technologies can be combined to generate speech translation systems. For example, after a segment of speech is transcribed into text using ASR, the text can be translated into another language through a translation module, and then TTS is used to generate speech output in the target language. This workflow supports cross-lingual communication and can also help hearing-impaired individuals communicate more easily across different languages.
By integrating speech-to-text and text-to-speech into multimodal AI systems, it not only expands the application scenarios of both technologies but also builds a smarter and more flexible bidirectional interactive experience, providing more humanized services for fields such as voice assistants, translation, and customer service.
Text-to-Music
In the field of ”text-to-music” in multimodal AI, AI can generate music that matches specific emotions and styles based on text inputs or descriptions provided by users. This technology is applied in scenarios such as music creation, game sound effects, and movie soundtracks, providing personalized music generation for different occasions. Representative models include OpenAI's Jukebox, Google's MusicLM, and AIVA. Below is a detailed introduction to this field.
Primary Models and Technologies
• Jukebox: The Jukebox model, introduced by OpenAI, is a neural network-based music generation tool capable of generating music in various styles, including pop, rock, and jazz, based on text descriptions. Jukebox can generate songs with lyrics and even mimic the style of specific artists.
• MusicLM: Google's MusicLM model can generate highly complex, emotion-matching music based on text descriptions. It can generate music clips of different lengths and offers a high level of control over music style and emotional expression, making it suitable for diverse creative needs.
• Bark: Developed by Suno, the Bark model is a generative audio tool capable of generating various types of audio, including music, vocals, and ambient sound effects, based on text descriptions. Bark can generate music clips with emotion and style, supports multilingual speech generation, and is suitable for creating sound effects and rich music content for different scenarios.
• AIVA: AIVA is an AI model primarily used for generating background music and soundtracks, capable of generating classical and light music based on input descriptions of emotion and style. AIVA is widely used for generating movie soundtracks and game music.
Note: In the field of text-to-music generation, OpenAI's Jukebox and Google's MusicLM are still experimental and not yet commercialized, while Bark and AIVA are among the few music generation AI platforms that have already been commercialized.AIVA It is a professional model focusing on commercial music generation, suitable for commercial music projects such as advertisements, film and television soundtracks, etc.Bark is more geared towards diverse audio generation. Although it can also be used in commercial scenarios, its application in the field of professional music creation is relatively limited, and it is mainly used as a sound effects and voice generation tool.
Technical Principles
• Sequence-to-Sequence (Seq2Seq) Models: Most text-to-music models adopt the Seq2Seq architecture, encoding text descriptions into feature vectors and then decoding them into corresponding audio sequences. The model learns the relationships between features such as emotion and style using a large amount of text and music data, thereby converting text content into music.
• Autoregressive generation models: Jukebox and MusicLM adopt autoregressive generation methods, generating music clips by progressively predicting the spectrum or waveform of the music. This method can control the continuity and details of the music more delicately, making the generated music more in line with human hearing habits.
• Emotional Feature and Music Mapping: To achieve consistency between music and emotion, the model generates music with corresponding emotional expression by identifying the emotion of the input text. The model uses emotion classifiers or style classifiers to help map text to appropriate music elements, thereby generating music content that fits the theme.
Application Scenarios
• Film and Game Scoring: Text-to-music is widely used in background music generation for films and games. Directors and developers can generate music that fits the context through simple text descriptions, greatly shortening the creation time of scoring.
• Personalized Music Creation: AI-generated personalized music can be used for social media and short video platforms. Creators can generate exclusive background music based on text descriptions to enhance the appeal of their content.
• Healing and Meditation Music: In the fields of health and psychological healing, text-to-music can generate soothing music based on the needs of meditation and healing scenarios, helping users relax and meditate. Users input specific emotions or theme descriptions, and the system generates matching music, helping to improve therapeutic effects.
• Real-time Sound Effect Generation: In interactive applications, text-to-music can be used to generate music with specific emotions based on real-time inputs. For example, in virtual reality (VR) or augmented reality (AR) experiences, AI can generate dynamic music based on the user's current emotions or scene requirements, enhancing the sense of immersion.
The future development directions of text-to-music include high control over specific styles and details, making AI-generated music better meet the needs of creators, and supporting multi-language descriptions to further expand its application scenarios.
Multimodal AI UI Tool: chatgpt-web-midjourney-proxy
Frustrations of Using Text Generation Tools
Among the various text-to-X multimodal needs mentioned earlier in the article (text-to-image, text-to-video, text-to-speech, text-to-music, and actually including text-to-code, though this is not considered a multimodal need, so I didn't mention it), the biggest problem actually lies in how to use them conveniently.
Among these multimodal needs, some can be completed through our commonly used local large model UIs (such as Lobechat), such as the text-to-image (OpenAI's dall-e-3 model) and text-to-speech (Azure TTS) mentioned earlier in the article. However, others can only be accessed through dedicated methods: for example, when generating images from text, if dall-e-3 and Stable Diffusion cannot meet the needs, Midjourney must be used; or for text-to-video, when models like Luma, Runway, or Kling are needed, they must be used in the way officially required. Using a single model is fine, but if you want to use multiple models, you might end up jumping all over the place, here one minute and there the next, which is a real headache to think about.
So, is it possible to use these generative models in the same UI interface? Yes, there actually is, and that is ”chatgpt-web-midjourney-proxy”.
Introduction to chatgpt-web-midjourney-proxy
The purpose of this project is to integrate ChatGPT with Midjourney to achieve the text-to-image function (project address:github.com/Dooy/chatgpt-web-midjourney-proxy). It mainly serves as a proxy interface, allowing users to access multiple API services through a unified UI, including OpenAI's ChatGPT, Midjourney's image generation, and other models (such as Suno's audio processing and Runway's video generation). In this way, users can create a customized frontend for ChatGPT while supporting multiple backends, such as Midjourney for image generation, and other APIs for multimodal tasks (such as audio-to-audio, image-to-video, and text-to-video):
It supports multiple text-to-image models such as MidJourney, dall-e, and IdeoGram. Among them, MidJourney supports face swapping and image blending in addition to drawing:
The project provides flexible deployment methods, including Docker and Serverless options, suitable for personal or server environments. In addition, it supports Vercel deployment, making it convenient for users to perform cloud configuration.
Deployment of chatgpt-web-midjourney-proxy
Docker Deployment
The docker run command format is as follows:
docker run --name chatgpt-web-midjourney-proxy -d --restart=always \
-p 6015:3002 \
ydlhero/chatgpt-web-midjourney-proxyIn the above command, you can actually directly use parameters-eto specify the API address and key of the multimodal service provider (or a self-built proxy for a specific service provider) (for example, by adding the following code to point to the MJ proxy address:
-e MJ_SERVER=https://your-mj-server:6013 \
-e MJ_API_SECRET=your-mj-api-secret \If you want to build your own MJ_SERVER proxy, you can use the following docker run format command to create it:
docker run --name mj6013 -d --restart=always \
-p 6013:8080 \
-e mj.discord.guild-id=discord服务ID \
-e mj.discord.channel-id=discord服务组ID \
-e mj.queue.timeout-minutes=6 \
-e mj.api-secret=abc123456 \
-e mj.discord.user-token=********** \
novicezk/midjourney-proxy:2.5.5Similarly:
-e LUMA_SERVER=`https://your-luma-server:8000` \
-e LUMA_KEY=your-luma-key \
-e SUNO_SERVER=`https://your-suno-server:8000` \
-e SUNO_KEY=you-suno-key \Note: LUMA_SERVER and SUNO_SERVER proxies also have their own setup methods, which I won't go into detail here. In fact, you don't necessarily have to complete the configuration of these environmental parameters when creating the Docker container; you can also log in to the UI interface to set them after the setup is complete.
After the Docker deployment is complete, directly usehttp://宿主机IP:6015to access it.
One-Click Deployment on Vercel
For friends using Vercel, you can directly click the following link for one-click deployment:One-Click Deployment on Vercel, and the relevant environment variables are explained as follows:
Serverless Personal Desktop Version Installation
This method uses local installation instead of deploying via Docker or Vercel (download link:Serverless Personal Desktop Version Download), and the current latest version is ”v2.21.9":
How to Use chatgpt-web-midjourney-proxy
Prerequisite Knowledge: Third-Party API Provider: OpenAi-HK
In the previous article introducing the use of Lobechat (see:Home Data Center Series: Unlocking the Full Potential of LobeChat: A Complete Guide from Setup to Practical Application), I mentioned a highly cost-effective third-party API provider: OhMyGPT. If you only look at the needs of Chat-type and text-to-code generation, OhMyGPT is completely sufficient.
However, when it comes to text-to-image, OhMyGPT only supporting OpenAI's dall-e-3 seems much weaker, let alone needs like text-to-video and text-to-music. Therefore, if you have a high demand for text-to-image and text-to-video functions, OhMyGPT is not very suitable. At this time, OpenAi-HK, a third-party API provider that simultaneously supports GPT4.0, dall-e-3, GPTs multimodal, Claude, drawing with Midjourney Flex Ideogram, music with Suno, video with Luma Runway Kling, and dancing with Viggle, becomes a very good choice (openai-hk official website address)：
As can be seen from the two figures above, OpenAi-HK indeed has a great advantage. Not only does it support many types of multimodal models, but the key is that it does not require VPN/proxy access, has a low top-up threshold, and has no time limit on use—you can use it until it runs out, which is very user-friendly.
After topping up, record your API key in the API KEY section of the console, and choose an API address that is the fastest for your network speed:
Note: For the screenshots in this article, I went ahead and topped up a huge sum of 10 RMB without hesitation~.
Hands-on Experience with chatgpt-web-midjourney-proxy
Accessing and Configuring chatgpt-web-midjourney-proxy
I deployed it using Docker, so I usehttp://宿主机IP:6015to access it:
Enter the settings interface from the bottom left:
“The ”Overview" section allows you to set the link and name of the avatar image below, as well as the language and background image of the workspace:
“The ”Model" section allows you to set the models and parameters for Chat-type communication. Generally, choosing gpt-4o-mini is fine. The context count, response count, and role settings can be configured as needed (normally I use Lobechat for Chat-type needs, so this part of the settings is not very useful to me):
“The ”Server" settings are the most important, as they are the key to successfully completing multimodal text generation requirements:
Once these settings are completed, you can officially use chatgpt-web-midjourney-proxy to start text-generation-based multimodal operations.
Text-to-Image
In ”chatgpt-web-midjourney-proxy”, the text-to-image models include Midjourney, dall-e-3, and IdeoGram.
DALL-E and Ideogram are AI models for generating images, but their design focus and application scenarios are different.
Developed by OpenAI, DALL-E can generate images in various styles based on text descriptions, making it suitable for diverse needs ranging from real-world scenes to abstract art. However, DALL-E has certain limitations when generating specific, concrete characters (such as anime characters), primarily due to copyright and intellectual property considerations. To avoid being too similar to specific copyrighted characters, DALL-E tends to be conservative when generating such content, which may result in images that are slightly blurry in detail or do not fully meet specific visual requirements.
Ideogram, on the other hand, is an AI tool that focuses more on the integration of text and images, making it highly suitable for generating image content that contains text, such as text art, advertising design, or brand logos. Its advantage lies in processing the fusion effect of images and text, so it performs better when there is a need to generate images containing clear text.
In contrast, Midjourney offers greater freedom in generating character images of specific styles, especially anime or fictional characters with fantasy or personalized artistic styles. Midjourney can usually provide more stylized and detail-rich images, aligning better with users' visual expectations in character creation.
As can be seen from the image below, choosing Midjourney offers a wealth of options:
In comparison, the options for DALL-E and Ideogram appear very rudimentary:
Ideogram:
Below, taking Midjourney as an example, we will generate an ”image of Android 18 wearing sexy clothes”.
First, find a template photo of Android 18, taking the image below as an example:
Then, generate the image according to the following parameters:
By default, 4 images will be generated. Click the U1, U2, U3, and U4 buttons below to select your favorite one for individual generation.
Then use the same parameters to generate an image of Bulma (using her appearance in the first episode of Dragon Ball as a template):
The generated image this time is as follows:
Overall, this operation of generating based on a template image is simple, and the actual results are quite satisfying. But what if I want to draw an image of Android 18 transforming into Super Saiyan 3? How do I tell the AI what Super Saiyan 3 is? It's a headache, I'll research it more when I have time later~.
Note 1: There are other parameters and options on the interface, such as ”upload image prompt”, ”sref”, etc., which I haven't had time to research yet. If you are interested, you can try them out.
Note 2: Midjourney can also swap faces and blend images (blending two images together). Taking face swapping as an example, if we swap the faces of my goddess Vivian Chow and the slightly chubby goddess Hong Jin-young, it looks like this:
Then:
The final result is okay, which can be called ”Hong Hui-min”:
In addition, the previously generated images are all in the ”Gallery” option, as shown in the red box below:
Note: It feels like sometimes submitting requests to ”generate images” fails. In this case, usually submitting it 2 more times works. I'm not sure if it's caused by some rate limit between OpenAI-HK and Midjourney.
Text-to-Music
This part of the function is mainly implemented through Suno's Bark model (fixed, cannot be customized). There are 2 modes: one is the description mode, which relies purely on text:
The other is the custom mode, where you can upload music samples:
The final generated music:
You can listen to it, it's actually not bad:快乐的星期天.mp3。
Text-to-Video
There are 4 models by default: Luma, Runway, Pika, and Kling. I will take Kling as an example to generate a video:
I feel the generated video is not that great, a bit stiff and a bit fake. I don't know if it's because I haven't mastered the prompting skills. I won't upload the video, you can try it yourself if you are interested.
Dance
This is actually similar to face swapping, except here a character photo is used to completely replace the person in the video template:
I'm too lazy to upload the video, so let's just look at a screenshot. A 16-second video was generated:
It feels very fake~.
Summary
Finally, there is a functional module called ”Real-time Voice Conversation”, which is actually a combined application of Whisper and TTS. This can also be achieved in LobeChat, so I won't bother with it.
Overall, the most practical feature in chatgpt-web-midjourney-proxy is generating images from text via Midjourney (otherwise, normal use of MJ would require messing around with Discord). As for other features like text-to-music, text-to-video, and dance, I personally feel that the functions provided in chatgpt-web-midjourney-proxy are too simple and can only be used for fun. If you really need to use them, you should honestly use them through the officially recommended methods of professional service providers.