Post

IUI-Instruction

This is our application IU.#

image

Video Instruction can access here: #

https://drive.google.com/file/d/1_GfEOhjjxeM8MI_nC4MIFwCVLz8iqiKW/view?usp=sharing

0. Access to the website: http://168.131.151.158:3000#

0. Access to the video material in your native language: https://drive.google.com/drive/folders/1xYVlZPwQmnSgni6dj4FXUXYfjNHxGzmC (for example, if you speak Chinese, open the Chinese folder) #

📹 STEP 1: UPLOAD VIDEO#

  1. Click "Add Media" button and choose your video [Please wait a little bit, it can take some time]
  2. Video will appear on Video Track in the timeline
  3. Play test to verify the video loaded correctly

Note: After finishing a task, sometimes the audio may not play. If that happens, please click the “Refresh” button.

🤖 STEP 2: VIDEO ANALYSIS#

Features:

  • Separate audio into Vocal (speech) and Background (music)
  • Transcribe speech to text
  • Generate automatic subtitles

How to use:

  1. Select the video element on timeline
  2. In the Properties Panel on the right, click "Run Analysis"
  3. Wait for processing (may take several minutes)
  4. After completion, you will see:
  • Vocal Track (speech audio)
  • Background Audio Track (background music)
  • Script Track (original subtitles)

Results:

  • The original video will be muted
  • Vocal and Background can play separately
  • Subtitles display synchronized with the video

🌍 STEP 3: SUBTITLE TRANSLATION#

Features:

  • Translate subtitles into different languages
  • Maintain original timing and structure

How to use:

  1. After obtaining Script Track from Analysis
  2. In Properties Panel, select:
  • Target Language (English)
  • Tone (e.g., Easy to understand, Educational, Casual, you can type any tone you want)
  1. Click "Translate"
  2. Wait for processing

Results:

  • New Script Track with translated language
  • Original script will be hidden (muted)
  • New subtitles display on video

🎤 STEP 4: TEXT-TO-SPEECH CONVERSION#

Features:

  • Convert translated text to new speech
  • Use AI to generate a natural voice
  • Synchronize with original timing

How to use:

  1. After obtaining the translated script
  2. In Properties Panel:
  • Select the Script Track that was translated
  • Enable/disable Multi Speaker (if multiple speakers)
  1. Click "Convert Translated Text to Audio"
  2. Wait for processing (may take a long time)
  3. Watch the video (You need to compare this video with the video from step 5 if it is a lipsync video)

Results:

  • Converted Audio Track with new voice
  • Original Vocal Track will be muted
  • New audio synchronized with video

💋 STEP 5: LIP SYNCHRONIZATION (applicable only to video with the “lip” mark)#

Features:

  • Synchronize lip movement with new voice
  • Create video with lips matching converted audio

How to use:

  1. After obtaining the Converted Audio
  2. In Properties Panel:
  • Select Audio Media (Converted Audio)
  1. Click "Run Lipsync"
  2. Wait for processing (may take a very long time)

Results:

  • Lipsynced Video Track
  • The original video will be muted
  • New video with lips synchronized to the converted voice

🏁 Last Step:#

Please complete the survey here:

https://docs.google.com/forms/d/e/1FAIpQLSf_cAjIQG_C0YnsMR9vHwIfdrsXkFPs47IIts4TfnomaZPMfw/viewform?usp=dialog

🙏🏻
Thanks for participating

@_sowwn

Students

https://forms.gle/ZaB1cqs7N5xtNi4r5

Teachers

https://forms.gle/FnS1L3k76hngkhev7