Karaoke generator

Karaoke Environment Manual

Karaoke Generator Banner

Welcome to the Karaoke Environment. This tool suite allows you to download videos from YouTube, separate vocals from instrumental music using AI models (Spleeter), generate timed lyrics/subtitles using speech recognition (Whisper), convert Traditional Chinese characters to Simplified Chinese (OpenCC), and render ready-to-sing Karaoke video files with burned-in subtitles.

📁 Environment Directory Structure

  • karaoke-setup.sh: Main setup script to initialize environment structure, Python virtual environment, and executables.
  • .venv/: Python Virtual Environment containing all required AI models, libraries (yt-dlp, whisper_timestamped, spleeter, opencc, tf_keras, torch), and CLI tools in .venv/bin/.
  • songs/: Output directory storing intermediate assets (downloaded .mp4/.wav, separated vocals, generated .srt files).
  • result/: Final Karaoke video outputs:
    • result/instrumental/: Pure backing tracks + original video with subtitled lyrics.
    • result/quarter-vocal/: Backing tracks + 25% low-volume original vocals + original video with subtitled lyrics.
  • mastered/: Location for finalized songs.
  • pretrained_models/: Cache directory for pretrained stem separation and speech recognition models.

🛠️ Setup & Initialization

To initialize or reproduce this environment, run:

./karaoke-setup.sh

This script will:

  1. Create all required directory structures (songs, result/instrumental, result/quarter-vocal, mastered, pretrained_models, resource, OpenCC, .venv/bin).
  2. Verify required system binaries (ffmpeg, ffprobe, opencc).
  3. Create a clean Python Virtual Environment (.venv) with prompt (karaoke).
  4. Install all Python dependencies (yt-dlp, whisper-timestamped, spleeter, opencc-python-reimplemented, tf_keras, h2, etc.).
  5. Generate CLI helper scripts directly into .venv/bin/ so they are on $PATH when .venv is activated.

🚀 Quick Start Guide

1. Generating a Karaoke Video from YouTube

Activate the environment and run Create-karaoke.sh with a YouTube link (song title parameter is optional; artist and song title will be auto-detected):

source .venv/bin/activate
Create-karaoke.sh "YOUTUBE_LINK" [SONG_NAME]

Examples:

# Auto-detect song title & artist from YouTube metadata:
Create-karaoke.sh "https://www.youtube.com/watch?v=dQw4w9WgXcQ"

# Or specify custom song title explicitly:
Create-karaoke.sh "https://www.youtube.com/watch?v=dQw4w9WgXcQ" "My Favorite Song"

Pipeline Steps:

  1. Metadata Extraction: Fetches title, artist, and track metadata from YouTube via yt-dlp.
  2. Video & Audio Download: Downloads original video (.mp4) and extracts high-quality audio (.wav).
  3. Audio Separation: Uses spleeter (2stems model) to split audio into vocals and accompaniment.
  4. Lyrics Generation: Uses whisper_timestamped (medium model with PyTorch GPU acceleration) to produce word-level aligned .srt subtitles.
  5. Chinese Conversion: Converts Traditional Chinese subtitles to Simplified Chinese using opencc.
  6. Video Rendering: Uses ffmpeg (with NVIDIA NVENC GPU hardware acceleration when available) to produce two MP4 files in result/:
    • result/instrumental/<Artist> - <Song Title>.mp4: Pure backing track with original video and subtitled lyrics.
    • result/quarter-vocal/<Artist> - <Song Title>.mp4: Backing track + 25% low-volume original vocals with original video and subtitled lyrics.

2. Fixing & Aligning Lyrics (fix_lyrics.py)

If Whisper misinterprets spoken words or introduces credit headers, use fix_lyrics.py to replace subtitle text with official lyrics while retaining Whisper’s precise timing timestamps.

source .venv/bin/activate
fix_lyrics.py <path_to_srt_file> <path_to_lyrics_txt>

Example:

fix_lyrics.py songs/My_Song/My_Song.wav.srt official_lyrics.txt

Output: Generates songs/My_Song/My_Song.wav_fixed.srt.

3. Trimming Video Tails (cutail.sh)

If the video output has trailing silence or extra padding at the end, use cutail.sh to trim a specified number of seconds off the end.

source .venv/bin/activate
cutail.sh <video_file.mp4> <seconds_to_cut>

Example:

cutail.sh "result/instrumental/My Favorite Song.mp4" 10

Output: Generates result/instrumental/My Favorite Song_cut.mp4.

🔄 Environment Upgrade & Maintenance

To update dependencies (such as yt-dlp for YouTube changes):

source .venv/bin/activate
upgrade.sh

🛠️ System Requirements & Dependencies

  • Python 3 / Virtualenv: Managed in .venv/
  • ffmpeg & ffprobe: Required for audio processing, video composition, and NVENC GPU encoding.
  • opencc: (Optional, recommended) Open Chinese Convert CLI tool for character set conversion.