Apply Now
top-right-arrow
Menu
Close

FINE-TUNED WHISPER TRANSCRIPTION

Rashika Maharjan, Prapti Pradhan, Arya Shrestha

This project is an AI-powered lecture analysis system that converts classroom audio recordings into text and generates meaningful summaries using Artificial Intelligence. It uses a fine-tuned Whisper-small speech recognition model for automatic transcription and a locally hosted Large Language Model (LLM) through Ollama to generate lecture summaries, extract key discussion points, and analyze syllabus topic coverage.

Developed using Python, Flask, Whisper, LoRA, CTranslate2, and Ollama, the system allows users to upload lecture recordings and syllabus documents to receive structured learning materials. It operates locally without sending data to external cloud services, providing a privacy-focused solution for students and educators.

 

About the Project

Fine-Tuned Whisper Transcription with Syllabus-Matched Summarization is an AI-based educational tool designed to simplify lecture reviewing and learning. The system converts recorded classroom lectures into text using Automatic Speech Recognition (ASR) and analyzes the transcript using a locally hosted Large Language Model.

It generates concise lecture summaries, identifies important discussion points, and compares lecture content with uploaded course syllabuses to determine which topics have been covered, partially covered, or not covered.

Objectives

  • Develop an automatic lecture transcription system using Whisper.
  • Fine-tune Whisper-small using LoRA.
  • Convert the model into CTranslate2 format for efficient inference.
  • Generate lecture summaries and key discussion points.
  • Match lecture content with course syllabus topics.
  • Provide a locally hosted and privacy-focused web application.

Key Features

  • Automatic speech-to-text transcription.
  • Fine-tuned Whisper model integration.
  • Lecture summarization using a local LLM.
  • Automatic extraction of key discussion points.
  • Syllabus-based topic coverage analysis.
  • Support for audio and syllabus file uploads.
  • Local processing without external cloud APIs.
  • Web-based interface for displaying results.

Technologies Used

  • Programming Language: Python
  • Speech Recognition: OpenAI Whisper-small
  • Fine-Tuning: LoRA / PEFT
  • Inference Optimization: CTranslate2, faster-whisper
  • Large Language Model: Qwen 2.5 (1.5B) via Ollama
  • Backend: Flask
  • Frontend: HTML5, CSS3, JavaScript
  • Libraries: Hugging Face Transformers, Accelerate
  • Development Environment: VS Code, Google Colab

Applications

  • Educational Institutions
  • Universities and Colleges
  • Online Learning Platforms
  • Lecture Recording and Review Systems
  • Self-Paced Learning
  • Academic Syllabus Monitoring

Expected Outcome

The project provides an intelligent lecture review platform that transforms lengthy audio recordings into searchable transcripts, concise summaries, important discussion points, and syllabus coverage reports. It helps students revise lectures efficiently and enables instructors to monitor syllabus coverage while maintaining data privacy through local processing.