i4Ds/whisper-finetune
i4Ds/whisper-finetune has 27 GitHub stars and 6 forks. GitHot measured 0 stars in 24 hours and 0 in 7 days. Last checked 2026-11-21 09:44 UTC. Last push 2026-09-17.
Measured star growth
- in 24h
- 0
- in 7 days
- 0
- in 15 days
- Collecting history
- in 1 month
- Collecting history
Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.
About
This repository contains code for fine-tuning the Whisper speech-to-text model.
Developer ToolsREADME.md
OriginalWhisper-Finetune
This repository contains code for fine-tuning Whisper speech-to-text models. It supports:
- Multi-dataset validation with macro averages
- WER, CER, NLL, log-probability, entropy, and calibration metrics
- Single-GPU and PyTorch DDP training
- LoRA training and LoRA checkpoint merging
- Timestamp training
- Prompt training
- Stochastic depth, SpecAugment, gradient checkpointing, and mixed precision
- W&B logging and local checkpointing
Installation
Clone the repository:
git clone https://github.com/i4ds/whisper-finetune.git cd whisper-finetuneCreate and activate a virtual environment (strongly recommended) with Python 3.11 or higher.
Install the package in editable mode:
pip install -e .Or using UV (very strongly recommended):
uv pip install -e .
Data
Please have a look at https://github.com/i4Ds/whisper-prep. The data is passed as a 🤗 Datasets to the script.
Timestamp handling
Training records may include Whisper timestamp tokens in the transcript, for example:
<|0.00|> Some text.<|3.64|><|3.66|> More text.<|7.78|>
The data loader behaves differently depending on whether no-timestamp training is enabled:
- With
no_timestamps=True, timestamp tokens are removed from the decoder labels and the<|notimestamps|>special token is added. - With
no_timestamps=False, timestamp tokens are kept in the decoder labels and the model is trained to predict them.
The two config keys interact, and the boolean wins:
no_timestamps = self.no_timestamp_training or torch.rand(1).item() < self.no_timestamps_rate
With no_timestamp_training: True, every sample is trained with <|notimestamps|> and
no_timestamp_rate has no effect. To train on a mixture, set no_timestamp_training: False
and use no_timestamp_rate alone — 0.5 means half the samples get <|notimestamps|>.
This matters for deployment. A model trained only without timestamps cannot predict them, and
Whisper's long-form decoding advances its 30 second window using exactly those predictions. Such
a model can score well on short clips and then emit full-width 30 second segments on long audio,
making the segment timings unusable for subtitles or for anything that consumes segment
boundaries. The transcript text itself can stay largely intact, which is what makes this easy to
miss: WER does not move, and no text is obviously absent - only the timings are wrong. Several configs under configs/ set no_timestamp_training: True
together with no_timestamp_rate: 0.5; that combination trains with no timestamps at all.
This mirrors the upstream behaviour in
jumon/whisper-finetuning, which this repository
started from. There the same short-circuit exists, but it is spelled out in the CLI help for
--no-timestamps-rate: "How often to use the no-timestamps mode. Only used if
--no-timestamps-training is NOT set." The note is worth keeping now that the two settings
live together in YAML, where both are easy to set at once.
Audio is normally padded or trimmed to Whisper's 30 second window. There is one special case in no-timestamp training: if the transcript ends with two consecutive timestamp tokens, such as:
... final text.<|24.28|><|24.94|>
the loader treats the last timestamp as the start of a following partial segment. In that case, with no_timestamps=True, the mel spectrogram is cut at 24.94 seconds and then padded back to the 30 second training window so the remaining time is represented as silence.
If the transcript ends with only one timestamp token, such as:
... final text.<|25.58|>
this special cut is not applied. With no_timestamps=True, the timestamp is still removed from the labels, but the audio follows the normal 30 second padding/trimming path.
Usage
Create a configuration file (see
configs/example_config.yamlfor a fully documented example)Run the fine-tuning script:
python src/whisper_finetune/scripts/finetune.py --config configs/example_config.yamlOn SLURM, use the provided batch script:
sbatch sc_sbatch.sh configs/example_config.yamlFor DDP, request multiple GPUs.
sc_sbatch.shdetects the allocated GPUs throughCUDA_VISIBLE_DEVICESand launchestorchrunwith one process per GPU:sbatch --gres=gpu:4 --cpus-per-task=32 sc_sbatch.sh configs/config_large_v3_best_muon_ddp4.yamlDDP uses
DistributedSampler, rank-0-only logging/evaluation/checkpointing, andmodel.no_sync()during gradient accumulation.training.accum_grad_stepsis the global accumulation window and must be divisible byWORLD_SIZE; withaccum_grad_steps: 8and 4 GPUs each rank uses 2 local accumulation steps. The effective batch is:batch_size * configured_accum_grad_stepsso it stays comparable between single-GPU and DDP runs.
(Optional) Merge LoRA weights into a standard Whisper checkpoint (saved via
save_model):python src/whisper_finetune/scripts/merge_lora_weights.py \ --input /path/to/best_model.pt \ --config configs/config_lora_only.yaml \ --output /path/to/last_model_merged.pt
Testing
Run the test suite:
pip install -e ".[dev]"
pytest
See tests/README.md for more details.
Deployment
We suggest using faster-whisper. To convert your fine-tuned model, use src/whisper_finetune/scripts/convert_c2t.py.
Further quality improvements may be possible by serving requests with whisperx.
Configuration
Modify the YAML files in the configs/ directory to customize your fine-tuning process. Refer to the existing configuration files for examples of available options.
Thank you
The starting point of this repository was the excellent repository by Jumon at https://github.com/jumon/whisper-finetuning
Contributing
We welcome contributions! Please feel free to submit a Pull Request.
Support
If you encounter any problems, please file an issue along with a detailed description.
Maintainer
- Vincenzo Timmel ([email protected])
Developers
- Vincenzo Timmel ([email protected])
- Claudio Paonessa ([email protected])
License
This project is licensed under the MIT License - see the LICENSE file for details.