SPARSEFIT: Few-shot Prompting with Sparse Fine-tuning for Jointly Generating Predictions and Natural Language Explanations

This work introduces SparseFit, a sparse few-shot fine-tuning strategy for generating natural language explanations (NLEs) with pre-trained language models (PLMs). SparseFit uses discrete prompts to jointly generate predictions and NLEs while fine-tuning only 6.8% of the model’s parameters, making it more efficient than full fine-tuning. Tested on three T5 model sizes and four datasets, SparseFit achieves competitive task performance and NLE quality, outperforming other parameter-efficient fine-tuning (PEFT) methods on average in both predictive accuracy and explanation quality.