Fine-tune a language model

Status

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Background

When to use it

On the OpenAI platform, supervised fine-tuning (SFT) is best suited to classification, nuanced translation, generating content in a specific format and correcting instruction-following failures.^[3] On the OpenAI platform the minimum is 10 training examples, improvements are seen from 50 to 100 examples, and the right number varies greatly by use case.^[3] Fine-tuning suits a table that already holds the answers: rows of past inputs with the label or text a person chose for each. (Meridian's view)

Fine-tuning can degrade a model's robustness to distribution shifts.^[1] RapidMiner's Finetune Text Classification operator has a train test ratio parameter that sets the share of rows used for testing the fine-tuned model.^[4] Hold back rows the model never trains on, and compare the tuned model with the untuned one on them before relying on it. (Meridian's view)

RapidMiner's tutorial that fine-tunes a sentiment classifier on only 400 training examples will run for many hours, depending on the hardware.^[4] RapidMiner's Finetune (OpenAI) operator estimates the price before training and does not start the job if the estimate exceeds a set limit.^[2] On the OpenAI platform, fine-tuned models remain available for inference until their base model shuts down.^[3] Keep the training rows alongside the tuned model, since they are what lets the tuning be repeated on another base model. (Meridian's view)

The method

Fine-tuning is typically accomplished through supervised learning.^[1] Instruction fine-tuning is a form of supervised learning used to teach large language models (LLMs) to follow user instructions.^[5] The additional training in fine-tuning can be applied to the entire neural network or only to a subset of its layers, in which case the layers not being fine-tuned are frozen.^[1]

On the OpenAI platform, training data is a file in JSONL format, with one complete JSON structure on every line.^[3] On the OpenAI platform, fine-tuning collects a dataset of examples, uploads it formatted in JSONL, and creates a fine-tuning job, which begins the training.^[6] In RapidMiner, the Finetune (OpenAI) operator returns a table describing the job rather than a model, and the model ID comes later from the Check Job Status operator.^[2] RapidMiner's Finetune (OpenAI) operator can fine-tune only foundation models, not models that have already been fine-tuned.^[2]

Low-Rank Adaptation (LoRA) freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture.^[7] LoRA was set out by Edward J. Hu, Yelong Shen and colleagues in a paper first submitted to arXiv on 17 June 2021.^[7] Compared with full fine-tuning of the 175-billion-parameter GPT-3, LoRA can reduce the number of trainable parameters by 10,000 times and the graphics processing unit (GPU) memory requirement by 3 times.^[7] A language model with billions of parameters may be LoRA fine-tuned with only several millions of parameters.^[1] Hugging Face's PEFT (Parameter-Efficient Fine-Tuning) library adapts large pretrained models by fine-tuning a small number of extra parameters instead of all of a model's parameters.^[8] RapidMiner's Finetune Text Classification operator performs either a full fine-tuning or PEFT / LoRA, which can dramatically accelerate the fine-tuning task.^[4]

Compared with the alternatives

A model's ability to temporarily learn from prompts is known as in-context learning.^[9] Unlike training and fine-tuning, which produce lasting changes, in-context learning is temporary.^[9] In the 2020 paper Language Models are Few-Shot Learners, GPT-3 was given tasks and a few demonstrations purely as text, without any gradient updates or fine-tuning, and sometimes reached competitiveness with earlier fine-tuning approaches.^[10]

On the OpenAI platform, fine-tuning can learn from more example inputs and outputs than fit in the context window of a single request, the most tokens a model can use at one time.^[6][11] On the OpenAI platform, a fine-tuned model can use shorter prompts with fewer examples, which saves on token costs at scale.^[6] Try a prompt with a handful of example rows first, and fine-tune when the examples no longer fit or the cost of sending them with every row adds up. (Meridian's view)

Sources

1. [1]Fine-tuning (deep learning) - Wikipediasecondary source
2. [2]Finetune (OpenAI) - Rapidminer Documentationdocumentation
3. [3]Supervised fine-tuning | OpenAI APIdocumentation
4. [4]Finetune Text Classification - Rapidminer Documentationdocumentation
5. [5]Large language model - Wikipediasecondary source
6. [6]Model optimization | OpenAI APIdocumentation
7. [7][2106.09685] LoRA: Low-Rank Adaptation of Large Language Modelssecondary source
8. [8]PEFT · Hugging Facedocumentation
9. [9]Prompt engineering - Wikipediasecondary source
10. [10][2005.14165] Language Models are Few-Shot Learnerssecondary source
11. [11]Context window - Wikipediasecondary source