Evaluating DeBERTaV3 with the Nonparametric Analysis¶

submited by

Style Pass

2024-04-18 16:30:07

For more background, see the notebook Nonparametric Analysis or our paper: Show Your Work with Confidence: Confidence Bands for Tuning Curves.

One approach to natural language processing takes a neural network pretrained with raw text and then fine-tunes it on labeled data. DeBERTa offers one such pretrained model, with two major versions: DeBERTa (He et al., 2021) and DeBERTaV3 (He et al., 2023).

Despite sharing their name, DeBERTa and DeBERTaV3 have significant differences. For example, DeBERTa is trained with a generative masked-language modeling objective (Devlin et al., 2018), while DeBERTaV3 uses the discriminative ELECTRA pretraining (Clark et al., 2020).

Both models have a number of hyperparameters to set when fine-tuning them. We can investigate these models individually and compare them to each other using optimal design analysis (OPDA).

In our case study, we compare fine-tuning two pretrained models, DeBERTa and DeBERTaV3, for natural language inference (NLI) on the MultiNLI dataset (Adina et al., 2018).