[1] Prompt Tuning of Pre-trained GPT
with Twitter Complaints Dataset

Series of experiments conducted for SIP for GPT

The experiment is started in 2023-08-22
THe experiment is finished in 2023-08-24

1. Introduction

2. Prompt Tuning Literature Reviews

Paper	Summary
Prefix-Tuning	train a tokens while freezing the parameters
P-Tuning v2	P-Tuning is not adequate for small-size models. P-Tuning V2 is an implementation of Deep Prompt Tuning
GPT Understands, Too	P-tuning employs trainable continuous prompt embeddings
Prompt Tuning	Add additional prompts to be trained.

3. Dataset for Finetuing

Labels = ['Unlabeled', 'complaint', 'no complaint'] DatasetDict({ train: Dataset({ features: ['Tweet text', 'ID', 'Label', 'text_label'], num_rows: 50 }) test: Dataset({ features: ['Tweet text', 'ID', 'Label', 'text_label'], num_rows: 3399 }) })

An example of data set is:

{ 'Tweet text': '@HMRCcustomers No this is my first job', 'ID': 0, 'Label': 2, 'text_label': 'no complaint' }

The prompt is the form of Tweet text : <id> <text> Label : <label>

4. Training Results

We follow Prompt tuning in PEFT with Pythia models.

4.1 Training Loss (Perplexity)

4.1.1 Prompt vs Prefix

Figure. Training Perplexity ($\downarrow$). We compare prompt tuning and prefix tuning over several models. The results indicate that prompt tuning shows faster convergence than prefix tuning.

4.1.2 Model Size

4.2 Generation

4.2.1 Prompt Tuning

Query: Tweet text : @openai No one knows the source of AI, but GPT can Label :

Answers:

Model Size	Try 1	Try 2	Try 3
`70m`	no complaintdescribe	no complaintdescribe	complaints2
`160m`	no complaintGeorg	no complaintogy	no complaintogy
`410m`	no complaint•	feature_class	Tweet.
`1b`	no complaint”,	no complaint	no complaint
`1.4b`	no complaint	complaint	no complaint
`2.8b`	no complaint	no complaint	no complaint
`6.9b`	no complaint	no complaint	no complainttext
`12b`	complaint	complaint	complaint

4.2.2 Prefix Tuning