Homework 2: Guided Generation

Now that you know how to prompt an LLM from HW1, we will be using some guided story generation techniques from Module 2. In this homework, you will be following a generation pipeline inspired by the Plan-and-Write system. In their work, they generated keywords from a title and then generated a story from the keywords. They tried both dynamic and static schemas to integrate the planning into their generation pipeline. This homework will focus on the “static” schema but use a finetuned LLM instead of their RNN model.

Learning Objectives

For this assignment, we will check your ability to:

• Setup data for finetuning LLMs to generate stories given varying amounts of context
• Finetune pretrained models
• Implement NLP evaluation metrics using existing libraries
• Compare the quality of guided vs unguided story generation
• Determine the adequacy of automated metrics like BLEU and ROGUE for creative evaluation

What to do

Getting Started

Like in the last homework, you will be using a Python notebook, but instead of using OpenAI’s suite of models, we’re going to use Qwen3-0.6B. Again, you can run in your local environment or upload it to Google Colab or DeepNote to do the assignment online.

You will be using a portion of the data from the original Plan-and-Write work. I have already setup the dataset in the notebook. You will be using the stories, their titles, and the keywords that Yao et al. extracted from the sentences and formatting them to finetune Qwen.

1) Setup the data

In the notebook, you are given a series of functions that will retrieve the story data for you.

• load_data will return a list of all of the data in the file. You will most likely not need to call this yourself.
• get_story will return a list of the sentences in the story.
• get_title will return the title of a story from a given line.
• get_keywords will return the keywords of a story from a given line. Note that there may be more than one keyword per sentence!

I have already divided up the data into a training split train_stories and testing split test_stories.

You will be setting up two datasets and finetuning a model for each:

• Unconditioned: Given a title, generate a 5-sentence story
• Conditioned: Given a title and keywords, generate a 5-sentence story where each keyword corresponds to a sentence in the story – this will be similar to the method in the paper

You are welcome to use any prompting techniques (e.g., zero-shot, few-shot, chain-of-thought).

2) Finetune two models

Now that your two datasets are ready, you will be using the Supervised Fine-Tuning (SFT) Trainer through HuggingFace. Alternatively, you are welcome to use the Unsloth integration since you will already be using Unsloth.

You might want to save both models. If you are using Google Colab, you can do this by mounting Google Drive (sample code is in the notebook). If you are using Deepnote with the student plan, the models will be available the next time you log in if you save the model to your workspace.

3) Generate & evaluate the stories

You will evaluate the stories in a few different ways: a) BLEU - precision using n-grams b) ROUGE - recall of n-grams

N-grams are a common unit in NLP for talking about words that appear next to each other, where the n denotes how many words. For example, the sentence “The dog was really happy” contains:

• The unigrams The, dog, was, really, and happy
• The bigrams The dog, dog was, was really, and really happy
• The trigrams The dog was, dog was really, and was really happy
• etc.

BLEU and ROUGE are common evaluation metrics used in NLP. BLEU was created to evaluate how accurate machine translation (computational translation of one human language to another) systems were. ROUGE was created to evaluate generated summaries of text.

You will implement BLEU and ROUGE using the following libraries:

• BLEU - https://www.nltk.org/api/nltk.translate.bleu_score.html
• ROUGE - https://pypi.org/project/rouge-score/

And you should calculate BLEU-1, BLEU-2, ROUGE-1, ROUGE-2, and ROUGE-L comparing both the controlled generation vs original story and uncontrolled generation vs original story. Specificially, the BLEU will be modified n-gram precision. Calculate these scores over each pair of sentences in the data and then average across the 20 stories. You will be implementing the BLEU and ROGUE functions to compare the sentences one-by-one in the stories and return an average across the 5 sentences.

Keep adjusting the prompt until you can consistently generate 5-sentence stories, but if you’ve tried a bunch of things and are still unable to get it to produce 5 sentences, evaluate on whatever sentences it generates. You can “pad” the story with empty strings to compare against with BLEU/ROUGE.

4) Analysis

Please answer the following questions in a separate document and save it as a pdf. Each answer should be a few sentences long.

1. Go through some of your generated stories from the testing set. By simply reading the generated stories:
   • a) What is the quality of the stories overall? (2 pts)
   • b) How do the compare across the conditions (controlled vs uncontrolled)? (2 pts)
   • c) Which of the two conditions produced better stories? Why? (2 pts)
2. Higher BLEU and ROUGE scores mean better matches. Were there any interesting patterns that you can see with the BLEU and ROUGE scores? (2 pts)
3. Did the BLEU/ROUGE correlate with your subjective analysis (just reading)? Why or why not? (2 pts)
4. Are BLEU/ROUGE sufficient metrics for evaluating these stories? Why or why not? (2 pts)

What to submit

You should submit

• your completed Python Notebook, and
• a pdf of your answers to the questions in part 4 (analysis) to Blackboard. You can work in pairs.

Grading

Recommended readings