Skip to main content
Warning: this assignment is out of date. It may still need to be updated for this year's class. Check with your instructor before you start working on this assignment.
This assignment is due on Monday, December 9, 2024 at 11:59PM EST.
You can download the materials for this assignment here:

Homework 5: COMET-ATOMIC Schema

Learning Objectives

  • Create a schema using a commonsense knowledge base.
  • Define the boundaries of your schema.
  • Determine when/what knowledge bases should be used for a given task.

Instructions

ATOMIC is a crowdsourced commonsense knowledge graph that is used for state-of-the-art commonsense reasoning tasks. In this homework, you will be working with COMET-ATOMIC-2020, a BART (encoder-decoder) model finetuned on an updated version of the original ATOMIC. You will take the information provided to you by COMET-ATOMIC-2020 and structure it into a schema used to track the state of a story. (A schema is a structured representation.)

Go to the Python notebook for more information.

Part 1: Creating a Schema

You will be filling in 4 methods in the code:

  • checkPrecondition: check whether or not a pre-condition passes
  • getPreconditions: query COMET to get relevant pre-conditions for a given event
  • getEffects: query COMET to get relevant effects for a given event
  • updateSchema: update the story state to reflect these new pre-conditions & effects

  1. (3 pts) Explain how you made your schema. (e.g., Why did you decide to design it the way you did? What are your inputs and outputs?) (1 paragraph) You can test how your schema does on the “testing call”. Once your schema is finalized, uncomment out the 5 story blocks, and then answer the following questions for each story:

    2-6. (1 pt each) What went well when processing this story? What went poorly? (2-3 sentences)

Part 2: Questions about COMET-ATOMIC

Answer each of the following with a couple of sentences:

  1. (2 pts) What types of stories might COMET-ATOMIC be good at tracking in general? In other words, what types of information is it good at modeling? (It might help to think about how COMET-ATOMIC compares to other knowledge bases.)
  2. (2 pts) What types of stories might COMET-ATOMIC be bad at tracking in general?

2a. (4 pts) Do you think any of the other knowledge bases mentioned in class could better model these? Which ones and why? If none of them could, why not?

What to Submit

  1. An iPython notebook called hw5.ipynb that runs your COMET-ATOMIC schema. Important: Save the output for the Story Tracking Questions!
  2. A pdf that has your schema explanation and the answers to the COMET-ATOMIC questions (Parts 1 & 2 above).

Submissions should be done on Blackboard.

Grading

  • Code - 4 points + 2 points extra credit
  • Schema Explanation - 3 points
  • Story Tracking Questions - 5 points
  • COMET-ATOMIC Questions - 8 points