Transformers and their Ability to Capture Meaning in Unique Vocabularies

Directed Study for my Master’s in Computer Science, Vector Institute Collaborative Specialization in AI
@ Western University, London, Ontario, Canada
Supervisor: Professor Dan Lizotte
Associate Professor, Department of Computer Science, Westen University

Overview

For my Master's directed study, I developed interpretability techniques to assess how Transformers capture the meaning of words from highly unusual and unique vocabularies. These unique vocabularies include:

Reddit communities (and other social media) that utilize memes, slang, and colloquial terms, which constantly evolve as internet culture shifts over time
Industries that employ specialized terminologies, with some terms having different meanings outside of those industries.

Transformers I'm Experimenting With

LLama 3 - 13B

Experimental Design

I will be developing 3 experiments to discover and evaluate what information LLMs captures when inputting unique vocabulary. Refer to the diagram below to get a high-level overview of how each experiment evaluates different layers of an LLM and the details of each experiment. (All experimental designs shown below were designed and proposed by myself and approved by my supervisor).

Go Home

Loading . . .

Transformers and their Ability to Capture Meaning in Unique Vocabularies

Directed Study for my Master’s in Computer Science, Vector Institute Collaborative Specialization in AI@ Western University, London, Ontario, CanadaSupervisor: Professor Dan LizotteAssociate Professor, Department of Computer Science, Westen University

Overview

Transformers I'm Experimenting With

Experimental Design

Directed Study for my Master’s in Computer Science, Vector Institute Collaborative Specialization in AI
@ Western University, London, Ontario, Canada
Supervisor: Professor Dan Lizotte
Associate Professor, Department of Computer Science, Westen University