kakadenotes2

• Examples: – Using a uniform distribution over m words gives a ppl of m . – Using the (estimated) unigram distribution has ppl about 1000 . – Shannon, in his paper ””Prediction and Entropy of Printed English” (’51), estimated 0.6 to 1.3 bits/character (using human prediction of letters). This translates to 4 . 5 bits/word, using 1 bit/character and 4 . 5 characters/word. This gives a ppl of 2 4 . 5 = 23 . – On the PTB dataset, the best ppl is about 55 - 60 (on the validation set). The best character level entropy rate is 1.2 bits/character. This translates to about 77 ppl in perplexity units per word (to see this use 2 ( 1 . 175 * 390000 / 74000) since there are 390000 characters in valida- tion set and 74000 words in the validation set). – There are other ’codings’ like BPE (byte pair encodings) and sub- words. One can translate perplexities between different codings, pro- vided they can faithfully represent the document/sequence. • Concerns: memory and long term dependencies may not be reflected in this metric? • Other ideas: RL, logic, meaning? • Datasets used for language modeling: – Penn Tree Bank (PTB): first collection. 1M words. 10K vocab sized (based on standardization) – WikiText-2 (2M words) and WikiText-103 (103M words). Scraped from articles on Wikipedia passing a certain quality/length threshold, on all topics. 300k vocab size, > 3 times each. – Google Billion Words: web crawl, assorted topics. 1B words. 800K vocab size. – Books corpus: 11k public-domain novels. 1B words. • Training: – GPUs/TPUs are needed. – Books/Billion words takes GPU weeks to a month to train (all standard models). TPU a few days. 3

4.1 Invariances and other observations Define: M = QK T which is of size T × T . • Suppose P = 0 (no positional encoding). The matrix M is shift invariant in that if translate the sequence by τ then ( i, j ) entry gets shifted to ( i + τ, j + τ ) (provided these are in bounds). Similarly, h i → h i + τ . • Lemma. Let Q, K, V ∈ R T × d embed , and let Π be a T -by- T permutation matrix. Then, σ (Π Q (Π K ) T )Π V = Π σ ( QK T ) V , where σ ( · ) is the row- wise softmax of a matrix. In particular, suppose P = 0 , then if we permute the sequence, i.e. x 7→ Π x , then h 7→ Π h . Proof. The LHS is σ (Π QK T Π T )Π V . It suffices to show that σ (Π QK T Π T ) = Π σ ( QK T )Π T . Indeed, letting π denote the permutation specified by Π , we have σ (Π QK T Π T ) i,j = e ( QK T ) π ( i ) ,π ( j ) ∑ T j 0 =1 e ( QK T ) π ( i ) ,π ( j 0 ) = e ( QK T ) π ( i ) ,π ( j ) ∑ T j 0 =1 e ( QK T ) π ( i ) ,j 0 = (Π σ ( QK T )Π T ) i,j . 5 Further comments: Transformer Observations: • hidden state interpretation and ’sequential’ training/scoring: the above de- scription is model for c Pr( w T +1 | w 1: T ) . We may be interested in the model predicting c Pr( w t +1 | w 1: t ) (for t < T where often T = 512 ), i.e. we may want to make multiple predictions simultaneously (say for training). For this, there is a way to use ’masking’ with an upper triangular matrix so that (for all t ≤ T ): Pr( X t +1 = j | w 1: t ) = [ softmax ( e O t )] j • computation: the transformer computations are very efficient due to the man- ner in which the matrix multiplications can be parallelized. In contrast, the LSTM fundamentally needs a for-loop over the history. (The LSTM is a circuit with greater depth.) 6