Transcript of YouTube Video: Stanford CS25: V1 I Transformer Circuits, Induction Heads, In-Context Learning

thank you all for having me it's

exciting to be here uh one of my

favorite things is talking about what is

going on inside neural networks or at

least what we what we're trying to

figure out is going on inside neural

networks so it's it's always fun to chat

about that

um

oh gosh i have to figure out how to how

to do things okay can i

what i i want okay there we go now now

we are advancing slides that seems wrong

um so i think interpretability means

lots of different things to different

people um it's a very a very broad term

and and people mean all sorts of

different things by it

um and so i wanted to talk just briefly

about uh the kind of interpretability

that i i spend my time thinking about um

which is what i'd call mechanistic

interpretability so

um most of my work actually has not been

on language models or on rnn's or

transformers but um on understanding

vision confidence and and trying to

understand how do the parameters in

those models actually map to algorithms

so you can like think of the parameters

of a neural network as being like a

compiled computer program and and the

neurons are kind of like variables or

registers and somehow there's there

there are these these complex computer

programs that are are embedded in those

weights and we'd like to turn them back

in to computer programs that that humans

can understand it's a kind of kind of

reverse engineering problem

um

and so this is this is kind of a

fun example that we found where there

was a car neuron and you could actually

see that um you know that we have the

car neuron and it's constructed from

like a wheel neuron

and it looks for in the case of the

wheel neuron it's looking for for the

wheels on the bottom and those are

positive weights and it doesn't want to

see them on top so there's negative

weights there and there's also a window

neuron it's looking for the windows on

the top and and not on the bottom and so

what we're actually seeing there right

is it's an algorithm it's an algorithm

that goes and turns um you know it's

it's just it's you know saying you know

well cars is has wheels on the bottom

and windows on the top and chrome in the

middle um and that's that's actually

like just the the strongest neurons for

that and so we're actually seeing a

meaningful algorithm and that's that's

not an exception that's that's sort of

the the general story that if you're

willing to go and look at neural neural

network weights and you're willing to

invest a lot of energy and trying to

first engineer them there's there's

meaningful algorithms written in the

weights waiting for you to find them

um and there's a bunch of reasons i

think that's an interesting thing to

think about one is you know just no one

knows how to go and do the things that

neural networks can do like no one knows

how to write a computer program that can

accurately classify imagenet let alone

you know the language modeling tasks

that we're doing no one knows how to

like directly write a computer program

that can do the things that gpd3 does

and yet somehow breaking descent is able

to go and discover a way to do this and

i want to know what's going on i want to

know you know how what is it discovered

that it can do in in these systems

there's another reason why i think this

is important which is uh is safety so

you know if we if we want to go and use

these systems in in places where they

have big effect on the world and

i think a question we need to ask

ourselves is you know what what happens

when these models have have

unanticipated failure modes failure

modes we didn't know to go and test for

or to look for to check for

how can we how can we discover those

things especially if they're they're

really pathological failure modes so the

models in some sense deliberately doing

something that we don't want well the

only way that i really see that we we

can do that is if we can get to a point

where we really understand what's going

on inside these systems

um so that's another reason that i'm

interested in this

now uh actually doing interpersonally on

language models and transformers it's

new to me i um before this year i spent

like eight years working on trying

reverse engineer confidence uh and

vision models um and so the ideas in

this talk um are are new things that

i've been thinking about with my

collaborators um and we're still

probably a month or two out maybe maybe

longer from publishing them um and this

is also the first public talk that i've

given on it so uh you know the things

i'm going to talk about um they made

they're i think honestly still a little

bit confused for me um and definitely

are going to be confused in my

articulation of them so if i say things

that are confusing um you know please

feel free to ask me questions there

might be some points for me to go

quickly because there's a lot of content

um but definitely at the end i will be

available for a while to chat about the

stuff um

and uh yeah also i apologize um if uh if

i'm unfamiliar with zoom and make make

mistakes um but

uh yeah so um with that said uh let's

dive in

um so i wanted to start with a mystery

um before we go and try to actually dig

into you know what's going on inside

these models um i wanted to motivate it

by a really strange piece of discover of

behavior that we discovered and and

wanted to understand

um

uh and by the way i should say all this

work is um uh you know is done with my

my colleagues anthropic and especially

my colleagues catherine and nelson

okay so on to the mystery

um i think probably the the most

interesting and most exciting thing

about um about transformers is their

ability to do in-context learning or

sometimes people call it meta-learning

um you know the gp3 paper uh goes and

and describes things as uh you know uh

language models are few shot learners

like there's lots of impressive things

about gp3 but they choose to focus on

that and you know now everyone's talking

about prompt engineering um and um

andrei caprathi was was joking about how

you know software 3.0 was designing the

prompt and so the ability of language

models of these these large transformers

to respond to their context and learn

from their context and change their

behavior and response to their context

and you know really seems like probably

the most surprising and striking and

remarkable thing about them

um

and

uh some of my my colleagues previously

published a paper that has a trick in it

that i i really love which is so we're

all used to looking at learning curves

you train your model and you you know as

your model trains the loss goes down

sometimes it's a little bit

discontinuous but it goes down

another thing that you can do is you can

go and take a fully trained model and

you can go and ask you know as we go

through the context you know as we go we

predict the first token and then the

second token and the third token we get

better at predicting each token because

we have more information to go and

predict it on so you know the first the

first con token the the loss should be

the the entropy of the unigrams and then

the next token should be the entry of

the biograms and it falls

but it keeps falling

and it keeps getting better

and

in in some sense that's our that's the

model's ability to go and predict to go

and do in-context learning the ability

to go and predict um you know to be

better at predicting later tokens than

you are predicting early tokens that is

that is in some sense a mathematical

definition of what it means to be good

at this magical in-context learning or

meta-learning that these models can do

and so that's kind of cool because that

gives us a a way to go and look at

whether models are good at in-context

learning

yeah if i could just ask the question

like a clarification question

please when you say learning there are

no actual parameters

that is the remarkable thing about

in-context learning right so yeah indeed

we traditionally think about neural

networks as learning over the course of

training by going and modifying their

parameters but somehow models appear to

also be able to learn in some sense um

if you give them a couple examples in

their context they can then go and do

that later in their context even though

no parameters changed and so it's it's

some kind of quite different different

notion of learning as you're as you're

gesturing that

uh

okay i think that's making more sense so

i mean could you also just describe in

context learning in this case as

conditioning as in like conditioning on

the first five tokens of a ten token

sentence

pretty cool tokens yeah i think the

reason that people sometimes think about

this as in context learning or meta

learning is that you can do things where

you like actually take a training set

and you embed the training set in your

context like if you just two or three

examples and then suddenly your model

can go and do do this task and so you

can do fuchsia learning by embedding

things in the context yeah the formal

setup is that you're you're just

conditioning on on on this context and

it's just that somehow this this ability

like this thing like there's there's

some sense you know for a long time

people were

were

i mean i i guess really the history of

this is uh

we started to get good at neural

networks learning right um and we could

we could go and train language uh train

vision models and language models that

could do all these remarkable things but

then people started to be like well you

know these systems are they take so many

more examples than humans do to go and

learn how can we go and fix this and we

had all these ideas about metal learning

develop where we wanted to go and and

train models

explicitly to be able to learn from a

few examples and people developed all

these complicated schemes and then the

like truly like absurd thing about about

transformer language models is without

any effort at all we get this for free

that you can go and just give them a

couple examples in their context and

they can learn in their context to go

and do new things um i think that was

like like that was in some sense the

like most striking thing about the gpd3

paper

um

and so uh this this yeah this ability to

go and have the just conditioning on a

context go and give you you know new

abilities for free and and the ability

to generalize to new things is in some

sense the the most yeah and to me the

most striking and shocking thing about

about transformer language models

that makes sense i mean i guess

from my perspective

i'm trying to square like

the notion of learning in this case

with you know if you or i were given a

prompt of like one plus one equals two

two plus three equals five

as the sort of few shot set up and then

somebody else put you know like five

plus three equals and we had to fill it

out in that case i wouldn't say that

we've learned arithmetic because we

already sort of knew it but rather we're

just sort of conditioning on the prompt

to know what it is that we should then

generate right

uh but it seems to me like that's

yeah i think that's on the spectrum

though because you can you can also go

and give like completely nonsensical

problems where the model would never

have seen um see like mimic this

function and give a couple examples of

the function and the model's never seen

it before and i can go and do that later

in the context um and i think i think

what you did learn um in a lot of these

cases you might not have you might have

um

you might not have learned arithmetic

like you might have had some innate

faculty for arithmetic that you're using

but you might have learned oh okay right

now we're doing arithmetic problems

um

got it in the case this is i agree that

there's like an element of semantics

here um yeah you know this is helpful

though just to clarify exactly sort of

what the

yeah what you remember

thank you for watching of course

so something that's i think really

striking about all of us

um

is well okay so we we've talked about

how we can we can sort of look at the

learning curve and we can also look at

this in context learning curve but

really those are just two slices of a

two-dimensional uh space so like the the

in some sense the more fundamental thing

is how good are we at producing the nth

token at a different given point in

training

and something that you'll notice if you

if you look at this and so when we when

we talk about the loss curve we're just

talking about if you average over this

dimension

if you if you like average like this and

and project on to the the training step

that's that's your loss curve um and um

if you the thing that we are calling the

incontext learning curve is just this

line um

uh yeah this this line uh down the end

here

um

and something that's that's kind of

striking is there's there's this

discontinuity in it um like there's this

point where where you know the model

seems to get radically better in a very

very short time step span at going and

predicting late tokens

so it's not that different in early time

steps but in late time steps suddenly

you get better

and a way that you can make this more

striking is you can you can take the

difference in in your ability to put the

50th token and your ability to predict

the 500 token you can subtract from the

the the 500th token the 50th token loss

and what you see

um is that over the course of trading

you know you're you're you're not very

good at this and you get a little bit

better and then suddenly

you have this cliff and then you never

get better than the difference between

these at least never gets better so the

model gets better at predicting things

but its ability to go and predict late

tokens over early tokens never gets

better

and so there's in the span of just a few

hundred steps in training the model has

gotten radically better at its ability

to go and and do this kind of in context

learning

and so you might ask you know what's

going on at that point

um and this is just one model but um

well so first of all it's worth noting

this isn't a small a small change and so

um

that you can we don't think about this

very often but you know often we just

look at loss goes more like did the

model do better than another model or

worse than another model but um you can

you can think about this as in terms of

gnats and that are are you know it's

just the information theoretic quantity

in that um and you can convert that into

bits and so like one one way you can

interpret this is it's it's something

roughly like you know the model 0.4 nas

is about 0.5 bits is about uh like every

other token the model gets to go and

sample twice um and pick the better one

it's actually it's even stronger than

that that's sort of an underestimate of

how big a deal going and getting better

by 0.4 an ounces so this is like a real

big difference in the model's ability to

go and predict late tokens

um and we can visualize this in

different ways we can we can also go and

ask you know how much better are we

getting at going and predicting later

tokens and look at the derivative and

then we we can see very clearly that

there's there's some kind of

discontinuity in that derivative at this

point and we can take the second

derivative then and we can um with well

derivative with respect to training and

now we see that there's like there's

very very clearly this this line here so

something in just the span of a few

steps a few hundred steps is is causing

some big change and we have some kind of

phase change going on

um and this is true across model sizes

um

uh you can you can actually see it a

little bit in the loss curve and there's

this little bump here and that

corresponds to the point where you have

this you have this change we we actually

could have seen in the lost curve

earlier too it's it's this bump here

excuse me so so we have this phase

change going on and there's a i think a

really tempting theory to have which is

that somehow whatever you know there's

some this this change in the model's

output and its behaviors and it's in a

in a in in these sort of outward facing

properties corresponds presumably to

some kind of change in the algorithms

that are running inside the model so if

we observe this big phase change

especially in a very small window um in

in the model's behavior presumably

there's some change in the circuits

inside the model that is driving that

at least that's a you know a natural

hypothesis so

um if we want to ask that though we need

to go and be able to understand you know

what are the algorithms that's running

inside the model how can we turn the

parameters in the model back into those

algorithms so that's going to be our

goal

um now it's going to recover us require

us to cover a lot of ground um in a

relatively short amount of time so i'm

going gonna go a little bit quickly

through the next section and i will

highlight sort of the the key takeaways

and then i will be very happy um to go

and uh you know explore any of this in

as much depth i'm free for another hour

after this call um and just happy to

talk in as much depth as people want

about the details of this

so

um it turns out this phase change

doesn't happen in a one layer attention

only transformer and it does happen in a

two-layer attention-only transformer so

if we could understand a one-layer

attenuate transformer and a two layer

only attention potentially transformer

that might give us a pretty big clue as

to what's going on

um

so we're attention only we're also going

to leave out layer and biases to

simplify things so you know you one way

you could describe a attention only

transformer

is we're going to embed our tokens

and then we're going to apply a bunch of

attention heads and add them into the

residual stream and then apply our

unembedding and that'll give us our

logits

and we can go and write that out as

equations if we want multiplied by an

embedding matrix

apply attention heads

and then compute the logics from the

unembedding

um

and the part here that's a little tricky

is understanding the attention ads and

this might be a somewhat conventional

way of describing attention and it

actually kind of obscures a lot of the

structure of attentionnets and i think

that oftentimes it's we we make

attention heads more more complex than

they are we sort of hide the interesting

structure

so what is this saying let's say you

know for every token compute a value

back a value vector and then go and mix

the value vectors according to the

attention matrix and then project them

with the output matrix back into the

residual string

um so there's there's another notation

which you could think of this as a as

using tensor products or using um using

uh

well i guess there's a few left and

right multiplying there's a few ways you

can interpret this but um

i'll just sort of try to explain what

this notation means um

so this means

for every you know x or our residual

string we have a vector for every single

token

and this means go and multiply

independently the vector for each token

by wv so compute the value vector for

every token

this one on the other hand means notice

that it's now on the a is on the left

hand side it means go and go and

multiply

the

attention matrix or go and go into

linear combinations of the values value

vectors so don't don't change the value

vectors you know point wise but go and

mix them together according to the

attention pattern create a weighted sum

and then again independently for every

position go and apply the output matrix

and you can apply the distributive

property to this and it just reveals

that actually it didn't matter that you

did the attention sort of in the middle

you could have done the attention at the

beginning you could have done it at the

end um that's that's independent um and

the thing that actually matters is

there's this wvwo matrix that describes

what it's really saying is you know

wvw describes what information the

attention head reads from each position

and how it writes it to its destination

whereas a describes which tokens we read

from and write to

um and that's that's kind of getting

more the fundamental structure and

attention an attention head goes and

moves information from one position to

another and the process of of which

position gets moved from and two is

independent from what information gets

moved

and if you rewrite your transformer that

way

well first we can go and write uh the

sum of attention heads just as as in

this form

um and then we can uh go and write that

as the the entire layer by going and

adding an identity

and if we go and plug that all in to our

transformer and go and expand

um we we have to go and multiply

everything through we get this

interesting equation and so we get this

one term this corresponds to just the

path directly through the residual

stream

and it's going to want to store uh

bigram statistics it's just you know all

i guess is the previous token and tries

to predict the next token

and so it gets to try and predict uh try

to store bi-gram statistics and then for

every attention head we get this matrix

that says okay well for we have the

attention pattern so it looks that

describes which token looks at which

token and we have this matrix here which

describes how for every possible token

you could attend to

how it affects the logics and that's

just a table that you can look at it

just says you know for for this

attention head if it looks at this token

it's going to increase the probability

of these tokens in a one layer attention

only transformer that's all there is

um yeah so this is just just the

interpretation i was describing

um

and another thing that's worth noting is

this

the according to this the attention on

the transformer is linear if you fix the

attention pattern now of course it's the

attention pattern isn't fixed but

whenever you even have the opportunity

to go and make something linear linear

functions are really easy to understand

and so if you can fix a small number of

things and make something linear that's

actually

a lot of leverage

okay

um

and yeah we can talk about how the

attention pattern is computed as well

um you if you expand it out you'll get

an equation like this

and uh notice well i think i think it'll

be easier

okay

the i think the core story though to

take away from all of these is we have

these two matrices that actually look

kind of similar so

this one here

tells you if you attend to a token

how are the logits affected

and it's you can just think of it as a

giant matrix of for every possible token

input token how how is the logic how are

the logics affected

by that token are they made more likely

or less likely

and we have this one which sort of says

how much does every token want to attend

to every other token

um one way that you can you can picture

this is

uh okay that's really there's really

three tokens involved when we're

thinking about an attention head we have

the

token that

we're going to move information to and

that's attending backwards

we have the source token that's going to

get attended to and we have the output

token whose logits are going to be

affected

and you can just trace through this so

you can ask what happens um how does the

the attending to this token affect the

output well first we embed the token

then we multiply by wv to get the value

vector the information gets moved by the

attention pattern

we multiply by wo to add it back into

the residual stream we get hit by the

unembedding and we affect the logits and

that's where that one matrix comes from

and we can also ask you know what

decides you know whether a token gets a

high score when we're when we're

computing the attention pattern and it

just says

you know embed embed the token

turn it into a query embed the other

token turn it into a key

and dot product to them and see you

that's where those those two matrices

come from

so i know that i'm going quite quickly

um

maybe i'll just briefly pause here and

if anyone wants to ask for

clarifications uh this would be a good

time and then we'll actually go and

reverse engineer and and say you know

everything that's going on in a

one-layer pendulum transformer is now in

the palm of our hands

it's a very toy model

you know one actually uses one layer

attention on the transformers but we'll

be able to understand the one layer

attention only transformer

so just to be clear so you're saying

that yes the the

quite key circuit is learning the

attention weights

and like essentially it's responsive

running the sort of attention between

different uh tokens

yeah yeah so

so this this matrix when it yeah you

know all three of those parts are

learned but that's that's what expresses

whether

a attention pattern is yeah that's what

generates the attention patterns gets

run for every pair of tokens and you can

you can you can think of values in that

matrix as just being how much every

token wants to attend to every other

token if it was in the context and we're

we're drawing positional weddings here

so there's a little bit that we're sort

of aligning over there as well but sort

of in in a global sense how much does

every token want to attend every other

token right

and the other circuit like the output

value circuit is

using the attention that's calculated to

guess

like affect the final outputs it's sort

of saying if if the attention head

assume that the attention head attends

to some token so let's set aside the

question of how that gets computed just

assume that it hence to some token how

would it affect the outputs if it

attended to that token

and you can just you can just calculate

that um it's just a big table of values

that says you know for this token

it's going to make this token more

likely this token will make this token

less likely

right okay

and it's completely independent like

it's just two separate matrices they're

they're not you know the the formulas

that might make them seem entangled but

they're actually separate

all right so to me it seems like the

lecture supervision is coming from the

output value circuit and the query key

second seems to be more like

unsupervised kind of thing because

there's no

i mean they're just i think in the sense

that every in in an yeah in a model like

every every neuron is in some sense you

know like

signals is is somehow downstream from

the ultimate the ultimate signal and so

you know the output value signal the

output value circuit is getting more

more direct is perhaps getting more

direct signal correct um but yeah

yes

we will be able to dig into this in lots

of detail in as much detail as you want

uh in a little bit so we can um maybe

i'll push forward and i think also

actually an example of how to use this

reverse engineer one layer model will

maybe make it a little bit more more

motivated

okay so

um just just to emphasize this there's

three different tokens that we can talk

about there's a token that gets attended

to

there's the token that does the

attention which are called the

destination and then there's the token

that gets affected yet it gets the next

token which its probabilities are

affected

um

and so something we can do is notice

that the the only token that connects to

both of these is the token that gets

attended to

so these two are sort of they're they're

bridged

by their their interaction with the

source token so something that's kind of

natural is to

ask for a given source token you know

how does it interact with both of these

so let's let's take for instance the

token perfect

which tokens for one thing we can ask is

which tokens want to attend to perfect

well apparently the tokens that most

want to attend to perfect are are and

looks and is and provides

um so r is the most looks is the next

most and so on

and then when we attempt to perfect and

this is with one one single attention

head so you know it'd be different if we

did a different intention attention ed

it wants to really increase the

probability of perfect and then to a

lesser extent super and absolute and

pure and we can ask you know what what

sequences of tokens are made more likely

by this

this particular

um set of you know this particular set

of things wanting to attend to each

other and becoming more likely well

things are the form

we have our tokens we attended back to

and we have some

some skip of some number of tokens they

don't have to be adjacent but then later

on we see the token r and it attends

back to perfect and increases the

probability of perfect

so you can you can think of these as

being like we're sort of creating

changing the probability of what we

might call might call skip trigrams

where we have you know we skip over a

bunch of tokens in the middle but we're

affecting the probability really of of

trigrams

so perfect our perfect perfect look

super

um we can look at another one so we have

the token large

um these tokens contains using specify

want to go and look back to it and it

increases probability of large and small

and the skip trigrams that are affected

are things like large using large

large contained small

and things like this

um if we see the number two and we

increase the probability of other

numbers and we affect probable tokens or

skipped diagrams like two one two

two

has three

um

now you're you're all in uh in a

technical field so you'll probably

recognize this one we have uh have

lambda and then we see backslash and

then we want to increase the probability

of lambda and sorted and lambda and

operator so it's all fall latex

and it wants to and it's if it sees

lambda it thinks that you know maybe

next time i use a backslash i should go

and put in some latex

math symbol

um

also

same thing for html we see nbsp for

non-breaking space and then we see an

ampersand we want to go and make that

more likely

the takeaway from all this is that a one

layer attenuating transformer is totally

acting on these skip trigrams

um

every everything that it does i mean i

guess it also has this pathway by which

it affects bi-grams but mostly it's just

affecting these skiff trigrams

um and there's lots of them it's just

like these giant tables of skip trigrams

that are made more or less likely

um

there's lots of other fun things that

does sometimes the tokenization will

split up a word in multiple ways so um

like we have indie

well that's that's not a good example we

have like the word pike and then we

we see the the token p and then we

predict ike

um when we predict spikes and stuff like

that um

or

these these ones are kind of fun maybe

they're actually worth talking about for

a second so we see

the token void

and then we see an l and maybe we

predict lloyd

um or r and we predict ralph

and c catherine

and but we'll see in a second then well

yeah we'll come back to that in a sec so

we increase the probability of things

like lloyd's lloyd and lloyd catherine

or pixmap

if anyone's worked with qt

um it's we see pics map and we increase

the probability of um p

xmap again but also

q

canvas um

yeah

but of course there's a problem with

this which is um

it doesn't get to pick which one of

these goes with which one

so if you want to go and make pixmap

pixmap

and pixmap q canvas more probable you

also have to go and create make pixmac

pixmap p canvas

more probable

and if you want to make lloyd lloyd and

lloyd catherine

more probable you also have to make

lloyd cloyd and lloyd lathron

more probable

and so there's actually like bugs that

transformers have like weird at least

and you know and these these really tiny

one-layer attention only transformers

there there's these bugs that you know

they seem weird until you realize that

it's this giant table of skip trigrams

that's that's operating

um

and the the nature of that is that

you're going to be

um

uh yeah you it sort of forces you if you

want to go and do this to go in and also

make some weird predictions

chris

is there a reason why the source tokens

here have a space before the first

character

yes um that's just the i was giving

examples where the tokenization breaks

in a particular way and okay um because

spaces get included in the tokenization

um

when there's a space in front of

something and then there's an example

where the space isn't in front of it

they can get tokenized in different ways

got it cool thanks

great question

um

okay so some just to abstract away some

common patterns that we're seeing i

think um

one pretty common thing is what you

might describe as like d

a b so you're you go and you you see

some token and then you see another

token that might precede that token then

you're like ah probably the token that i

saw earlier is going to occur again

um or sometimes you you predict a

slightly different token so like maybe

maybe an example the first one is two

one two

but you could also do two

has three

and so three isn't the same as two but

it's kind of similar so that's that's

one thing another one is this this

example where you've a token that

something it's tokenized together one

time and that's split apart so you see

the token and then you see something

that might be the first part of the

token and then you predict the second

part

and

i think the thing that's really striking

about this is these are all in some ways

a really crude kind of in context

learning

and and

in particular these models get about 0.1

nouns rather than about 0.4

of incontex learning and they never go

through the phase change so they're

doing some kind of really crude and

context learning and also they're

dedicating almost all their attention

heads to this kind of recruiting context

learning so they're not very good at it

but they're they're they're dedicating

their um their capacity to it

uh i'm noticing that it's 10 37 um

i i want to just check how long i can go

because i maybe i should like super

accelerate because this is

chris uh i think it's fine because like

students are also asking questions in

between such uh you should be good

okay so maybe my plan will be that i'll

talk until like 10 55 or 11 and then if

you want i can go and answer questions

for a while after after that

yeah it works

fantastic

so you can see this as a very crude kind

of in context learning like basically

what we're saying is it's sort of all

this flavor of okay well i saw this

token probably these other tokens the

same token or similar tokens are more

likely to go and occur later and look

this is an opportunity that sort of

looks like i can inject the token that i

saw earlier i'm going to inject it here

and say that it's more likely that's

like that's basically what it's doing

and it's dedicating almost all of its

capacity to that so you know these it's

sort of the opposite of what we thought

with rnn's in the past like used to be

that everyone was like oh you know rnn's

it's so hard to care about long distance

contacts you know maybe we need to go

and like use dams or something no if you

if you train a transformer it dedicates

and you give it a long a long enough

context it's dedicating almost all of

its capacity um to this type of stuff um

just kind of interesting

um there are some attentions which are

more primarily positional um usually we

in the model that i've been training

that has two layer or it's only a one

layer model has twelve attention units

and usually around two or three of those

will become these more positional sort

of shorter term things that do something

more like like local trigram statistics

and then everything else becomes these

skipped programs

um yeah so uh some takeaways from this

uh yeah you can you can understand one

layer eventually transformers in terms

of these ov and qk circuits um

transformers desperately want to do

in-context learning they desperately

desperately desperately want to go and

and look at these long distance contacts

and go and predict things there's just

so much so much entropy that they can go

and reuse out of that

the constraints of a when they are

intentionally transformer force it to

make certain bugs but it wants to do the

right thing

um

and if you freeze the attention patterns

these models are linear

okay

um

a quick aside because so far this type

of work has required us to do a lot of

very manual inspection like we're

walking through these giant matrices but

there's a way that we can escape that we

don't have to use look at these giant

matrices if we don't want to

um we can use eigenvalues and

eigenvectors so recall that an

eigenvalue

and an eigenvector just means that if

you if you multiply that vector by the

matrix um it's equivalent to just

scaling

and

uh often

in my experience those haven't been very

useful for interpretability because

we're usually mapping between different

spaces but if you're mapping onto the

same space either values either vectors

are a beautiful way to think about this

um so we're going to draw them um

on a

a radial plot

um and we're going to have a log

uh radial scale because they're gonna

vary their magnitude's gonna vary in by

many orders of magnitude

um okay so we can just go and you know

our ob circuit maps from tokens to

tokens that's the same vector space and

the input and the output and we can ask

you know what does it mean if we see

eigenvalues of a particular kind well

positive eigenvalues and this is really

the most important part mean copying so

if you have a positive eigenvalue it

means that there's some set of of tokens

where if you if you see them you

increase their probability

and if you have a lot of positive

eigenvalues um you're doing a lot of

copying if you only have positive

eigenvalues everything you do is copying

um now imaginary eigenvalues mean that

you see a token and then you want to go

and increase the probability of

unrelated tokens and finally negative

eigenvalues are anti-copying they're

like if you see this token you make it

less probable in the future

well that's really nice because now we

don't have to go and dig through these

giant matrices that are vocab size by

vocab size we can just look at the

eigenvalues

um and so these are the eigenvalues for

our one layer attention only transformer

and we can see that you know

for

many of these they're almost entirely

positive these events are are sort of

entirely positive these ones are almost

entirely positive and really these ones

are even almost entirely positive and

there's only two

that have a significant number of

imaginary and negative eigenvalues

um and so what this is telling us is

it's just in one picture we can see you

know okay they're really you know

10 out of 12 of these of these attention

heads are just doing copying they just

they just are doing this long distance

you know well i saw a token probably

it's going to occur again type stuff um

that's kind of cool we can we can

summarize it really quickly

okay

um

now the other thing that you can yeah so

this is this is for a second we're gonna

look at a two-layer model in a second

and we'll we'll see that also a lot of

its heads are doing this kind of copying

or stuff they have large positive

eigenvalues

um you can do a histogram like you know

one one thing that's cool is you can

just add up the uh the eigenvalues and

divide them by their absolute values and

you get a number between zero and one

which is like how copying how copying is

just the head or between negative one

and one how copying is just the head you

can just do a histogram you can see oh

yeah almost all the heads are doing

doing lots of copying

you know it's nice to be able to go and

summarize your model you know uh and i

think this is this is sort of like we've

gone for a very bottom-up way and we

didn't start with assumptions about what

model is doing we tried to understand

its structure and then we were able to

summarize it in useful ways and now

we're able to go and say something about

it

um now another thing you might ask is

what what do the the eigenvalues of the

qk circuit mean and in our example so

far they haven't been that they wouldn't

have been that interesting but in a

minute they will be and so i'll briefly

describe what they mean a positive

eigenvalue would mean you want to attend

to the same tokens

and imagine your eigenvalue and this is

what you would mostly see in our models

we've seen so far means you want to go

in and attend to a unrelated or

different token

and a negative eigenvalue would mean you

want to avoid attending to the same

triplet

so that will be relevant in a second

um yes so those are going to mostly be

useful to think about in in multi-layer

potentially transformers when we kind of

change the attention heads and so we can

ask you know well i'll get to that in a

second yeah so there's a table

summarizing that um unfortunately this

this approach completely breaks down

once you have mlp layers mlp layers you

know now you have have these

non-linearities and so you don't get

this property where your model is mostly

linear and you can you can just look at

a matrix but if you're working with only

attention only transformers this is a

very nice way to think about effects

okay so recall that one that you're

intentionally transformers don't undergo

this phase change that we talked about

in the beginning like right now we're on

a hunt we're trying to go and answer

this mystery of how what the hell is

going on in that phase change where

models suddenly get good at in context

learning um we want to answer that and

one layer attention only transformers

don't undergo that phase change but two

layer attenuation transformers do so

we'd like to know what's different about

two layer attention only transformers

um

okay well so in our in our previous when

we're dealing with one layer attention

transformers we're able to go and

rewrite them in this

this form and it gave us a lot of struct

ability to go and understand the model

because we could go and say well you

know this is bi-grams and then each one

of these is looking somewhere and we

have this matrix that describes how it

affects things and

and yeah so that gave us a lot of a lot

of ability to think about this thing

these things and we we can also just

write in this factored form where we

have the embedding and then we have the

attention heads and then we have the

unembedding

okay well

um

oh and for simplicity we often go and

write wov for wowv because they always

come together it's always the case like

it's it's in some sense an illusion that

w o and wv are different matrices

they're just one low rank matrix they're

never they're they're always used

together and similarly w q and w k it's

sort of an illusion that they're they're

different matrices um they're they're

always just used together and and keys

and queries are just sort of they're

just an artifact of this of these low

rank matrices

so in any case it's useful to go and

write those together

um okay great so um a two-layer

attention only transformer what we do is

we we go through the embedding matrix

then we go through the layer one

attention heads then we go through the

layer two attenuates

and then we go through the unembedding

and for the the attention is we always

have this identity as well which

corresponds just going down the residual

string so we can

uh go down the residual stream or we can

go through an attention head

next step we can also go down the

residual stream or we can go through an

attention head

um and there's this useful identity uh

the mixed product identity that um any

tensor product or or other ways of

interpreting this um obey which is that

if you have an attention head

um and we have say you know we have the

weights and the attention pattern and

the wov matrix and the attention pattern

the attention patterns multiply together

and the ov circuits multiply together

um and they behave nicely okay great so

and we can just expand out that equation

we can just take that big product we had

at the beginning we just expanded out

and we get three different kinds of

terms so one thing we do is we get this

this path that just goes directly

through the residual stream where we

embed and unembed and that's going to

want to represent some bigram statistics

um then we get things that look like

the attention head terms that we had

previously

and finally

we get these terms that correspond to

going through two attention heads

and

now it's worth noting that these terms

are not actually the same as they're

because the attention head uh the

attention patterns in the next in the

second layer can be computed from the

outputs of the first layer there those

are also going to be more expressive but

at a high level you can think of there

as being these three different kinds of

terms and we sometimes call these terms

virtual attention nets because they they

don't exist in the sun like they aren't

sort of explicitly represented in the

model but um they in fact you know they

have an attention pattern they have no

circuit they're sort of in almost all

functional ways like a tiny little

attention head and there's exponentially

many of them

um

it turns out they're not going to be

that important in this model but in

other models they can be important

um right so one one thing that i said it

allows us to think about attention in a

really principled way we don't have to

go and think about um

you know i think there's like people

people look at attention patterns all

the time and i think a concern you could

have is well you know

there's multiple attention patterns like

you know the information that's been

moved by one intention it might have

been moved there by another attention

ahead and not originally there it might

still be moved somewhere else um but in

fact this gives us a way to avoid all

those concerns and just think about

things in a single principled way

um okay in any case um an important

question to ask is how important are

these different terms well we could

study all of them how important are they

um

and it turns out um you can just

there's an algorithm you can use where

you knock out attention

knock out these terms and you go and you

ask how important are they

um and the it turns out the by far the

most important thing is these individual

attention head terms in this model far

by far the most important thing the

virtual tension heads basically

don't matter that much

they only have an effect of 0.3 not

using to be the above ones and the

bigrams are still pretty useful so if we

want to try it on channel's model we

should probably go and focus our

attention on you know the virtual

attention heads are not going to be the

best way to go in and go in our uh focus

our attention especially since there's

there's a lot of them there's 124 of

them for 0.3 knots it's very little that

you would understand for prosthetic one

of those terms

so the thing that we probably want to do

we know that these are bigram statistics

so what we really want to do is we want

to understand

the

the individual tension head terms

um this is the algorithm i'm going to

skip over it for time

we can ignore that term because it's

small

um

and it turns out also that the layer two

attention heads are doing way more than

layer one attention so that's that's not

that surprising like the layer two

intense are more expressive because they

can use the layer one attention to

construct their attention patterns

okay so uh if we could just go and

understand the layer two attention heads

we probably understand a lot of what's

going on in this model

um

and the trick is that the attention

heads are now constructed from the

previous layer rather than just from the

tokens so this is still the same but the

attention head the attention pattern is

more more complex and if you write it

out you get this complex equation that

says you know you embed the tokens then

you're going to shuffle things around

using the attention edits for the keys

then you multiply by w uk then you

multiply shuffle things around again for

the queries and then you go and multiply

by the embedding again because they were

embedded and then you get back to the

tokens um

uh

but let's actually look at them so

uh one thing that's remember that when

we see positive eigenvalues in the ob

circuit we're doing copying so one thing

we can say is well 7 out of 12 and in

fact the ones with the largest

eigenvalues um are doing coffeeing so we

still have a lot of attention they're

doing copying

um

and yeah the qk circuit so one one thing

you could do is you could try to

understand things in terms of this more

complex qk equation you could also just

try to understand what the attention

patterns are doing empirically so let's

look at one of these copying ones

um i've given it the first paragraph of

harry potter and we can just look at

where it attends

and something really happened

interesting happens so almost all the

time

we just attend back to

the first token we have this this

special token at the beginning of the

sequence

and we usually think of that as just

being um a null attention operation it's

a way for it to not do anything in fact

if you if you look the value vector is

basically zero it's just not copying any

information from that

um

but when whenever we see repeated text

something interesting happens so when we

get to mr

tries to look at and it's a little bit

weak then we get to d

and it tends to errors

that's interesting

and then we get to ers

and it tends to lean

um and so it's not attending to

the same token it's attending to the

same token

shifted one forward

well that's really interesting and

there's actually a lot of attention nets

that are doing this so here we have one

where now we hit the potter's pot and we

attend deters maybe that's the same

tension i don't remember when i was

constructing this example

um it turns out this is a super common

thing so you you go and you you look for

the previous example you shift one

forward and you're like okay well last

time i solved this this is what happened

probably the same thing's gonna happen

um and we can we can go and look at the

effect that the attention head has on

the logits most of the time it's not

affecting things but in these cases it's

able to go and predict when it's doing

us this thing of going and looking

forward to go and predict the next token

um so we call this an induction an

induction head looks for the previous

copy looks forward and says ah probably

the same thing that happened last time

it's gonna happen you can think of this

as being a nearest neighbor's it's like

an in-context nearest neighbor's

algorithm it's going and searching

through your context finding similar

things and then predicting that's what's

gonna happen next

um

the way that these actually work is uh i

mean there's actually two ways but in a

model that uses rotary attention or

something like this you only only have

one

and

you shift your key first you have

an earlier attention head shifts your

key forward once you you like take the

value of the previous token and you

embed it in your present token

and then you have your query and your

key go and look

at

uh

yeah try to go and match so you look for

the same thing

um and then you go and you predict that

whatever you saw is going to be the next

token so that's the the high-level

algorithm um sometimes you can do clever

things where actually it'll care about

multiple earlier tokens and it'll look

for like short phrases and so on so

induction heads can really vary in in

how much they of the previous context

they care about or what aspects of the

previous context they care about but

this general trick of looking for the

same thing shift forward predict that is

what induction has been

um lots of examples of this

and the cool thing is you can now

you can use the qk eigenvalues to

characterize this you can say well you

know we we're looking for the same thing

shifted by one but looking for the same

thing if you expand through the

attention notes in the right way that'll

work out

um and we're copying and so an induction

head is one which has both positive ov

eigenvalues and also positive qk

eigenvalues

um and so you can just put that on a

plot and you have your induction heads

in the corner see

here ov eigen values your qk eigenvalues

i think actually ov is this axis qk is

this one access doesn't matter um and in

the corner you have your your icon

values

or your um your induction heads

um yeah and so this seems to be uh well

okay we now have a natural hypothesis

the hypothesis is the way that that

phase change we're seeing the phase

changes this is the discovery of these

induction heads that would be um the

hypothesis uh and these are way more

effective than regular you know than

this first algorithm we had which was

just sort of blindly copy things

wherever it could be plausible now we

can go and like actually recognize

patterns and look at what happened and

break that similar things are going to

happen again that's a way better

algorithm

um

yeah so there's other attention heads

that are doing more local things i'm

gonna go and skip over that and return

to our mystery because i am running out

of time i have five more minutes okay so

what what is going on with this in

context learning well now now we've

hypothesis let's check it um so we think

it might be induction heads

um

and there's a few reasons we believe