Transcript of YouTube Video: Jacob Andreas | What Learning Algorithm is In-Context Learning?

hi everyone uh I'm Jacob I work on uh

mostly natural language processing at

MIT

um today I'm going to be talking about

something a little different which is uh

the worst possible way to fit uh or to

estimate the parameters of a linear

model and what this can teach us about

uh sort of modern NLP systems and

language generation systems

um here we go okay so to start off right

everybody's favorite introductory

estimation problem linear regression we

have some x's we have some y's a new X

comes along and we want to be able to

predict uh the corresponding y from this

x under whatever the data data

generating process is

um right and so as every undergraduate

now knows uh the way to do this is you

take all of your x's and you put them

into a matrix you take all your y's you

put them into a matrix and then you open

up chat GPT and you paste your access

and your y's into chatgpt and you ask it

to give you a solution to this problem

um I actually I wasn't going to do this

for real but I did this when I was

putting these slides together and it

produces this very long very

authoritative sounding answer there's a

bunch of you know sort of intermediate

computations and math here and if you

take the actual predictor that it gives

you at the end and you drop it back into

this problem

um it looks a little bit like this so

not necessarily a reliable way to uh to

estimate linear regression models right

now and you know more generally I guess

this is obviously a kind of ridiculous

thing to do we have already uh very very

old uh very very good algorithms for

solving these kinds of problems and so

it doesn't seem like the sort of thing

uh that we might necessarily uh need to

ask a language model to do or or where

it would be an interesting question to

ask whether language models text

generation systems uh like the kind that

we have today are able to do it at all

um nevertheless uh the kinds of things

that we can do with these big open-ended

text generation systems uh are becoming

you know sort of like more and more

General uh these models themselves are

becoming more and more capable right so

here's an example from what's now you

know an oldish paper 2020 uh showing

that these models can do real sort of

open-ended uh text generation so here's

a particular language model and we'll

talk more specifically about what these

language models are in a minute uh but

that is being prompted right it's asked

to predict what text should come next

after uh this text that we have in light

gray up at the top of the slide so we

give it the title of uh sort of

hypothetical news article we give it the

subtitle of a hypothetical news article

and then we ask it to generate the text

of the article itself and it gives you

this long paragraph right that uh are

you know both sort of compatible with

the title describing the sort of thing

uh that you would expect to be in an

article with that title written in the

style of a news article uh and more or

less internally coherent um it's also

worth noting that uh this article is or

the sort of factual content of this

article is wrong uh that it's describing

you know as prompted a sort of real

split that happened in the Methodist

Church but the identities of the

individual sort of groups that came out

of the Schism uh while plausible

sounding are totally made up

um now we can do more with these text

generation systems than just generate

long blocks of text

um one of the other surprising

capabilities uh that we've seen as

people have started to play around with

these things is that they can do

instruction following right so you can

take again a sort of generic uh next

word prediction system uh you can say

give it as input the text explained the

moon landing to a six-year-old in a few

sentences

um and a good enough text predictor will

actually respond by following this

instruction and by generating some text

that specifically explains the moon

landing or answers a question why do

birds migrate south for the winter

um and so on and so forth

um

and maybe the last sort of surprising

capability that we've seen in these

models as they have scaled up and as

people have started to play around with

them more

um is a phenomenon that has is coming to

be referred to as in context learning

and so the What's Happening Here is that

rather than just giving this model uh

the sort of beginning of a document that

we want it to complete and rather than

giving it an explicit textual

description of the task that we wanted

to perform we just give it some examples

of that task being performed on other

kinds of inputs so here we're asking it

to do some sort of grammatical error

correction task and we do this by

placing you know as the input to this

model the string poor English input I

eated the purple berries good English

output I ate the purple berries poor

English input thank you for picking me

as your designer I'd appreciate it but

good English output and we fill in all

of these pieces like this so again all

of this text that we're showing in light

gray on the slide is uh text that we as

a human user of this system are

providing and notice that the sort of

structure of this input right is that we

have some human written bad examples

good examples some human written inputs

and outputs

um uh We've Ended here with an input and

what the model is going to generate next

is a desired output and in this case

what we get out of the model is in fact

a corrected version of the last sentence

that we asked for and so we can specify

tasks that we want these language

generation systems these next word

prediction systems to perform uh again

not sort of in in the form of natural

language instructions but in the form of

tiny tiny data sets for these new tasks

that we're asking these models to

generate and this turns out uh you know

and this was something that was not even

necessarily planned out ahead of time

but discovered in these models uh around

2020 uh and it's something that you can

do for a wide variety of tasks uh you

can do it even for simple arithmetic

kinds of things you can do it for

spelling correction you can do it for

translation you can do it for that

grammatical error correction and we'll

see a couple more examples later on um

um and I think you know the important

thing to note here is that in some cases

this is uh

more effective right than providing a

precise natural language description of

the task that you want the model to

perform that they actually get better at

doing these kinds of things when you

give them uh explicit examples in the

input uh and just ask them to generate

more examples or generate you know the

output pieces of those examples uh from

that same distribution um and so this

you know like I said before this

phenomenon is coming to be referred to

as in context learning and what this is

meant to evoke is that there is a kind

of learning that looks like machine

learning as we're used to thinking of it

or parameter estimation is we're used to

thinking of it but that is happening now

not as part of training one of these

prediction systems but actually uh in

the course of figuring out how to

generate text or figuring out how to

complete one of these documents

um and so what this talk is going to be

about the sort of big question that I

want to ask today is basically what on

Earth is going on here uh what is the

relationship between this particular

phenomenon and learning as we're used to

thinking about it in a machine learning

setting and what can we say about the

the sort of you know reliability of

these Pro this process or the limits of

this process

um I want to note also before I go on

here that uh everything that I'm going

to be talking about today was led by my

student accurac with a big collaboration

from some folks at Google and Stanford

as well uh Dale Sherman's and Denny Joe

at Google and Tanya ma who's joined

between Google and Stanford

and the sort of broad uh outline of this

talk is to First figure out uh or you

know discuss just what this in context

learning phenomenon is uh and what

possible hypotheses we might build about

what's going on when models are doing

this text generation process

um and then we're going to ask with the

kinds of models that we have today

algorithmically what kinds of things

might they be able to do what kinds of

things might they not be able to do when

presented with these sorts of in-context

learning problems

um and this is all going to be kind of

in principle arguments just from the

structure of uh language models as they

exist today uh you know without looking

at any real models and then we'll

actually go out and train some models

and try to figure out what's what's

really going on under the hood and then

talk about sort of the implications of

this uh more more broadly for research

on NLP research on language models

so starting with this first question uh

what exactly is this uh in context

learning phenomenon and how might we be

able to describe this behavior

um and so to talk about this right let's

start by being a little bit more precise

about what these language models are

what something like uh you know chat apt

or whatever is

um and fundamentally what one of these

models is uh is just a next word

prediction system right like the thing

that sits on your phone as you're

writing a text message and tries to

generate what word you're going to say

next

um and so what that means in practice is

that we're modeling a distribution over

strings typically natural language

strings but now in practice kind of

anything that you can write in Unicode

um and that that distribution is

represented uh via just the sort of

normal chain rule distribution one token

at a time with some kind of parametric

model that's just trying to predict

every word given all of the words that

came before right so concretely right

thinking about what one of these in

context learning examples looks like uh

we're going to try to model a joint

distribution over for example input

output pairs with some separator tokens

and some input markers and whatever by

just asking what's the probability of

the first input and then the probability

of some separator symbol given that

first input and then the probability of

the first output and so on and so forth

all right and as you might expect uh the

very beginning of this generation

procedure the sort of first step in this

in this decomposition is going to be

super high entropy the model doesn't

know what it's supposed to be doing at

all if it has actually learned by the

learn the task that we're asking it to

perform by the end when we're asking it

to predict the X's it'll know the

distribution of x's and when we're

asking it to predict the Y's it'll you

know know sort of precisely the

distribution of y's given X's

um so how do we actually model this

distribution uh in practice with you

know sort of in in the kinds of language

models that we're using today

um and pretty much every language model

that is in in widespread use that you'll

encounter that people are working with

in in the research Community right now

um is a neural network and it's a neural

network of a very specific kind called a

Transformer

um there's a lot of low-level details

about these Transformer models that I'm

not going to talk about at all right now

but that but sorry but there are some

high level details that are important

for uh for sort of understanding how

these models work and so I want to talk

through those things

um at a high level when I'm given an

input like this one or that contains

goat misspelled Arrow goat spelled

correctly comma snake misspelled Arrow

um what we're going to try to do uh I

guess some of the alignments are off

here but hopefully the correspondence is

right the first thing that we're going

to do is we're going to take each of

these natural language uh tokens right

the word goat or the word or you know

the the symbol Arrow or whatever

um and we're going to assign them what's

called an embedding we're gonna you know

map each of these words using some fixed

dictionary that isn't self learned to

some high dimensional parameter vector

and so if you've heard of word

embeddings right every you know what

we're giving this model is input uh is

an embedding of each of these words in

sequence

um and the first thing that we're going

to do once we've represented each of

these words in the input uh as some high

dimensional Vector

um is that we're going to perform what's

called attention so every word uh is

going to sort of pick up its or you know

at every position in this input we're

going to pick up the embedding that

we've computed so far

um and we're going to take

a linear combination of all of the word

embeddings that are in the sentence uh

up until this word that we've predicted

right now where the sort of Weights in

that linear embedding are themselves

some learned function of the input so

effectively you know when we're looking

at this word goat it's going to decide

how strongly you know and the the reason

this uh is called an attention mechanism

think of it as deciding how strongly it

wants to look at all of the other words

uh in the input up until this point uh

pool them according to sort of how

strongly it's looking at each of them

and then build some new representation

in context of this word goat that

incorporates all of this other

information from all of the other words

that have come before um and we can do a

little extra processing on top of this

so we don't like lose the identity of

the the word that we had as input but

you know at a high level think of this

the first thing that a Transformer does

this attention mechanism uh is just

taking some weighted combination of all

of the other representations that we've

built up to this point

um and we're going to do this in

parallel for every token in the input

right so we're going to compute a

representation of goat in this way and

go gets to look at all three tokens to

the left we're going to compute a

representation of for example this last

Arrow character and that last Arrow

character gets to look at the entire

sentence that we've seen so far and so

we're going to get as a result of this

entire process a sequence of now how

these attention vectors that are derived

from our original input

once we've done this uh we're just going

to apply some other uh learned

non-linear transformation to each of

these new vectors that we've built up uh

this now doesn't look at any other piece

of the input at all it's happening again

sort of locally in parallel at every

step of the input

um you know and works basically the way

a normal feed forward neural network

works if you've seen those uh but we're

doing that now for every word

simultaneously and these two steps

together uh this attention step followed

by this multi-layer perceptron this sort

of generic non-linear transformation

we're going to refer to as a Transformer

layer and we can actually take these

Transformer layers and we can stack a

bunch of them on top of each other in

sequence so we can do this once and then

we can do it a second time where now

when we do that attention step we're

attending not to the original word

embeddings but to all of the your you

know all of the representations that we

built at the last step of this procedure

uh and you know in practice for big

models you're doing this tens of times

or hundreds of times uh to eventually

build up a representation of the entire

input sequence that you have and what

we're going to get here at the end uh

again is a sort of sequence of uh of the

outputs of those final uh MLP steps uh

one for every token in this input

sentence and the last thing that we're

going to do is we're going to take the

very last hidden representation the last

output from the last of these

Transformer layers and we're just going

to try to decode it into a distribution

over possible next tokens using

basically something that looks like a

log linear model right so again every

word in my vocabulary just like at the

embedding layer is associated with

some vector and we're just going to sort

of compute dot products between those

vectors and this last hidden

representation and then exponentially

normalized to get a distribution over

y's and so what we hope right in the

context of these kinds of in context

learning problems is that this

distribution that we get over the end

places a lot of Mass on the correct

spelling of snake and not a lot of Mass

on on all of the other words in our

vocabulary

good so how uh having defined this model

architecture do we train them uh and you

know in the most basic sense the

training is super easy you go out and

you scrape as much text Data as you can

get your hands on uh you crawl the

entire internet you pay a bunch of

people to sit in a room and write you

know sort of nicely structured documents

for you

um and then you just do maximum

likelihood estimation with this model on

that gigantic uh gigantic data set

um a lot of the challenges here are

really engineering challenges uh more

than uh than kind of modeling challenges

and a lot of the reason that you know

sort of open AI uh is able to train

these models and people in Academia are

not uh has to do with just like the cost

of uh computational resources that you

need to really do this at a scale that

makes it effective

um one thing that I'll say that's not

going to be part of this talk at all is

that in modern most modern language

models especially the ones that sort of

look and talk like chat Bots there's

another step that goes on after this

process of sort of learning uh via

something that looks like reinforcement

learning from human feedback

that's not really going to be relevant

for what we're talking about today the

relationship between that and uh some of

the sort of capabilities that I was

talking about at the beginning is still

a little unclear

um

but is part of kind of the modern recipe

okay so this is the basic uh Transformer

architecture

um what happens when we actually do this

when we you know fit this uh maximum

likelihood objective uh to the entire

internet

um and what happens is you get a model

that is very very good at a very large

number of different things right so here

are some examples uh from a recent paper

showing uh good performance on the bar

exam although apparently these bar

numbers are actually really sketchy and

this is you know maybe not something to

take seriously

um but pretty good performance if not

like top human level performance across

a wide variety of uh things requiring

both a little bit of reasoning and uh

quite a lot of knowledge right the LSAT

the SAT the GRE uh both the math

sections and the verbal sections

um and you know this has been in the

news I assume many people have seen this

before

um and I think you know it was

surprising surprising is maybe not even

a strong enough word but uh you know to

many of us in the field both that uh

just doing this at a large enough scale

would get you here

um and that we were going to be able to

do this kind of as quickly as possible

and that the large enough scale turns

out to be you know only all of the text

that Humanity has ever written and not

something much more than that

okay good so among the things that

happens when you train a model in this

way like we were talking about before

um is this in context learning

phenomenon right so here now we can talk

a little bit more precisely about what

ICL uh is or what it would take to be a

good in context learner and it's

something like being able to take a

sequence of inputs that looks like this

um and you know have as your output

distribution something that places a lot

of probability uh on uh in this context

the French word for cheese and you know

maybe other sort of plausible variations

on it

um similarly if we want to do not

machine translation but sentiment

analysis we want to be able to plug in

some uh examples of sentences associated

with with sentiment uh assign them

natural language labels uh and and be

able to predict the appropriate ones I

guess there's some text getting cut off

here but uh but in the right way

um and finally uh one of the kind of

surprising things that happens is that

you can do this not just with

um you know sort of well-defined

problems that you expect to see

occurring many times in the training

data like the sentiment analysis problem

or the machine translation problem uh

but with weird made up problems that

surely nobody in the world has ever

asked one of these models to solve

before uh so you know this thing on the

slide this particular combination of

symbols and meanings

um is something that ekken made up the

first time he gave a version of this

talk uh and uh models are at least the

you know particular language model that

he uh asked this question to was able to

produce the right output here uh the

first time

so coming back to the original question

at the very beginning of the talk

um what is it that's actually going on

here under the hood in virtue of which

these models are able to solve problems

like this

um and I think you know the initial

reaction from most people in the

language processing and machine learning

communities uh was that learning was

probably not the right way to describe

what was actually going on in these

models and maybe better to think of it

as something like uh identification and

here's a quote from Reynolds and

McDonald that I think captures this sort

of hypothesis about what's going on here

pretty nicely

um fuchsia prompting is not really

learning of skills but just locating the

skills that the model already has this

is most obvious for translation where

you can't learn a new language from five

examples and you know I think the

translation case is a clear example of

uh of where this is the right way of

thinking about these models in fact you

can't learn French from uh

five examples like skipping ahead here

you know certainly if there's an English

word that you've never seen before and

you can produce the right French output

some of the knowledge about the

relationship between English and French

does not reside in the sort of in

context training set here but has to

come from the model's background

knowledge

but when we think about these more kind

of algebraic puzzle solving problems um

it's a little less clear that this is

the right way of thinking about things

uh right I think we can be pretty

confident that at least then maybe not

now this particular never particular

example didn't appear in the training

data for any of these models you know

maybe other things with a similar flavor

a similar grammar induction structure

um but uh but not exactly this one and

so competing with that hypothesis that

we had on the previous slide is another

hypothesis that the Zim context learning

thing is quote unquote real learning

right that it's capable of sort of

taking a training set that fully

determines the behavior that you want to

get out and using that kind of training

set to index within some reasonably

large hypothesis class the actual

function that you want this model to

implement even if that's a function that

it never saw executed anywhere at

training time

and so the thing that we set out to do

in this project was basically to answer

that question to figure out

um whether it is possible even in

principle uh for something like one of

these Transformer models uh to make a

version of this hypothesis true to get

them to do real learning and then to

actually figure out what's going on in

Real Models Yeah question

um that is a great question I am

uh not sure we could try it

yeah I mean and like this is I I so I'm

not going to be able to like produce off

of my top of my head though like full

exploration that we did here but no I

mean that's a good point and you know uh

there are also examples of cases where

providing this kind of extraneous

information actually confuses the model

and

um uh you know I'm sure it's also not

the case that uh you can get 100

accuracy on things in this General class

um

other questions before we go on

Okay cool so yeah the the first question

that we're gonna ask here then is

whether it's possible for these

Transformer type models uh to do real

learning whatever that means uh even in

principle so to sort of take this

hypothesis that we had before

um and figure out whether there's some

class of uh functions and some class of

models for which this is true

um and we're going to do this by looking

actually at those uh linear models that

we looked at at the very beginning of

the talk uh just because this is a case

where we know exactly what the kind of

space of algorithmic solutions looks

like it's very easy to generate data and

it's a simple enough problem that you

don't have to go out and use something

trained on the entire internet you can

actually do all of the learning yourself

um one other sort of important caveat to

make here uh is that you know in some

sense this question of uh can a

Transformer Implement function X where

even function X is you know training

some little machine learning model on

the inside

um was already answered decades ago we

know that neural networks are general

purpose function approximators when made

sufficiently large and trained on enough

data so really the question here is

whether uh you can do but but the

constructions that uh you use to do that

kind of universal approximation are kind

of ridiculous they blow up uh

exponentially in the the size of the

input and the complexity of the function

so really what we're trying to figure

out here

um is whether you can do this not just

at all but with models that are the kind

of size and shape of the ones that we're

actually using uh today and of the size

and shape that are uh showing all those

real world results that I was showing

you before

um citation is cut off here but it is to

paper by Garg at all at Stanford who

around the same time we were doing this

uh asked a similar set of Behavioral

questions about just like what kinds of

functions extensionally are you able to

learn in this framework okay so uh the

way we're going to go about looking at

this right rather than taking our

translation problems from before and

asking models to solve translation

problems by via this word embedding step

or whatever we're just going to generate

some input output pairs we're going to

sample uh you know a random weight

Vector here I guess I'm showing this as

one dimensional but these are going to

be higher dimensional problems later on

uh and we're just going to train the

model right in exactly whoops uh in

exactly the same way and I guess for now

we're not even training models we're

just asking how we can parameterize them

to begin with so can we uh parameterize

some Transformer in such a way that you

know we give this zero as input it

predicts a zero as output

um one thing to note here you know we

can also and in all in the experiments

that I'm going to show later on we are

going to train these models both to

predict wise or y's given x's and to

predict X's uh given the whole history

of interactions that have happened

before uh here we're not going to worry

about actually modeling the input

distribution at all we're just going to

worry about modeling that conditional

distribution

um and I think it's actually an open

question how much this matters for uh

for real training things

um good so you know

not to belabor this point too much uh

you know we've had linear regression and

solutions to linear regression problems

since the 1800s uh we know that there

are lots of algorithms that you might

pack into one of these Transformers that

will solve them

um and you know you can do this by just

sort of directly in inverting this

relevant Matrix you can do this by

gradient descent on uh some kind of

least squares objective or something

else that looks like that and so the

question is whether we can take any of

these algorithms and pack them into one

of these models like this

um

and you know it turns out that you can

do this uh

the details are sort of mechanical and

I'm just going to give a sort of high

level flavor for one of these but it

turns out that you can show uh both that

if you're iterative trying to fit these

linear models via SGD or via batch

gradient descent you can do that you can

do that using a relatively shallow model

uh and you know using a model that has

depth that scales proportionally to the

number of steps of SGD that you want to

do here

um using you know sort of a reasonable

number of these attention mechanisms a

reasonable number of these layers and

second you can do this not just via SGD

but just sort of by directly

constructing uh that final predictor via

a sequence of rank 1 updates to

um uh that uh xdx inverse Matrix

um you know again there's a lot of sort

of low-level mechanical stuff but just

to give a high level like Flavor of how

this works

um we're going to define a little

calculus of operations that we can

Implement inside a transformer for sort

of moving data around and applying

linear transformations to it and then

once you have these things you can just

chain them together uh and Implement uh

really any algorithm uh that you choose

that bottoms out in these operations

right uh so what are the things that we

need to do for example paradigmatically

to implement uh gradient descent or even

just the first step of gradient descent

on this objective

um well one thing that we're going to

need to be able to do if uh most of the

processing is going to get done by these

multi-layer perceptron units these feed

forward layers that we were talking

about before is just to consolidate our

data right so we need to get our x's and

our y's into the same kind of part of

the representation space so that the

model can work on them uh we're going to

call this a move operation and it turns

out that you can implement this move

operation uh pretty simply using uh that

attention mechanism that we saw before

they can just sort of pick up a piece of

the Hidden State uh from one input and

move it to the next time step so the

first thing that we're going to do for

sdd here for example is just accumulate

the X's uh into our y's

another thing that you need to do here

is take offline Transformations right

for example on this first step we have

some initial guess at the weight Vector

we need to dot that by our initial

feature Vector to figure out what our

error is and so we need some way of

doing affine Transformations uh and you

know affine Transformations are in some

sense like the only thing that these

multi-layer perceptron uh units can do

and so this is very very easy to

implement using that and MLP layer that

we were showing before

um and then the last thing and the thing

that turns out to be the sort of

fussiest piece of this

um is Computing dot products between

pieces of these feature vectors or

between feature vectors across time you

need this for example to scale your

input by your prediction error when

you're doing an SGD step

um and it turns out that you can also do

this uh by exploiting a bunch of very

low level details in the way these MLP

layers are defined in practice for sort

of real world Transformer models and

this is like I I don't know very inside

baseball

um probably not worth going into too

much details and if they would make

sense you've already read this paper but

basically you can get uh the

non-linearity in that MLP to pretty well

approximate uh element-wise

multiplication of a couple of these

vectors they're scaling by a scalar

um if you get things very close to zero

and do things in the right way

um and once you do this right once

you've sort of defined uh Transformer

chunk that does this move operation a

Transformer chunk uh that does this

affine operation and Transformer chunk

that does this thought operation

um then all you need to do to show that

a Transformer can Implement SGD is write

the program in terms of these operations

uh that actually uh does so right so

here for example is the one that does

SGD you can write a corresponding one uh

for doing the sort of sequence of rank

one updates with the the Sherman

Morrison formula

um and that's all you need to do to

generate these kind of in-principle uh

demonstrations that Transformers can do

real in context learning

um one other nice kind of detail here is

that all of those three things that I

was showing before you can actually

consolidate into a big generic kind of

like uh read transform and write

operation uh that you can you can

implement it in a sort of generic way

um and there has been some follow-up

work since we put this out sort of

developing just generalized versions of

that like uh generic Transformer read

attend write operator uh to make in some

sense nicer programming languages for

describing what goes on or what what

kinds of functions you can Implement

with these Transformers uh and coming up

with tweaks to this architecture

good and you know so uh maybe

unsurprisingly but it's nice to see that

in fact uh it is possible to

parameterize these models uh so that

they Implement real algorithms that

we've written down ahead of time uh and

thus it's possible at least in theory

that these Transformers that we're

seeing out in the real world for at

least some subset of these uh in context

learning problems uh are doing real ICL

um and so you know of course the natural

question now is what is happening in

practice when we actually train these

models on the distribution that we

assumed we were going to get in the

previous part of this talk uh do they

actually converge to the kinds of

solutions that we were expecting to find

here um and so we can ask this in a

couple of different ways uh one is you

know just do you get the same kinds of

generalizations that would be predicted

if you were implementing some version of

real learning as we've described it

before and second what's the

relationship between a model's ability

to discover the solution and uh lower

level details about the capacity of the

model how big it is and other

implementational things

um so we're going to try to answer both

of these things and we're going to do it

right by now just taking this basic

learning setup that we assumed for

constructing some in context Learners

before and actually now just training

models on this data right so we're going

to sample

um some weight vectors we're going to

sample some input vectors we're going to

generate some y's using the weight

vectors and the input vectors that we

sampled and in particular right we're

going to construct a sequence or a data

set of sequences where each of these

sequences uh was generated by a single

one of these weight vectors right so uh

you know here I have a bunch of y's that

were generated from some particular W

sampling my x's from from some normal

distribution that's going to stay the

same

um and I'm going to present the model

with the sequence and ask it to make a

prediction at every step of the sequence

and then when it gets to the end of the

sequence I'm going to sample a new

weight vector and I'm going to give it a

sequence of X Y pairs that were

generated by that new weight Vector

um and you know again we can do this

over and over again this data is fake

it's easy to generate we can generate

lots of it and you can train uh actually

smallish models as we're going to see

later on to do pretty well on this

objective

so what happens when you do this

um and so the first thing that we're

going to look at here is just the

quality of the predictions that the

model is making

um so notice here that when we're

actually

um oh the axis labels got chopped off

but uh uh we'll we'll explain these in a

minute

um when we are

uh

uh sampling from the model right at no

point are we asking it to explicitly

exhibit uh an inferred parameter Vector

we're just asking it to predict uh you

know our y's from our X's

um and for all of the plots that I'm

going to show now we're going to look at

eight dimensional problems so

unsurprisingly right with fewer than

eight examples in the input the model

can't possibly know exactly what uh

noiseless linear function is trying to

learn and so you get some high error

rate that declines uh gradually over

time

um so this is just the you know sort of

uh squared error for the predictions

made by the model all of the other lines

that I'm going to show right now are not

agreement with ground truth but

agreement with other uh prediction

algorithms that we can fit to these

datas the the these data sets that we're

handing to our Transformer model to do

ICL right uh so we might oh yeah

question

sorry

yeah

you just won just one Epoch yeah so it

just sees you know X Y X Y X Y and

sequence without repetition yeah

other questions about the setup

okay

um good so what we're going to ask now

is you know not just what's the fit of

this model to the sort of ground truth

labeling function but to other learning

algorithms that we might train on the

same data uh we might hypothesize and in

fact people in the NLP Community have

hypothesized that the right way to think

about ICL is that it's doing something

like nearest neighbors so we can fit

some nearest neighbors models and those

don't actually seem to do a very good

job at all of describing uh the kinds of

predictions that this Transformer makes

you can do SGD and you know various

versions of of SGD and gradient descent

where you look at all the examples in a

batch or you look at them one at a time

um all of these things early on seem to

agree relatively well with the model's

predictions uh and you know sort of

diverge around that critical example and

then start to fit again better later on

um you can ask about you know more sort

of Old School uh I guess not old school

but but standard regression algorithms

here we're looking at uh regularized

you know Ridge regression here uh this

is a noiseless problem so you shouldn't

need this in principle nevertheless this

gives us a really pretty good fit to our

data but maybe unsurprisingly right if

you just do ordinarily squares uh taking

the mid-norm solution in this uh less

than eight regime uh this is almost a

perfect fit to the predictions that

these models are making so they really

do seem to be uh behaving uh now without

saying about the sort of anything about

the computations that support them uh or

support that but behaving like

um uh like these OLS type models um and

this you know maybe should not surprise

us right if this is really our data

generation process we know that the Min

Bayes risk uh predictions are going to

come from a weight Vector that looks

like this and so if we could really

drive this

loss function all the way down to zero

in some sense what we would have to do

would be to make predictions that look

like this in this noiseless regime

um and you know we can sort of stress

test this a little bit by changing the

data generation process such that the

minrisk predictor looks a little bit

different uh we can make it for example

we can add some noise to our y's uh so

that now the right thing is to behave uh

as though you're regularized a little

bit right and we know in particular for

uh particular uh particular distribution

of weight vectors uh particular

distribution of noise being added to

these y's uh that there's a nice closed

form for uh the grand truth predictor

here in terms of these two things and so

you know you can do uh something that

looks like Ridge regression uh with a

parameter that's determined by these two

uh

noises

um and when you do this and you ask

these questions about what's the sort of

best fit to the model uh you again see

exactly the pattern that you want to see

here right that when there's no noise at

all uh the fit to OLS is better than

everything else as you start to add

noise either to the data generation

process or sorry either do your exes uh

or sorry either to your W's or to your

uh wise given W's

um you see this nice pattern where uh

the predictor that that best fits the

predictions that you actually get out of

this Transformer are exactly the ones

that you would expect if you were

uh you know so this is just saying that

this sort of finding before is robust

that basically these models at least at

the scale that we're training them do

actually give you exactly the predictor

that you want for these kinds of linear

regression problems and this agrees with

uh that uh Garg at all paper that I was

talking about before that does similar

kinds of experiments also in the

presence of uh you know sort of sparse

X's sparse W's things like that

um a more interesting question that we

can ask here

um is whether this always happens or

whether in the sort of uh presence of

stronger computational constraints than

we have assumed up to this point uh

models are still able to sort of

perfectly fit these distributions or

whether they do something different

um and the cool thing that happens here

is that you actually get different fits

to different algorithms as a function of

model size so as we make you know for

the sort of very shallowest models if we

only give them one layer uh they are not

perfectly described but best described

as doing something that looks like a

single step of I think I've covered this

up on the legend but of actually bash

gradient descent on uh the input as you

make these models bigger uh they look a

little bit less like they're doing

gradient descent and a little bit more

like they're doing uh proper properly

squares uh and you know as you make them

uh big enough like we've been doing

before uh you converge to that OLS

solution uh you can also look at this as

a function of uh the hidden size of the

model the size of these embedding

vectors that we're passing up and here

you see less clear but also a similar

kind of trend uh where for very very

small hidden sizes

um uh you have a slightly better fit but

not a great fit from these SGD type

predictors uh and at bigger sizes you

have a better fit from uh from the right

ones

um and so we can think of there being a

couple kind of uh phase phases or

regimes in uh model parameter space uh

or sorry in in model architecture space

that describe the kinds of algorithmic

solutions that these models find to

um to this regression problem

um and we're going to look uh later on

specifically at uh at this sort of SGD

regime uh and try to figure out better

what's going on there

the last question that we are going to

ask about these train models before we

do that

um is just whether we can figure out

anything at all about what's going on

under the hood uh you know so far all of

the characterization of real models that

we've done uh has been uh sort of

extensional right in terms of their

functional form and not in terms of

their uh internal computations

um but it's reasonable to ask given that

we have these sort of constructions

lying around for what intermediate

quantities you might need to compute in

order to solve these problems whether we

can see any evidence that trained models

are actually Computing the relevant

intermediate quantities

um and so to do this we're going to do

what's called I guess now in the the ml

literature A probing experiment we're

going to take our original trained model

we're going to freeze its parameters and

we're going to fit some teeny little

model uh either uh just like a single

linear readout or a little tiny little

multi-layer perceptron and we're going

to train this probe model to try to

predict what the real uh

optimal W hat was uh for the problem

that's being presented in the input from

the internal states of the Transformer

model so basically can I just look at

one of these hidden States and recover

with some predictor that's not powerful

enough to like solve the layer

regression problem on its own uh the

predictor that we think the model is

using uh to actually make predictions

um and so you can do this for different

kinds of intermediate quantities that

that you might want to look for here the

natural one is just this weight Vector

right we said before we're never

actually showing the model any weight

vectors we're never asking it to

generate a weight Vector but if you try

to probe for this weight Vector in its

intermediate States uh you find that you

can do that and you can actually do it

pretty well with linear readout right

near the end of the network which I

think is exactly what we would have

expected from the little construction

that we gave before

um as some sanity checks here right you

can try doing this on a Model that is

trained to perform some task that

requires you to look at all of the

inputs and outputs but is not linear

regression and for those what we're

calling control tasks

um uh no matter where you look in the

network you don't acquire the ability to

recover W as accurately as we were here

so some evidence that you know in the

course of making the predictions that we

were looking at before

um our weight Vector actually gets

encoded by uh by the models that are

making these predictions which is a cool

thing to find

um we can ask about other relevant

intermediate quantities right just like

the the product of our X Matrix and our

y Matrix and if we do this the story is

a little muddier

um it's definitely not linearly encoded

maybe it's non-linearly encoded and the

evidence that it's non-linearly encoded

is the sort of gap between this control

task and the real task which is not as

big as we were looking at here

um but to the extent that there's

anything going on

um it's going on much earlier in the

network that the model sort of computes

this around layer 7 or layer eight and

then hangs on to it

um and you know again this is a little

bit more speculative I wouldn't take

this as uh like dispositive of of models

Computing this quantity but it is what

we would expect if it were implementing

one of these algorithms that we looked

at before because this is a sort of

First Step that you need to do in order

to compute the W later on

um good so to come back to all of these

questions uh that we were sort of asking

at the beginning of this talk uh right

what is in context learning and the sort

of main hypotheses that we want to

discriminate here are whether it's task

identification or quote unquote real

learning um and in the context of a very

very very simple real learning problem

uh We've shown that you know it's

possible at least in principle that

models are are really doing it that you

can get smallish Transformers uh to

implement real learning algorithms and

we've presented both sort of Behavioral

evidence and at least preliminary

um uh representational evidence that

this is actually what's being

implemented by these models at least at

the scale that we're looking at

so in I guess the little bit of time

that I have left before I switch over to

questions

um one thing that's been really fun this

was obviously a question that was like

very much on people's mind uh I guess a

year ago last summer when we started

doing this project and there's been a

ton of work that's come out in this

space both kind of concurrently with uh

with what we were doing here uh and

since this paper came out

um so you know some natural questions

that you might have at the end of

everything that I've been showing here

is whether these constructions we've

given are actually the right ones or

whether especially given that you know

we were getting pretty good results from

like two layer Transformers four layer

Transformers whether we can do these

things more efficiently whether we can

say anything uh you know sort of

theoretically about the conditions under

which we're actually going to recover

the the real learning solution

um and how this relates to data

distributions both in these kinds of

synthetic models and in Real Models um

and what's nice is that we're starting

to get answers actually to all of these

questions so shortly after

um uh we did this uh there was a paper

actually also from another group at

Google that I guess had not been talking

to our Google collaborators uh showing a

very similar thing showing that you

could get standard Transformers uh to

implement in this case uh fitting these

linear linear regression problems via a

single step of gradient descent uh and

they do it in a very different way they

use the attention mechanism actually to

compute all of the dot products that you

need to compute and this allows them to

do it in a single layer with a single

attention head

um and so if you think back to those

kind of experimental results we had

before right showing that in the single

layer regime we're looking more like a

one step of gradient descent model than

anything else here is now some evidence

or here is an example of a way in which

you can solve this problem by

parameterizing a Transformer that would

generate exactly that behavior for you

um there's one kind of important caveat

here which is that this is not actually

using the kind of real models that

people train and practice but something

with a slightly simpler and slightly

less expressive attention mechanism and

this might account for the little Gap

that we were seeing between our SGD

predictors and uh and the real Model

Behavior

um even cooler very recently some folks

at Berkeley I think Spencer is now at UC

Davis but showed that not only can you

do this but you are under certain

conditions guarantee to converge to

exactly this linear self-attention one

step of gradient descent solution so uh

you know we can say now a little bit

more precisely uh the conditions under

which in context learning really is real

learning and and you can actually uh

guarantee this up front using this

linear self-attention model

um finally all of these interesting

questions about data sets

um so one and I think this is a very

recent paper but but cool thing that

came out right

um You can imagine that if you only ever

trained uh your model at training time

on outputs from a single weight Vector

uh the right thing to do kind of no

matter what is to not do any in context

learning just memorize that weight

vector and use that weight Vector to

predict whatever a wise you're going to

see at test time

um and so there's a sort of question of

whether like how much diversity in that

training set you really need to see

before you switch over to being uh a

real learner as opposed to something

that's memorized a fixed family of wheat

vectors

um and so now some empirical evidence

that you do get exactly that kind of

again phase transition as a function of

the diversity of the data set that for

small W's you remember a small number of

W's you memorize all the W's for a

sufficiently large number uh even finite

of W's that you get to see over and over

again during training time eventually uh

you learn to uh to do in context

learning instead I think this is

especially interesting because it sort

of goes against that

um you know Min Bay's risk story that I

was talking about before when you're in

the regime of having a finite number of

uh W's that you've ever seen at training

time uh probably the right thing to do

is eventually you know at least sort of

in a Bayesian sense

um is to just memorize that large finite

set of W's models don't do that and at

some point they learn the the solution

that generalizes instead

um and finally we can start to ask these

kinds of questions about Real Models

thinking back to the very beginning of

the talk right we have this kind of back

and forth between is this just task

identification uh is this uh real

learning uh and now some empirical

evidence that in fact uh it's it's a

mixture of both and that you can buy

changing the label distribution or uh

the kinds of instructions that you

provide up front uh actually induce

models to behave more like task

identifiers or more like uh in context

Learners uh again just by sort of

manipulating the inputs

um finally oh man everything moved

around on the slide but one thing that

we started to look at uh that's a sort

of next direction that I'm super excited

about

um is generalizing to more interesting

kinds of prediction problems right so we

can ask now not just to produce a single

categorical label for these things uh

but more structured kinds of outputs

that maybe start to look a little bit

more uh like the kind of real machine

translation examples and and other text

generation examples uh that we saw at

the beginning of this uh of this talk

um uh what we're looking at specifically

is in context learning of uh finite

automata and other sort of formal

languages and here it turns out that at

least empirically right these models are

also very very good at doing this uh in

context you can get close to perfect

accuracy

um and interestingly this seems to be a

task that separates uh these

Transformers the sort of current

architecture from a lot of the other uh

work that has been done in recent years

uh trying to propose some new new kinds

of models that are that are easier to

train uh have lower computational

budgets things like that

um and so

ICL right this this in context learning

thing seems not only to be a sort of

surprising property of these uh these

models but not something that you

necessarily get for free at scale in any

sufficiently large neural sequence

predictor uh there is something special

about Transformers and so the question

is what what is that thing uh and

ultimately if we can answer that

question uh we will hopefully know what

we need to do to to figure out what the

next generation of model architectures

looks like

um with that I will wrap up uh you know

as always most of the credit goes to

ecken and and the rest equally

distributed among our collaborators this

was a super fun collaboration

um and happy to answer any questions

foreign