Why Transformers Want Positional Encoding For Time Sequence: A Visible Information

Whereas digging into basis fashions for time collection, I spotted that I might probably not perceive them with out first understanding transformers. I didn’t wish to use these fashions as black containers, so I began tracing the concepts backward, from basis fashions to transformers, and from transformers to self-attention. What made the transition attention-grabbing is that though transformers have been initially constructed for language, the core thought carries naturally to time collection. The 2 modalities are very completely different, however they share one thing basic: each are sequences, and in each instances, order modifications which means.

In language, canine bites man could be very completely different from man bites canine.

Time collection are not any completely different. A temperature of $30^circ$ yesterday and $20^circ$ immediately tells a special story from $20^circ$yesterday and $30^circ$ immediately. The values stands out as the similar, however their order modifications the which means of the sequence.

The query is that if self-attention seems to be in any respect observations directly, how does a transformer know which remark got here first, which got here later, or how far aside two observations are?

That query led me to positional encoding.

What stunned me most was how such a easy mathematical thought might give a Transformer a way of order. The precise methods have developed significantly since then, however the underlying drawback stays the identical.

This text is my try and construct that instinct from the bottom up, beginning with a easy time collection and following the trail from uncooked observations to self-attention and at last to positional encoding.

From scalar observations to vector representations

Think about a easy time collection containing the temperature recorded over 5 weekdays:

Every remark $x_t$

A easy manner to do that is thru a realized linear projection i.e. embedding:

giving us a sequence of vector representations: $e_1, e_2, e_3, e_4, e_5$

An embedding is a deep, summary illustration of the collection within the type of a multidimensional numerical vector that encodes its options and that the mannequin understands. [1]

Every $e_t$

The necessary phrase right here is realized. The mannequin is just not given a predefined vector illustration for a temperature similar to $18^circ$. The parameters $W_e$

At this level, $e_t$

How self-attention builds context?

Self-attention permits every remark to make use of info from the remainder of the sequence.

Suppose we wish to replace Friday’s illustration. The mannequin first creates three realized projections from each $e_t$

The matrices $W_Q$

For Friday, its question $q_5$

Every comparability produces an consideration rating:

which measures how related remark ‘j’ is when updating Friday’s illustration. The scaling issue $sqrt{d_k}$

These scores are handed by way of a softmax operate to transform them into consideration weights:

Lastly, these weights are used to mix the worth vectors:

So $e_5$

In brief:

Queries and keys be taught which observations are related to at least one one other. Values carry the knowledge that’s mixed to type the brand new illustration.

What occurs if we shuffle the sequence?

Now comes the necessary query.

Suppose the identical 5 temperature observations are rearranged.

The values themselves haven’t modified; solely the order has. After the realized projection, we nonetheless have the identical set of worth representations, simply rearranged.

Self-attention can nonetheless examine every illustration with all of the others. The identical question, key, and worth projections are utilized, and the identical sorts of pairwise relationships can nonetheless be computed.

What has disappeared is the temporal construction.

Nothing inside $e(27^circ)$ says that it initially got here from Thursday. Nothing inside $e(18^circ)$ says that it occurred after $e(27^circ)$. Additionally, if Wednesday and Friday have the identical temperature worth, the realized projection will map them to the identical embedding vector. With out positional info, the mannequin due to this fact has no strategy to distinguish which embedding got here from Wednesday and which got here from Friday.

That is the important thing limitation:

Self-attention can be taught which observations are associated, however with out an extra positional sign, it has no built-in strategy to know the place these observations occurred within the sequence.

What ought to positional info inform the mannequin?

If self-attention doesn’t know the order of the observations, then the following query is: what sort of positional info can be helpful?

At a minimal, we’d need the mannequin to know:

• Which place an remark belongs to?
  Place 2 must be distinguishable from place 20.

• Which remark got here earlier than or after one other?
  The mannequin ought to have the ability to distinguish $t-1$

• How far aside are two observations?
  In time collection, the distinction between $t-1$

• That close by positions are associated in a structured manner.
  Place 10 and place 11 shouldn’t seem like two utterly unrelated identifiers.

• That the illustration stays helpful over longer sequences.
  Ideally, the positional scheme ought to nonetheless present significant construction because the sequence grows.

For time collection, the third property is very helpful. A mannequin might care about an remark one step in the past due to short-term dependence, or seven steps in the past due to a weekly seasonal sample.

So positional info ought to do greater than merely assign a novel label to every timestep. It ought to give the mannequin a structured illustration of order and relative distance.

How can we signify place?

We now know what info is lacking. The subsequent query is methods to signify it.

A easy strategy to signify place can be to assign every timestep a quantity:

However feeding the uncooked place straight into the mannequin is just not ultimate. The values continue to grow with sequence size, and a single quantity doesn’t give the mannequin a wealthy illustration of positional relationships.

One of many authentic Transformer’s options was sinusoidal positional encoding, the place every place is represented utilizing sine and cosine capabilities at completely different frequencies.

Why sine and cosine?

Begin with the best two-dimensional instance:

As $t$ modifications, the positional vector strikes easily round a circle that permits close by positions to have completely different however nonetheless associated representations.

Extra importantly, transferring ahead by the identical variety of steps produces the identical form of change within the positional illustration. For instance, an offset of seven positions has the identical mathematical relationship whether or not we transfer from place 3 to 10 or from place 20 to 27. That’s helpful for time collection as a result of relative distance usually issues:

can signify very completely different temporal relationships.

The complete sinusoidal positional encoding extends this concept throughout many dimensions:

Completely different dimensions use completely different frequencies. Some change rapidly throughout close by positions, whereas others change way more slowly.

One helpful manner to consider that is as many clocks working at completely different speeds. Collectively, their readings give each place a structured positional signature.

So as an alternative of giving timestep