An Introduction to Holographic Reduced Representations (HRR)

In the previous post An Intuitive Introduction to Hyperdimensional Computing , we explored the fascinating world of Hyperdimensional Computing (HDC), a brain-inspired paradigm that uses high-dimensional vectors to represent and manipulate information. Today, we're going to dive into one of the earliest and most influential frameworks within this family:

Holographic Reduced Representations (HRR), developed by Tony Plate.

HRR is a specific flavor of Vector Symbolic Architecture (VSA) that provides a powerful and mathematically elegant way to encode complex, structured data—like sentences or relational databases—into a single, fixed-size hypervector. It shares the core HDC principles of efficiency and robustness, but it has its own unique operational toolkit that sets it apart.

The "Holographic" Metaphor holographic" principle of distributed storage."

First, why the name "holographic"? The analogy to optical holograms is surprisingly fitting. In a hologram, every part of the recording medium contains information about the entire scene. If you break off a small piece of a hologram, you don't get a piece of the image; you get a complete, albeit blurrier and lower-resolution, version of the entire image.

HRR works the same way. Information is not stored in any single dimension or subset of dimensions of the hypervector. Instead, it's distributed holistically across all components. This distributed representation is the source of its robustness: if you introduce some noise or even zero out a few components, you get a gracefully degraded version of the original concept, not a catastrophic failure. This is a foundational property of all HDC systems, and HRR is a prime example.

The Building Blocks of HRR

While some HDC models use binary vectors, HRR typically uses dense, real-valued hypervectors. The components are usually drawn independently and identically (i.i.d.) from a Gaussian distribution with a mean of and a small variance, such as where is the dimensionality. This setup ensures two key properties:

  • Quasi-orthogonality: Any two randomly generated vectors will have a dot product very close to zero.

  • Stable Norm: The length ( norm) of the vectors is concentrated around a stable value.

Bundling: Vector Addition (+)

Just like in our general overview of HDC, HRR uses simple element-wise vector addition to represent sets or unordered collections. This operation is often called superposition. If you have hypervectors representing and , their bundled representation is simply:

The resulting vector ​is similar to both ​and ​, which you can verify by checking their dot products.

Binding: Circular Convolution (⊛)

Here's where HRR introduces its signature move. To bind two concepts together into an ordered, associative pair (like a key and a value), HRR uses circular convolution.

Circular convolution is a way of "blending" two vectors, say and , to produce a third vector that is dissimilar to both and . The component of the resulting vector is defined as:

Intuitively, you can think of it as anchoring one vector, reversing the other, and then sliding it around in a circle, taking the dot product at each step.

The most important feature of circular convolution is that it's invertible. This allows us to decode information from a combined representation.

Decoding: Unbinding with Correlation

So, if we have a bound vector , how do we get back if we know ? We use the approximate inverse of circular convolution, which is circular correlation (often denoted ). Circular correlation is nearly identical to convolution, but without reversing the sliding vector.

To recover a noisy version of A, we compute:

Here, is the approximate inverse of . For vectors with components drawn from a symmetric distribution around zero (like a Gaussian), the approximate inverse is simply a permutation of the original vector's elements. The convolution of a vector with its approximate inverse, , produces a vector that approximates the identity element for convolution—a vector with a large value at the first index and small, noisy values elsewhere.

Therefore, the result is a noisy version of the original vector . To get the clean result, we simply search our codebook of known hypervectors for the one that has the highest dot product with .

A Structural Example: Representing a Sentence

Let's see how we can use bundling and binding to encode a simple sentence: "The boy threw the ball."

  1. Define Atomic Vectors: We first need random hypervectors for each unique concept: . We also need vectors for the grammatical roles: .

  2. Bind Roles to Fillers: We use circular convolution to bind each role to its corresponding word.

  3. Bundle the Phrases: We bundle these bound pairs using vector addition to form a single hypervector representing the entire sentence.

    This single vector S now contains all the relational information of the sentence in a compressed, distributed form.Query the Representation: Now, we can ask questions. For example, "Who was the agent?" To find out, we query the sentence vector S with the role vector using circular correlation.

Because convolution distributes over addition, this becomes:

The first term simplifies to a noisy version of . The other two terms are effectively noise, as they are convolutions of four different quasi-orthogonal random vectors. The resulting vector will be most similar to our original hypervector, giving us the answer to our query.

Conclusion

Holographic Reduced Representations provide a powerful and mathematically grounded framework for encoding symbolic knowledge into vector space. By using vector addition for superposition and circular convolution for binding, HRR can represent complex relational structures in a robust, distributed format that is amenable to simple, neurally plausible computations. It stands as a foundational model in the broader landscape of Hyperdimensional Computing and continues to inspire new approaches to building more intelligent, efficient, and robust AI systems.