Dreaming with cubes | In the direction of information science

It is vital to me (and plenty of others). As a result of in a way, it watched me develop from elementary college to (quickly to be!) school commencement. An plain a part of the sport’s attraction is its infinite replayability via world technology. Within the present version of the sport, Minecraft makes use of a mixture of various noise capabilities. Procedurally generate [1] The world is within the type of chunks, i.e. $16 instances 16 instances 384$ Blocks are likely to kind (kind of) “pure” wanting terrain, which supplies a lot of the immersion to the sport.

My purpose for this venture was to transcend hard-coded noise and see if I may educate a mannequin to “dream” in voxels. We leveraged current developments in vector quantized variational autoencoders (VQ-VAE) and transformers to construct a pipeline that generates 3D world slices that seize the structural essence of the sport’s panorama. The precise output is $4$ Chunk ( $2×2$ grid), which appeared like Minecraft terrain.

As an apart, this is not a wholly new concept. Chunk GAN [2] Supplies an alternate method to tackling the identical purpose.

3D generative modeling challenges

in video [3] Beginning in January 2026, Computerphile featured Lewis Stuart highlighting key points round 3D technology. We encourage our readers to look at it. However to summarize the important thing factors: 3D technology is troublesome as a result of good 3D datasets are arduous to search out or just do not exist, and including the dimension of levels of freedom makes issues way more troublesome (assume classical). three body problem [4]). Be aware that whereas lots of the considerations are transferable to the final concept of ​​3D technology, the video explicitly addresses diffusion fashions (which require labeled information). One other difficulty is just scale. be $512 × 512$ picture ($2^{18}$ pixels) would virtually actually be thought of low decision by fashionable requirements, however a 3D mannequin of the identical constancy would require: $2^{27}$ Voxel. Extra factors shortly suggest increased computing necessities, which might shortly make such duties infeasible.

To beat the shortage of 3D information that Stuart talked about, I turned to Minecraft. In my view, that is the most effective supply of voxel information out there for terrain technology. Through the use of scripts to teleport round pre-generated worlds, we pressured the sport engine to load and render hundreds of distinctive chunks. I used a separate extraction script to extract these chunks instantly from the sport’s area recordsdata. This resulted in a dataset with excessive semantic consistency. Not like a group of random 3D objects, these chunks signify a steady, flowing panorama the place the “logic” of the terrain (how a river mattress slopes, how a mountain peak climbs) is preserved throughout the chunk boundaries.

To bridge the hole between the complexity of 3D voxels and the constraints of contemporary {hardware}, we could not merely feed uncooked chunks into the mannequin and hope for the most effective outcomes. I wanted a approach to condense tens of millions of blocks of “noise” right into a significant, compressed language. This led me to the center of the venture. It’s a two-stage generative pipeline that first learns to “tokenize” 3D house after which learns to “converse” it.

Knowledge preprocessing

An vital however not apparent commentary is that a good portion of Minecraft’s chunks are stuffed with “air” blocks. The principle purpose this isn’t trivial is that air is just not technically a block and can’t be positioned or eliminated like different blocks within the recreation, however moderately there is no such thing as a block at that time. In fashionable Minecraft, the vertical span is usually air, so moderately than being taken into consideration fully. $384$ restricted to peak degree $y in [0, 128]$. Should you’re accustomed to Minecraft world technology, you will know that blocks have damaging properties. $y$-Values, at all times $-64$ And at this level I’ve to apologize. As a result of after we carried out this structure, this was fully out of our minds. The mannequin introduced on this article works equally effectively when contemplating bigger vertical spans, however resulting from an unlucky oversight on my half, the outcomes I current are from restricted spans of blocks.

In the case of block limits, chunks include a lot of blocks that seem occasionally and don’t contribute to the final form of the terrain, however are needed to take care of participant immersion. At the very least for this venture, I made a decision to restrict the blocks to the highest 30 blocks that make up chunks by frequency.

Pruning your vocabulary, so to talk, helps, nevertheless it’s solely half the battle. As talked about earlier than, the Minecraft world is primarily composed of “air” and “stone”, so the dataset suffers from a moderately excessive class imbalance. To keep away from the mannequin merely predicting free house to realize a “path of least resistance”, i.e. low loss, we carried out a weighted cross-entropy loss. By scaling the loss based mostly on the antilog frequency of every block, we pressured VQ-VAE to prioritize structural “minorities” corresponding to grass, water, and snow.

$textual content{Weight[block]} = frac{1}{log(1.1 + textual content{likelihood[block]})}$

In layman’s phrases, the rarer a block kind is in a dataset, the extra penalized the mannequin is for failing to foretell it, inflicting the community to deal with snow and riverbeds as vital because the huge expanses of stone and air that dominate most blocks.

Structure overview

This mermaid sequence diagram [6] Supplies a hen’s eye view of the structure.

Uncooked voxel issues and 3D house tokenization

A naive method to constructing such an structure includes constructing chunks by studying block by block. There are numerous the explanation why this isn’t perfect, however an important downside is that with out really offering semantic construction, it could possibly shortly grow to be computationally infeasible. Think about constructing a Lego set with hundreds of components. $1 × 1$ brick. Though doable, it will be too gradual and would not actually have structural integrity. Horizontally adjoining components will not be related, primarily leading to constructing a sequence of disjointed towers. The best way Lego offers with this downside is by utilizing massive blocks like the long-lasting blocks. $2 x 4$ Bricks that occupy house, normally requiring a couple of $1 × 1$ items. This lets you fill the house quicker and will increase its structural integrity.

For the system, the codeword is $2 x 4$ Lego blocks. The aim of utilizing VQ-VAE (vector quantized variational autoencoder) is to construct a codebook, a set of structural signatures that can be utilized to reconstruct a whole chunk. Assume constructions like flat patches of grass or chunks of diorite. In my implementation I allowed the next codebook $512$ distinctive code.

To implement this, I used 3D Convolutions. Whereas 2D convolution is prime to picture processing, 3D convolution permits the mannequin to be taught kernels that slide within the X, Y, and Z axes concurrently. This is essential in Minecraft, the place a block’s relationship to the blocks under it (gravity/help) is simply as vital as its relationship to the blocks subsequent to it.

Additional data

Crucial part at this stage is the “VectorQuantizer”. This layer sits on the “bottleneck” of the community, forcing steady neural alerts to snap into a set “vocabulary” of 512 realized 3D shapes.

One of many largest obstacles in VQ-VAE coaching is “useless” embeddings, or codewords that the encoder by no means selects, successfully losing mannequin capability. To resolve this, we have added a approach to “reset” invalid codewords. If the codeword utilization drops an excessive amount of, the mannequin is pressured to reinitialize the codewords by “stealing” vectors from the present enter batch.

brick by brick

It is nice to have quite a lot of blocks, however they do not imply a lot if they do not match collectively effectively. Subsequently, to make efficient use of those codewords, we used GPT. To make this work, we captured the latent grid generated by VQ-VAE right into a set of tokens. This primarily flattens the 3D world right into a 1D language. GPT then appears at eight chunks price of tokens to be taught Minecraft’s spatial grammar and obtain the semantic consistency talked about above.

To perform this, I used informal self-attention.

Lastly, throughout inference, the mannequin makes use of top-k sampling, together with temperature for management. unstable technology Creativity within the subsequent technology loop:

By the tip of this sequence, GPT has “written” a structural blueprint that’s 256 tokens lengthy. The following step is to move these to the VQ-VAE decoder and $2×2$ A recognizable grid of Minecraft terrain.

consequence

On this rendering [6]the mannequin clusters the leaf blocks effectively and mimics the tree construction of the sport.

On this case [6]In , the mannequin makes use of blocks of snow to cowl the stones and grass to replicate the slices of highlands or tundra discovered within the coaching information. Moreover, this rendering reveals that the mannequin has realized find out how to generate caves.

On this picture [6]the mannequin locations water in depressions and bounds them with sand, indicating that they internalize the spatial logic of the shoreline moderately than arbitrarily distributing blocks of water throughout the floor.

Maybe essentially the most spectacular result’s the inner construction of the chunks. As a result of this implementation makes use of 3D convolution and a weighted loss perform, the mannequin really generates steady underground options corresponding to caves, overhangs, and cliffs.

The result’s recognizable, nevertheless it’s not a whole clone of Minecraft. VQ-VAE compression is a “lossy” compression, so block boundaries could also be barely “blurred” or blocks could float. Nevertheless, for fashions working on extremely compressed latent areas, the power to take care of structural integrity throughout the house is $2×2$ I imagine that the Chunk Grid has been an important success.

Reflection and future efforts

This mannequin efficiently represents “desires” in voxels, however there may be appreciable room for growth. Future iterations could revisit all the vertical span. $y in [−64,320]$ That is to accommodate the large jagged peaks and deep “cheese” caves that characterize fashionable variations of Minecraft. Moreover, increasing the codebook past 512 entries permits the system to tokenize extra complicated and area of interest constructions, corresponding to villages and desert temples. Maybe most fun is the potential for conditional manufacturing, or “biomerization” of GPT. This permits customers to information the constructing course of with particular prompts corresponding to “mountains” or “sea”, turning random desires into artistic instruments with course.

Thanks for studying! Should you’re within the full implementation or wish to check out the weights your self, be happy to test it out. repository [5].

citations and hyperlinks

[1] Minecraft Wiki Editor, World Era (2026), https://minecraft.wiki/w/World_generation

[2] x3voo, ChunkGAN (2024), https://github.com/x3voo/ChunkGAN

[3] Lewis Stuart for Computerphile, Producing 3D Fashions by Diffusion – Computerphile (2026), https://www.youtube.com/watch?v=C1E500opYHA

[4] Wikipedia Editor, Three-Physique Drawback (2026), https://en.wikipedia.org/wiki/Three-body_problem

[5] spaceybread, glowing-robot (2026), https://github.com/spaceybread/glowing-robot/tree/grasp

[6] Picture by writer.

Notes on datasets

All coaching information was generated by the writer utilizing an occasion of Minecraft Java Version operating regionally. Chunks had been extracted from procedurally generated world recordsdata utilizing a customized extraction script. No third-party datasets had been used. As the info was generated and extracted by the authors from their very own recreation situations, no exterior license restrictions apply to their use on this analysis context.