NCM: Beyond Next-Token Prediction: From Sequential Generation to Whole-Picture Refinement

There are two ways we learn a new subject. They are either going sequentially chapter by chapter or first going through all the chapters to make a complete picture of how much to learn, Relation between various chapters, A rough complete picture of the subject. This is how I do learning. Overall picture first then going into each chapter in details. Usually this is not the method used in coaching. Anyway I came to know of new way of generating text other than the LLM which relies on this method called NCM as explained below.

---

For several years, the development of Artificial Intelligence has largely revolved around Large Language Models (LLMs). Whether it is ChatGPT, Claude or Gemini, most of these systems rely on a similar underlying principle: autoregressive token generation.

In simple terms, an LLM generates text one token at a time. Each new token depends on the tokens that came before it.

This approach has been remarkably successful. However, it also introduces a fundamental computational bottleneck. Even when a GPU can perform millions of operations simultaneously, conventional autoregressive generation is inherently sequential.

Now, a new approach called NCM (Next-Concept Modeling) is exploring a different way of generating language.

Instead of generating text token by token, NCM generates hierarchical representations of information, progressively refining them into complete text.

Could this be the beginning of a new generation of AI architectures? And more importantly, could it eventually replace conventional LLMs?

1. The problem with conventional LLMs

Consider the sentence:

“The Indian economy is growing rapidly.”

A conventional autoregressive LLM generates this sentence in a sequence:

1. The
2. Indian
3. Economy
4. Is
5. Growing
6. Rapidly.

Each subsequent token is predicted using the preceding context.

This creates a sequential dependency. The model cannot ordinarily finalize the next token until the previous token has been generated.

There are several consequences:

• Sequential inference latency.
• Repeated computation during generation.
• Significant memory-bandwidth requirements.
• KV-cache requirements that increase with context length.
• Limited parallelism across future output tokens.

The underlying Transformer architecture can perform considerable parallel computation during training and prompt processing. However, autoregressive inference still has a sequential generation bottleneck.

NCM attempts to address this problem by changing the representation of the information being generated.

2. What is NCM?

NCM stands for Next-Concept Modeling.

The central idea is to generate information at multiple levels of abstraction rather than directly predicting individual text tokens.

Imagine writing an article.

Before writing individual sentences, you establish the overall subject and structure. Then you develop the paragraphs and finally refine the details.

NCM applies a similar coarse-to-fine approach to language generation.

Its architecture uses two important components:

A. Multiscale residual vector-quantized variational autoencoder (VQ-VAE)

This component converts text into a hierarchy of discrete representations, called codes.

The representations operate at different levels of resolution. Coarser levels capture broader information, while finer levels add additional details.

B. Transformer-based hierarchical code prediction

A Transformer predicts these codes progressively.

Instead of predicting every text token sequentially, the model predicts multiple codes in parallel within each level.

The process can be understood as:

1. Generate a coarse representation of the information.
2. Generate additional codes to refine that representation.
3. Continue refining it at progressively higher resolutions.
4. Reconstruct the final text.

The important distinction is that generation still happens sequentially across levels, but multiple representations can be generated simultaneously within a level.

This is fundamentally different from ordinary next-token prediction.

3. Why could NCM be significantly faster?

The biggest potential advantage of NCM is inference speed.

Conventional LLMs require a generation step for every new token. NCM instead generates a hierarchy of representations in a smaller number of sequential stages.

The early stages are particularly efficient because they operate on compact representations.

The original research reports impressive speed improvements on an NVIDIA H100 GPU.

Some reported results include:

• Approximately 21 times faster on the LM1B benchmark.
• Approximately 10 times faster on OpenWebText.
• Approximately 31 times faster on XSum summarization.

These comparisons are against selected diffusion language models, not frontier autoregressive systems such as GPT or Claude.

The researchers also report using approximately four to five times fewer training tokens.

These results are promising, but they should not be interpreted as proof that NCM is 21 times faster than ChatGPT.

Independent benchmarking has also reported speed improvements against matched autoregressive models in certain workloads, while showing that the advantage can change substantially under different batching conditions.

The important point is that NCM introduces a new way to reduce the sequential generation bottleneck.

4. Memory: Can NCM reduce RAM and VRAM requirements?

Memory efficiency is another important question.

However, it is necessary to distinguish between three different types of memory.

Model weights

These contain the learned parameters of the neural network.

NCM still uses a Transformer, along with additional components for hierarchical representation and reconstruction. Therefore, it does not eliminate the memory requirements associated with model parameters.

KV cache

Conventional Transformers use a key-value cache to retain intermediate attention information during autoregressive generation.

As the context becomes longer, KV-cache requirements can become substantial.

NCM changes the generation process and the sequence of representations being processed. This could alter KV-cache requirements and potentially offer memory advantages.

However, there is not yet sufficient evidence to claim a specific percentage reduction in VRAM compared with an equally capable conventional LLM.

Long-term conversational memory

This is an entirely different problem.

An AI remembering information from previous conversations requires additional memory and retrieval mechanisms.

NCM does not automatically provide persistent memory or the ability to remember an entire user’s history.

Consequently, faster inference should not be confused with better long-term memory.

5. NCM versus conventional LLMs

The fundamental differences can be summarised as follows:

Feature	Conventional LLM	NCM
Generation method	Token by token	Hierarchical codes
Parallelism	Limited across future tokens	Parallel within each level
Generation process	Autoregressive	Coarse-to-fine
Sequential steps	Many	Relatively few
Global structure	Emerges during generation	Generated at coarse levels
Streaming	Naturally supported	Less naturally supported
KV cache	Established implementation	Different requirements
Long-term memory	Requires additional systems	Also requires additional systems

One limitation is particularly relevant to applications such as ChatGPT.

Conventional LLMs can stream responses almost immediately, displaying words as they are generated.

NCM’s current hierarchical approach requires the generation and reconstruction process to complete before the final text is available. This can make interactive streaming more challenging.

Therefore, faster overall generation does not necessarily mean a better interactive experience in every application.

6. Does NCM outperform conventional LLMs?

There are three different comparisons that should not be confused.

First, NCM versus diffusion language models.

Second, NCM versus conventional autoregressive Transformers.

Third, NCM versus frontier models such as GPT, Claude and Gemini.

The original research demonstrates promising results, particularly against selected diffusion language models.

Independent testing has also reported approximately four times faster inference in certain single-request workloads against a matched autoregressive model, with approximately twice the speed in a summarization task.

However, the results also show that conventional autoregressive models can perform better under certain high-batch workloads.

Furthermore, questions about output quality, reasoning capabilities, fixed output structures and generation artifacts remain important.

NCM has not yet demonstrated that it can outperform frontier-scale LLMs across general-purpose reasoning, coding, mathematics and long-context tasks.

7. Is this the end of traditional LLMs?

Not necessarily.

There are three possible directions in which this technology could develop.

Scenario A: NCM replaces conventional autoregressive generation

If NCM can scale to much larger models without sacrificing quality, it could become a serious alternative to traditional Transformer-based language generation.

This would have significant consequences for AI infrastructure.

A reduction in sequential generation steps could improve inference throughput and reduce latency.

Scenario B: Hybrid architectures

This is perhaps the most interesting possibility.

Imagine combining NCM with a conventional autoregressive model.

NCM could generate the broad structure of an answer, while an autoregressive model could refine individual sentences and details.

Such a system could combine parallel generation with the flexibility and precision of conventional language models.

Scenario C: Specialised applications

NCM could become particularly useful for applications where the entire output structure matters more than incremental streaming.

Examples include:

• Document summarisation.
• Translation.
• Structured text generation.
• Report generation.
• Planning and content organisation.

It does not necessarily have to replace conventional LLMs to become commercially important.

8. The biggest unresolved challenges

Despite its promise, NCM still has several important limitations.

Scaling: The reported research experiments extend only to relatively small model sizes, with scaling experiments up to approximately 725 million parameters. Much larger models are needed to establish whether the architecture can compete with frontier LLMs.

Accuracy: Faster generation is not useful if the quality of the output deteriorates.

Training: The representation-learning and generation components introduce additional training considerations.

Streaming: Hierarchical generation is less naturally suited to conventional token-by-token streaming.

Long-context reasoning: There is no established evidence that NCM offers a significant advantage in reasoning over entire books, large document collections or lengthy conversations.

Memory efficiency: The actual VRAM and energy advantages need more comprehensive comparisons.

Production deployment: Large-scale serving, batching and infrastructure requirements remain to be established.

These are not necessarily fundamental obstacles, but they need to be addressed before NCM can be considered a replacement for conventional LLMs.

9. The bigger question: Does intelligence require sequential generation?

Perhaps the most interesting aspect of NCM is not its speed.

For years, the dominant approach to language generation has been predicting the next token based on previously generated tokens.

NCM explores a different possibility.

Instead of constructing an answer one token at a time, a model can first establish a broad representation and progressively refine it.

This raises a more fundamental question:

Does intelligence require sequential generation of information, or can a system first establish a global representation and then progressively fill in the details?

Human thinking itself does not always resemble writing a sentence one word at a time. We frequently understand the overall meaning of a situation before finding the words to express it.

This does not mean that NCM replicates human cognition. It does, however, provide an interesting alternative computational approach to language generation.

10. What should we watch next?

There are five important milestones that will determine whether NCM becomes a major architectural development.

1. Scaling beyond one billion parameters: Does the architecture continue to improve as model size increases?
2. Fair comparison with GPT-style models: Can it match conventional models at equivalent parameter counts, compute budgets and output quality?
3. Memory and energy efficiency: Does parallel generation reduce total inference costs?
4. Long-context reasoning: Can it handle entire books, research papers and complex conversations?
5. Hybrid architectures: Can NCM and conventional Transformers complement one another?

These experiments will determine whether NCM is simply a faster specialised architecture or the beginning of a broader transition in AI.

Conclusion

NCM is an interesting development because it challenges one of the fundamental assumptions behind modern language generation: that information must be generated one token at a time.

Its coarse-to-fine hierarchical approach offers a potentially more parallel way of generating language.

However, it is too early to declare the end of conventional LLMs. Scaling, output quality, memory requirements and production performance still need to be demonstrated.

More importantly, there are two separate revolutions taking place in AI research.

The first is the generation revolution: changing how an AI produces information.

The second is the memory revolution: changing how an AI stores, retrieves and uses information over long periods.

NCM primarily addresses the first.

Combining faster generation, persistent memory and sophisticated reasoning could ultimately be much more consequential than simply making existing LLMs faster.

The future of AI may not be about generating the next word faster. It may be about finding entirely new ways to represent, process and generate information.