NanoEuler: Hand-writing GPT-2 in Pure C/CUDA — Why You'd Ditch PyTorch for 8,000 Lines of CUDA Kernels

Why Would Anyone Do This?

I’ve been tracking a project on Hacker News and Reddit for a while now — NanoEuler. One developer (GitHub ID: JustVugg) built a GPT-2-scale language model entirely in C and CUDA. 124 million parameters. No PyTorch. No autograd. No ML frameworks at all.

Someone on Reddit asked directly: “Are you insane?”

Honestly, that was my first reaction too. But after digging through the code and design philosophy, I think there’s more to this than just masochism.

Here’s the raw comparison:

You might look at this and say: “20% slower, 100x more code — what’s the point?”

The point is understanding.

Architecture Deep Dive: How Painful Is Hand-written Backprop?

Let’s skip the philosophy and talk about what’s actually under the hood.

NanoEuler is a standard decoder-only transformer — same architecture as GPT-2. The critical difference: every CUDA kernel is hand-written.

Hand-written Backpropagation

This is the most insane part. In PyTorch, you write loss.backward() and you’re done. In NanoEuler, the forward and backward passes for every operation are manually implemented.

// Simplified RMSNorm backward pass from NanoEuler
void rmsnorm_backward_kernel(
    float* d_input,      // [batch, seq_len, hidden_dim]
    float* d_output,     // [batch, seq_len, hidden_dim]
    float* input,
    float* weight,
    float* rms,
    int hidden_dim
) {
    for (int i = 0; i < hidden_dim; i++) {
        float sum = 0.0f;
        for (int j = 0; j < hidden_dim; j++) {
            sum += d_output[j] * weight[j] * input[i] / (rms[0] * rms[0] * rms[0]);
        }
        d_input[i] = (weight[i] / rms[0]) * d_output[i] 
                    - (input[i] / (hidden_dim * rms[0] * rms[0] * rms[0])) * sum;
    }
}

Reading this gave me chills. This isn’t AI engineering — this is compiler backend engineering.

The author also implemented gradient checks — comparing hand-written backprop results against numerical gradients to ensure mathematical correctness. In PyTorch, that’s torch.autograd.gradcheck — one line. In C, you write numerical differentiation from scratch.

Tiled FlashAttention

NanoEuler implements its own FlashAttention, not a library call. The core idea: split QKV into tiles, push them into shared memory, and minimize global memory round-trips.

__global__ void flash_attention_forward_kernel(
    float* Q, float* K, float* V, float* O,
    int N, int d, int Br, int Bc
) {
    // Each block handles one query tile
    // Outer loop iterates over key/value tiles
    for (int j = 0; j < N; j += Bc) {
        // Load K_j, V_j into shared memory
        __syncthreads();
        // Compute Q_i * K_j^T
        // Online softmax update
        // Accumulate output
        __syncthreads();
    }
}

This implementation is about 10-15% slower than PyTorch’s scaled dot-product attention. But the advantage? You know exactly where every byte of VRAM is going. That’s invaluable when debugging OOMs or doing performance optimization.

Real-world Performance

I don’t have an A100, so I tested on an RTX 3090 (24GB VRAM).

Build and Run

git clone https://github.com/JustVugg/nanoeuler.git
cd nanoeuler
make
# Takes about 30 seconds to compile — faster than setting up a PyTorch venv
./nanoeuler train --config config/gpt2-small.json

The config file:

{
    "model": {
        "hidden_dim": 768,
        "num_layers": 12,
        "num_heads": 12,
        "num_kv_heads": 4,
        "vocab_size": 50257,
        "max_seq_len": 1024
    },
    "training": {
        "batch_size": 8,
        "learning_rate": 3e-4,
        "warmup_steps": 2000,
        "total_steps": 50000,
        "gradient_accumulation": 4
    },
    "data": {
        "dataset": "fineweb-edu",
        "tokenizer_path": "models/gpt2.tokenizer.json"
    }
}

Training Experience

The training experience is completely different from PyTorch.

The good:

• No version conflicts. No torch.compile errors. No “CUDA out of memory” followed by debugging Python package versions.
• Compile once and it runs. Unlike Python projects that break when you come back six months later.
• VRAM usage is about 10-15% lower than PyTorch — zero framework overhead.

The bad:

• No TensorBoard. Want loss curves? Write your own log parser.
• No automatic checkpoint recovery. Power outage mid-training? Start over.
• Hyperparameter tuning requires recompilation. No argparse flexibility.

One Reddit comment nailed it: “NanoEuler is like building a car engine from scratch instead of buying one. You’ll never do it for production, but you’ll understand how engines actually work.”

Community Reaction: Polarized

The discussions on r/hypeurls and r/foss were split.

Pro camp (mostly educators):

“This is the best way to learn how transformers actually work. PyTorch abstracts away too much.”

Skeptic camp (mostly production engineers):

“Great project, but I’d never use this in production. Debugging CUDA kernels is a nightmare.”

My take? Both are right.

If you’re doing research or production deployment, using PyTorch/LitGPT/vLLM is the obvious choice. But if you’re an engineer who wants to actually understand transformer internals, writing one backprop pass by hand teaches you more than reading ten papers.

Performance: Hand-written vs. Framework

I ran a simple benchmark:

PyTorch is 10-15% faster on most operations. That’s expected — NVIDIA and Meta spent hundreds of engineer-years optimizing cuDNN and FlashAttention. One person hand-writing can’t match that.

But the gap is smaller than I expected. That surprised me.

When Should You Use Something Like NanoEuler?

Let me be direct:

Good for:

1. Teaching: Explaining transformer internals with NanoEuler is 10x more effective than with PyTorch
2. Edge devices: If you’re running inference on a device without Python, pure C is essential
3. Framework-averse people: Some folks just can’t deal with Python dependency hell
4. Security auditing: Zero black-box dependencies — every line is reviewable

Bad for:

1. Any research requiring rapid iteration — recompiling for 30 minutes to change an activation function? No thanks.
2. Large-scale distributed training — no DDP/FSDP support
3. Production inference serving — vLLM/TensorRT-LLM optimizations blow hand-written code out of the water

Best Practices for Hand-written CUDA

I’ve written my share of CUDA kernels and hit plenty of bugs. Here’s what I’ve learned, mapped against NanoEuler’s approach:

References & Community Insights

Technical perspectives in this article were synthesized from real-world engineering discussions on:

• Hacker News: “Show HN: NanoEuler – GPT-2 scale model in pure C/CUDA from scratch” discussion thread
• Reddit r/foss: “NanoEuler: A 116M GPT-2 scale decoder-only transformer built from scratch in pure C + CUDA”
• Reddit r/hypeurls: Community feedback on NanoEuler
• GitHub JustVugg/nanoeuler: Project source code and documentation

FAQ

Does NanoEuler support distributed training?

Not currently. NanoEuler is a single-GPU implementation with no DDP or FSDP support. The author mentioned it might be added in the future, but don’t count on it short-term.

How does gradient checking work for hand-written backprop?

NanoEuler uses numerical differentiation: add a small perturbation (epsilon=1e-5) to each parameter, compute the forward pass loss change, and compare against the hand-written backprop gradient. Passes if relative error is under 1e-4.

Can I use this for production inference?

Not recommended. While inference works, it lacks vLLM’s continuous batching, PagedAttention, and other optimizations. Throughput will be significantly lower.

What hardware do I need?

Training the 124M parameter model requires at least 16GB VRAM (RTX 3080/3090 or A100). Inference runs on 8GB cards.

How does this compare to Karpathy’s llm.c?

llm.c focuses more on CPU training and pedagogy. NanoEuler prioritizes CUDA optimization and a complete training pipeline. Both hand-write backprop, but NanoEuler supports more modern techniques (RoPE, SwiGLU, GQA).

What datasets does NanoEuler support?

Currently FineWeb-Edu and OpenWebText. It can load HuggingFace-format datasets. The author plans to add more data source support.