I built a pipeline that does document extraction, embedding, and summarization, all running on a single RTX 4070 with 12GB of VRAM sitting under my desk. No cloud GPU, no API calls to OpenAI. Here’s what actually happened when I tried to make that work in practice, instead of in a demo notebook.

The first mistake was assuming I could just load everything at once. I had a 7B instruct model for summarization, a separate embedding model (bge-large), and a layout model for PDF parsing, all loaded into memory simultaneously because that’s how the tutorial did it. That lasted about four requests before I got CUDA out of memory. Tried to allocate 340.00 MiB. Turns out running three models on 12GB works fine until your batch size goes above 1, and then it doesn’t work at all. I ended up writing a dumb little scheduler that loads and unloads models based on which stage of the pipeline is active. It’s not elegant, it’s basically torch.cuda.empty_cache() and prayer, but it dropped my peak memory from 11.8GB to about 7GB.

Quantization is where I spent most of my time. I started with the fp16 version of Mistral-7B-Instruct because that’s the default everyone points you to, but it barely fit alongside anything else. Switching to a GGUF Q4_K_M quant through llama.cpp cut memory use almost in half and the output quality difference, for my use case of summarizing scraped articles, was not something I could reliably tell apart in blind tests. I ran the same 30 articles through both and had a coworker guess which summary came from which model. He got it right 14 out of 30 times, which is basically a coin flip. That’s when I stopped worrying about quantization loss for this particular task and just used Q4 everywhere I could.

Batching was the actual lesson, more than any specific model choice. My initial version processed documents one at a time through the whole pipeline: extract, embed, summarize, repeat. Throughput was awful, something like 40 seconds per document even for short ones, because the GPU sat idle between tiny operations and the constant model swapping I mentioned earlier added its own overhead. Once I restructured it to batch the embedding step across 16 documents at a time and only load the summarization model after all embeddings were done, throughput went from 40 seconds a document to about 6. Same hardware, same models, just less thrashing.

I also underestimated how much CPU-side work becomes the bottleneck once the GPU stops being the problem. PDF parsing with pdfplumber on multi-column academic papers was eating more wall-clock time than the actual model inference for a while. I switched to PyMuPDF for raw text extraction and only fell back to the layout model for documents where PyMuPDF returned garbage (I check this crudely by looking for weird whitespace ratios and averaging word lengths). That single change removed a bottleneck that had nothing to do with AI at all, and it’s the kind of thing that never shows up in “how to build an AI pipeline” tutorials because it’s boring.

Ollama was genuinely useful for prototyping but I moved off it for the production pipeline because I needed more control over context window management and concurrent request handling than its API gives you easily. I switched to running llama.cpp’s server directly with a custom queue in front of it, written in about 80 lines of Python using asyncio. It’s not as polished, but I know exactly what’s happening when something breaks, and something always breaks eventually.

The other thing worth saying is that 12GB is a real constraint, not a suggestion. I kept reading forum posts saying “just offload some layers to CPU” like it’s free. It’s not. Offloading even 8 layers of a 7B model to CPU on my machine (a Ryzen 5700X) added close to 3x the inference time. For a background batch job that’s fine. For anything closer to real-time, it’s not usable, and you’re better off running a smaller model fully on GPU than a bigger model split across GPU and CPU.

If I were starting over, I’d build the scheduler and the batching logic before touching model choice at all. The model swap was maybe two hours of work total. Getting the pipeline to not choke on its own memory management took most of a weekend.