One major consequence of the ramapocalypse, I think, is an even higher focus on small efficient models. I personally believe that the multi-trillion parameter models are fundamentally missing things and the push to smaller, more efficient will drive evolutionary structural changes that will lead to future gains
I'd assume the closed weight models are all working on shrinking their parameter counts anyways. They too benefit from smaller models. It'd be foolish for these SOTA labs to not be working at reducing parameter counts.
With the benefit of LLMs already being proven, in a couple of years we will have vastly better hardware for inference I guess.
I feel like now hardware is stagnating a bit, because the software side has moved too fast for the hardware to catch up. Once we settle on sone good, optimal software architecture for the models, dedicated hardware will easily increase throughout by 10x or 100x, for a fraction of the most.
LLMs seems quite simple, maybe we'll be able to print at home our own chips with the models.
I would say even without rampocalypse there would still be the strong incentive to innovate at the edge and under more extreme constraints. The incentives are just even stronger now.
I'm looking forward to seeing what types of new things people create over the coming years once there is less obsession with massive unwieldy LLMs. I think the incentives are just too strong to ignore.
Let's not forget the Bitter Lesson. Small models sound really nice but at some point you're just fighting the laws of information theory. Efficiency gains on the small model side are nice, but efficiency gains + giant model tends to be even better...
I think the path of least resistance will end up being the cheapest and that is scaling up the parameters a ridiculous amount until you get highly capable models that can develop/distill/design the RAM efficient models. Going straight for low param is foolish and just a cope by smaller labs because they don't have the compute/talent to train the large ones.
This is 100% true for pretrains, likely true for RL as well although maybe there is some benefit to smaller activated params there. There is of course 0 benefit to small dense models relative to large sparse ones that are equally as memory efficient if you have enough computers.
I'm having an extremely hard time thinking of companies that have prospered due to software optimization. Most of them were swept away by hardware advances, instead.
The 1980's US car industry comes to mind. Nearly wiped out because they refused to make efficient vehicles. SpaceX is arguably showing how a rethink towards efficient can take over an entire industry. I am sure there are strong examples in software as well but they aren't coming to mind.
I think when successful, optimization really just means 'finally built right' and people forget the ridiculously inefficient ways before.
Many of the current internet titans started by making things more efficient and accessible. Google for search, Facebook for connecting to people online, Microsoft for working with PCs at a reasonable price, Amazon for buying online as well as AWS. There are examples in other industries as well, Toyota is famous for it for example. There are probably counter examples but efficiency gains can be a huge deciding factor making companies successful.
I think finding significant efficiency gains with LLMs and the like may lead to qualitatively better products. Looking at people's experiences to DSV4F makes me believe that even more than before too.
I don't think people are realizing that speed can allow for categorically different user experiences that are more than just "worse than frontier capabilities but faster".
The headroom for hardware advances is a lot lower now than it has been for most of the industry's existence, when Moore's law held strong. Now we find ourselves limited by cost, physics, fab capacity, and complexity of spinning up more fab capacity.
I honestly hope to see this across all applications, games, services, operating systems, etc. We've been in a period of wasteful RAM usage for over a decade. Constraints, whatever their origin, can be a good thing.
If China makes half decent RAM I would bet more on things like 128GM of RAM being the default on low spec laptops 10 years from now.
While I do love optimized software, the hardware side, especially for PCs, has been stagnating for way too long. At least now we have a valid use case for doubling available RAM every 2-3 years again.
I had a reasonably beefy Lenovo consumer line laptop that I bought in 2011, 8GBs of RAM. Its screen hinge broke and I couldn't repair it but I'm fairly sure it was otherwise still usable in 2023-24, once the HDD was replaced with an SSD. I think even now entry level laptops are sold with 8GB of RAM.
By comparison a PC from 2000 was utterly unusable in 2012-13.
I count on a 128GB baseline in 10 years. Beyond the current atmosphere of despair, I really want to see what Apple especially is cooking. Local AI is right up their alley and the current scarcity is unacceptable for them in so many ways.
I got the feeling laptops gonna feel very different in 2036.
> NeMo Switchyard, an open source library for smart routing
> When deployed, NeMo Switchyard can intelligently direct each request to the most capable and suitable model for the job
How do routers like this handle prompt caching when you send the second request?
Sticky models per session? but then the second message of that session won't be sent to a suitable model, and will only be sent to the same model as previous one.
I've seen ones that are configurable to pick a trade off point between lower cost (cache stickiness) and routing performance (best model for that turn).
But yeah I'm skeptical all this overhead is worth it.
Caching should be possible as long as all the models use the same shared cache. The models don't even need to be running on the same server if the shared cache is distributed.
I have a feeling people reading this are thinking that a model router would be used to route between different providers. And in that case, a shared cache would be impossible, although some caching would still be effective. I think, ideally, a router like this is in front of a set of models hosted in one place.
Prompt caching isn’t about caching the literal text of the prompt. It’s about caching the result of running prefill on the prompt (or, equivalently, the result of generating the prompt one token at a time by autoregressive inference, or some combination of the above in the case of speculative decoding). This is often called the “KV” cache, and it is very model-specific.
The repo is probably a better entrypoint to it, bit more concise description than the press releases: https://github.com/NVIDIA-NeMo/Switchyard (Notably: "Experimental software. Not for production use."). Unclear if they actually want you to deploy it or not, press release says yes, README says no, do with that what you will.
Doesn't seem to mention "cache" in the README nor the docs, but the code has mentions of it (https://github.com/search?q=repo%3ANVIDIA-NeMo%2FSwitchyard+...), I'm not sure what their thinking is there. "Good luck" essentially? Seems to be per-provider at best, but weird position for a routing library to take.
To keep it simple, forget about routers and imagine you're in Cursor using GPT for a while, reaching a cache of says 200k.
You decide to switch to DeepSeek in the same session via the model picker, and continue as usual. What happens is that the cache for DeepSeek is created with the 200k + the incremental message. After this, cache can be kept warm for both models; two instances of the cache exists, one for GPT and one for DS.
You switch back to GPT. The whole session is sent to the model with the 200k original from GPT and the incremental messages you sent to DS. The 200k is read from cache and the incrementals are new, and then added to the cache.
Let's say every second message you switch between GPT and DS; cache was 200k and each incremental message is 1k. If you kept going with only GPT, cache hit rate would be 200k/(200k+1k) = 99.5%. When you switch between two models with warm cache, hit rate instead becomes 200k/(200k+2k) = 99%.
Model routers work the same way. Keep the cache warm, replicate it in two places. For this reason, when you set up your model pool for routing, you want to keep the model pool small and differentiated.
To elaborate, because I don't think some of the people reading this understand the reason, typically a lot or most of the cost in "agentic" API usage is cached read + generation. Cached read costs scale with turn count, which multi-model switching doesn't increase, and of course generation gets cheaper if you do some of it with a cheaper model. When you switch models the "catching up" batch of messages is just a single prefill and then that goes into cache. You don't even need to have the same chat history across models so long as the view from each model's perspective looks like a series of appends.
The main problem with model routing in my experience is that to work well the router needs to be pretty strong, maybe even moreso than any of the actual models in service. There are probably clever solutions to this but I haven't seen any that look better than just using sub-agents.
Given model A with cache C(a) and model B with C(b)
Isn't this not true because the moment you switch models from A to B, you need to provide C(b) the latest conversation diff since C(b) last updated, say many turns ago?
If you take 10 turns with a model A, it has to read the cache 10 times and write a lot of tokens (the expensive part). Switching to model B is just prefilling the diff + your new message, which is still just one turn. So total number of turns does not increase for a long session even with many switches.
I didn't understand this before reading the sibling comments so I'm not sure I got it fully right but I think the total cost becomes like this:
* Input tokens: Pay for both models
* Output tokens: Pay for the model that generates
* Cached tokens: Pay per turn, so in total a weighted average over both models?
Since output tokens are the most expensive, I can see how this is an overall win for many use cases as benchmarks also show. The hard part is routing correctly.
See sibling answer but essentially the effectiveness of cache is not diminished by having a separate one per model (relative to the win of doing more turns and generation with a cheaper model).
I love the wave of new small model releases. Pleasantly surprising that an NVIDIA model runs so well on Apple Silicon using MLX! I was using nemotron-3.5-lightning:30b-mlx with OpenCode on my old (cheap) Mac this morning and no bad experiences except for running slowly.
They conveniently decided not to include the Qwen range of models in the Artificial Analysis graph, except the out-of-league Max variant. At least be brave and honest.
Not by the looks of it, but it got me thinking, currently in the middle of Level1Techs coverage on the model and switchyard and he mentions "how easy it is to customize it". Fully admitting that I haven't yet read the docs, my issue with that is "we can train LORAs for 35B as well, why use this (according to benchmarks) worse model for customization instead of a slightly bigger better one?"
Assuming I eat my words after going through the docs and this is actually a more efficient model / loras adapt better, I don't see as much value in it as is, as a REAP of it (remove least-important experts, domain-locked tests show ~98% retained accuracy) to something like 20B-A3B (rouhgly matching gpt oss, which while a good model, is outdated knowledge-wise and not as good with tool in my xp).
Having a 20B-A3B model at q4 that has a lora to be your local orchestrator (delegating coding to server/cloud models) and ci/cd runner does start sounding like an appealing proposition to me, as that would fit in 16gb vram easily (fitting many consumer gpus and 24gb macs).
The suite is across many categories, not only coding, and most of the tasks are low-horizon (or what the opposite of long-horizon is), where the max thinking time is around 10 minutes.
Gemini models are really smart, unfortunately they don't play well with any harness, so hard to use in practice.
But try them out for one-shot tasks, they are really good. Don't use them for coding in a harness, but you can ask them to generate code/planning (still, for coding only other models are indeed recommended).
loading the model would be similar vram footprint, correct, however the size of KV is based on 'active' params, not total params. So while at 1k ctx both will be in the same ballpark vram footprint-wise, at 100k the story will be very different. 27B at q4 kv for 256k takes ~8gb, while 35B at q4 kv around ~3.5gb, so at full precision kv those would be ~32gb and ~14gb (all ballparks, if you want exact numbers, its not hard to test).
As for the "cost", here i think the interesting arguments are around speed vs accuracy/"getting the job done", not literal $ cost per token.
With the benefit of LLMs already being proven, in a couple of years we will have vastly better hardware for inference I guess.
I feel like now hardware is stagnating a bit, because the software side has moved too fast for the hardware to catch up. Once we settle on sone good, optimal software architecture for the models, dedicated hardware will easily increase throughout by 10x or 100x, for a fraction of the most.
LLMs seems quite simple, maybe we'll be able to print at home our own chips with the models.
I'm looking forward to seeing what types of new things people create over the coming years once there is less obsession with massive unwieldy LLMs. I think the incentives are just too strong to ignore.
This is 100% true for pretrains, likely true for RL as well although maybe there is some benefit to smaller activated params there. There is of course 0 benefit to small dense models relative to large sparse ones that are equally as memory efficient if you have enough computers.
Many on HN are in deep denial about this imo.
I'm having an extremely hard time thinking of companies that have prospered due to software optimization. Most of them were swept away by hardware advances, instead.
I think when successful, optimization really just means 'finally built right' and people forget the ridiculously inefficient ways before.
They weren’t quantitatively better than previous companies. They were qualitatively better.
I don't think people are realizing that speed can allow for categorically different user experiences that are more than just "worse than frontier capabilities but faster".
And I'd argue "hardware advances" are more proof of optimization.
While I do love optimized software, the hardware side, especially for PCs, has been stagnating for way too long. At least now we have a valid use case for doubling available RAM every 2-3 years again.
I had a reasonably beefy Lenovo consumer line laptop that I bought in 2011, 8GBs of RAM. Its screen hinge broke and I couldn't repair it but I'm fairly sure it was otherwise still usable in 2023-24, once the HDD was replaced with an SSD. I think even now entry level laptops are sold with 8GB of RAM.
By comparison a PC from 2000 was utterly unusable in 2012-13.
I got the feeling laptops gonna feel very different in 2036.
> When deployed, NeMo Switchyard can intelligently direct each request to the most capable and suitable model for the job
How do routers like this handle prompt caching when you send the second request?
Sticky models per session? but then the second message of that session won't be sent to a suitable model, and will only be sent to the same model as previous one.
But yeah I'm skeptical all this overhead is worth it.
I have a feeling people reading this are thinking that a model router would be used to route between different providers. And in that case, a shared cache would be impossible, although some caching would still be effective. I think, ideally, a router like this is in front of a set of models hosted in one place.
Prompt caching isn’t about caching the literal text of the prompt. It’s about caching the result of running prefill on the prompt (or, equivalently, the result of generating the prompt one token at a time by autoregressive inference, or some combination of the above in the case of speculative decoding). This is often called the “KV” cache, and it is very model-specific.
Doesn't seem to mention "cache" in the README nor the docs, but the code has mentions of it (https://github.com/search?q=repo%3ANVIDIA-NeMo%2FSwitchyard+...), I'm not sure what their thinking is there. "Good luck" essentially? Seems to be per-provider at best, but weird position for a routing library to take.
prompt-cache won't work with these
You decide to switch to DeepSeek in the same session via the model picker, and continue as usual. What happens is that the cache for DeepSeek is created with the 200k + the incremental message. After this, cache can be kept warm for both models; two instances of the cache exists, one for GPT and one for DS.
You switch back to GPT. The whole session is sent to the model with the 200k original from GPT and the incremental messages you sent to DS. The 200k is read from cache and the incrementals are new, and then added to the cache.
Let's say every second message you switch between GPT and DS; cache was 200k and each incremental message is 1k. If you kept going with only GPT, cache hit rate would be 200k/(200k+1k) = 99.5%. When you switch between two models with warm cache, hit rate instead becomes 200k/(200k+2k) = 99%.
Model routers work the same way. Keep the cache warm, replicate it in two places. For this reason, when you set up your model pool for routing, you want to keep the model pool small and differentiated.
First principles of model routing: https://try.works/first-principles-of-model-routing
role-model router and protocol: https://github.com/try-works/role-model
note: edited to keep the answer to the below message clearer
The main problem with model routing in my experience is that to work well the router needs to be pretty strong, maybe even moreso than any of the actual models in service. There are probably clever solutions to this but I haven't seen any that look better than just using sub-agents.
Given model A with cache C(a) and model B with C(b)
Isn't this not true because the moment you switch models from A to B, you need to provide C(b) the latest conversation diff since C(b) last updated, say many turns ago?
I didn't understand this before reading the sibling comments so I'm not sure I got it fully right but I think the total cost becomes like this:
* Input tokens: Pay for both models * Output tokens: Pay for the model that generates * Cached tokens: Pay per turn, so in total a weighted average over both models?
Since output tokens are the most expensive, I can see how this is an overall win for many use cases as benchmarks also show. The hard part is routing correctly.
Source?
- problem: massive deluge of information because of AI
- solution: human beings should adopt a minimalist style of communicating in writing.
- e.g. this entire website page can be ten bullet points.
We can’t have legitimate debate if we have to assume a few bad actors are cloning their voice by the thousands, poisoning debate.
If we can pin one account to a real person, we won’t get rid of LLM-content and misinformation, but at least we can hold them accountable.
Or do you have something else in mind ?
I'm not sure people really want that
Assuming I eat my words after going through the docs and this is actually a more efficient model / loras adapt better, I don't see as much value in it as is, as a REAP of it (remove least-important experts, domain-locked tests show ~98% retained accuracy) to something like 20B-A3B (rouhgly matching gpt oss, which while a good model, is outdated knowledge-wise and not as good with tool in my xp).
Having a 20B-A3B model at q4 that has a lora to be your local orchestrator (delegating coding to server/cloud models) and ci/cd runner does start sounding like an appealing proposition to me, as that would fit in 16gb vram easily (fitting many consumer gpus and 24gb macs).
https://aibenchy.com/compare/meta-muse-glimmer-30b-xhigh/nvi...
but
Qwen 3.8 27B is dropping this week...
Even though that model is already "old", qwen was way ahead everyone else in that size category before this Meta model.
Also, probably for non-Chinese usage, using a non-Chinese model might lead to better results.
The suite is across many categories, not only coding, and most of the tasks are low-horizon (or what the opposite of long-horizon is), where the max thinking time is around 10 minutes.
Gemini models are really smart, unfortunately they don't play well with any harness, so hard to use in practice.
But try them out for one-shot tasks, they are really good. Don't use them for coding in a harness, but you can ask them to generate code/planning (still, for coding only other models are indeed recommended).
Happy to hear what would make the website more useful.
I will add model sizes (total/active params) for each model, good point.
The cost is similar vram footprint I guess (?)
As for the "cost", here i think the interesting arguments are around speed vs accuracy/"getting the job done", not literal $ cost per token.
If the entire model fits in vram, won't the tps be comparable?