Generative AI deployment and hosting options
As we consider which types of use cases we are looking to pursue to provide business value, we must consider the infrastructure on which we will deploy and host our systems. With the new normal of leveraging cloud resources, we tend to assume that capacity is not a concern anymore, but is this right? Let’s dissect this thought – is the biggest model the right solution for all use cases? Realistically speaking, LLMs are nice and easy to test and get initial results, but when considering scale and productionalization, they are not as appealing as you would think. Some of the limitations are GPU availability, cost, and latency. This realization is steering the market into more specialized smaller models that solve a specific use case.
Designing product architecture for LLMs requires careful consideration of several factors. Cost optimization strategies like Mixture-of-Depths can be employed to dynamically allocate resources...
