Prompt compression and API cost reduction

This part is dedicated to a recent development in resource optimization for when employing API-based LLMs, such as OpenAI’s services. When considering the many trade-offs between employing a remote LLM as a service and hosting an LLM locally, one key metric is cost. In particular, based on the application and usage, the API costs can accumulate to a significant amount. API costs are mainly driven by the number of tokens that are being sent to and from the LLM service.

In order to illustrate the significance of this payment model on a business plan, consider business units for which the product or service relies on API calls to OpenAI’s GPT, where OpenAI serves as a third-party vendor. As a particular example, imagine a social network that lets its users have LLM assistance to comment on posts. In that use case, a user is interested in commenting on a post, and instead of having to write a complete comment, a feature lets the...