In my spare time, I’m developing a weight-lifting workout app that will use a locally-deployed Large Language Model (LLM) (or an LLM that runs server-side) which will do the following:

Typically, large language models require significant GPU power to fine-tune and run. However, in May 2023, researchers from the University of Washington published a paper called “QLoRA: Efficient Finetuning of Quantized LLMs” that establishes several techniques that can save significant memory when fine-tuning and running an LLM without compromising on performance. Suddenly, fine-tuning and running an LLM on consumer-level hardware became a reality. In fact, the researchers go on to say they “[…] estimate that, with an iPhone 12 Plus, QLORA can finetune 3 million tokens per night while the phone is charging. While finetuned 7B models do not reach the quality of ChatGPT, we believe that the quality is good enough to enable novel applications that have not been possible before due to privacy or LLM quality issues.”