Avoid Re-encoding Reference Images in Vision-LLM When Comparison Criteria Are User-Defined

Hi everyone, I’m working with a Vision-LLM (like Qwen-VL / LLaVA / llama.cpp-based multimodal models) where I need to compare new images against reference images. The key part of my use case is that users define the comparison criteria (e.g., fur length, ear shape, color patterns), and I’m using image-to-text models to evaluate how well a new image matches a reference according to these criteria. Currently, every time I send a prompt including the reference images, the model re-encodes them from scratch . From the logs, I can see: llama-server encoding image slice... image slice encoded in 3800–4800 ms decoding image batch ... Even for the same reference images, this happens every single request , which makes inference slow. Questions: Has anyone dealt with user-defined comparison criteria