Throughout benchmarks score fashions on reasoning and multilingual expertise, akin to BigBench, MMLU, and ARC Problem, the MoE-instruct mannequin, though with fewer parameters than rivals (6.6 billion) carried out higher than Llama 3.1-8B-instruct, Gemma 2-9b-It, and Gemini 1.5-Flash. Nevertheless, it couldn’t match the efficiency of OpenAI’s GPT-4o-mini-2024-07-18 (chat).

Nevertheless, the corporate identified that the mannequin remains to be basically restricted by its measurement for sure duties.

“The mannequin merely doesn’t have the capability to retailer an excessive amount of factual data, subsequently, customers might expertise factual incorrectness,” it mentioned, including that this weak point might be resolved by augmenting Phi-3.5 with a search engine, significantly when utilizing the mannequin beneath RAG settings.