Sign upSign InSign upSign InMember-only storyWenqi GlantzFollowLevel Up Coding--1Sharezephyr-7b-alpha, a new HuggingFace open source Model, emerged and raised quite some eyebrows over the past week.In this article, let’s explore zephyr-7b-alpha through the lens of EDD (Evaluation Driven Development). Using a multi-document RAG pipeline built with LlamaIndex, we will compare it with the OpenAI model GPT-3.5.zephyr-7b-alpha is a 7b parameter GPT-like model fine-tuned from mistralai/Mistral-7B-v0.1. According to HuggingFace MT Bench, a benchmark suite for evaluating the quality of machine translation (MT) systems, zephyr-7b-alpha outperforms both Llama-2-70b-chat-hf and Mistral-7B-Instruct-v0.1. See the comparison table below.LlamaIndex has performed a LLM Compatibility Tracking, from which we learn that zephyr-7b-alpha is so far the ONLY open source 7b model that does well on advanced RAG tasks.As can be seen from the screenshot above, even though zephyr-7b-alpha outperforms the other two 7b models in most of the categories, it still has its limitations as a Data Agent. Let’s make a note of it as we move on to our evaluation POC in the next section.BAAI/bge-base-en-v1.5 is a text embedding model developed by the Beijing Academy of Artificial Intelligence (BAAI). It is a large language model trained on a massive dataset of text and code. It can generate text embeddings for various NLP tasks, such as retrieval, classification, clustering, and semantic search. It is the current number 2 embedding model on HuggingFace’s MTEB (Massive Text Embedding Benchmark) Leaderboard, right after its more capable sibling bge-large-en-v1.5.We will be using BAAI/bge-base-en-v1.5 as the embedding model for our evaluation POC.----1Level Up CodingMom, wife, software architect with a passion for technology and crafting quality productsWenqi GlantzinBetter Programming--4Arslan AhmadinLevel Up Coding--19Arslan AhmadinLevel Up Coding--27Wenqi GlantzinBetter Programming--Jerry LiuinLlamaIndex Blog--1Heiko HotzinTowards Data Science--18Wenqi GlantzinBetter Programming--2Damian GilinTowards AI--5Shivam SolankiinTowards Generative AI--2Fabio MatricardiinArtificial Corner--HelpStatusAboutCareersBlogPrivacyTermsText to speechTeams