Building the open source standard for evaluating LLM Applications
The fragmented and proprietary evaluation tools today are leading to significant inefficiencies and confusion among developers. The world needs a standard everyone can rely on and that is why we are building Ragas as the open-source standard. We have 4k stars on GitHub, 1.3k members in our discord community, and over 80+ external contributors. We also have partnerships with key AI companies like Langchain, Llamaindex, Arize, Weaviate and more to help create a standard. We already process 5 million evaluations monthly for engineers from companies like AWS, Microsoft, Databricks, and Moody’s and it is growing at 70% month over month. We are building LLM application testing and evaluation infrastructure for Enterprises.
Active Founders
Jithin James
Jithin James (jjmachan), believes that life's purpose is found in the toil in mastering a craft and the fulfilment one gets from using it for the greater good. For him, this means excelling in software development, particularly in open-source and AI tooling, to empower fellow developers and help materialize their ambitious visions into reality. His journey is a testament to skill, dedication, and a commitment to community enrichment.
Shahul ES
Shahul (aka ikka) discovered his fascination for AI in his sophomore year while pursuing his computer science degree. This early interest laid the foundation for his journey to becoming a Kaggle GrandMaster and led to his significant contributions to open-source AI, particularly in initiatives like OpenAssistant AI.
Company Launches
TL;DR
We are building Ragas — an open-source evaluation and testing infrastructure for LLM application developers to deploy their applications in production with confidence.
About us
We’re Jithin and Shahul! Having met in college, we’ve collaborated on various projects for almost a decade now.
Jithin takes care of building the software and infrastructure. He was an early employee at Bento ML, where he built and maintained tools like Bentoctl, Bentoml, and Yatai. Shahul is responsible for AI research and engineering. He is a Kaggle Grandmaster and a lead contributor to different open-source AI projects, including Open-Assistant AI.
Problem
Before 2023 software used to be written in code but with the emergence of foundational models software and applications are going to be compound systems containing code, prompts, and other components. This introduces several new problems
- How do you select the best model or component suitable for your application from the abundance of available resources?
- How do you test these systems and ensure continuous quality?
- How do you derive insights from production to measure and improve your system?
As early adopters of this technology to build applications, we faced this problem while we were building RAG systems early last year.
Solution
We at Ragas make use of model-graded evaluations and testing techniques to ensure quality. This includes automated synthesis of test data points, explainable metrics, and adversarial testing.
We started by building this for RAGs, which is the most popular application of LLM as of today. Ragas is now the default open-source standard for evaluating RAG applications, processing over 4.7 million responses last month and used by engineers from enterprises like AWS, Microsoft, Databricks, Moody’s, UHG, and Tencent.
Our Ask
YC Sign Photo
