Sentry for AI Agents
Zubin Koticha, Founder
Alexis Gauba, Founder
Ben Hylak, Founder
https://www.youtube.com/watch?v=pN82WxN-_G0
Today, we're excited to launch Raindrop Deep Search
Itโs like Deep Research for your Production AI Data
Search for anything, and Raindrop automatically trains little models to accurately classify any topic or issue, across millions of events.
๐ง๐ต๐ฒ ๐ฃ๐ฟ๐ผ๐ฏ๐น๐ฒ๐บ
Weโve heard from thousands of AI engineers and theyโre struggling to track issues with their agents.
Imagine a user reports a problem: your agent is saying it canโt search the web for documentation. You need to know if this is a one-off problem or a much bigger issueโฆ but how? Keyword search, or even semantic search, doesnโt tell the full story.
๐๐ฎ๐ปโ๐ ๐๐ฒ ๐ท๐๐๐ ๐๐๐ฒ ๐ง๐ฟ๐ฎ๐ฑ๐ถ๐๐ถ๐ผ๐ป๐ฎ๐น ๐๐๐ฎ๐น๐?
Offline evals work well as unit tests. But since theyโre running on preset data, you have no visibility into whatโs actually happening in production.
Online evals just run these unit tests on a tiny sample of production data, leaving you blind to how widespread problems are.
๐๐ป๐๐ฟ๐ผ๐ฑ๐๐ฐ๐ถ๐ป๐ด ๐ฅ๐ฎ๐ถ๐ป๐ฑ๐ฟ๐ผ๐ฝ ๐๐ฒ๐ฒ๐ฝ ๐ฆ๐ฒ๐ฎ๐ฟ๐ฐ๐ต
Thatโs why we built Deep Search. Itโs like Deep Research for your production data.
How Deep Search works:
1. Describe the issue (eg. agent failing to search the web)
2. Deep Search finds examples out of millions of events
3. Refine search with feedback
4. Start tracking the issue
Deep Search runs across all of your production data to give you an accurate metric of issue frequency.
Traditional classification systems require humans to manually label thousands of data points. So to achieve this, Raindrop Deep Search introduces a new research breakthrough, bespoke few-shot classifiers, which only need a few examples.
Itโs essentially bootstrapping weaker systems from stronger systems, ultimately training custom small models that analyze millions of events a day. You can think of it like creating materialized views for natural language.
Once you start tracking the issue you can use Raindrop to dive into traces and tool calls to find the root cause. And you can quickly confirm whether your fixes are effective by monitoring issue frequency and receiving real-time Slack alerts.
You can try out Deep Search at raindrop.ai.
Weโre excited to hear what you think!
