Perplexity AI Releases WANDR: An Open Benchmark to Consider Analysis Brokers Needing to Search Broadly and Deeply

Analysis brokers as we speak already deal with actual data work. Groups delegate battle mapping, due diligence, and literature opinions. Nonetheless, most benchmarks check a single reply relatively than a big assortment backed by proof. Perplexity targets gaps with new open benchmarks.

Confusion is launched wandle (Broad and deep research). An open benchmark and analysis harness. It’s constructed round 500 life like and difficult knowledge assortment duties for data work. WANDR is a broader sibling to Perplexity’s DRACO benchmark for deep analysis. DRACO asks whether or not brokers are producing lengthy experiences which can be correct, full, and goal. WANDR as an alternative asks if it may well construct a big assortment containing the proof.

What’s WANDR?

At its core, WANDR checks two requests collectively. vast This implies discovering a big and sometimes limitless set of eligible entities. deep This implies completely researching all organizations to assist every declare with proof. Combining each adjustments the agent drawback. It is not sufficient to simply give a number of compelling examples right here. A complicated narrative constructed on incomplete analysis additionally falls quick.

To seize this, WANDR makes use of composables. Modifier key hierarchy. One activity could require firm(n) -> worker(m) -> url(ok). Which means that there are n eligible firms, every with m staff, and every with ok assist pages. All full paths within the tree are verified individually. The identical construction can characterize a flat listing, nested search, or matrix.

Examples of particular duties

To ascertain this hierarchy, the launched ceo_cfo_appointments activity. At the least 70 U.S.-based firms are being requested to take action. Every particular person should have a primary introduced CEO or CFO appointment between March 1 and April 30, 2026. One authoritative appointment web page is supplied for every agent. The subtask provides a listing permissions web page for every firm. This activity requires a complete of 140 source-supported data.

Particularly, two hierarchies and one submitted document would appear to be this:

# Process hierarchies
firm(70) -> company_appointee(1) -> url(1)   # 70 appointment data
firm(70) -> url(1)                           # 70 itemizing data

# One document the grader re-fetches and re-checks (values are illustrative)
{
  "merchandise":     "Instance Corp - new CFO",
  "url":      "https://issuer.instance.com/press/cfo-appointment",
  "excerpts": ["Example Corp today named Jane Doe as Chief Financial Officer, effective April 2026."],
  "reply":   "Jane Doe appointed CFO; introduced April 2026"
}

Practical duties generated at scale

WANDR builds duties primarily based on real-world utilization, not only a single instance. This begins with anonymized patterns seen in manufacturing relatively than artificial prompts. A semi-automated pipeline then converts these patterns into duties. The pipeline runs by 4 levels: seeding, authoring, admission, and curation. Makes use of interleaved writer and critic loops with mechanical linting.

Because of this, the median activity requires 50 members and 245 data total. Throughout all 500 duties, WANDR requires 170,495 source-backed data. The duties are divided into 167 low issue, 166 medium issue, and 167 excessive issue. Issue is decided not solely by dimension but additionally by the work completed on every document.

How WANDR evaluates brokers

Not like fastened reply keys, WANDR grades every declare towards the proof cited. Each document contains an merchandise, a URL, a particular excerpt, and a solution. The grader re-fetches the web page through the analysis. Checks if the web page is on the market and in scope. Subsequent, confirm that the excerpt truly seems and helps all of your necessities.

These binary document verdicts are rolled up in a hierarchy. accuracy Measure the standard of what your system sends. recollection Measure quality-adjusted completeness and fill in gaps with zeros. mushy The rating offers partial credit score to incomplete members. troublesome The rating solely counts members whose full subtree is right.

Benchmark outcomes

Utilizing this technique, Perplexity ran six manufacturing methods with all 500 duties. Led by our proprietary Search as Code (SaC) system. Nonetheless, no system comes near fixing the benchmark.

With extra effort, Perplexity reaches 0.447 in mushy F1. xhigh setting. The price of the whole setup is over 4 orders of magnitude. Costs vary from $0.03 per activity to $324.83 per activity.

Past the leaderboard, 4 findings stand out. First, partial advances are widespread however by no means absolutely lined. All methods have mushy recall lower than mushy precision. Second, scale makes the issue exponentially worse. Deeper hierarchies are most dangerous as a result of they add factors of failure at every department. Third, discovery is the primary structural bottleneck. The highest-level detection completion ranges from 0.611 to 0.951 for the whole system. A lot of the lacking quantity is defined by underdelivery relatively than duplicate merging. Fourth, discovering obtainable pages is often straightforward. The troublesome half is making it full proof. On Perplexity, 41.4% of pages fail to fulfill substantive necessities. Moreover, 57.5% of the excerpts don’t assist the total declare. Smooth F1 is 0.531 for the search-only verify and drops to 0.363 for the total judgment.

Particularly, search-as-code matches the form of this activity properly. Brokers can programmatically categorical logic corresponding to retrieval, filtering, fanout, joins, deduplication, and stopping. Deterministic computation then handles operations which can be repeated outdoors the mannequin context.

Utilization and examples

In actuality, WANDR maps to jobs that your staff is already automating. Market analysts want all eligible rivals and matching proof for every. Due diligence groups require dozens of firms, plus possession, administration, and financing. Expertise discovery requires many candidates, and every candidate wants a supported profile web page. WANDR exactly checks these broad and deep assortment patterns at an expert scale.

The analysis is completed on a record-by-record foundation, permitting the staff to pinpoint the placement of failures. A scoring tree separates losses for discovery, enhancement, or proof extraction. This diagnostic helps engineers enhance weak levels one by one.

Vital factors

• WANDR is an open benchmark with broad and deep duties requiring 500 proofs.
• The duty makes use of a credential key hierarchy that’s validated for every go.
• No references are required for grading. The grader retrieves and checks the cited proof.
• Perplexity Search as Code leads with mushy F1 at 0.363 and onerous F1 at 0.133.
• Discovery and full proof stay the most important failure factors.

Please verify technical details and Click here for the report. Additionally, be at liberty to observe us Twitter Do not forget to affix us 150k+ML subreddit and subscribe our newsletter. hold on! Are you on telegram? You can now also participate by telegram.

Have to companion with us to advertise your GitHub repository, Hug Face Web page, product launch, webinar, and so forth.? connect with us