Benchmarks

Graph-Me-Maybe against text RAG and GraphRAG, on the same documents, with the same model. Results are published here with everything needed to rerun them.

In progressThe full run is under way. This page shows numbers only once they are measured.

What we measure on

Office document setTwo invented companies (Korean and English): Excel, PowerPoint, Word, PDF and scans, with questions written against the exact cell, slide or paragraph that answers them.

MMLongBench-Doc (NeurIPS 2024)Long PDFs — reports, papers, brochures, financial filings — with questions on text, tables, charts, figures and layout, and some that cannot be answered. Reported by recent vision-language models.

MultiHop-RAG (COLM 2024)News articles where the answer spans several of them. Used by the GraphRAG paper — the home ground of graph methods.

Systems

Graph-Me-Maybe; text RAG (text extraction, chunks, vector search); Microsoft GraphRAG, standard and fast indexing; LightRAG. Every system runs its own default settings and the same model, Claude Haiku 5.5. A different model, Claude Opus 5.5, grades the answers without knowing which system wrote them.

What we report

Accuracyby question type — including the types where we lose

Source accuracywhether the answer pointed to the right document and page

Build time and costtime, model calls, tokens and list-price dollars to index the documents

Stored sizewhat each system keeps on disk, against the documents

Per questiontokens and seconds to answer

Rerun it

The harness, the question sets, the raw answers, the judge’s verdicts and the run date will be published with the results. Public datasets are downloaded from their sources, not redistributed.

Limits

One model family for every system; an LLM judge, spot-checked by hand; our own office set is written by us. These are stated beside every result.