~/divyam/stories citescope-citation-finder
ai junction · ai systems
CiteScope: A Citation Finder That Knows When It Can't Answer
Give it a research abstract and it returns the papers that abstract should cite, or abstains with a reason when its 26,601-paper corpus can't help. Classical retrieval built from scratch, measured against dense and hybrid search.
- when
- October 2026
- role
- AI Engineer
- stack
- PythonNLTKNumPysentence-transformersOpenAlex APIStreamlit
Overview
CiteScope is a should-cite finder. Paste a research abstract and it returns the papers that abstract ought to be citing.
It also does something most search engines don't. When the corpus has no good candidates, it shows no list at all. Instead it explains why: which of the abstract's distinctive terms the corpus never covers, how rare they are, and what the nearest material actually is.
Problem
Search engines return something for every query. For citations, that is actively harmful. If an abstract comes from a field the corpus doesn't cover, or relies on work it doesn't hold, five confident-looking but wrong suggestions are worse than none.
So CiteScope has two jobs: rank the corpus for an abstract-length query, and, before showing anything, decide from the retrieval evidence alone whether the corpus can serve that query at all.
Solution
The corpus is 26,601 papers from OpenAlex: 16,330 information-retrieval papers from 2015 onwards, plus 10,271 works they cite. Each paper's own reference list is the ground truth, and the retriever never sees it.
The ranking core is classical information retrieval written from scratch, with no IR library: tokenising and stemming, an inverted index with word positions and separate title, abstract and venue zones, tf-idf with cosine scoring, heap-based top-K, champion lists, index elimination and a small citation-based quality prior.
BM25, dense retrieval with MiniLM embeddings, and a hybrid that fuses the tf-idf and dense rankings sit behind the same interface, so every approach is measured the same way.
Abstention reads the shape of the result, not an LLM's opinion: how much of the query the corpus covers, how the scores fall away, how high the top score is, how rare the query's terms are, and how alike the top results are. A small logistic model, about thirty lines with no library, weighs these signals into one of three verdicts: confident, partial or insufficient.
Evaluation
Retrieval was measured on 400 held-out abstracts. Hybrid search reached an nDCG@10 of 0.184 against 0.152 for plain tf-idf, a 21% improvement (p < 0.001, paired randomisation test). Tuned tf-idf alone reached 0.169.
Abstention was tested on 379 cases the model never saw: 180 answerable abstracts, 49 whose references sit outside the corpus, and 150 abstracts from biology and economics. When CiteScope abstains it is right 95.4% of the time, and it turned away 148 of the 150 out-of-field abstracts.
Against an embedding baseline that simply thresholds the top similarity score, overall accuracy is statistically the same (F1 0.893 against 0.890). They fail differently, though: CiteScope wrongly refuses 4% of answerable queries, the baseline 10%. And every CiteScope refusal names the terms behind it.
Highlights
- Inverted index, tf-idf, BM25, champion lists and top-K written from scratch
- Hybrid search: nDCG@10 of 0.184 against 0.152 for plain tf-idf on 400 held-out queries
- Abstains on retrieval signals alone, and is right 95.4% of the time it does
- Turned away 148 of 150 out-of-field abstracts
- Every refusal explains itself: missing terms, how rare they are, and the nearest material
- Leakage-safe corpus building, so test abstracts never see their own citations
- Full evaluation and ablation suite, a Streamlit demo and 32 unit tests
Limitations
About half of each query's references are not in the corpus at all, so recall has a ceiling of 0.491 and every number should be read against it.
The hardest case is only partly solved. When a paper uses the field's vocabulary but relies on work the corpus lacks, the signals look normal, and only 39% of those cases are caught.
Repository
python citescope.py cite --file my_abstract.txt # the engine on your own abstract
python citescope.py search --text "..." --explain 3 # tf-idf ranking with score breakdowns
python citescope.py eval # rebuild every number above
streamlit run app.py # local web UIOutcome
The ranking was the expected part. The harder question was when not to answer, and answering it from retrieval evidence alone, with a reason a person can check, turned out to be the most useful thing the system does.