Funes & Pravda - Spider for discovering politician sources

Research & Development
,
1

Funes & Pravda

Hey everyone! Johan here, with something new: Imagine you have a straw. Into the straw, you push the internet. Out of the straw comes a list of all political position holders.

What would this straw look like?

That’s the question I’m chasing: can one automated pipeline handle the long tail of governments and institutions out there? The orgs where writing a bespoke scraper doesn’t make economic sense — regional water boards, municipal councils, specialised agencies — but collectively represent a huge gap in our structured data.

The approach so far is split into two parts:

  • Pravda — the evidence layer. A shared service that snapshots web pages and stores the raw artifacts.
  • Funes — the extraction experiments. It feeds Pravda snapshots into LLMs and extracts structured human/position pairs.

Pravda

Before we can ask a model “who runs this water board?”, we need the page. Not just the HTML, we need the rendered document. We learned from Poliloom, and are taking the opportunity to move the “capture webpage” part into a separate service.

That’s Pravda: a small FastAPI service that turns a URL into some artifacts: plaintext, rendered HTML, a full-page screenshot, and an archive of resources the page requested. It stores it all in content-addressed storage. It is basically a thin wrapper around playwright.

The goal is to have a single evidence layer that other projects, like Funes, but soon also Poliloom, can make use of to store the artifacts that extracted data is sourced from.

Funes

Right, so with the storage out of the way, let’s get back to “who runs this water board?”. What can we extract automatically from these pages, without writing a scraper?

We tackled a variant of this challenge before. Poliloom is extracting structured data from web pages, but only for pages that we know are about a certain politician. Can we do what Poliloom is doing, but for arbitrary pages? And can we identify what pages we should look at?

I think we can.

To get an idea of the random quirks of the internet, we started with a synthesized test dataset. Leadership pages from international organizations: 1049 URLs across 487 organizations.

LLMs are pretty good at structured data extraction these days, but what input do you give them? Because not every webpage is created equal, we’re leaning into multi-modality. We start with the plain text extracted from the page, however, some pages have barely any text. In those cases we also attach the screenshot.

There is pages out there that contain a forty headshot photos with names superimposed in fancy fonts. The remedy is a couple of signals that decide whether to attach a screenshot: if the plaintext is very short, or if the page has a lot of images.

We also tried letting the model decide for itself whether it needed the image, through function calling. It turns out, it will not think it needs it. Ever.

Currently, extraction is a single structured response: we ask the the model to return the page_type and the holders[] list in a single call. A holder is a human with a position. page_type is there for us to make sense of the quality of URLs we pass in.

Of the ~1000 pages processed, holders were extracted from 60%. Across those ~600 pages, ~10,000 holders were identified. Screenshots were attached in roughly one fourth of extractions, split between thin-text and image-dense pages.

The rest of the pages on which no holders were found were overwhelmingly classified as other. Looking at these, it turns out that there’s quite a couple generic pages in our test set.

Next steps

Actually look properly at the extracted data! So far, we’ve only evaluated this machinery by taking samples and comparing with source documents manually. The samples look promising, but to say anything meaningful about the quality of this LLM extraction, we’d have to compare with a known set.

Something else to dive deeper into is the page_type signal. For the testing I did currently, I wanted to know what sort of pages I was passing in. However, the real question we’re trying to answer is more along the lines of “should we recurrently scrape this page?”, the answer to which is probably contained in the page having holders or not.

Also, what other pages does a page link to? Do any of those on the same web property maybe also contain holders? Can we “intelligently” spider?

Then there is the Pravda side of the story. How does it handle all the blocks the modern internet has to offer? Think bot detection, geo-fencing and stuff like that.

There is many open questions, as I’m sure you are aware of. As always, I’d love to hear what you think, let me know! :smiley:

2 Likes
2

Many thanks for this! One question, how strong would the LLM need to perform against the known set in your view?

1 Like
3

Thanks @frederikrichter! In my view there’s 2 dimensions to that question that are similar but not the same.

The first one would be “Faithfulness” or “copy fidelity”. How likely is a name to come back altered? For short proper-name strings at small list sizes, mutation is rare but not negligible. The risk grows with output length and with how “familiar” the model thinks a corrected version of the name is.

The other question I have is: How likely are names to be dropped (getting a incomplete list)? There’s 2 factors at play there. Position bias, and Length-dependent omission in the output. LLMs perform best for information that is at the start or end of context, with information getting “lost in the middle”. They also get worse the longer lists get, basically, completeness collapses as the list to emit grows.

To keep the model “Faithful”, we can keep context short, and validate extracted data against the input to make sure no strings roll out that were not in the original document. To perform well on longer lists, there is different strategies we can apply. One thing that comes to my mind is to have the LLM generate CSS selectors for the elements that contain holder information.

Now all this gets a lot harder when there’s no textual info on a page, and we’re using images as input. Here the problem of “Faithfulness” is amplified. Language priors can outweigh the visual signals, leading to the model “reading” a plausible word, rather than the actual word. I think we can still do things like use different models (think OCR) and compare their output. I’m curious to see how well LLMs perform there. We should be able to test this by comparing the outputs we get for the screenshots of pages where we also have the textual data.

Now to come back to your question, how strong would the LLM need to perform. I think that there is no single answer here, and that the question we should try to answer is more along the lines of “What strategy do we apply when?”. I think the goal should be to find the right strategies that allow us to stay true to the source documents.

Does that answer your question?

1 Like
4

Yes, and thank you!!

5

This is interesting. One very manual approach would be to take small screenshots and then have people double keying. See https://journals.openedition.org/jtei/739

Another thought would be could you build two AI tools that are independent and only add people where both models agree?

6

Hey everyone! Time for an update on the spider-into-structured-data experiment. Last time I described the straw: push the internet in, get political position holders out. Turns out the problem is easier to reason about when you use multiple straws.

From one big question to many small ones

The first version asked one agent one big question per page: what kind of page is this, and who holds which positions? Whether the model got plain text or a tiled full-page screenshot was decided by counting words and images.

Instead of that, now we think in small jobs.

inspect: extract what we came for, store the outcome. discover: look at a captured page and decide which links on it are worth visiting next. repair: Broken pages (Cloudflare challenges, timeouts, empty app shells) get routed aside to see if we can do something to make them work still.

So it’s all jobs emitting some data or some other jobs, it makes the system easy to reason about and analyze. Every job is small, stored, and leaves a full transcript behind.

Besides the architecture switch, we now tell the model what we’re looking for. Every dataset carries a brief — the people sought, for which organization. It’s what allows the model to decide whether to return a certain person, or what links on a page look interesting. It makes the spidering bounded. A spider that knows what it’s looking for can decide when it’s done.

The screenshot is dead, long live the image

Last time, we tried letting the model decide for itself whether it needed to see the page as an image. It would not think it needs it. Ever.

In hindsight, of course it didn’t. We handed it plain text and nothing else — no indication that images existed, let alone what they might contain. Inferring from something that is missing from context is a impossible task. There was no decision to make.

Now the agent gets a compact, structured outline of the rendered page plus a tool to request any of the embedded resources individually. And it does ask for them, because now it can see what it’s missing. The modality decision stopped being a code heuristic and became part of the agent’s job.

What we learned

We ran a full dataset (~470 international organizations) twice, a day apart. Two findings stand out.

The spider works, and it’s bounded. Most pages in the second run were found by the discovery agent — and the yield per visited page drops as the crawl moves outward. The frontier shrinks.

The jitter is in the interpretation of the brief. I sampled pages where the results differed and analyzed the transcripts. Differences trace back to the model re-interpreting the brief’s boundary: are committee co-chairs “prominent leadership”? Today yes, tomorrow no.

That decision has to be made somewhere. Scoping the goal is what makes the problem tractable at all, and an explicit, written-down objective is something we can inspect and improve.

The loop, all the way up

So how do you improve it? The way we’ve been improving everything else: run the pipeline, review what comes back, distill the failures into synthetic fixtures, and run them in an eval suite to sharpen the prompt. Each agent its own eval loop.

It’s the same trick at every level. We’re improving the prompts this way. But we can apply the same loop to the briefs. Fixtures sharpen agents; reviewed runs sharpen briefs. The same machinery transfers to Poliloom, which has the same shape: brief, capture, extract, review.

Next steps

  • An answer to the jitter. Maybe it’s frequency: what gets extracted in every run is what the model is confident about, so recurrence across recrawls could be the confidence signal — with entity resolution delegated to zavod and nomenklatura, where it belongs.
  • Repair: broken snapshots currently park and wait. Bot detection and friends live there.
  • PDFs. Discovery already finds them by the dozen, but the pipeline can’t read them yet.

As always, curious what you think!