PoliLoom – Loom for weaving politician's data

[Monneyboi]

@frederikrichter, @leon.handreke & others I’ve forgotten. Thank you for the kind words and support!

Devlog #3

Welcome back! Strap in, this is a long one.

So the main thing we figured out is that we have to many positions to send to the OpenAI structured data API. This led us into similarity search territory, time to have fun with vectors.

Let me explain how our tactic changed over time. First we did this:

1. Fetch the Wikipedia article for a politician
2. Fetch positions that the politician could have based on his citizenship
3. Pass those positions to the structured data OpenAI API.
4. Get back mostly correct positions for the politician

To me, this is the greatest way to do it. The LLM can only output positions that we have a Wikidata ID for.

However, looking at our positions data:

select countries.name, count(position_country.position_id) as pos_count from countries left outer join position_country on position_country.country_id = countries.id group by countries.id order by pos_count desc;

We see that we have quite a couple positions for some countries:

FR	78871
ES	13977
BR	5890
US	4264
...

And the OpenAI structured outputs API has some pesky limitations that only allow us 500 enum properties

What now?

So we’ll have to find the positions that are most likely to be in our Wikipedia article. Enter similarity search. If you’re interested in how that works, Here is part of my talk that tries to explain it.

So now, we started of with embedding all the positions, what we did next was:

1. Fetch the Wikipedia article for a politician
2. Embed the whole article
3. Fetch positions that the politician could have based on his citizenship
4. Select the 500 positions that are most similar to the article.
5. Pass those positions to the structured data OpenAI API
6. Get back mostly wrong positions for the politician

It turns out that there is too much info in the articles to give us a reliable set of similar positions, leading to the model outputting incorrect positions, as we force it to use any of the positions we pass it.

For example, when searching for similar positions for Aurélien Pradié, “Mayor of Pradiers” will be very similar. Looking at the proof, the model has a good understanding of what positions Aurélien has had, but is forced to output tokens that match the structured data schema:

LLM extracted data for Aurélien Pradié:
  Positions (5):
    Deputy of the French Second Republic (2017 - present)
      Proof: Aurélien Pradié (French pronunciation: [oʁeljɛ̃ pʁadje]; born 14 March 1986) is a French politician who has represented the 1st constituency of the Lot department in the National Assembly since 2017.
    Mayor of Pradiers (2014 - 2018-01-05)
      Proof: In the 2014 municipal elections, Pradié was elected mayor of Labastide-Murat, when his party list received over 70% of the vote.
    Mayor of Pradiers (2016 - 2018-01-05)
      Proof: Following the merger of communities, he became mayor of Cœur-de-Causse (the new merged commune) in 2016.
    deputy to the National Legislative Assembly (France) (2016 - 2018)
      Proof: He has also held a seat in the Regional Council of Occitania since 2021, previously in office from 2016 to 2018.
    deputy to the National Legislative Assembly (France) (2021 - present)
      Proof: He has also held a seat in the Regional Council of Occitania since 2021, previously in office from 2016 to 2018.

The new strategy

So we finally arrive at the strategy I was trying to avoid. What I think we’ll have to do now is:

1. Fetch the Wikipedia article for a politician
2. Prompt the language model to extract arbitrary positions, have it return whatever.

Then for every position:

2. Check if we have a exact match with our Wikidata positions.
3. If not, fetch the 500 most similar positions
4. Pass those positions to the structured data OpenAI API.
5. Prompt the model for the correct Wikidata position, or None
6. Get back mostly correct Wikidata positions for the Wikipedia positions

That’s the theory at least.

The benefit of this tactic is that it can be generalized to other properties as well. We’ll have to do the same for the BirthPlace property for example.

Other news

While Claude was fleshing out this enrichment process, I had Claude start work on the NextJS GUI in a second working directory. Now we also have a basic-gui branch containing the start of the confirmation GUI.

Which does not much yet, however, it does Mediawiki auth (With help from Bryan Davis, thanks!)

Thank you for reading this far, I’ll try to keep the next one shorter .

Again, if anything I say sounds weird or I’m going in a direction that you think is not fruitful. Please let me know!

That was it! Have a great day!