I’ve been testing a matcher for company and vessel names against OFAC, the trade.gov CSL, the UN and the EU lists, and I’d like feedback on the method more than the tool.
The thresholds are chosen on training pairs. The rates are then measured on a separate set that someone else wrote without seeing the method: 400 names that are on a list and 200 similar names that are on no list. At the strict setting it finds 332 of the 400 (95% interval 79 to 86%) with no alert on the 200 (0 to 2%).
Two things I’m unsure about. First, how to write near-matches that are hard enough without copying the method’s own blind spots. Second, vessels. When a counterparty has an IMO number, a listed vessel with the same number is a match whatever its name, and a similar name with a different number is set aside but still shown. How do you handle a vessel renamed after designation, when the list only has the old name and no number?
The test sets and the code are public: GitHub - ArslaneSempai-ui/cascade-screening: Sanctions screening for company and vessel names against OFAC, CSL, UN and EU lists, with published, blind-tested accuracy. · GitHub (source-available, not open source).
1 Like
Thanks for publishing the rates with their intervals, and for testing on names someone else wrote. Most screening tools give a threshold and nothing to judge it by.
On your vessel question: in the lists I follow, a ship entry with an old name and no IMO is rare. About 17 of some 5,000 entries have no IMO, and most are small fishing boats. The case I run into far more often is the reverse. The list has the IMO, but the ship has since taken a new name, and the paperwork only carries that. One ship shows it well.
On Aug 7 the crude tanker ASTRAL sat stopped in the Gulf of Suez, broadcasting her listed name on a Sierra Leone number. That’s the last I heard of that number. From Aug 9 a Russian number started sending her IMO, 9274800, and her 249 by 44 metre size. It called her LABRADOR. VesselFinder has her as LABRADOR under the Russian flag too.
None of the six authorities that list her uses that name. The EU, Australia, New Zealand and Switzerland have her as ASTRAL, and the UK and Canada as YANGTZE. Of your four lists, only the EU has her at all. OpenSanctions does carry LABRADOR, from Ukraine’s War and Sanctions site. What ties the new name back for me is the AIS static message, which carries the name and the IMO together. Someone types that IMO in by hand, so a match is a lead, not proof.
That may help with your first question too. Real pairs like this can’t share your method’s blind spots, because nobody wrote them. ASTRAL against LABRADOR is one ship with nothing in common but the IMO. A Cyprus bulk carrier is also called LABRADOR (IMO 9415222, on no list), which is one name on two ships. Every rename pair in your test set keeps the old name in the string, like “Barka NOGAT 12 (ex WARTA 7)”. How does your strict setting score ASTRAL against LABRADOR, with the IMO on both sides and without it?
(Disclosure: these numbers come from a vessel tracker I built.)
Thanks Josh, this is exactly the kind of case I was hoping for.
First, a correction to my post: the held-out names were not written by a person. A separate AI agent wrote them without seeing the matcher, and every name is invented. Since then I’ve measured on real company names from the GLEIF register, and recall on real spelling variants was much lower than on the invented set. It’s in verification/GLEIF.md in the repo.
I ran your ship through the published version, with the lists of 26 September:
- IMO 9274800 is on none of the five lists I load, and ASTRAL isn’t in my EU file either. My EU source is the consolidated financial sanctions file, which doesn’t seem to carry the EU’s vessel designations, so the IMO rule never gets a chance.
- By name it’s worse than a miss. ASTRAL, with or without the IMO, comes back as a strong candidate for an unrelated company, through a weak alias “AO AZ URAL”. YANGTZE does the same with “Su Yang”. LABRADOR only gets a possible candidate, also unrelated.
So on this ship the tool finds nothing true and raises two false strong alerts. Two fixes follow: load the lists that carry vessel IMOs, and stop a weak alias or one short word from reaching the strong level.
I’d gladly take you up on the data. Real renames (old name, new name, IMO, dates) would be the vessel test I can’t write myself. Could you share a sample under terms that let me publish aggregate results, with credit to HullCipher?
Thanks Arslane, and thanks for running it so quickly and sharing the results openly. Renamed ships are hard for any name matcher, so I’m glad this one was useful.
On the EU miss: the financial sanctions file has no ships. The EU publishes its designated vessels as a separate list, and that is where ASTRAL and the IMO are.
Yes to the renames. Email me at josh@hullcipher.com and we can sort out the details there. One caution for any rename set built from AIS: crews type the broadcast name by hand, so some apparent renames are typos.