Our crawler
What FyndEmShieldBot is, what it does on your site, and how to make it stop.
CentroidAI builds brand-protection software. Part of it reads public web pages to find counterfeit, diverted and unauthorized medicines being offered for sale, on behalf of the company that owns the brand. If you are looking at your logs and wondering who we are, this page is the answer.
We identify ourselves on purpose. Our collectors used to send an ordinary browser user agent. They no longer do. We would rather you know it is us, be able to look us up, and be able to tell us to go away, than be indistinguishable from a shopper.
How to recognize us
Every request carries this user agent:
Mozilla/5.0 (compatible; FyndEmShieldBot/1.0; +https://centroid-ai.com/crawler)
and these headers:
| Header | Value |
|---|---|
From | crawler@centroid-ai.com |
X-Crawled-By | FyndEm Shield 1.0 (https://centroid-ai.com/crawler) |
Anything claiming to be FyndEmShieldBot without these is not us.
What it does
- Requests public pages and reads what they return.
- Reads images that appear on those pages, in order to compare them with the genuine product artwork filed with regulators.
- Requests pages at a modest rate, with a pause between requests.
What it does not do
- It does not attempt to sign in, and does not use any account.
- It does not submit forms, add anything to a basket, or buy anything.
- It does not attempt to bypass a login, a paywall, a rate limit or bot protection. If you refuse the request, we record the refusal and move on.
- It does not follow links off the page it was asked to look at, and does not try to map your whole site.
- It does not collect anything for advertising, and we do not sell what it reads.
How long we keep what it reads
Not long, and less than you might expect. Copies of images and page captures are held privately for a limited period and then deleted automatically. A specific item is kept beyond that only when a named person records a reason, normally because it is evidence in an active case. What survives afterwards is the record that we saw a particular page on a particular date, together with a cryptographic fingerprint, not the copy itself.
Where the source’s own terms require earlier deletion, those terms win. Material read through an online marketplace’s programming interface is deleted as soon as the underlying listing is no longer publicly available.
We do not use any of it to train artificial intelligence models.
How to block it
Add this to your robots.txt:
User-agent: FyndEmShieldBot Disallow: /
You can also block the user agent at your edge or firewall. Either works, and we will not work around it. We do not switch to a disguised user agent when we are refused.
Would rather talk to a person? Write to crawler@centroid-ai.com. If our traffic is a problem for your site, tell us and we will slow down or stop. If you think we have something of yours we should not have, say so and we will look at it. You do not need to send a legal notice to get our attention, though our Terms of Use sets out a formal copyright takedown route if you want one.
Why this exists at all
Falsified medicine is sold through ordinary-looking websites and marketplace listings. Finding it means looking at those pages the way a customer would, and keeping enough of a record that a regulator or a court can act on it. We think that work is worth doing, and we think it should be done openly rather than from behind a fake browser string.
CentroidAI, Inc. · Knightdale, North Carolina, United States
crawler@centroid-ai.com ·
Privacy Policy ·
Terms of Use