FixedLink

Crawler information

FixedLinkBot

You are probably here because you found this in your server logs.

FixedLinkBot/1.0 (+https://fixed.link/bot; link preview fetcher)

What it does

FixedLink is a URL shortener. When someone creates a short link, we fetch the destination page once and read its <head> so the short link unfurls with the right title, description and image when it is pasted into Slack, LinkedIn or a message. We re-read it occasionally in case the page changed.

Separately, a customer who has proved they control a domain can ask us to index it, so they can see which of their own pages have short links and which do not. That reads your sitemap and then the pages it lists. It only ever happens for a domain whose owner has published a DNS TXT record proving control of it, and it only covers that hostname.

We never execute JavaScript, never submit forms, never follow links behind a login, and never store page content beyond the metadata described above.

How it behaves

  • It reads /robots.txt before crawling and obeys it, including Crawl-delay.
  • If /robots.txt is unreachable or returns a server error, we do not crawl. We treat that as your server asking for quiet.
  • It requests one page at a time per host, at least a second apart.
  • It gives up on a page after 8 seconds and reads at most 512KB of it, so it will not hold a connection open.
  • It follows at most 5 redirects and does not crawl off your hostname.

Slowing it down

Put this in your robots.txt and we will wait ten seconds between requests.

User-agent: FixedLinkBot
Crawl-delay: 10

We cap the delay we will honour at 30 seconds. Past that the honest answer is to crawl fewer pages rather than the same pages more slowly, so use Disallow to narrow what we look at.

Stopping it

This blocks the crawl entirely.

User-agent: FixedLinkBot
Disallow: /

One thing to know before you do. This also stops us reading the page metadata for links pointing at your site, which means short links to your pages will unfurl with no title, no description and no image wherever they are shared. Most sites would rather keep that and restrict the crawl to the parts they care about:

User-agent: FixedLinkBot
Disallow: /admin/
Disallow: /search
Allow: /

Something else wrong

If FixedLinkBot is behaving badly - ignoring these rules, requesting too often, or hitting something it should not - mail abuse@fixed.link with a few log lines and we will fix it. Include the timestamps; they let us find the customer and the property behind the request.