UnitListedBot

UnitListedBot is the crawler behind UnitListed, a search engine for self-storage facilities, RV parks and laundromats that are for sale. It reads public broker listing pages so buyers can search them in one place, and it links back to the original listing on every result.

How it behaves

It obeys robots.txt

Before the first request to a host we fetch and parse its robots.txt, and we honour Disallow rules and Crawl-delay for our user-agent. If robots.txt disallows a path, we do not fetch it. If your robots.txt is unreachable we treat the host as off-limits rather than assume permission.

It crawls slowly, once a night

One request at a time per host, with a delay of at least one and a half seconds between them, and a longer delay if your robots.txt asks for one. A full run happens once every 24 hours. There is a hard page budget per run, so a change on your side cannot turn into a flood on ours.

It backs off when you push back

A 429 or a 403 is treated as rate limiting, not as something to retry through. We back off exponentially and stop hitting the host for the rest of the run.

It reads listing pages, nothing else

We follow links to public listing pages that a visitor could reach without logging in. We do not attempt logins, do not submit forms, do not touch anything behind a paywall, and do not collect personal data beyond the broker contact details you publish on the listing itself.

It never crawls sites whose terms forbid it

LoopNet, Crexi, BizBuySell and CoStar are blocked in our code, independently of any configuration, because their terms prohibit automated access. That block cannot be switched on by editing a database row.

How to block it

Add this to your robots.txt and we stop on the next run, within 24 hours:

User-agent: UnitListedBot
Disallow: /

If you would rather not wait, or you want listings that are already indexed taken down, email admin@unitlisted.com and we will remove them and stop crawling the source. We do not ask for a reason and we do not argue the point.

Why you might not want to

Every listing we show links back to your page and names the listing broker. We do not host your photographs at full size, we do not reproduce your descriptions in full, and we do not sell your listing data. Buyers who find a property here go to you to transact.

Operated by Aevorn LLC. See our terms and privacy policy. The sources we currently index are listed on our sources page.