Crawler policy

If you've landed here from your server logs — hello. Here's exactly what our crawler does, and how to stop it in one line if you'd rather we didn't.

How to block us

Add this to your robots.txt and we will stop on our next visit:

User-agent: FindEquipmentSalesBot
Disallow: /

Or email takedown@findequipment.com.au and we'll add a permanent domain-level suppression within 48 hours — no justification needed, and it survives future crawls. See our takedown page.

What we do

• We read and obey robots.txt, including Crawl-delay, and we honour X-Robots-Tag, noindex, noarchive and noimageindex directives.
• One request at a time per site, with a minimum 2-second gap between requests, and a hard cap of 60 pages per crawl.
• We back off on 429 and 503 responses and honour Retry-After.
• We use conditional requests where your server supports them, so unchanged pages cost you almost nothing.
• We prefer your sitemap.xml over crawling pagination, because it's cheaper for both of us.

What we don't do

• We do not rotate IP addresses or disguise our user-agent to get around a block.
• We do not attempt to bypass CAPTCHAs, logins, paywalls or hotlink protection.
• We do not copy your photographs to our servers. Images are displayed from your own URLs, or not at all.
• We do not reproduce your long-form listing descriptions. We store structured specifications and, at most, a short factual excerpt, then link to you for the rest.
• We do not index private-seller listings or collect personal contact details.

What we store

Structured fields only: make, model, year, price and stated GST basis, odometer or hours, condition, stock number, location, category, specification values, image URLs, and the URL of your listing page. Plus the timestamp of each observation, so we can show buyers how current the price is.

Why we're crawling you

Find Equipment Sales is an independent search platform for buyers of trucks, prime movers, trailers and heavy machinery. Every listing links directly to your own page — we don't take enquiries on your behalf, we don't charge you anything, and we don't sell placement. If you'd like to supply a proper feed instead of being crawled, we'd prefer that too: contact@findequipment.com.au.