About LBRCrawl
LBR's page reader.
What it is
LBRCrawl reads public web pages about places in Spain, the Canary Islands and Portugal. We use what it reads to build accurate travel guides that take neurodivergent and sensory needs into account.
Every request it sends names itself with this user agent:
LBRCrawl/1.0 (+https://lbrcrawl.com/)How it behaves
It reads your
robots.txtbefore it first fetches anything from your site, and reads it again regularly.It obeys every group in your
robots.txtthat names LBRCrawl, and every group that names the name it used before.If one of those two names has no group of its own, your
User-agent: *groups apply to it instead. Where the answers differ, the stricter one wins. So aUser-agent: LBRCrawlgroup cannot let it fetch more than yourUser-agent: *groups allow, unless you also write a group for its old name.If your
robots.txtcannot be reached because of a server error or a timeout, it fetches nothing from your site.One exception: when a page redirects it to another address, it goes there without checking
robots.txtfor that address. If the new address is on another host name, such as www.example.com instead of example.com, it also goes there without pausing.It sends one request at a time to your site and pauses between requests. It pauses for longer when your
robots.txtsets aCrawl-delay.If your site answers
429 Too Many Requests, it stays away from that page for at least an hour, or for as long as yourRetry-Afterasks, up to one day.It does not log in, fill in forms or run JavaScript.
It usually comes back to a page about once a month: sooner when the page has changed, and less often when it has not.
The surest way: robots.txt
Add this to your robots.txt. LBRCrawl reads your robots.txt before its first visit and reads it again regularly. While the copy it last read disallows a page, it does not fetch that page.
User-agent: LBRCrawl
Disallow: /To keep it out of one folder only:
User-agent: LBRCrawl
Disallow: /private/What we do with your pages
We keep a copy of each page we read, so that every fact we take from it can be checked against its source.
Or ask us directly
No file to edit? Type your website's address and press the button.
When your request reaches us, we:
stop fetching your site and its www. form,
robots.txtincluded;stop picking it from our list of sites to read;
keep a note that you asked: the address and the time, nothing about you;
record it even while we are paused.
example.com covers www.example.com, not shop.example.com.
We cannot promise how long this takes. robots.txt works whether or not you send this.
We do not ask for your name or your email address. Pages we read before your request are kept.
En español
LBRCrawl es el rastreador web de LBR Systems. Lee páginas públicas sobre lugares de España, Canarias y Portugal para crear guías de viaje precisas que tienen en cuenta las necesidades sensoriales y de neurodivergencia.
Respeta su robots.txt: lo lee antes de visitar su sitio por primera vez y vuelve a leerlo con regularidad. Para bloquearlo, añada:
User-agent: LBRCrawl
Disallow: /¿Prefiere no editar ningún archivo? Escriba la dirección de su sitio web y pulse el botón. Cuando recibimos su solicitud, la añadimos a la lista de sitios que LBRCrawl deja en paz. No podemos prometer cuánto tarda. El robots.txt funciona tanto si envía esto como si no. Solo guardamos la dirección que escriba y la hora. No pedimos su nombre ni su correo electrónico. Las páginas que leímos antes de su solicitud se conservan.
example.com incluye www.example.com, pero no shop.example.com.
Em português
O LBRCrawl é o rastreador web da LBR Systems. Lê páginas públicas sobre lugares em Espanha, nas Canárias e em Portugal, para criar guias de viagem precisos que têm em conta as necessidades sensoriais e de neurodivergência.
Respeita o seu robots.txt: lê-o antes de visitar o seu site pela primeira vez e volta a lê-lo com regularidade. Para o bloquear, acrescente:
User-agent: LBRCrawl
Disallow: /Prefere não editar nenhum ficheiro? Escreva o endereço do seu site e carregue no botão. Quando recebemos o seu pedido, acrescentamo-lo à lista de sites que o LBRCrawl deixa em paz. Não podemos prometer quanto tempo demora. O robots.txt funciona quer envie isto quer não. Guardamos apenas o endereço que escrever e a hora. Não pedimos o seu nome nem o seu e-mail. As páginas que lemos antes do seu pedido são guardadas.
example.com inclui www.example.com, mas não shop.example.com.