# BeaconNet web crawl policy # Public pages are fine to index. Internal operations tooling is not. User-agent: * Allow: / Disallow: /internal/noc.html