NunoBot

Overview

NunoBot is an automated HTTP client operated by Nuno. It visits public pages on a client's website when Nuno runs an authorized web assessment for that client. The assessment helps identify technical, structured-data, content, and AI-discoverability improvements.

NunoBot is not a general-purpose search crawler and does not build a public search index. Content fetched by NunoBot is not used to train foundation models.

Request identity

Direct assessment requests identify themselves with the following HTTP User-Agent header:

User-Agent: NunoBot/1.0 (+https://discovernuno.com/docs/nunobot)

The product token used for robots.txt matching is NunoBot. A User-Agent string alone can be copied by another requester; if you need help validating traffic, contact us using the address below.

Cryptographic verification

Direct NunoBot requests use Web Bot Auth HTTP message signatures. Participating network providers can use these signatures to confirm that a request was sent by Nuno rather than relying on the User-Agent header alone.

Signature-Agent: "https://api.spinlink.io/.well-known/http-message-signatures-directory"

NunoBot's signed public-key directory is available at https://api.spinlink.io/.well-known/http-message-signatures-directory. It publishes only the public Ed25519 verification key; the private signing key is not exposed. Cryptographic verification does not override robots.txt or other website access controls.

What NunoBot accesses

NunoBot may request the following public resources during an assessment:

• robots.txt and sitemap files used to discover permitted pages
• The public homepage and all permitted public pages discovered for the assessment
• Public HTML, response headers, metadata, headings, links, image markup, scripts, and JSON-LD structured data

NunoBot attempts to fetch every permitted public page it discovers for the assessment. Some technical checks may make additional requests to site-control files or re-check a subset of those pages. NunoBot does not sign in, submit forms, complete transactions, or attempt to bypass authentication, paywalls, CAPTCHAs, or access controls.

Data handling

Page responses are processed to produce assessment findings. Nuno may retain the assessed URLs, page titles, selected metadata and headings, structured-data findings, technical check results, and a timestamped website-content snapshot for the client.

Raw HTML is processed during the assessment but is not stored as part of the assessment report. NunoBot does not intentionally collect form submissions, account credentials, authentication headers, payment information, or private account content.

Crawl behavior

• NunoBot checks robots.txt and evaluates rules for the NunoBot product token before fetching assessed pages.
• If your robots.txt file does not mention NunoBot, NunoBot follows the rules listed under User-agent: *.
• NunoBot waits at least one second between primary page requests. If your robots.txt file asks crawlers to wait longer, NunoBot follows that delay.
• Pages that return an error, block, or rate-limit response are not retried through a proxy during the identified NunoBot page crawl.

Enable NunoBot for an assessment

If you have authorized Nuno to assess your public website, add the following group to your robots.txt file. This explicitly allows NunoBot to access public pages:

User-agent: NunoBot
Allow: /
Crawl-delay: 1

You can keep sensitive or operational areas excluded by listing them after the allow rule. NunoBot does not need access to account, checkout, staging, or administration pages.

User-agent: NunoBot
Allow: /
Crawl-delay: 1
Disallow: /account/
Disallow: /checkout/
Disallow: /staging/
Disallow: /wp-admin/

Publish the file at https://your-domain.com/robots.txt, clear any website or CDN cache, and send the public URL to your Nuno contact. Nuno will confirm access before starting the assessment.

A robots.txt allow rule does not override a firewall, CDN bot challenge, or password protection. If your security provider still blocks NunoBot, contact us before creating an exception. Do not create a permanent security bypass based only on the User-Agent header, because that header can be copied by other requesters. We can coordinate a narrowly scoped, time-limited assessment window instead.

Control access with robots.txt

To block all NunoBot page crawling, add this group to your site's robots.txt file:

User-agent: NunoBot
Disallow: /

To block only a section of the site:

User-agent: NunoBot
Disallow: /private-section/

Changes are applied the next time an assessment reads your robots.txt file. robots.txt is a crawl preference rather than an authentication mechanism; protect non-public resources with appropriate access controls.

Questions or removal requests

If you believe NunoBot accessed a site without authorization, want an assessment stopped, or need help identifying requests, email hello@discovernuno.com. Include the affected hostname, approximate request time, timezone, and relevant server-log entries when possible.