- From: med-ab <notifications@github.com>
- Date: Thu, 20 Jun 2024 09:34:06 -0700
- To: whatwg/fetch <fetch@noreply.github.com>
- Cc: Subscribed <subscribed@noreply.github.com>
Received on Thursday, 20 June 2024 16:34:10 UTC
what about X-Robots-Tag for example where rules specified without a user agent are valid for all
```js
console.log(
new Headers([
['x-robots-tag', 'noindex'],
['x-robots-tag', 'unavailable_after: 25 Jun 2099 15:00:00'],
['x-robots-tag', 'somebot: nofollow'],
['x-robots-tag', 'otherbot: noindex, nofollow'],
['x-robots-tag', 'noarchive'],
]).get('x-robots-tag')
==
'noindex, unavailable_after: 25 Jun 2099 15:00:00, somebot: nofollow, otherbot: noindex, nofollow, noarchive'
)
```
now is the noarchive rule for all or just for otherbot ?!
--
Reply to this email directly or view it on GitHub:
https://github.com/whatwg/fetch/issues/973#issuecomment-2181105460
You are receiving this because you are subscribed to this thread.
Message ID: <whatwg/fetch/issues/973/2181105460@github.com>
Received on Thursday, 20 June 2024 16:34:10 UTC