- From: Gary Gozlan <notifications@github.com>
- Date: Thu, 20 Jun 2024 10:22:38 -0700
- To: whatwg/fetch <fetch@noreply.github.com>
- Cc: Subscribed <subscribed@noreply.github.com>
Received on Thursday, 20 June 2024 17:22:42 UTC
> what about X-Robots-Tag for example where rules specified without a user agent are valid for all
>
> ```js
> console.log(
> new Headers([
> ['x-robots-tag', 'noindex'],
> ['x-robots-tag', 'unavailable_after: 25 Jun 2099 15:00:00'],
> ['x-robots-tag', 'somebot: nofollow'],
> ['x-robots-tag', 'otherbot: noindex, nofollow'],
> ['x-robots-tag', 'noarchive'],
> ]).get('x-robots-tag')
> ==
> 'noindex, unavailable_after: 25 Jun 2099 15:00:00, somebot: nofollow, otherbot: noindex, nofollow, noarchive'
> )
> ```
You always have to put the `*bot` last in your array.
i.e. `noindex, noarchive` should be the first 2 in your output
--
Reply to this email directly or view it on GitHub:
https://github.com/whatwg/fetch/issues/973#issuecomment-2181183295
You are receiving this because you are subscribed to this thread.
Message ID: <whatwg/fetch/issues/973/2181183295@github.com>
Received on Thursday, 20 June 2024 17:22:42 UTC