- From: Gary Gozlan <notifications@github.com>
- Date: Thu, 20 Jun 2024 10:22:38 -0700
- To: whatwg/fetch <fetch@noreply.github.com>
- Cc: Subscribed <subscribed@noreply.github.com>
Received on Thursday, 20 June 2024 17:22:42 UTC
> what about X-Robots-Tag for example where rules specified without a user agent are valid for all > > ```js > console.log( > new Headers([ > ['x-robots-tag', 'noindex'], > ['x-robots-tag', 'unavailable_after: 25 Jun 2099 15:00:00'], > ['x-robots-tag', 'somebot: nofollow'], > ['x-robots-tag', 'otherbot: noindex, nofollow'], > ['x-robots-tag', 'noarchive'], > ]).get('x-robots-tag') > == > 'noindex, unavailable_after: 25 Jun 2099 15:00:00, somebot: nofollow, otherbot: noindex, nofollow, noarchive' > ) > ``` You always have to put the `*bot` last in your array. i.e. `noindex, noarchive` should be the first 2 in your output -- Reply to this email directly or view it on GitHub: https://github.com/whatwg/fetch/issues/973#issuecomment-2181183295 You are receiving this because you are subscribed to this thread. Message ID: <whatwg/fetch/issues/973/2181183295@github.com>
Received on Thursday, 20 June 2024 17:22:42 UTC