- From: Mike Wilcox <mike@mikewilcox.net>
- Date: Mon, 12 Jul 2010 09:43:14 -0500
On Jul 12, 2010, at 8:39 AM, Nils Dagsson Moskopp wrote: > >> That's a little different. Google purposely uses unstandardized, >> incorrect HTML in ways that still render in a browser in order to >> make it more difficult for screen scrapers. They also "break it" in a >> different way every week. > > Assuming this is true (which I find difficult to believe), wouldn't a > screen scraper based on the HTML5 parsing algorithm defeat this > purpose ? Honestly, I don't know. But W3 defaulted to an HTML5 validator: http://validator.w3.org/check?uri=http%3A%2F%2Fwww.google.com%2Fsearch%3Fsource%3Dig%26hl%3Den%26rlz%3D%26%3D%26q%3Dhtml5%26aq%3Df%26aqi%3D%26aql%3D%26oq%3D%26gs_rfai%3D&charset=%28detect+automatically%29&doctype=Inline&group=0 On Jul 12, 2010, at 7:55 AM, Julian Reschke wrote: > > Any evidence for this? And how do you know the reason for CNET isn't the same? > > VERY unconvinced. Such skepticism! I really didn't expect that I said anything controversial. We all know Google is a black box, and I don't, nor have I ever worked for them, so I can't testify that what I say is a fact... Google does not publish a full API for their search results in spite of the fact that they publish most everything else. That's because their search results / API is their golden egg ? they can't give that away. And they need to protect it. I've tried scraping results with YQL and it failed. I made other attempts that had more success only to find they did not work a week later. Studying the validation results more closely however does not back up my original statement ? the markup does look as I described, but the things I expected to break the validator are not what I expected. The things I had in mind was for the most part unclosed tags, which I didn't know validated. I knew they were allowed, but thought the validator would complain. Besides the protecting of their API, Google also will scratch and claw to save every byte. They are the gold standard of a high performance website. While this may or may not explain the things that don't validate, what it does say is that nothing coming from google.com is accidental. Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.whatwg.org/pipermail/whatwg-whatwg.org/attachments/20100712/2efb59cb/attachment-0001.htm>
Received on Monday, 12 July 2010 07:43:14 UTC