Web Proxy Traffic Profile (HTTPS, Inspection, etc.)

Hi,

It was good to attend a Saturday's design group meeting and to meet many of you. A few folks wanted me to follow up on the list with some of the stats I quoted regarding HTTP, HTTPS and HTTPS Inspection (MITM) usage. I've also thrown in some regarding Video, FTP-over-HTTP and blocks since they came up in various discussions.

As a bit of background I work for Cisco on their Cloud Web Security (CWS) product. CWS is a large cloud-based HTTP(s) proxy which provides safe Web browsing for corporate, SMB, government and educational customers. We provide policy-enforcement, reporting and a number of anti-malware technologies including Web Reputation, commercial Anti-Virus engines and some secret sauce. We're also integrating technology from the recent SourceFire acquisition - they're the guys behind Snort IDS, ClamAV, FireAMP and a bunch of other stuff.

Anyway, here are some illustrative stats - all taken from last Wednesday, a typical day chosen because it's recent and being in the middle of the week means it avoid the weekend biases.
- We processed 5.1 BN plain HTTP transactions and 1 BN CONNECT requests for HTTPS resources. That makes HTTPS connects 16% of the total (by requests) although that does include some traffic like Webex and Gotomeeting which tunnel over CONNECT but don't use HTTP underneath.
- With 140 TB of HTTP data and 46 TB of HTTPS, HTTPS is consuming around 25% of our bandwidth.
- Of the 186 TB of total traffic, 28TB went to YouTube, 14TB to various MPEG video and 12TB to other Flash video. This means video is consuming around 30% of our total bandwidth
- We ran HTTPS inspection on 19.7M CONNECTs (2% of the total number) processing 45M underlying requests (2.3 reqs per connect)
- On those inspected transactions we blocked 22,337 requests due to bad reputation scores - cases where the URLs in question are strongly associated with threats including serving malware, running botnet C&C centres, hosting phishing, fake escrow or similar malicious content or being heavily associated with spam e-mails.
- Of those which passed our request-based security filtering, we identified 183 as carrying viruses using our commercial scanners and a further 116 potential viruses using our own "Outbreak Intelligence" tech (the false-positive rate using this is higher but the false-negative rate lower)
- We also blocked 1.1M inspected requests due to customer policies
- This is in contrast to the 24M WebRep and 207,091 anti-virus blocks we did on uninspected requests - the percentage of malicious content found on HTTPS still remains a fraction of that served over the equivalent volume of unencrypted transactions.
- We also processed 100,000 FTP-over-HTTP requests.

Although I feel these statistics should be broadly representative of traffic in an work environment, we do have some selection biases because of the profile of our customers, but it. In particular we have biases:
- Towards US and EMEA traffic and away from LatAm, Africa and APAC because of our customer base
- Away from porn and other adult material - partly because it's blocked but mostly because people are less likely to browse those resources at work even when they're allowed

I hope some of these numbers help inform the discussions around proxy usage,

Regards,

Richard

Received on Monday, 10 March 2014 11:58:03 UTC