Comcast "Principal Engineer" gets caught intentionally
LYING about the cause of download problems

(or Comcast upper management is lying to me)
September 8, 2014 (Revision 7.c updated 09/26/2014) - Technology Blog Index
duckware.com/comcastlied

  1. Executive Summary


I am seeing download speed problems accessing my server, caused by packet loss somewhere in the network. A Comcast 'self-proclaimed' TCP/IP and WireShark 'expert' claimed that the problem was my server, for not supporting TCP/IP "fast retransmit". [see §2]

But my server clearly DOES support "fast retransmit". [see §3] This proof was provided to Comcast's 'expert', along with numerous other factual problems with his conclusions. [see §4] The Comcast 'expert' went silent and refused to respond to requests to speak to his manager.

Kevin Burns, Principal Engineer
'self-proclaimed' TCP/IP 'expert'
kevin_burns@cable.comcast.com

Scott Allison, VP Operations
scott_allison@cable.comcast.com
410-933-0579

Tony Werner, Chief Technology Officer
tony_werner@cable.comcast.com
215-286-8551
I followed up with Comcast corporate, but corporate insisted that Comcast was right and threatened that there would be "no further communication" on this issue. [see §2]

So, this Comcast 'expert', knowing that my server DOES support "fast retransmit", knowingly lied to Comcast upper management claiming once again that my server does not support "fast retransmit" -- or Comcast upper management knows the truth and is knowingly lying to me.

Given that I have since confirmed that packet loss exists within Comcast's network somewhere between Dallas, TX (where I know packets from my server enter Comcast's network) and Maryland, that provides a convincing motive for why Comcast is knowingly lying to me. [see §6]

Comcast's latest trick: When Comcast knows that its network is dropping packets, just lie, and blame the server!


  2. Overview of the issue

The download problem: I was experiencing significant download problems under Comcast's new 105Mbps Blast Internet service connecting to my server (where I normally see over 120Mbps speeds). Each drop / red circle is a dropped packet:


Download speed test run September 8, 2014 at 9:17 PM

By connecting a PC directly to the cable modem (a SB6121) via 1Gbps ethernet (eliminating local router issues), and using WireShark, the problem was verified to be lost packets, likely caused by network congestion. Downloads in the mornings are normally good, afternoons OK, evenings bad, late night horrible. But downloads from my server from a Verizon FiOS internet connection are always good. [see §7]
On September 4, Comcast came to my house, cut off and replaced the ends on all cables both inside and outside the house, tweaked upload power levels, and verified that there was nothing wrong on the 'physical' side of things. That fixed nothing. Comcast never hooked up a computer to the modem to try to replicate the lost packet issue (after telling me they would). So a pair of wire cutters is all that was needed for Comcast corporate to confirm that the problem was with my server -- amazing! Because hours after this visit Scott Allison's "no further communication" letter was overnighted to me.
Comcast's 'expert' opinion: A Comcast 'expert' declared that the problem was that my server did not implement the "fast retransmit" algorithm within RFC2581, section 3.2 "TCP Congestion Control":


Note that Comcast -- indirectly -- admits that the problem IS network congestion! Comcast just does not say where.

But Comcast is factually wrong: But Comcast's conclusion was factually wrong. [see §3] I provided proof that my server clearly implements "fast retransmit". [see §4] I also pointed out in no uncertain terms that Comcast's 'expert' had actually made numerous other technical blunders in his analysis -- explaining how he had ignored the latency induced by CMTS bufferbloat, and provided WireShark captures that proved my points, and asked for a new 'expert'. [see §5]

Silence: Well, Comcast did not take me pointing out their factual mistakes very well -- The 'expert', the Comcast National Support desk, and the national We_can_help@, from that point on, all refused to respond to any contact requests after that!

Comcast's so-called 'expert' was "Kevin Burns", a self-proclaimed 'expert' at both TCP/IP and WireShark.

Comcast Threat: So, I wrote a letter to the CTO of Comcast, Tony Werner, which resulted in a letter dated September 2, 2014 (right) from Comcast's Scott Allison (VP, Operations) restating the same old trash (no "fast retransmit" support). But this letter went way further -- Comcast threatened that since they/Comcast were right, "there will be no further communication from Comcast on this matter".

Comcast knows better: There is only one person inside of Comcast that has claimed that my server does not support "fast retransmit", and that is Kevin Burns. But Mr. Burns already knows that my server DOES implement fast retransmit. [see §4]

So why is Comcast management restating the same old trash that Comcast's 'expert' knows is false!?

Comcast is intentionally LYING, but who? The only possible conclusion is that either (1) Kevin Burns is intentionally lying to his managers (he knows my server supports "fast retransmit", but is unable to admit that to management, since it would contradict his earlier analysis, as an 'expert', that my server has a "fast retransmit" problem), or, (2) Comcast management knows all of this and is intentionally lying to me!


  3. The proof my server DOES implement "fast retransmit"

WireShark Capture Proof: Here is a WireShark capture of my server properly recovering from a lost packet using "fast retransmit":


The time from transmission of the 3rd duplicate ACK (line 5068; time 1.341344) until the time that the retransmitted packet is received (line 5769; time 1.398557) is 57.2ms.
The RTT to my Linux server hovers around 52.7ms. The TCP_RTO_MIN (minimum retransmission timeout) for Linux (all versions since 2.0) is known to be 200ms, something any real TCP/IP expert would have known. And I have confirmed that TCP_RTO_MIN is set to 200ms on my server.
You don't have to be an expert to understand that 57.2ms is much smaller than the minimum 200ms retransmission timer, and that the only way the server could have retransmitted a packet earlier than the minimum 200ms retransmission timeout, is if the server actually implements fast retransmit, as per RFC5681:
"After receiving 3 duplicate ACKs, TCP performs a retransmission of what appears to be the missing segment, without waiting for the retransmission timer to expire.
Proof from the Linux Source Code: Given the numerous references in the Linux TCP source code specifically to RFC5681 and RFC2581, it is very clear that Linux running on my server fully implements "fast retransmit". Especially this comment and define within Linux's tcp.h:
/* After receiving this amount of duplicate ACKs fast retransmit starts. */
#define TCP_FASTRETRANS_THRESH 3
Conclusion: It is an undisputable fact that my Linux server implements "fast retransmit".


  4. The Comcast 'expert' KNEW my server implements "fast retransmit"

Comcast's 'expert' absolutely knew on August 20, 2014, that my server implements "fast retransmit". The proof comes in the form of an email reply from Comcast's 'expert', containing my evidence to him (WireShark captures) that my server implements "fast retransmit":


Comcast's 'expert' read this email, as he replied to it. He knew. A retransmit in well under the 200ms minimum retransmission timeout, and near the RTT, means only one thing: "fast retransmit" implemented.

So my server not only implements "fast retransmit", but Comcast's 'expert', Kevin Burns, absolutely knew this fact as well.

Kevin Burns is between a rock and a hard place. His original analysis that my server does not support "fast retransmit" was horribly wrong, which raises questions about his 'expert' status. But, after reading my email (seen above) with WireShark capture evidence, this 'expert' knowingly insisted to me that my server did not support "fast retransmit", which means he knowingly lied to me. An 'expert' that is willing to lie has just destroyed their credibility.

The only remaining question is: Did this 'expert' also lie to Comcast upper management, or is upper management in on it, and also lying to me?


  5. Comcast's 'expert' is clueless about bufferbloat's impact

VIDEO: Review the Bufferbloat Demo at IETF 86 with a Comcast Director and BufferBloat.net's Dave Taht.

Comcast's 'expert' apparently does not comprehend how (or believe that) RTT times can increase from around 50ms to over 250ms -- even after he was spoon fed the reason:


Comcast's 'expert' does not comprehend the latency impact of CMTS bufferbloat, even after the situation was explained to him:


Again, Kevin Burns is between a rock and a hard place. He is going to have to explain to management why he, supposedly a TCP/IP 'expert', could not comprehend (even after it was explained) that a large receive window causes buffering, where that buffering will take place (the CMTS), and how that buffering impacts RTT times.

Latency induced by bufferbloat: All real TCP/IP experts know about bufferbloat. Bufferbloat is queueing within network devices that causes latency and RTT to increase, sometimes very dramatically.

Seen visually, the increase in RTT times is obvious: The red "retransmitted" packet below must wait for all of the other blue packets ahead of it in the network to be transmitted to the modem/user first. And how many blue packets are allowed into the network (in the network when the 3rd dup ack is received by a server and acted on) is directly related to the RWIN receive window size (and cwnd):


For example, if the blue packets represents 3,000,000 bytes of queueing within the CMTS on a 120Mbps internet connection, the red packet will have to sit and wait around ±200ms (3000000*8/120000000; ignoring protocol overhead) within the CMTS egress queue.

Bufferbloat's impact can be monitored: For example, if you download a large file and 'max out' your Internet connection (chart below left), there absolutely will, with 100% certainty, be some bufferbloat (chart below right, buffering within the CMTS, that impacts RTT times (if there is no buffering, your connection is NOT maxed out). The only question is how much will the impact be? The answer depends upon the amount of buffering and the speed of your internet connection. Using WireShark, the following download ran with an advertised window size around 32000000, causing RTT times to increase from around 50ms to 50+150ms in just 1.5 seconds (until there were a bunch of lost packets):

 

Again, Kevin Burns is between a rock and a hard place. He is going to have to explain to management why he, supposedly a TCP/IP 'expert', incorrectly allocated ALL RTT time increases (seen in a WireShark capture) to a customer's 'server', and did not allocate a single millisecond to Comcast owned equipment (the CMTS). It is as if his knowledge of TCP/IP comes only from books and LAN testing, and not real-world WAN testing.


  6. A motive for Comcast lying: Comcast itself is dropping packets!

Congestion is the problem: Lost packets under TCP/IP are a key signature of congestion -- namely, there is not enough bandwidth "in the pipe" between you and the server. The only question remaining then is: Where 'in the pipe' is the congestion and where are packets being dropped?

Comcast blames my hosting provider: Comcast absolutely refuses to acknowledge that they are the cause of the dropped packets, and blames my hosting provider, 1and1. And my hosting provider blames Comcast. With multiple networks involved (who all refuse to dig deeper to find the root cause), finding the 'where' is virtually impossible.

The Download path/route: But, 1and1 had provided a 'reverse' traceroute to me (the path from my server to me), which was a critical clue -- the path goes through a peering location in Dallas, TX -- which again, Comcast's 'expert' clearly knew due to this email:


Comcast Speed Test: So on a hunch, I ran Comcast's own speed test from Maryland to a Comcast server in Texas, and WireShark instantly captured dropped packets:


Comcast itself is dropping packets from Comcast TX to Comcast MD:
Source: 69.241.80.90/speedtest.greenspoint.tx.houston.comcast.net
Destination: 76.100.x.x/c-76-100-x-x.hsd1.md.comcast.net
There is no other network. This is Comcast talking to Comcast, and dropping packets!

The weak link: If Comcast does not have enough capacity in its own network from Texas to Maryland to prevent dropped packets, then this 'weak link' might just explain where packets are being dropped when downloading from my server (as 1and1 packets are routed to and enter Comcast's network in Dallas, Texas).


Comcast's Nationwide Fiber Optic Network

A final thought: The presumption is that Comcast's network is 'congested' and dropping packets. But we don't know the ultimate reason for the dropped packets until Comcast cooperates, which given Comcast's history, appears unlikely. Another possibility is Comcast software (a firewall, advanced queue management, etc) that is actively monitoring and limiting socket throughput.


  7. Both Verizion FiOS and ATT DSL works just great -- NO dropped packets

Verizon FiOS does not have this problem: When I use a Verizion FiOS Internet connection to perform a download speed test on the same file from my server, no packets are dropped -- even at the exact same time/second that Comcast is dropping packets from my server, Verizon FiOS is NOT dropping packets:


Verizon FiOS has no packet loss from my server

Clearly, the dropped packets issue is NOT caused by my hosting provider. Rather, Comcast, or a Comcast peering parter, is the prime suspect for dropping packets.

And since we know the Comcast network is 'congested' on the Texas to Maryland route, (and packets from my server enter Comcast's network in Texas), Comcast is the very likely the cause of all problems (packet drops).

Update: ATT DSL does not have this problem: Testing under ATT DSL also shows that it drops NO packets at the same time that Comcast drops packets [2014/09/26].


  8. A comment about 'dropped packets'


It is very distressing that Comcast's 'expert' (a Principle Engineer) stated as an excuse to me that "Packet Loss is to be expected on the Internet". With that attitude, it is no wonder that Comcast's Dallas TX to Maryland link and losing packets! Rather, in a well run network, packet loss is rare, because packet loss kills speed. Just look at the Verizion FiOS network. [see §7]

Packet loss under TCP/IP means 'congestion' and is a signal to TCP to slow down -- which indicates poor capacity planning on the part of network engineers. Usually, network planners closely monitor utilization and packet loss, and augment network capacity well before packet loss becomes a problem.

Read Observations of an Internet Middleman, which is a scathing blog post by Mark Taylor of Level3 about ISP's that allow packets to 'drop on the floor' at peering points. Mr. Taylor talks about ISPs that "refuse to augment capacity" and "are deliberately harming the service they deliver to their paying customers" concluding that ISP's "are not allowing us to fulfil the requests their customers make for content".

It IS all about money and profit. This one sentence from the blog sums it all up: "In countries or markets where consumers have multiple Broadband choices (like the UK) there are no congested peers." When there is no competition, an ISP can drop all the packets they want (not spending money on network upgrades means more profit) -- and customers can't switch away, even if they wanted to.

Comcast, it is way past time for full network transparency! Publish a 'weathermap' for the entire Comcast network (eg: OVH weathermap). Or better yet, publish all utilization and packet loss information for the entire Comcast network. If Comcast is serious about getting off the "World's Most Hated Companies" list, Comcast will make this information public.


  9. Conclusion


Automaton
Comcast intentionally lied and got caught.

Comcast's network from Texas to Maryland is 'congested', somewhere, and is dropping packets, negatively impacting my download performance, and likely that of other paying customers.

We_can_help@cable.comcast.com: Misnomer. Should be "We_Will_Ignore_You".

Comcast upper management: Automatons.

Comcast TCP/IP expert: Oxymoron.

Will Comcast listen, and learn? Maybe its time to put that #1 strength to good use!

Regardless, since Comcast threatened me, Maryland's Attorney General, Consumer Protection Division, is now involved, as a complaint (Ref #244409) was formally filed against Comcast on 09/09/2014.

Comcast needs to 'own up' to what they have knowingly done (lied!), and make it right.

  10. Comcast is monitoring this web page

UPDATE: September 16, 2014. Welcome Comcast (Aurora/Englewood/Denver, Colorado)! Two IP addresses, known to be associated with Comcast's "National Support Desk", have been monitoring this web page every day, and sometimes multiple times a day, since this page was first published.

Interesting that Comcast wants to spend to the time to monitor this web page, but does not want to spend the time to fix packet loss problems.

Some content may be copyrighted by their respective owners and is reproduced
in this paper under Title 17 Section 107 (Fair Use) of the copyright code.
The rest of this document is Copyright © 2014-2017 Jerry Jongerius