Comcast "Principal Engineer" gets caught intentionally
LYING about the cause of download problems
(or Comcast upper management is lying to me)
September 8, 2014 (Revision 7.c updated 09/26/2014) - Technology Blog Index
duckware.com/comcastlied
I am seeing download speed problems accessing my server,
caused by packet loss somewhere in the network.
A Comcast 'self-proclaimed' TCP/IP and WireShark 'expert' claimed that
the problem was my server, for not supporting TCP/IP "fast retransmit".
[see §2]
But my server clearly DOES support "fast retransmit".
[see §3]
This proof was provided to Comcast's 'expert', along with numerous
other factual problems with his conclusions.
[see §4]
The Comcast 'expert' went silent and refused to respond to requests to speak to his manager.
|
Kevin Burns, Principal Engineer
'self-proclaimed' TCP/IP 'expert'
kevin_burns@cable.comcast.com
Scott Allison, VP Operations
scott_allison@cable.comcast.com
410-933-0579
Tony Werner, Chief Technology Officer
tony_werner@cable.comcast.com
215-286-8551
|
|
I followed up with Comcast corporate, but corporate insisted that Comcast was right
and threatened that there would be "no further communication" on this issue.
[see §2]
So, this Comcast 'expert', knowing that my server DOES support
"fast retransmit", knowingly lied to Comcast upper management
claiming once again that my server does not support "fast retransmit"
-- or Comcast upper management knows the truth and is knowingly lying to me.
Given that I have since confirmed that packet loss exists within Comcast's
network somewhere between Dallas, TX (where I know packets from my server enter Comcast's
network) and Maryland, that provides a convincing motive for why Comcast
is knowingly lying to me.
[see §6]
Comcast's latest trick: When Comcast knows that its network is dropping
packets, just lie, and blame the server!
The download problem: I was experiencing significant download problems
under Comcast's new 105Mbps Blast Internet service connecting to my server
(where I normally see over 120Mbps speeds). Each drop / red circle is a
dropped packet:
Download speed test run September 8, 2014 at 9:17 PM
By connecting a PC directly to the cable modem (a SB6121) via 1Gbps ethernet
(eliminating local router issues), and using
WireShark,
the problem was verified to be lost packets, likely caused by network congestion.
Downloads in the mornings are normally good, afternoons OK, evenings bad, late
night horrible. But downloads from my server from a Verizon FiOS internet
connection are always good.
[see §7]
On September 4, Comcast came to my house, cut off and replaced the ends on all
cables both inside and outside the house, tweaked upload power levels, and
verified that there was nothing wrong on the 'physical' side of things. That
fixed nothing. Comcast never hooked up a computer to the modem to try to
replicate the lost packet issue (after telling me they would). So a pair of
wire cutters is all that was needed for Comcast corporate to confirm that
the problem was with my server -- amazing! Because
hours after this visit
Scott Allison's "no further communication" letter was overnighted to me.
Comcast's 'expert' opinion: A Comcast 'expert' declared that the
problem was that my server did not implement the "fast retransmit"
algorithm within
RFC2581,
section 3.2 "TCP Congestion Control":
Note that Comcast -- indirectly -- admits that the problem
IS network congestion! Comcast just does not say where.
But Comcast is factually wrong:
But Comcast's conclusion was factually wrong.
[see §3]
I provided proof that my server clearly
implements "fast retransmit".
[see §4]
I also pointed out in no uncertain terms that Comcast's
'expert' had actually made numerous other technical blunders in his analysis --
explaining how he had ignored the latency induced by CMTS bufferbloat,
and provided WireShark captures that proved my points, and asked for a new 'expert'.
[see §5]
Silence:
Well, Comcast did not take me pointing out their factual mistakes
very well -- The 'expert', the Comcast National Support desk, and the
national We_can_help@, from that point on, all refused to respond
to any contact requests after that!
Comcast's so-called 'expert' was "Kevin Burns", a self-proclaimed
'expert' at both TCP/IP and WireShark.
Comcast Threat:
So, I wrote a letter to the CTO of Comcast, Tony Werner, which resulted in
a letter dated September 2, 2014 (right) from Comcast's Scott Allison (VP,
Operations) restating the same old trash (no "fast retransmit" support).
But this letter went way further -- Comcast
threatened that since they/Comcast were right,
"there will be no further communication from Comcast on this matter".
Comcast knows better:
There is only one person inside of Comcast that has claimed that my server does
not support "fast retransmit", and that is Kevin Burns.
But Mr. Burns already knows that my server DOES implement fast retransmit.
[see §4]
So why is Comcast management restating the same old trash that Comcast's 'expert'
knows is false!?
Comcast is intentionally LYING, but who?
The only possible conclusion is that either
(1) Kevin Burns is intentionally lying to his managers
(he knows my server supports "fast retransmit", but is unable to admit that to
management, since it would contradict his earlier analysis, as an 'expert', that my
server has a "fast retransmit" problem), or,
(2) Comcast management knows all of this and is intentionally
lying to me!
|
3. The proof my server DOES implement "fast retransmit"
|
|
WireShark Capture Proof:
Here is a WireShark capture of my server properly recovering from a lost packet
using "fast retransmit":
The time from transmission of the 3rd duplicate ACK (line 5068; time 1.341344)
until the time that the retransmitted packet is received (line 5769;
time 1.398557) is 57.2ms.
The RTT to my Linux server hovers around 52.7ms.
The TCP_RTO_MIN (minimum retransmission timeout) for Linux (all versions since 2.0)
is known to be 200ms, something any real TCP/IP expert would have known. And I have
confirmed that TCP_RTO_MIN is set to 200ms on my server.
You don't have to be an expert to understand that 57.2ms is much smaller than the minimum 200ms
retransmission timer, and that the only way the server could have retransmitted a packet
earlier than the minimum 200ms retransmission timeout, is if the server actually
implements fast retransmit, as per
RFC5681:
"After receiving 3 duplicate ACKs, TCP performs a retransmission
of what appears to be the missing segment, without waiting for the
retransmission timer to expire.
Proof from the Linux Source Code: Given the numerous references in the
Linux TCP source code
specifically to
RFC5681
and
RFC2581,
it is very clear
that Linux running on my server fully implements "fast retransmit".
Especially this comment and define within Linux's tcp.h:
/* After receiving this amount of duplicate ACKs fast retransmit starts. */
#define TCP_FASTRETRANS_THRESH 3
Conclusion: It is an undisputable fact that my Linux server implements "fast retransmit".
|
4. The Comcast 'expert' KNEW my server implements "fast retransmit"
|
|
Comcast's 'expert' absolutely knew on August 20, 2014, that my server
implements "fast retransmit". The proof comes in the form of an email
reply from Comcast's 'expert', containing my evidence to him (WireShark
captures) that my server implements "fast retransmit":
Comcast's 'expert' read this email, as he replied to it. He knew.
A retransmit in well under the 200ms minimum retransmission timeout, and near
the RTT, means only one thing: "fast retransmit" implemented.
So my server not only implements "fast retransmit", but Comcast's 'expert',
Kevin Burns, absolutely knew this fact as well.
Kevin Burns is between a rock and a hard place. His original analysis that my
server does not support "fast retransmit" was horribly wrong, which raises
questions about his 'expert' status. But, after reading my email (seen above) with
WireShark capture evidence, this 'expert' knowingly insisted to me that my server did not
support "fast retransmit", which means he knowingly lied to me. An 'expert' that
is willing to lie has just destroyed their credibility.
The only remaining question is: Did this 'expert' also lie to Comcast upper management,
or is upper management in on it, and also lying to me?
|
5. Comcast's 'expert' is clueless about bufferbloat's impact
|
|
VIDEO: Review the Bufferbloat Demo at IETF 86
with a Comcast Director
and BufferBloat.net's Dave Taht.
Comcast's 'expert' apparently does not comprehend how (or believe that) RTT
times can increase from around
50ms to over 250ms -- even after he was spoon fed the reason:
Comcast's 'expert' does not comprehend the latency impact of CMTS bufferbloat, even
after the situation was explained to him:
Again, Kevin Burns is between a rock and a hard place. He is going to have to
explain to management why he, supposedly a TCP/IP 'expert', could not comprehend
(even after it was explained) that a large receive window causes buffering, where that
buffering will take place (the CMTS), and how that buffering impacts RTT times.
Latency induced by bufferbloat: All real TCP/IP experts know about bufferbloat.
Bufferbloat is queueing within network devices that causes latency and RTT to increase,
sometimes very dramatically.
Seen visually, the increase in RTT times is obvious:
The red "retransmitted" packet below must wait for all of the
other blue packets ahead of it in the network to be transmitted to the
modem/user first. And how many blue packets are allowed into
the network (in the network when the 3rd dup ack is received by a server
and acted on) is directly related to the RWIN receive window size
(and cwnd):
For example, if the blue packets represents 3,000,000 bytes of queueing within the CMTS on
a 120Mbps internet connection, the red packet will have to sit and wait around
±200ms (3000000*8/120000000; ignoring protocol overhead) within the CMTS egress
queue.
Bufferbloat's impact can be monitored:
For example, if you download a large file and 'max out' your Internet connection
(chart below left), there absolutely will, with 100% certainty, be some
bufferbloat (chart below right, buffering within the CMTS, that impacts RTT times
(if there is no buffering, your connection is NOT maxed out). The only question
is how much will the impact be? The answer depends upon the amount of buffering
and the speed of your internet connection. Using WireShark, the following
download ran with an advertised window size around 32000000, causing
RTT times to increase from around 50ms to 50+150ms in just 1.5 seconds (until
there were a bunch of lost packets):
Again, Kevin Burns is between a rock and a hard place. He is going to have to
explain to management why he, supposedly a TCP/IP 'expert', incorrectly allocated
ALL RTT time increases (seen in a WireShark capture) to a customer's 'server', and
did not allocate a single millisecond to Comcast owned equipment (the CMTS).
It is as if his knowledge of TCP/IP comes only from books and LAN testing, and
not real-world WAN testing.
|
6. A motive for Comcast lying: Comcast itself is dropping packets!
|
|
Congestion is the problem:
Lost packets under TCP/IP are a key signature of congestion -- namely,
there is not enough bandwidth "in the pipe" between you and the server.
The only question remaining then is: Where 'in the pipe' is the
congestion and where are packets being dropped?
Comcast blames my hosting provider:
Comcast absolutely refuses to acknowledge that they are the cause of
the dropped packets, and blames my hosting provider,
1and1. And my hosting provider blames Comcast. With multiple networks
involved (who all refuse to dig deeper to find the root cause), finding
the 'where' is virtually impossible.
The Download path/route:
But, 1and1 had provided a 'reverse' traceroute to me (the path from my server
to me), which was a critical clue -- the path goes through a peering location
in Dallas, TX -- which again, Comcast's 'expert' clearly knew due to this email:
Comcast Speed Test:
So on a hunch, I ran Comcast's own
speed test
from Maryland to a Comcast server in Texas, and WireShark instantly
captured dropped packets:
Comcast itself is dropping packets from Comcast TX to Comcast MD:
Source: 69.241.80.90/speedtest.greenspoint.tx.houston.comcast.net
Destination: 76.100.x.x/c-76-100-x-x.hsd1.md.comcast.net
There is no other network. This is Comcast talking to Comcast, and dropping packets!
The weak link:
If Comcast does not have enough capacity in its own network from
Texas to Maryland to prevent dropped packets, then this 'weak link' might
just explain where packets are being dropped when downloading from my
server (as 1and1 packets are routed to and enter Comcast's network in Dallas,
Texas).
Comcast's Nationwide Fiber Optic Network
A final thought: The presumption is that Comcast's network is 'congested' and
dropping packets. But we don't know the ultimate reason for the dropped packets
until Comcast cooperates, which given
Comcast's history, appears unlikely. Another possibility is Comcast software (a firewall,
advanced queue management, etc) that is actively monitoring and limiting socket
throughput.
|
7. Both Verizion FiOS and ATT DSL works just great -- NO dropped packets
|
|
Verizon FiOS does not have this problem: When I use a Verizion FiOS
Internet connection to perform a download speed test on the same file from my server,
no packets are dropped -- even at the exact same time/second
that Comcast is
dropping packets from my server, Verizon FiOS is NOT dropping packets:
Verizon FiOS has no packet loss from my server
Clearly, the dropped packets issue is NOT caused by my hosting provider. Rather, Comcast,
or a Comcast peering parter, is the prime suspect for dropping packets.
And since we know the Comcast network is 'congested' on the Texas to Maryland route,
(and packets from my server enter Comcast's network in Texas), Comcast
is the very likely the cause of all problems (packet drops).
Update: ATT DSL does not have this problem: Testing under ATT DSL also
shows that it drops NO packets at the same time that Comcast drops packets
[2014/09/26].
|
8. A comment about 'dropped packets'
|
|
It is very distressing that Comcast's 'expert' (a Principle Engineer) stated as an excuse to me
that "Packet Loss is to be expected on the Internet". With that attitude, it
is no wonder that Comcast's Dallas TX to Maryland link and losing packets! Rather,
in a well run network, packet loss is rare, because packet loss kills speed.
Just look at the Verizion FiOS network.
[see §7]
Packet loss under TCP/IP means 'congestion' and is a signal to
TCP to slow down -- which indicates poor capacity planning on
the part of network engineers. Usually, network planners closely monitor
utilization and packet loss, and augment network capacity well before
packet loss becomes a problem.
Read
Observations of an Internet Middleman,
which is a scathing blog post by Mark Taylor of Level3 about ISP's that allow
packets to 'drop on the floor' at peering points. Mr. Taylor talks about ISPs that
"refuse to augment capacity" and "are deliberately harming the service they deliver
to their paying customers" concluding that ISP's "are not allowing us to fulfil
the requests their customers make for content".
It IS all about money and profit. This one sentence from the blog sums it all up: "In
countries or markets where consumers have multiple Broadband choices (like
the UK) there are no congested peers."
When there is no competition, an ISP can drop all the packets they
want (not spending money on network upgrades means more profit)
-- and customers can't switch away, even if they wanted to.
Comcast, it is way past time for full network transparency! Publish a
'weathermap' for the entire Comcast network
(eg: OVH weathermap).
Or better yet, publish all utilization and packet loss information for
the entire Comcast network.
If Comcast is serious about getting
off the "World's Most Hated Companies" list, Comcast
will make this information public.
Automaton
|
Comcast intentionally lied and got caught.
Comcast's network from Texas to Maryland is 'congested', somewhere, and is dropping packets,
negatively impacting my download performance, and likely that of other paying customers.
We_can_help@cable.comcast.com: Misnomer. Should be "We_Will_Ignore_You".
Comcast upper management: Automatons.
Comcast TCP/IP expert: Oxymoron.
Will Comcast listen, and learn? Maybe its time to put that
#1 strength to good use!
Regardless, since Comcast threatened me, Maryland's Attorney General, Consumer Protection Division,
is now involved, as a complaint (Ref #244409)
was formally filed against Comcast on 09/09/2014.
Comcast needs to 'own up' to what they have knowingly done (lied!), and make it right.
|
10. Comcast is monitoring this web page
|
|
UPDATE: September 16, 2014. Welcome Comcast (Aurora/Englewood/Denver, Colorado)! Two IP
addresses, known to be associated with Comcast's "National Support Desk", have been monitoring
this web page every day, and sometimes multiple times a day, since this page was
first published.
Interesting that Comcast wants to spend to the time to monitor this web page,
but does not want to spend the time to fix packet loss problems.
|
|
