"But a customer quite legitimately changing their settings had exposed a bug in a software update issued to customers in mid-May, causing "85% of our network to return errors"
They are careful to make clear that the customer did nothing wrong and that the problem was a bug in their software.
I know — as I've said, the main blame lies with the bbc. However, as it's reported, it comes across very much as Fastly trying to save face. Maybe the blame is entirely on the bbc, maybe Fastly were naive in thinking that giving them this information wouldn't result in irresponsible headlines.
What verbiage exactly are you looking for from Fastly here? I'm hearing "Nobody else did anything wrong, it was 100% a software bug on our end, and we're sorry about that." How much more responsibility are you asking them to take before you would no longer be considering them to "save face"? I'm trying to come up with an ironic exaggeration here, but I can't, because it kinda seems like Fastly has already taken 100% full responsibility and there's no room left for exaggeration.
> Early June 8, a customer pushed a valid configuration change that included the specific circumstances that triggered the bug, which caused 85% of our network to return errors.
Is it necessary to refer to "a customer" at all in this statement? What would be problematic if the above were rewritten as something like:
> Early June 8, a configuration change triggered a bug in our software, which caused 85% of our network to return errors.
The advantage is that you wouldn't get ignorant reporting that "one customer took down the internet". I'm not sure there are disadvantages that net outweigh that.
Yes, because it is explaining that it was a valid *customer* configuration, which is a separate set of concerns from, say, infrastructure config.
The important adjective "valid" means it was completely normal/expected input and thus not the fault of the customer.
It's perfectly clear you've come at this with a pre-determined agenda of "I bet fastly, like most other public statements after corporate booboos I've seen, will try to shrug this one off as someone else's fault" after reading the BBCs title and haven't bothered to read it at all until now.
> We experienced a global outage due to an undiscovered software bug that surfaced on June 8 when it was triggered by a valid customer configuration change.
> Is it necessary to refer to "a customer" at all in this statement? What would be problematic if the above were rewritten as something like:
That's literally what happened. They even say it was a valid configuration change, it's very blameless.
Saying "a configuration change" loses critical context. I would have assumed that this was in some sort of deployment update, not something that a customer could trigger. Why would you want less information here?
OK, I'm replying to your comment since it's the least aggressive — thanks for that!
I'll fully retract my statement. This is 100% the BBC's fault, 0% Fastly's.
Can I make one small suggestion that might help to prevent this kind of misleading reporting in future, though? What if Fastly produce the detailed statement they have, with as much accurate technical detail as possible AND a more general public-facing statement that organisations such as the BBC can use for reporting, that doesn't include such detailed information that can easily be misconstrued?
Most of the replies to yours haven’t been aggressive. Ironically it’s your comments that have come across the worst by using terms like “aggressive”, “blame” and “fault” in the first place. Calling other people’s comments aggressive is pretty unfair. One might even say hypocritical.
I hate being part of a dogpile, so yeah sorry about that, I just open up things to reply to, and then come back later and write it up just to find that I'm one of 10 people saying the same shit.
edit: FWIW I had a very negative initial reaction to the headline as well.
No worries. I probably didn't take it very well because my intentions were genuine and I really wasn't trying to level anything beyond the very mildest criticism towards Fastly. I recognise, however, that even that was misplaced — I think the BBC headline just got me too worked up!
In what way is the BBC at fault for this? Their title is objectively true. A _valid_ configuration setting that was used by a customer _did_ cause fastly to have an outage.
It's not limited to one specific customer (i.e this customer isn't the only customer who could have caused the issue, presumably), but it _was_ something the customer (legitimately) did. It wasn't a server outage. It wasn't a fire. It wasn't a cut cable.
"a customer quite legitimately changing their settings (BBC: one fastly customer) had exposed a bug (BBC: triggered internet meltdown) in a software update issued to customers (fastly admitting, when combined with 'legitimately', that fastly are at fault) in mid-May".
Not me — I adore the BBC. I've always paid my licence fee gladly, and I've been waxing lyrical about the latest BBC drama on Twitter just this very hour. On this issue, I believe they've made a mistake.
Whatever happened to nuanced opinion, where you can see good and bad in the same entity? Why do some people insist so strongly on absolutes?
Fastly, the cloud-computing company responsible for the issues, said the bug had been triggered when one of its customers had changed their settings.
Fastly senior engineering executive Nick Rockwell said: "This outage was broad and severe - and we're truly sorry for the impact to our customers and everyone who relies on them."
But a customer quite legitimately changing their settings had exposed a bug in a software update issued to customers in mid-May, causing "85% of our network to return errors", it said.
The headline accurately portrays the story given the limit on headlines.
The wording "a configuration change triggered a bug" in this context sounds (to me) like it was a configuration change made by Fastly to something on their backend.
The wording which was actually used makes it clear that that was not the case.
They are careful to make clear that the customer did nothing wrong and that the problem was a bug in their software.