It's actually surprising it isn't down more often—internally, everyone has write access to prod and the rule is that if you deploy something to prod you need to be able to roll it back.* Apparently, though, someone has failed on the second item.

* Or so I was told in a job interview with the big A a few years back.

I find this extremely hard to believe. (Not calling you a liar, but I think something must have gotten lost in translation).

The possibility for theft and fraud would be so massive if every dev at Amazon had write access to production that I find it nearly impossible to believe this is true.

Developers probably have access to most production systems. Credit card processing and source of truth on orders that get shipped are most likely segregated. (actually PCI dictates that physical and data access controls be in place so only essential employees can access card data)

Who cares if I can access the credit card processing system if I can insert random code elsewhere in the system that redirects you to my phishing page whenever you enter credit card information?

Given that you would be an Amazon employee with a solid audit trail leading back to you in that scenario, I'd say it's pretty likely you'd be caught and prosecuted rather quickly.

Yes, I and my coworkers could've sold the realtime trades of a petroleum multinational to the highest bidder, including ones that hadn't happened yet. That would've been easy, and would've been worth 100's of millions to someone. Not getting caught and having your life ruined -- that was scary and would've been hard. Now, if I was working for a sovereign power, like China, and my life was there anyhow, then pulling stuff like that in the US wouldn't be so hard.

When did amazon start selling petroleum futures?

Amazon is not a petroleum multinational. Guess again.

The bits that need high security such as production databases have extra layers of access and tracking. But most devs can push changes to the retail website.

Very often at a lot of companies devs can read production databases and upload the contents offsite.

I imagine he means the ability to deploy code to prod at will, not full access to their database.

Does the code you deploy have access to the database?

No.

It's true.

One of the reasons I left Amazon was that I was given the job to deploy code regularly (about weekly) at 1am or so, and one evening, there was a problem due to work of another team, so it escalated and we spent 6 hours dealing with it. We rolled the change back right away, but for contractual reasons their code had to be fixed and deployed and there was an interdependency. Fortunately, it wasn't my team's mistake, but I had to be there to help test it, etc.) So, it's finally working at 7am, and I stuck around for 30 minutes to make sure it kept working before going to sleep around 7:45AM.

I emailed my boss about it, and of course he was getting emails the whole while as the tickets status was changing.

Still, the fact that I showed up at 10:15 for the 10AM meeting that morning was "unacceptable" and I got chewed out. (~2 hours sleep!)

I made the mistake of thinking that my HR rep might be someone to talk to about this, because I wasn't sure how to make it clear to him that it was kinda unreasonable (Especially since I told him I'd be late for the meeting)... and that's when I found out that everything I told her was written up in an email & sent to him.... resulting in getting chewed out yet again for going to HR!

The lesson: as a programmer, never work for a boss who can't program, or at least, be very wary of it!

I have to say, it sounds to me like the lesson isn't about bosses who can't program, so much as "don't have a terrible boos". There are plenty of fields I know nothing about, but if I was managing people in that field, I would expect that on 2 hours sleep they wouldn't be effective, and I also wouldn't expect them to work both night and day shifts. It's common sense.

I have to say, from experience, that you shouldn't count on an HR rep for anything. At all.

45 minutes of downtime so far, we're seeing mostly 503 responses with an occasional 200 getting through. We've seen a few other smaller outages for amazon.com in the past but this is definitely the longest in at least the last 3-4 years. Details at http://reports.panopta.com/amazon/server/96291

I hope we get a nice detailed postmortem on this one.