Until the US federal government pays civilian tech talent competitively, this is always going to be an issue.
Your typical hands-on-keyboard blue team engineer in federal government is a GS-12 getting paid around $68,000 per year (or $99k in very high cost of living areas like DC). They have expensive health benefits, 13 days of PTO a year, put a huge chunk of their paycheck (almost 5%) into a mandatory pension plan that consistently underperforms the market, and can literally go to jail for making mistakes at work depending on the statutory context they work in.
The best people in these jobs burn out fast and quit or they end up having to abandon IC work for GS-14/15 jobs (max pay is around $190 for those) in order to keep up with cost-of-living and justify their careers.
As a result, you have almost zero genuinely capable principal/senior engineers in government who have the authority to architect complex IT systems for security. Instead you get contractors who charge the taxpayers enormous overhead costs and cut corners wherever possible.
If there's one letter to write your congress person to improve government - my vote would be for civil service reform to attract and retain actual top tech talent. They've done it for doctors and lawyers (both of whom can get paid well above the $190k GS pay ceiling), but engineering is still not treated as a comparably skilled professional trade.
I was fine for the pay structure on its own. I gave up when I was rejected for not having the hyper specific domain experience they wanted for the pay they were asking for. This was primarily a CRUD job btw and I was qualified by any other standard.
I tried so hard to get into gov't tech but ultimately gave up. Jumping from the private sector to public seems impossible to me as an outsider.
A friend of mine, who is a lawyer and does HR for the federal gov't, spent about a week helping me get my fed resume tightened up and I still got nothing. I don't even care about the pay cut. It just seems like interesting work.
My spouse is a federal civilian employee with a scientific background and a special pay rate accommodation for it. Relative to her industry the pay discrepancy is still about 20% lower. In software roles the top end of the career is roughly the starting pay of junior developers everywhere else. It's not just lower, it's horribly lower.
They can absolutely make adjustments if congress needed or wanted to. The DoD as of 2019 does direct officer commissions for cybersecurity roles bringing people in as majors IIRC (it's still not as good pay as civilian cybersecurity roles but the gap is smaller and it has the prestige and lifetime benefits of being an officer).
With that attitude, the pay will always be lower. Letting the dogma be self-reinforcing isnt the winning strat. The difference isnt even benefits, retirement, and pension. Maybe in the 80/90s or even 00s that was the case, but it's a dead philosophy carried by dead justifications.
The (employment) contract value should be the same, not the pay.
There is less inherent risk for public sector jobs than for companies that can go bankrupt. Hence for the contract value to be the same, the pay needs to be a bit lower.
> a GS-12 getting paid around $68,000 per year (or $99k in very high cost of living areas like DC)
One valid sounding concern that I’ve heard is that the WASHINGTON-BALTIMORE-ARLINGTON, DC-MD-VA-WV-PA GS Locality Area underpays folks in DC by including farflung areas like PA and WV that skew the cost-of-living analysis. Whether that’s an intentional cost-cutting move or bureaucratic incompetence I’m not sure, but in the end the DC-area federal government pay ranges I’ve seen have struck me as quite low.
Totally. I think comp is a necessary but not sufficient precondition for fixing government technology. The actual solutions (good authentication and least privilege systems, robust monitoring, rapid intrusion detection and response, secure by default system architectures) all take talented people to execute and the government doesn't have enough of those in-house. Instead most systems are built with a 7-figure contract to Booz Allen and friends and then maintenance and sustainment is left as an exercise to the reader.
This is a common misperception but it’s not that simple. Here’s an old study discussing how it varies based on the field, where the lower level jobs do tend to pay better but higher-skill jobs have the opposite trend:
Since the Obama era, this has gotten worse because there were a ton of people trying to score political points saying they were cutting waste by freezing civil servants’ salaries and that really got ugly in tech jobs because salaries were booming once things like the Silicon Valley wage collusion lawsuit and high demand for security, DevOps, etc. started raising the ceiling for the private sector. In 2010 the top end of the GS scale was competitive once you factored in benefits, hours, etc. but a decade later that just wasn’t the case. I knew multiple people who were trying to stay in the public sector but it was literally 2-3 times more money if they went private even though their skills were considered mission critical for their agencies.
This sabotages contract work, too, because there isn’t anyone qualified to guide or review the work and that tends to burn orders of magnitude more money than simply paying more directly would.
That study is a decade old and covers a very limited 4 year period right after the big 2008 recession where the private sector took big losses and had a glut of college graduates competing for entry level jobs. Even then it shows specialized and highly educated workers doing far better in the private sector.
On the other hand, private companies treat security as an almost unnecessary expense, cutting corners. And playing roulette with whether they get hacked.
I think our whole paradigm of computing is unfit for the adversarial world of today. Our systems are like loaded guns where you need to hold 1000 safeties (some of them hidden) for it to (probably) not fire. It's absurd how hard it is to make anything.
Oh, we're good on safeties. The problem is people for whom an additional click an hour or some thoughtfulness making some decisions is a breaking software issue.
This is a funny statement considering that the Fed isn't hiring anymore than any other tech corp. Across the board tech hiring in the U.S. is at an all time low relative to candidate population.
Tech could drop salaries to 40K/year and get just as many resumes discarded in the trash.
Good point, but with the mass layoffs and salary balancing going on, the government may find itself in a relatively more competitive place than it used to
In terms of benefits, here's an anecdotal comparison with a senior engineer (5-10 years experience) at a mid-level start up I worked at.
* Federal Pay (GS-12): $100,000
* Startup Pay: $150 base + $25 k bonus + equity
* Federal Health Insurance (United mid-tier plan, no family): $2,500/year
* Startup Insurance (United mid-tier plan, no family): $0/year
* Federal Leave: 20 days (after 4 years in federal government)
* Startup Leave: Unlimited
* Federal Sick Leave: 13 days
* Startup Sick Leave: Unlimited
The pension I'm talking about actually isn't the TSP (which is fine, but slightly more expensive than comparable Vanguard funds).
All federal employees must contribute 4.4% of their salary to the FERS now which is taken out of their base pay just like their health/dental/fegli. It used to be 0.8% but congress gutted it a few years ago.
FERS takes decades before it's more than pocket change and the same money invested in the market would yield higher expected returns without requiring you to work 20 years in gov to benefit from it.
True that! I use probably 15 days of "unlimited" leave and still manage to feel guilty about it.
The frustrating thing for people in fed jobs is that if you hit your 13 days that's it (during your first 3 years in government). It can be impossible to get PTO until you build up hours again. You have to either quit, negotiate LWOP (often seen as a performance adverse metric on your record), or work. So if you land a sweet concert ticket, see a flight deal, have a friend get married, etc. you better hope you've banked up the leave for it. Since you gain hours every 2 weeks (4, 6 or 8 depending on service) you also start out in government with virtually no leave and can't actually take a 2 week trip until you've been there almost a full year.
> It can be impossible to get PTO until you build up hours again. You have to either quit, negotiate LWOP (often seen as a performance adverse metric on your record), or work.
I’m not sure if this is your actual experience, or if you’re just reading the docs, but…
Most supervisors totally understand the limited leave for folks in their first two years, and they will frequently grant advance leave (basically leave that gets repaid when earned) for folks who are performing at an acceptable level.
It’s not a shit show unless someone wants to take a lot of leave before earning it.
Weddings, concerts, even helping family for health stuff… all that’s usually covered under advanced leave when necessary.
I would say that the leave situation as a fed is much easier than in an “unlimited leave” situation.
The real shitty part, imho, is “time and attendance”. Kicking out early for your kids ball game, for example, will cost leave. As a business owner, I like that I can just stop working and do whatever.
It is extremely easy to burn out cause if you’re the best and have aspirations to move up, you’re just fucked. You will be blocked at every single opportunity while others around you fail upward.
I guarantee that someone in the org saw a password file and said “yo? wtf? Let’s get a proper secrets vault going we can do it ov…..” *punched in the clit, thrown out a window*
I call BS. I've never heard of anybody in government "going to jail" for some sort of mistake. Sure, there's all kinds of threats and regulatory control but when it comes down to it barely anybody is held to any kind of responsibility. It's practically impossible to fire someone in the government for incompetence and that's coming from engineers I know in government who work with essentially weaponized incompetence.
This is a dated example but since "you've never heard of it", it's still relevant. I worked at Ford Aerospace/Loral and Boeing on space shuttle contracts. Part of the training was a video interview with a sysadmin who left a job on a Friday, went to a different role on Monday and then remembered a script he'd need for his new job. Same employer, just different government contracts. He logged in to his old system and copied it across since his access hadn't been cut yet. Five year sentence in federal prison. Now you've heard of it happening. Happy to help.
Yes, he shouldn’t have accepted bribes, but in the private sector this would have been extremely unlikely to result in jail time.
Even if jail time isn’t a common thing, it’s far closer to happening to the average person working in the government than it is to those working in the private sector. The private sector simply fires bad employees. The government seeks to be made whole.
I'm not really impressed by someone going to jail for accepting bribes, even if it's less likely to happen in the private sector.
Show me someone going to jail for bringing down prod or making the wrong architecture call or choosing the wrong platform/backend/language or even just getting burnt out and spending a week on the clock re-watching all of Star Trek: Voyager. I want to go, "Holy shit, that could have been me!", not "Well no shit he went to jail."
The company I work for (high profile private sector U.S. defense contractor) has security people (FSOs and such) that are constantly concerned about being held legally responsible for actions (or inactions) related to theirs and other's work (specifically those with personal or facility security clearances). They regularly claim that they can be held responsible for the failures of others.
Their hesitation leads me to believe these legal repercussions happen more often than not. Would be interesting to see some data on the claims. My guess is the people being held responsible for these things aren't your average developer taking down prod.
That's a separate issue. There are criminal and administrative penalties for mishandling classified information that apply to anyone with a clearance, regardless of whether they are a government employee or private contractor. As long as you follow all the rules yourself you won't be punished for someone else's actions.
There's an excellent book about this topic. Three Felonies a Day by Harvey Silverglate. Convictions for white-collar crimes aren't about stopping significant crime, they're about building statistics by sacrificing the most convenient bodies for expedient wins.
"Improve" government by scaling it back down to where it was when pennies from tarrifs could pay for it instead of 25% Federal income tax that already gives you mediocre results.
Counterintuitively, scaling government down goes hand in hand with increasing the attractiveness of the civil service.
Right now if a government agency wants to do something like make a webform where you can apply for a passport, they have zero web developers on staff who can do it. Instead they must pay a team of non-technical officials and lawyers to make and adjudicate an RFP. Then pay a contracting firm to put a developer behind a government computer to do the actual work. Putting this contractor in a seat can easily cost the taxpayer $500k a year despite the contractor only receiving $130k of that money. The rest goes to the HR department, IT Department, C-Suite, lawyers, lobbyists, and shareholders at the contracting firm. The government has their own HR/Lawyers/IT too, but the contractor can't use those so the tax payer ends up double-paying overhead and missing out on economies of scale on every contract.
This is one of the many reasons government websites are always $50 million dollar boondoggles that an intern could have done better. The government ends up spending millions of dollars feeding leeching middle-men before they can hand that money to a mediocre dev deep in the bowels of Accenture's cheapest subcontractor.
If an agency just could hire a few strong web developers directly and then assign them to whatever task is needed during a particular sprint, we'd see a massive reduction in cost and increase in the quality of engineers working on our country's most important work. But most agencies are literally not allowed to spend more than $120k on an in-house engineer, while no one bats an eye on them spending 5 times that on an Accenture contract placement.
> If an agency just could hire a few strong web developers directly and then assign them to whatever task is needed during a particular sprint,
Isn’t that what usds [0] is for? I think there’s always an alignment challenge for service needs that are outside an organization’s primary knowledge domain. Without knowledge of what the “strong web devs” can and can’t do then the results are often not great [1].
USDS is great! I know people who have made a huge impact there and if I personally were to go into government from tech it's where I'd look. They are situated at the White House which allows them to be hired at a higher level than normal federal jobs (up to GS15, though still lower than comparable private sector work) and then they get sent out to various agencies by the White House to try and fix things. In practice though, USDS is a tiny tiny drop in the bucket compared to what federal agencies actually need. Maybe if every agency had a digital service of their own the model could work.
The federal government is an enterprise with 4 million employees (more than half in DoD as military or civilian). So the handful of people at USDS are basically only sufficient to swoop to fix the most dire of dumpster fires.
But then who would pay for all of Israel's bombs? Think of the foreign nation whose citizens are happier and healthier than you with single payer healthcare?
I'm not sure more money => more talent in quite the direct relationship you're suggesting here. If this were true, the cryptocurrency industry would be the most secure in the world, since they pay their engineers the most.
Stealing crypto money is an order of magnitude more difficult than stealing internal data from an average government office. So in a weird way, yes, the cryptocurrency industry is more secure.
Nice they are doing their job and glad they exist.
But how to fix ? Most US Gov agencies are underfunded, it is either beef up security or provide services. Really a tough choice, and the outlook looks like they may lose even more funding.
Doesn't the US outspend in terms of dollars almost every single developed country on the planet at absolutely everything, even in per capita statistics, from military, police to education and healthcare? How could it be underfunded?
In total the US executive is very well funded, but individual agencies aren't always well-funded and often lack the specific skillset to properly utilize funding even if they have it, especially wrt IT systems when that's outside their main field of expertise.
Omar a minimum we need to deduplicate insane degrees of replication. Why does virtually every government agency have dedicated law enforcement? Narrow it down, reallocate budget, eliminate the wasteful overhead.
It’s not all or nothing here. We spend too much and get too little out of it.
Law enforcement is incredibly difficult, especially with the number and types of laws on the books. You want multiple different law enforcement agencies so they can specialize based on the field of law they are in.
Currency counterfeiting is a different set of laws than interstate financial fraud, which is a different set of laws than throwing a Snickers wrapper on the ground in Yosemite.
The federal government does too much. We could simply eliminate many of those agencies and save money for taxpayers, or redirect those resources to more important functions. There is excess capacity, it's just in the wrong places.
As for IT functions, all of that should be centralized under the GSA with proper security controls. There's no benefit to having every agency maintain its own IT infrastructure. Most of those staff are redundant and could be laid off.
> As for IT functions, all of that should be centralized under the GSA with proper security controls. There's no benefit to having every agency maintain its own IT infrastructure. Most of those staff are redundant and could be laid off.
This works for email. Would you work at a company where you had to build and deploy apps on infrastructure controlled by someone in a different department and location, whose boss gave them the mandate to standardize as much as possible to reduce costs? (Hope you like Oracle…)
That was your fairly typical on-premise corporate hosting. Things have gotten better with the cloud but it's still hardly a free-for-all, use-whatever-you-want-with-no-oversight situation.
Cloud environments helped, but the kind of massively centralized environment they’re talking about in the cloud work still tends to mean “get 4 levels of approvals and you can get a t3.medium using our AMI with Java 6”. My point is just that successful IT needs people who understand the mission and share your incentives – you can outsource email and other generic services but most people here work on things which aren’t one-size fits all.
Sure, I have worked at companies that manage their IT infrastructure that way. It's fine, way more efficient and secure than letting every department do their own thing.
I worry for having a centralized IT infrastructure as that now puts every single agency at risk from a single attack. No one would call a neighborhood "secure" if every home used the same key.
It's getting centralized anyways, just not inside any federal agency. Instead they're outsourcing it to companies like MS and Google who provide hosted services. This gives the agency cover so long as they do their part (like making use of MFA, using encryption on email). Then they can offer a claim of making their best-effort and going with industry "standards" (standard as in common, not as in ISO, ANSI, or others).
You can only spend money on what your budget specifically allows. If you’re in the military, the fact that you are authorized to procure $1B aircraft doesn’t mean you can hire a $200k IT security engineer to protect your HR system and you can go to jail if you try to pay for an application upgrade out of that budget unless it’s directly linked to that program.
If you’re not in the military, the fact that someone else has a big budget doesn’t help you any more than your neighbor having a Mercedes helps pay your internet bill.
There are general budgets and people build in support costs, of course, but it’s terribly easy to find people who have been asking for budget to replace something years before its end of life but keep getting turned down in the congressional budgeting process. Politicians want to fund things their constituents like, but the unloved internal support app is just as much of a risk to have on your network.
When political candidates vow to "trim the fat" of the US government, the military is typically off limits, but the other government departments certainly aren't.
The military? Yes. They get more money than they know what to do with - I heard a story about how there's a base where all they do is build M1 Abrams tanks on an assembly line, then disassemble the tanks, and re-assemble, and then disassemble, Ad Infinium, in order to spend all the money they're allocated. It's always a political winner for Congress to give more money to the military, so their budget has become astronomical and only continues to grow.
Every other agency and branch of the US government? Absolutely not.
Look, I heard a lot of things from the guy at the street corner too, but that doesn't make it true.
We'll start with the fact that M1s aren't built on a "base", they're built at the Joint Systems Manufacturing Center in Lima. Government owned, contractor (GDLS) run, not a base.
What they do do is refurbish older tanks, which one I suppose could distort into "disassembling", if one wanted to make a rather distorted claim.
The waste contention for that base comes from an Army proposal to temporarily shut down the factory in 2013, which was supposed to save ~$1B. GDLS explained that, sure, can do, but spinning up production again is going to cost ~$1.5B, and restarting production in 2017/18 was always planned. It's not as simple as "the politicians always allocate money to the military".
I'm not sure about that particular story, but many similar stories are rooted in a very real issue of maintaining domestic supply chain expertise.
While centrally managed economies can just mandate that state-owned factories continue to exist, private markets won't do this. If you don't order tanks and missiles, the factories that make tanks and missiles will cease to exist, and the market will reallocate resources elsewhere.
Do you have any source for the claim that there is a factory with the sole purpose of assembling and disassembling M1 Abrams tanks without actually delivering or upgrading any of them?
That sounds very implausible, bordering on conspiracy theory.
It’s bullshit. I write software for autonomous tanks. Part of our contract is to have procedures for literally everything, which includes how to take apart and put back together the robot in order to be able to maintain them.
We of course run these procedures while writing and formally verifying them, before handing them to the customer, not running them would be folly.
Whatever this story is sounds like the worst kind: a small nugget of truth surrounded in a giant ball of shit.
Even more than that, a customer doesn’t just hand us a pile of money and say “talk to you again in 2 years” far from it. They literally “status the status” once or more weekly. If the answer to the question of “what is the plan this week” has the answer of “we’re out of work so we’re just charging you pad our profits” won’t play.
Also, building tanks is fucking hard, I sincerely doubt any company in the industry isn’t using every penny they have to solve problems and deliver a good product.
For those that scoff at the last bit: companies are in the business of making money, and if you deliver an inferior product, you won’t get another contract.
The US government is broadly disallowed, for political reasons, from doing anything. So they have to buy services on the open market, where they get charged through the nose.
Our state legislature recently voted down adding a new Data Analyst position to one of their departments. That department cannot function without that position, so instead it has to use it's funding to buy that same position from a 3rd party contractor for 3x the price or more.
The result is that we pay more than anyone else for basically everything we do.
I generally agree that the government often hamstrings itself. But consider also that you can't terminate state employees easily, and their benefits packages often cost more than 100% of their salary. The salary itself and the working conditions often don't attract the best talent. Thus it's not always so cut and dry in terms of what's the best outcome for the public interest.
My understanding is that regardless of funding, the US federal government has standardized pay scales that top out way below what private industry pays, so even well funded agencies can only possibly get junior developers/IT or people that are willing to take a significant (50-80%) pay reduction. The very most you can possibly make as a GS15 in 2024 is 191,900, and they have locality-adjusted pay with most localities being below that.
They might also generally still drug test? I don't even do drugs, but I'm not going to pee in a cup for someone to effectively do charity lol. Good luck recruiting a professional with decades of engineering experience when you treat them like they're a 16 year old working at Taco Bell. Even someone with 0 years doesn't have to deal with that kind of treatment in industry.
Drug testing is mostly limited (for civilians) to those with access to sensitive, secret, or TS information. In those orgs, you have higher odds of being drug tested as a contractor in the same team than as a federal civilian.
Regarding pay, it's actually pretty bad. A typical IT worker will be a GS-11 to GS-13 depending on location and degree (possibly lower in some locations, maybe higher in some high COL areas). GS-13 in many places is restricted to management and SMEs, though they're bumping up a lot of the "working level" grades because they realize they can't compete in hiring.
To pick a high COL area where you might find GS-13 working level IT folks, San Diego GS-13's max out at $153k. If they're actually GS and not another pay system (has a different pay raise method but usually maps to some GS grades, like Acqdemo) then it takes 18 years to go from GS-13 Step 1 to GS-13 Step 10. Most likely they aren't starting at Step 1 in any grade, let's say they start at Step 4, then it's 12 years to max. Once maxed, they only get the general pay increase every year. There are few technical GS-14 positions (this is changing, but not rapidly) even in high COL areas so the only "promotion" option for many is to go from a GS-13 technical role to a GS-13 management role (same pay) and then leverage that into a GS-14 management or technical role, if someone dies and a position opens up. GS-15 technical roles are pretty rare.
Charity? I sympathize somewhat, but I’m also disgusted by the utter lack of respect for government and societal service in general. That shit means something.
I wish to believe there are still people that don’t care about making Yet Another few hundred thousand and just want to actually contribute to society instead of working on ad tech or whatever bullshit.
Regardless of whether or not one personally enjoys the work one is doing, if one really is contributing to society, one should get fairly compensated for it.
Additional requirements not common in the private sector, such as rigorous drug testing, ethics codes, requirements on gift reporting, increased surveillance, etc., should come with additional benefits to compensate. Instead, government workers submit to these requirements and a substantial pay cut.
That's mostly because conservatives 1) desire tax cuts at any cost and 2) want to demolish the entire administrative state. The stability and consistency that comes with a well-funded civil servant class are an obstruction to their stated goals.
I vouched your comment, because I think you're precisely making the relevant point in the first two paragraphs.
However, I think you're wrong, at least in part, in your third paragraph. I mean, I think the word "mostly" is wrong in that paragraph. Politicians from all political factions are (quite reasonably) under pressure to lower the cost of doing the work of government, and (quite reasonably) to raise the integrity of the process. Combined with some of the dysfunction inherent in agent-principal problems, I think that's more than enough to cause the problem you're talking about. I experience this firsthand in a jurisdiction that has much less of the "demolish the entire administrative state" that afflicts the American right wing (which I'm guessing is your point of reference).
Mind you, I am not claiming that the problem is not badly worsened by American right-wing politics. I wouldn't know. I'm just claiming that the problem is semi-intrinsic to the situation, and I strongly doubt that it's "mostly" caused by those particular political issues.
I'm confused. You're complaining about the use of the word "charity"?
Background: You make an argument that at least some people should consider putting contributions to society ahead of "making yet another few hundred thousand". I agree with you, at least broadly, and I think the up-thread poster is not disagreeing.
Summary: We're discussing the act of taking a personal financial hit, for the good of society.
The word for that is "charity". That's what that word means.
---------
I also am sympathetic to the GP's point, about which you are so "disgusted", but I think there's room to disagree there.
I am sympathetic because professionally I do work that many people think is "good for society", I currently earn approximately median income (below mean) for my age/gender/nationality, far far below software engineer pay, and I am treated with unbelievable disrespect by my employer, the government. If I was not trapped in this job by personal circumstance (for now), the disrespect part would definitely factor into my decision making about staying in this allegedly-virtuous job. If you're gonna pay people below market, and you treat them badly, that's not a combination that gets you quality employees. Even if there's some social purpose.
Doing something out of a sense of duty should not require a vow of poverty along with it unless we plan on committing to lifetime benefits and support for the people who take that path (like providing food and housing, because the low end of the GS scales are literally below poverty rates as it is).
>Doing something out of a sense of duty should not require a vow of poverty along with it
The wages offered are hardly poverty - just not competitive with the private sector.
Besides, "doing something out of a sense of duty", when duty meant something, has also often meant doing it for free, or even doing it on one's own dime, and it absolutely meant accepting a pay cut.
If you are a GS-5 (typical entry level government roles, 5 rungs up from the actual bottom of the pay scale, since it's literally impossible to get applicants for a GS-1 role if you tried) and support a family of four you are currently at 2023 rates within 3 digits of income from the poverty line.
If we push it lower how are we not expecting that to require poverty? What legion of people in the US do you reckon even have "their own dimes" to spend on being full time volunteer public servants and can afford to serve from a sense of duty? Retirees?
Giving up 50% or more of your income can be a completely different life. It's not "only" making 300k instead of 400k. Based on the other comment saying G13 or lower is more likely, it's making 115k or less and barely being able to afford a house near not great schools where your kids will probably get a worse education than you did (after all, you presumably have a CS degree since the government fixates on degrees and credentialism).
Not all tech jobs are ads. I work in networking equipment and it pays much, much better.
Anyway, my point was they don't even give respect to the people who do that, and still treat you like their property. Same with the vaccine mandates (especially for remote workers): whether you got it isn't the point. My employers have never asked because it was never any of their business.
You'd have to look at purchase power, not dollars, to see how much they can actually do with all that spending. You get a lot further with $5 in a place where wages are $1, than with $20 in a place where wages are $15.
The US is really really big. People way underestimate how huge it is in terms of land area. You could fit Europe inside the US. So the reason Germany or France underspend the US has more to do with the area they have to cover and the number of people they have in their borders.
The argument is density. Things tend to be cheaper per capita, when density is high.
I don't buy it thought. I think the reason why the spend is less in Europe is due to higher salary equality - good people take job in government because the salary is only 50% higher in private sector (for tech, even less for other areas).
There are only a couple of US states less dense than e.g. Norway, and even in those states the vast majority of the population live in small, higher density areas.
The areas where the vast majority of Americans actually live are fairly high density.
Some cost might come down to density, but not much.
I'm guessing the poster was thinking of "Europe" as more like "the EU" or "Europe minus Russia"; 1.1 million of Russia's 6.6 million square miles are part of Europe's 3.93; take it away, and "Europe" drops to more like 2.8 million square miles, a smidge less than the contiguous US.
Why would you ignore the largest state in that land area calculation? Alaska still isn’t quite enough to push the US up, but it’s good for another 20%-ish increase in land area over the contiguous number.
>Why would you ignore the largest state in that land area calculation?
As a non-contiguous later addition, with a small population, where statistically nobody lives there per sq mile, and is not pertinent to the discussion of population density as related to infrastructure problems?
Except in what's holding US bureucratic efficiency down (what we were discussing), and requires spending inflated federal budgets for little returns, Alaska is a big factor relative to its size...
The problem with such "lessons learned" is that they are usually not lessons learned - I bet there were numerous people within the organization who knew about these issues before the test.
The actual issue is probably that these people are (a) ineffectual at communicating and prioritizing concerns clearly; or (b) good at communicating, but are not being listened to; or (c) they are listened to, but the organization has no practical means to fix this - no money, unable to recruit talent, etc.
Most techies often assume (b), but (a) is at least as common. The last issue - (c) - might be superficially true, although it's usually not correct in a deeper sense: there is plenty of discretionary and wasteful spending in any sufficiently large bureaucracy. Central resource allocation is just a hard problem.
Anyway, my point is that the problems that need fixing are almost never just technical. Recommendations such as "implement sufficient controls to detect malicious activity" seldom get to the root cause. They are still useful in temporarily overcoming organizational obstacles, but it usually doesn't last.
Private companies get breached often too (see AT&T).
Pretty much everyone gets breached.
The only ones I don't think get breached deep are the really big software engineering companies where most of the company are also software engineers... like Google.
Software is too complex to be secure without a massive team IMO.
> Software is too complex to be secure without a massive team IMO.
We could do better as an industry though. Modern operating system design makes it far too easy to shoot yourself in the foot.
Imagine a world where all we all use memory-safe/null-safe/type-safe languages, applications and data are strictly sandboxed, access to data is only granted using capabilities-based security, application-level security patches are automatically applied by the OS, data was always encrypted while at rest and while in transit, and passwords are completely replaced with passkeys / smartcards (for users) and X.509 certificates (for servers). While this isn't a panacea, it would solve a great number of the most common security vulnerabilities.
Each of these pieces exist individually. There's no reason why we can't have all of these things today, other than support for legacy applications and retraining engineers. However, it's nearly impossible to get away from legacy software needs.
But if you want low hanging fruit... stop writing C/C++, and get rid of passwords. These are the biggest flaws in the stack.
There is no incentive for the industry to do better. As for government penalties being an incentive:
If it was popular amongst the voters to hold corporations seriously responsible, you would see politicians campaign on it and win. It's not nearly as popular as virtually anything else based on empirical data.
Another way to say this is if 80% of all voters, regardless of party prioritized this as the #1, #2, and #3 issue, politicians would pass laws. It makes the politician look good and solidifies their reelection. Likewise, politicians that vote against those laws would almost certainly not be reelected.
Eh, a massive team makes its own major issues. See the many Google leaks over the years.
Assuming software can be secure (and hence not doing proper defense in depth, limiting the types and nature of information processed, etc). is the bigger issue IMO.
Standardization by centralized administration, reduction of rogue external and DIY options, concentrating efforts on delivering managed services based on a common and repeatable platform, strict change control, clear training enumerating what is and what is not allowed, and a security team that actually digs in to make cross-cutting concerns more audited automatically, practical recommendations, least privilege continuously ratcheted down, and advise on new features and new projects.
I remember one time Satya said the red teams reported to him which Microsoft services they were currently in. He would then ask the heads of those services if they had detected any breaches. Sometimes there would be services that had been breached for years, undetected. Must have been hard for Satya to keep a straight face.
One phrase that struck with me from their security training: "Assume Breach".
The red team would beach them then wait years before telling them?
The goal the the red team should be to increase security. Waiting years before telling them means there's a years-long delay before security can be improved. That goes against the goal.
i liked that Solo told the backstory to this that made sense. iirc it was an intentional vuln built in as a protest by the architect who was forced to work on the death star
I work with a security officer who joined our company after a career in the DoD and other similar places, always in computer security.
He is so incompetent that I initially thought that we have communication problems (I am French). But no - he simply has absolutely no idea about cybersecurity and the teams he "oversees" from that perspective are losing their minds.
I had no idea that you could work for years in such sensitive US environments and have completely no knowledge.
He is good at saying generalities, though, with complicated words.
I'm not a huge fan of how red teaming is generally conducted. It's sometimes necessary, but the CISA report seems to indicate that the organization wasn't responding to their requests the way they wanted, leading to a communication breakdown right from the start. The vulnerability was patched and the red team's initial compromise was contained, so they targeted them with phishing, owned their domain controllers, then maintained access for months while pivoting to partner organizations, then published a public report.
It's hard to establish constructive dialogue after that, allot of bad feelings and burnt bridges - and sometimes HR. It's tough because there's generally allot of dynamics at play, but I'm sure the impact of this testing was felt by people within the targeted org.
> The vulnerability was patched and the red team's initial compromise was contained, so they targeted them with phishing, owned their domain controllers, then maintained access for months while pivoting to partner organizations, then published a public report.
You're missing a few steps between the pivot to partner organizations and the report. The public report was made on 11 July 2024. They revealed the breach to the target last year, June 2023, and then began a collaboration effort at that point, running through September 2023. They also don't name and shame in this public report, we don't know what the target organization was.
My next sentence is really a comment on that step.
I generally find that working incrementally alongside the teams provides better outcomes. This is not to say you can't use black-box testing or perform unscoped testing, but collaborating throughout the process gives you additional visibility that can save a lot of time, especially in large-scale and diverse environments. People generally respond in a more positive and collaborative manner as well.
"It said real adversaries may have instead used prolonged password-praying attacks rather than phishing at this stage"
Did the author mistype Password-Spraying or is there a seperate type of attack known as Password-Praying? Googling doesn't reveal any other hits on this term.
I really really want to root for CISA, but just a few months ago they leaked a trove of critical infrastructure documents that they had collected from partners, that if they hadn’t collected wouldn’t be in the wrong hands currently.
Long ago I worked on a government contract at a civil agency, which ran WordPerfect Office on DG minis. The main contractor won a contract with another division in that agency, setting up a slightly spiffier version. Somebody at the COTR's office at the other division encouraged or perhaps dared us to break in. It took about two hours. We let them know at once, but I think that with a bit of discretion we could have maintained our presence for a long time.
Who on earth is still using solaris in this 2024th year of our Lord Jesus Christ!?!?!? I legitimately thought it had been EOL'd at least half a decade ago, guess I was wrong about that.
Your typical hands-on-keyboard blue team engineer in federal government is a GS-12 getting paid around $68,000 per year (or $99k in very high cost of living areas like DC). They have expensive health benefits, 13 days of PTO a year, put a huge chunk of their paycheck (almost 5%) into a mandatory pension plan that consistently underperforms the market, and can literally go to jail for making mistakes at work depending on the statutory context they work in.
The best people in these jobs burn out fast and quit or they end up having to abandon IC work for GS-14/15 jobs (max pay is around $190 for those) in order to keep up with cost-of-living and justify their careers.
As a result, you have almost zero genuinely capable principal/senior engineers in government who have the authority to architect complex IT systems for security. Instead you get contractors who charge the taxpayers enormous overhead costs and cut corners wherever possible.
If there's one letter to write your congress person to improve government - my vote would be for civil service reform to attract and retain actual top tech talent. They've done it for doctors and lawyers (both of whom can get paid well above the $190k GS pay ceiling), but engineering is still not treated as a comparably skilled professional trade.