I've done most of the spectrum from Perl webdev to chip design, with a good chunk of embedded C and C++ in the middle. I basically agree, and I think there's roughly these types of challenge:

- Inadequate or high-effort tools. This is half the challenge of the embedded world, and the cause of "I've destroyed my tools with my tools!" https://www.usenix.org/system/files/1311_05-08_mickens.pdf

- Complicated business requirements ("do this every second Tuesday if there is an R in the month, but not if the customer is using the US VAT system and not if the product is a type of cake")

- Critical system requirements (remember, using Java to control your nuclear reactor violates the license). People working in this environment can easily get elitist about it, until you look at their code and find Toyota spaghetti.

- Problems that require Actual Maths or Actual CS, with known solutions. Rare in the webdev world until someone remembers about ACID; more common in the embedded world if you need control systems.

- Problems with unknown solutions. Actually extremely rare; often the first line of attack involves Matlab or Excel or some other "not real programming" technique. Often the smart domain experts you need to solve the problem are not also very good at software engineering.

IME the first two take up 80-90% of the working day. The third is only applicable to some industries.

Different people are better at different types of challenge. This doesn't matter unless you're trying to construct some sort of total ordering over human beings in order to feel better about yourself.

> - Complicated business requirements ("do this every second Tuesday if there is an R in the month, but not if the customer is using the US VAT system and not if the product is a type of cake")

And then you realise there's no consistent definition of "cake" in the system. And when people in the business say "US VAT system" they actually mean "Regulatory system XYZ", not actually anything to do with the US.

Sigh.

Plus the requirement is not neatly in one place ("do X every Y except if Z"), but strewn all over a document (or several), and different requirements contradicting each other, with no priority defined.

Funny. Most of my career was about problems with no known solutions. I wonder why people keep calling them "rare". After all, why even bother solving a solved problen?

And yes, most useful prototyping tools for me were far from C++ and alike - I used Maxima, Mathematica, various Lisps with tons of batteries. Also not a "real programming".

So there's two things:

#1 Just because you're constantly tackling interesting problems doesn't mean they're super common. Most of us seem to be working on an application meeting some trivial business need.

#2 Just because you don't know the answer doesn't mean the answer doesn't exist. The world of development is filled with 20 somethings getting paid huge sums of money to learn and deliver something completely alike to all the other things they think are really different.

You probably fit into one of these categories, if it's the first then congrats, because that sounds like an interesting life.

> doesn't mean they're super common

I still cannot understand what justifies re-solving a solved problem over and over again? Solved once? Automate it.

> Most of us seem to be working on an application meeting some trivial business need.

OP says it's somehow "not trivial". For some weird reasons.

Anyway, if the workflow is trivial, if required components are standard, if no customisation is required - it does not need any engineering at all. List those initial requirements, infer the implementation automatically. It is hard to comprehend why the CRUD world is so reluctant to do such an easy thing.

> doesn't mean the answer doesn't exist

It may exist in some highly secretive labs, but in most cases it cannot be found in any of the papers or preprints.

So just because something is trivial doesn't mean that it doesn't require time to implement the rules. You can be entering a bunch of convulted rules into a GUI based system or you can be writing them in code.

I largely actually prefer the code way, believing it's much more flexible, sane and safer. However, the problem is the average person who understands these rules is scared of the code, and the average coder is much more interested in solving a lot of inconsequential problems.

This makes it much harder to pay someone in who's just going to sit down and churn out your CRUD and when folks find these guys, they're either keeping them happy or think they're easily replaceable and will have a fall later on.

I'm doing devops at a big org at the moment. They have several dev teams working on bespoke low-traffic resource managment applications. Most of these have 5 developers, devops, delivery manager, user researcher, designer in addition of going view a technical review process, pen testing, performance review, and functional testing.

Basic math suggests that's going to come out at about £350,000 per project over 3 months. A million quid later, they've all passed reviews without any of these so called highly technical people realising they're dealing with a common problem and that they're developing 3 of them. Also they're all shit because they're done by MEAN stack tech hipsters who haven't quite realise they're leaking important data.

> You can be entering a bunch of convulted rules into a GUI based system or you can be writing them in code.

There are more options than just these two. The best way is to write your rules in a nice, readable, dense DSL, designed specifically for that domain experts who know the rules but are afraid of code. And such a DSL can be very much free form and forgiving, helping a lot along the way, so the experts won't need much assistance.

With such an approach, developers (i.e., those who are not afraid of code) are either not needed or only concerned with maintaining this DSL, while the experts can code their rules directly. It eliminates unnecessary elements of a chain, and cuts costs quite significantly.

> It is hard to comprehend why the CRUD world is so reluctant to do such an easy thing.

It is odd, isn't it? I'm tempted to say if you think it's so easy, go forth and do it: automating away the production of CRUD would earn you billions. SAP is probably the nearest existing product. But every attempt so far just creates a different group of developers (e.g. SAP).

If it's so easy then it would be already automated. Expectations rise when something is easy until it no longer is easy.

In a sane and rational world - yes, it would have been automated. But in our world everything is done the most twisted way possible, either deliberately or out of some deeply ingrained human stupidity.

I cannot stop shaking every time I interact with pretty much anything designed by supposedly high profile specialists. There is no single day I do not run into an astonishing stupidity. Daily commute? London Oyster system is such a thoroughly shitty design that it is nearly unbelievable. Shopping? It never works as one would expect, and is always far too twisted - all that Amazons, Ebays and alike. Healthcare? Just super stupid, stupider than the trashier of the comedies. Yes, they "automate". They decided that a doctor should not type or write, to save a valuable time. They're expected to record everything on a voice recorder and pass it to an assistant for typing. Guess what? They type or write first, because it's easier, and then read it aloud. Shall I go on?

This world is a pile of stupid shit, throughout. And expecting that something that definitely should have been automated or eliminated altogether would have happened is just totally naive.

And if someone really do automate all the CRUD needs for 80% of the possible users, they'd never know anything about it. So why bother? They must grow up and automate it themselves, it's dead easy.

If you are specific enough then most problems have no known solutions. I work for government and we see a lot of solutions that solve most of our problems but there is always a catch. That cool cloud stuff? Can't use it. The problem in reality is we need a solution for X that also satisfies our privacy and security policy. From this perspective there is no known solution. We have to develop our own solution.

>> - Inadequate or high-effort tools.

This makes sense for embedded: you want to save every penny since it's a high volume business, the chip manufacturer wants to save every penny , there's a huge amount of variety in chips - so in the end development is hard. Without all those requirements, heck maybe something like the arduino/mbed would have become popular among professionals.

But as for webdev ? not so much. The critical factors are time-to-market and development cost. And in the domain of web based business apps, we already have useful high-level tools[1], and it seems to be technically possible to improve the situation in consumer facing apps too.

So where are the tools ?

[1]http://ww2.mendix.com/rs/mendix/images/gartner-magic-quadran...

For webdev, the biggest inadequate tool is the browser itself: ancient versions, vendor quirks, mandatory single language. This is mostly due to it being the place where various competing interests have fought one another to a standstill.

But a good, high level tool solves/abstracts all those issues for it's developers(with the cost of some performance decrease) . The business oriented tools manage to do so.

You hit the nail on the head with the Actual CS problems. Storing data is probably the only complex area of web deevelopment, but it's something that 90% of developers seem to punt on.

Really, there is nothing wrong with being a "PHP Programmer," or even a WordPress "programmer," if you can find people to pay you for it. Well done, that counts as succeeding in life.

On the other hand, if someone has spent years feeling bad that they don't know lower-level programming, they need to get over themselves and spend a few weekends learning assembly or something. There's no reason to feel bad about this, take a bit of time to expel your ignorance.

Completely agree, I'm currently a PHP programmer (and quite a lot of Python and JS and some other stuff).

I grew up programming and had my BASIC, Pascal, C++, Delphi and C# phases, I like web development (generally, the tooling sometimes sucks but as a platform you have global reach), I'm also well past the point where I worry about "real" programming or whether I'm a "real" programmer - I can solve any problem I run into by research or buying a good book, there are domains I'm completely oblivious too (for example 3D programming and quite a lot of the machine learning stuff).

The systems I build solve problems and provide value, that value might be in making someones life easier (which is what my current side project will do) or automating a business process (my favourite was an engineer management system I built for a small company a few years ago, they've now grown to ~20 engineers and are still using it, they've filed 3 bugs in 3 years - literally the best work I ever did and at the time I thought it was average).

Programming is such a vast field and there is more out there than you could learn deeply in a life time so I try to stay abreast of what's going on by watching conferences online and not worry too much about measuring my e-peen. (latest one is Code Mesh https://www.youtube.com/playlist?list=PLWbHc_FXPo2jB6IZ887vL... which has some stunning stuff).

If I have a particular strength it's that I enjoy the process of meshing what the client thinks they need with what they actually need and finding the intersect, often clients don't realise what they can gain from software and the simple business suggestions can be huge, I wrote a system for a company that used NCR pads they had to get custom printed, this required the field engineers to keep the pads in stock, they where filling them out by hand for every job and it was expensive and error prone, I suggested instead that we generate the PDF's with everything prefilled in and duplicate pages which are sent to the engineers by email, they can get them printed wherever they are either on site at the clients or any place that does print from disk, this saved them a lot of money, it hadn't occurred to the client that this was possible - so now they assign jobs to an engineer, the system generates all the paperwork for that week and they just require a signature from the client who keeps one copy while they keep the other.

> Well done, that counts as succeeding in life.

Yes, as defined by most people.

Succeeding in life in the eyes of a select group of people including yourself, is something else entirely.

> I've been doing web client and server programming for some time now, and I can without a doubt claim that it poses its own set of challenges that compare and oftentimes exceed in complexity to those encountered in the "lower level" programming fields.

I'm a math major turned 'full stack' dev with an eye on embedded dev as my prize- I needed this comment.

It's extremely true. I occasionally joke about how web programming isn't really programming, but it really is nothing but a joke.

Making a LED blink on a microcontroller is just about as difficult as showing an intermittent alert in a browser window. It may seem harder because you need to know a little more about the abstraction layers underneath, but those are things that you can pick up just as quickly as JavaScript.

Designing a high-throughput data interface system in FPGA (which I'm doing now) is at least as hard as doing a high-throughput back-end infrastructure for a web service. I can barely do the former -- I sure as fuck can't do the latter right now, regardless of how "real" my programming skills are. When I read through a paper on the latter topic, or watch a presentation, I can mostly understand what it's about and appreciate the technical prowess, but doing something like that myself would likely be a quick lesson in humility.

If you like what you're doing and you're creating programs that do useful, or at least spectacular things, you're a real programmer.

The difference is that with microcontrollers, what you see is what you get. The API is defined by the hardware, and most of the time you're just poking numbers into registers. There's nowhere else to go.

With web programming a lot of the difficulty is self-inflicted by the industry. To get a web page up you need to know at least two different abstractions (HTML and CSS) which aren't clean or well-designed and have more warts than a skin care clinic.

Then, to fix their limitations you need to use js.

Then, because there's a lot of repetition in web dev, you have to use a handful of js frameworks, which currently outnumber the stars in the universe, and each has a production half-life of around six months.

And then there's the server side.

And if you get all of that working you have to run at scale, with adequate security.

And can we have it in green and make the logo bigger?

So of course it's hard. It's like being trapped in a novel by Kafka written in XML.

> The difference is that with microcontrollers, what you see is what you get. The API is defined by the hardware, and most of the time you're just poking numbers into registers. There's nowhere else to go.

No it's not. In real life, it's fairly rare to write 100% bare-metal applications, and even when you do, you're often writing against some "homegrown" abstraction layer. Sometimes you're expected to write such an abstraction layer, but writing only against the hardware-defined "API" is usually not strategically sound, unless you have a very good reason to assume that the underlying hardware will never change. This is sometimes true for very resource-constrained microcontrollers, and virtually always true for things that have ARM in their name and stronger. Not to mention for embedded, but relatively high-power devices (see e.g. the Linux kernel).

Sure, the drivers are hardware-specific, but the applications are usually programmed against their (unitary) interface.

I'm not arguing against the rest of what you mention -- I'm painfully disappointed by the neverending spaghetti of technologies that modern web is, and I also think it's largely self-inflicted. However, web dev is treated as much more of a commodity than other branches of programming; left to their own devices, there are a lot of web developers who would produce very good client-side code.

Shit like bootstrap isn't there because web developers are incompetent. It's there because much of their job involves churning out support code for products managed by people who know nothing about technology and the WWW, whose quality standards are dubious at best and who value product conformance over performance because they sell perception, not quality.

Just admit it. Most of the challenges in the frontend development are only there because of a horrible overengineering of this entire stack. It was supposed to be easy.

The Web is a full-spectrum application delivery mechanism that allows for completely untrusted, actively malicious programs to be run, and it evolved from a mechanism to display static documents. Of course it's going to be strangely designed.

Given those requirements, your only choices are Java Sandboxed Apps, Flash and the Web. Which would you choose?

> and it evolved from a mechanism to display static documents

It evolved from a deliberately dumbed-down mechanism. Much more sane options existed back than, but for reasons unknown (laziness is quite likely) a choice of a technological foundation was very much random.

I do remember web from its very first days (used to work in an institute with some very tight connections with CERN), and even then I was wondering why such a good idea is going such a non-optimal way.

And, well, it was not quite "evolved", it followed the trends set exactly by this lot - by the web devs. So I cannot see who else should I blame for this mess.

> Which would you choose?

If I had a choice, it would have been Tcl/Tk in a sandbox.

Agreed. I've done over 10 years in both embedded programming and full-stack web, and I often wish for the simplicity of embedded programming.

Of course, embedded programming has gotten a lot more complicated too. When I started, we had to fit our programs into 4K of ROM. These days an embedded program often includes a full Linux stack.