This is exactly why I found DE is not a long term career path.
I work in data, I occasionally do DE roles, but if you aren't moving over to the app side or business/domain side, you are leaving a lot of money on the table long term.
The funny thing, being in data, if you take the time to understand it.. you are tremendously valuable to the business/domain side.
Otherwise you 1000% risk becoming part of the background infra.
No one celebrates when the faucet dispenses water or the power switch works.
They do lose their collective #$#@ when the water or power fails though.
Data Engineering has a tendency to be treated that way.
DQ is a big part of it as well. Everyone wants it, no one wants to pay the hours/dollars to do it.. No one is sure what they want, but probably what you propose is too simple. Can't AI do it? Etc.
The funny thing, being in data, if you take the time to understand it.. you are tremendously valuable to the business/domain side.
Otherwise you 1000% risk becoming part of the background infra. No one celebrates when the faucet dispenses water or the power switch works. They do lose their collective #$#@ when the water or power fails though. Data Engineering has a tendency to be treated that way.
DQ is a big part of it as well. Everyone wants it, no one wants to pay the hours/dollars to do it.. No one is sure what they want, but probably what you propose is too simple. Can't AI do it? Etc.