At my company, the big thing that I think would help is if people would stop treating data engineers as if they are a cheap way to get both software engineering + data analytics in one package. They’re not. Data engineers are brilliant at data things like writing complex pipelines and wicked queries, but they just aren’t as good at writing secure code. Meanwhile, software engineers are the opposite.
Data engineering teams need at least part of their team to be designated software or systems engineers that are experts in stuff like how to create a robust cloud environment. This is not only beneficial for security, but also for cost optimization, performance, operations, etc! It drives me crazy when I see a data engineer or marketing team told they have to fend for themselves when trying to figure out how to build a cloud environment.
Data engineering teams need at least part of their team to be designated software or systems engineers that are experts in stuff like how to create a robust cloud environment. This is not only beneficial for security, but also for cost optimization, performance, operations, etc! It drives me crazy when I see a data engineer or marketing team told they have to fend for themselves when trying to figure out how to build a cloud environment.