Thanks for sharing this analysis Matt! All that buzz about no-code tools making software engineers obsolete (that never happened) seems to be making sense in the context of data engineers.
I also quite like how assertive you are in saying that most businesses don't need data engineers -- indeed, they just need to invest in purpose-built tools. That said, most businesses are also very skeptical about investing in a set of tools they don't fully understand and don't have in-house resources to manage. Would love to know your thoughts on that!
I think this is probably the hardest part of any technology oriented industry, finding the best way to apply their hard earned experience to allow others to avoid the historical mistakes.
Ultimately it is on us, the practitioners to win businesses over, one at a time. If there is truth to any of these notions of data engineering progress, then the adoptions will become more common, easier and require less in-house resources to be successful.
Examples of where this worked - CRM systems being ubiquitous
Where it didn't work - ERP systems largely failing due to complexity
I suppose modern data tools need to consistently deliver value in various contexts, otherwise they run the risk of having a "high risk" label like ERP.
Would love to hear back from you, I don't have strong convictions that that is the right way to think about it.
It is on practitioners to win businesses over -- I'm 100% with you on this. The biggest gap I see (and that I'm trying to fill) is the ability for individuals at companies without a data practice (or even a data foundation) to get expert, unbiased answers to their data questions -- moreso about tools and technologies but also about processes and people.
Individuals spend a lot of time and resources evaluating tools but very few understand the need for 7 different tools to build a data stack, and how those tools talk to each other. And a bigger issue at hand is that people think that the time between buying and deriving value should be miniscule (as is the case with regular SaaS tools) which, as you know, is not true. Non-data teams don't have the resources to quickly understand how the tools are implemented and how they can be involved in the process -- this, I believe, is the second big issue at hand.
I could go on but my thinking is this: "Now that we have purpose-built tools for every layer of the stack, it's time to invest in enabling people across teams to gain an understanding of how these tools work and what it takes to adopt them successfully."
Yes very good points. A well resourced implementation is likely a successful endeavour, but still huge variance based on who runs the project, and expectations on time to value. I think value of these is only truly realised over quite a few months and years.
"what it takes to adopt them successfully" this is very much the crux, and you are right in saying that we are a long way off for this not to involve risk. Consolidation of certain tools into features in other tools, rinse and repeat implementations and lower technical barriers will make these more successful (but maybe take the sharp edge off of the expected outcomes as well - see the dreaded but industry standard Salesforce)
Matt, I think you’re spot on with this article. I am also an industrial engineer who transitioned into this role: Data Engineers: Solutions oriented engineers, Data. I think the ability to help organizations achieve business outcomes and optimize processes will be critical for data people to excel at in the next few decades.
Nathan thanks for the feedback, glad to hear that this resonated. Definitely agree, and I suppose a great opportunity for those in our position to try and determine which direction is optimal in terms of skills and demand.
One of the most insightful articles I've read in a long while!
Thanks Bart-Vee!
Thanks for sharing this analysis Matt! All that buzz about no-code tools making software engineers obsolete (that never happened) seems to be making sense in the context of data engineers.
I also quite like how assertive you are in saying that most businesses don't need data engineers -- indeed, they just need to invest in purpose-built tools. That said, most businesses are also very skeptical about investing in a set of tools they don't fully understand and don't have in-house resources to manage. Would love to know your thoughts on that!
Hey Arpit, thanks for the response!
I think this is probably the hardest part of any technology oriented industry, finding the best way to apply their hard earned experience to allow others to avoid the historical mistakes.
Ultimately it is on us, the practitioners to win businesses over, one at a time. If there is truth to any of these notions of data engineering progress, then the adoptions will become more common, easier and require less in-house resources to be successful.
Examples of where this worked - CRM systems being ubiquitous
Where it didn't work - ERP systems largely failing due to complexity
I suppose modern data tools need to consistently deliver value in various contexts, otherwise they run the risk of having a "high risk" label like ERP.
Would love to hear back from you, I don't have strong convictions that that is the right way to think about it.
It is on practitioners to win businesses over -- I'm 100% with you on this. The biggest gap I see (and that I'm trying to fill) is the ability for individuals at companies without a data practice (or even a data foundation) to get expert, unbiased answers to their data questions -- moreso about tools and technologies but also about processes and people.
Individuals spend a lot of time and resources evaluating tools but very few understand the need for 7 different tools to build a data stack, and how those tools talk to each other. And a bigger issue at hand is that people think that the time between buying and deriving value should be miniscule (as is the case with regular SaaS tools) which, as you know, is not true. Non-data teams don't have the resources to quickly understand how the tools are implemented and how they can be involved in the process -- this, I believe, is the second big issue at hand.
I could go on but my thinking is this: "Now that we have purpose-built tools for every layer of the stack, it's time to invest in enabling people across teams to gain an understanding of how these tools work and what it takes to adopt them successfully."
Yes very good points. A well resourced implementation is likely a successful endeavour, but still huge variance based on who runs the project, and expectations on time to value. I think value of these is only truly realised over quite a few months and years.
"what it takes to adopt them successfully" this is very much the crux, and you are right in saying that we are a long way off for this not to involve risk. Consolidation of certain tools into features in other tools, rinse and repeat implementations and lower technical barriers will make these more successful (but maybe take the sharp edge off of the expected outcomes as well - see the dreaded but industry standard Salesforce)
Thanks Matt, hope to see more resources on successful implementation of tools.
Matt, I think you’re spot on with this article. I am also an industrial engineer who transitioned into this role: Data Engineers: Solutions oriented engineers, Data. I think the ability to help organizations achieve business outcomes and optimize processes will be critical for data people to excel at in the next few decades.
Nathan thanks for the feedback, glad to hear that this resonated. Definitely agree, and I suppose a great opportunity for those in our position to try and determine which direction is optimal in terms of skills and demand.