Context, and the Lack Thereof
Where we come from, we’d call you crazy
Enterprise data systems can often be quite distinct from their smaller startup cousins. This post takes a look at how our conversations around data systems and techniques need to be more sensitive to context, specifically when conversationalists have varying backgrounds.
A puddle-deep dive into the world of enterprises and how they compare to startups.
Does it generalise?
A key thing I come back to with working it data, writing about it and especially reading about it, is the following: does this generalise? Does this apply to a general context?
We all have a specific set of experiences and sufficient time to learn from others. This gives each of us the perspective that forms our opinions on the world.
So occasionally, we will say something like1:
This tweet led to fascinating conversations with many people, experts and non-experts alike.
What is even more fascinating, is that the context that caused this statement to resonate so strongly is entirely presumed, or even unknown.2 The context, the specific experiences that led to the specific insight, is unknown.
This I would argue is a key issue with a few of the rifts in data discussions. Data mesh, modern data stack, bundling, data modelling, dbt, human middleware, contracts, purpose, gods, etcetera.3
How many of these rifts are due to a lack of generalised context?
Let’s have a look.
Who are we?
Data people do come from a pretty consistent set of contexts:4
The monolithic startup driven by a charismatic CEO. The meshy enterprise and its tangled mess of systems. The methodical R&D group with its tinkering innovators. The mutinous midsize company and its C-Suite fights over whose department initiatives are really driving growth.
Across all of these complexity busting teams, possibly the clearest dimension that I've noticed is the different, almost opposed operating models between the scrappy startup/scaleup mode and the sprawling enterprise mode when it comes to implementing data solutions and systems.
Here are a few key ways of describing characteristics that really distinguish:
CTO vs CIO5 responsible for data
Data needs to be useful vs data needs to be accurate
Growing vs stable (dying)
Emergent vs traditional
Product market fit vs risk averse
This is worth discussing because we have long since reached the point where these conversations feel quite like worlds colliding
(the enterprise data-mesh car crash through the fence of startup analytics world),
and this was like two previously un-contacted tribes with each their own gods and deities coming together and not loving the look of each other. People just screaming right past each other. Here is me screaming:
My Background
I have always worked in and with smaller businesses, the median people number being ~150. I briefly worked as a solutions engineer for a company that nearly exclusively sold products to big enterprise companies, multinationals, banks.
This experience was not agreeable and shocked me in a few ways. Primarily, how incredibly complicated the enterprise customer operations can be, and how dysfunctionally the enterprise customers ran their technology projects.
Complicated is fine, complicated is not inherently complex, and just requires careful consideration, rather than rocket science, to figure out what to do. But in these older businesses, what to do is staggeringly complicated.
And so if you have worked in a startup, you may experience enterprise as static, as if nothing happens.
Well, things happen, just nothing changes,
At least not satisfyingly so, and certainly not quick enough for any feedback loops to develop, and absolutely not long enough for anyone to be around for long enough to be held accountable for their actions.
What ends up happening is pretty organic, often incredibly misguided, and very likely even counterproductive without realising it:
Not only do systems expand well beyond their original goals, but as they evolve they tend to oppose even their own original goals. … For example, incentive reward systems set up in business can have the effect of institutionalizing mediocrity. This leads to the following principle. Systems tend to oppose their own proper function.
Working in a company selling software was great, but selling to such established mega-corps was not my style of toasted cheese sandwich.
This blog has simmered as a comparison between Startup-land and "your typical lower tier enterprise". The result is a massive generalisation, highlighting the best of startups and comparing them to the worst of enterprise. I have a minor sliver of context, and I'm adding it to the mix. I'm not an enterprise/startup expert, but I do have experience, which means I have some context to share.
Many people in this space go their whole lives not looking under the hood of an enterprise, or startup, and this is for them.
So what about startups
Anyone reading this probably has an idea of startup land.
Startups have the benefit of a mostly clean slate, minimal baggage and an attitude of must take the time to get the tech systems right. Data is very often part of the strategic differentiation (guided or misguided as it may be).
Single data system, single tech, migrate quickly, do it now, cutting edge, sorry cancel the contract we turned that off.
Often wrong but always correcting, both the data stack and the company. Startups pivot on a cent(dime/penny), making extremely confident moves, with everyone in the team ready to have their role/responsibility severely disrupted, and be thrilled about it (unless you get fired).
Some great points I seem to have copied from a slack about what to expect at a startup if you’ve only worked in an enterprise:
You will ALWAYS be building the plane as you fly it. Take time to step back and really think about your foundation and the scale it needs to operate at.
My advice would be to try to be as ‘T’ shaped as possible. Don’t worry about being amazing at lots of things - just be good enough at lots of things and best-in-team at one thing in particular.Just do things, permission is unaffordable overhead.
Prioritize aggressively because if you do the wrong things the company dies.
Hot on the trail of customer needs and wants, iterating through the early stages of getting product market fit, and then once the scaleup stage is reached, all hands to the optimisation engine.
Startups are relatively simple - everyone is aligned. If not then it is probably just a small enterprise.
Summary
Uncompromised, as everything matters, and the business will probably fail
Any single failure could be fatal, but only can be minimally mitigated
Aligned incentives
Fragile, like a newly lit fire with massive potential 🧨
Scale-ups?
Scale-ups are what happens when a startup does exceedingly well (product market fit or excessive funding) and transitions from chaos into chaos on a mad hiring spree. The metamorphosis is described here:
The growth of a company from startup to enterprise has a lot in common with the growth of a caterpillar into a butterfly. And those parallels are especially meaningful when you’re in the goopy, amorphous pupa phase.
It can be disorienting to go from a stage where the emphasis is on quick execution for immediate results to a stage where ideas take longer to incubate and impact is measured in years, not days.
Often enough a chaotic nightmare where no one knows what truly matters. The inherent value of this stage is being tested by the current lack of growth VC funding.
Pouring fuel on the fire now that you won’t smother it 🔥
So then what about enterprise
Enterprise is entirely different. With endless silos, centralising and decentralising efforts, complicated by mergers, locations, borders, timezones and subdivisions. The focus of data is on basic coherence, control, and standardisation.
I'm not talking about Apple here. FAANG doesn't have this issue, they just build solutions. CV buffering for engineers, they have other issues. They have issues relating to hiring people they don't need to prevent other FAANGs from having access to them, and other rumours about deliberately depressing profits.6
I am talking about your mid-tier banks, in forgotten regions. Companies that need technology to survive, but have a complicated relationship with it. Think COBOL stories. Think of General Electric post-merger with one of the 100's of companies in one of the very many sectors. Declining profits, fired CEOs, disillusioned staff. Failed ERP implementations. The long tail of companies that just exist on momentum and middle management careering.
At a true enterprise, one can expect many legacy tech stacks, each with a plan and migration timeline. Keep it reliable, supported, staffed and contracted, ideally a single vendor.
What this leads to is Zombie Projects, with incredibly disillusioned teams running them:
when a project has failed or has ballooned in size to the extent that it will never be completed, and everyone knows it. When that happens, and no one wants to face the facts, and so the project continues to move forward, then it becomes a BS project.
Individuals start to realise that the project has died but will remain funded, and the system supports itself. They realise that despite whether they work or not, nothing happens and nothing matters, other than the charade itself. A reminder that this really drains motivation.
That intro is probably a bit harsh, What does the enterprise do well?
Running incredibly complex, massively scaled businesses, often successfully, credit where it is due.
Hero Warship
Another recognisable enterprise pattern is hiring saviours.
Person X, accolade Y, education Z (MBA) is heralded as the big new hire with the big new plan, assigned a big new budget to (finally) achieve the big broad goal.
Hero comes off the back of recent success in the exact same situation at a competitor. They start with an investigation into the “business needs” through many committees, leading to a feature hit-list-box-ticking exercise. Eventually, only massive companies like Oracle or IBM are in the running.
Cynically, this project is entirely for the purpose of leveraging it into a new role at a new bank (Failing Up), years before the project has any chance of fruition.
Another version of this game: In walk the management consultants. Strategise the flavour of the month (Modern Data Mesh), sell the dream, plan the diagram:
optimise cut reduce expand single source modern truth AI prediction profit, drops mic
The contract staffing company arrives, waterfall chart in hand and by the time the paint has dried most of the original stakeholders have jumped ship as digitisation data AI blockchain cost management experts, and are spamming the success of the project before it has even begun!
What are you sinking about
I think of enterprise as a system that has built so many incredibly reliable systems to prevent the business from failing that they become rigid.
Incredibly strong. Incredibly inflexible. Entire functional areas begin to decay, while the rigidity remains.
Put people in charge of rules, meetings, and forms, and their first inkling will be “there should be more rules, meetings, and forms”.
The rigidity and strength prevent flexibility, and so change becomes increasingly unlikely. This eventually leads to a state of dysfunction. The structure remains, but the function gradually dissolves. Idiosyncrasies like change review boards outright inhibiting changes.
Broken systems, operating in an organic state of failure:
The system continues to function because it contains so many redundancies and because people can make it function, despite the presence of many flaws. After accident reviews nearly always note that the system has a history of prior ‘proto-accidents’ that nearly generated catastrophe.
Summary
Compromised, primarily due to risk aversion
Severe misalignment
Hot embers, of a big fire 📛
Looking Ahead
As we have seen, the enterprise has largely taken note of the successes (and failings) of the organic brew of recent data tech.
My guess is that over the next few years, we will see this accelerate (they don’t read the salty tweets), but it will largely squish the pulp out of it in the outsourced and cross-matrix-managed divisional regional organisational chaos, and after decades of implementation, will abandon it to the pile of supported and business-critical but largely ignored technology.
Alongside we’ll see ever more of the offering of cargo-cult as a service from the consultancies forcing their way into the fray, with AI names and outsourced quasi-bundled solutions.
Often these kinds of trends are misguided, but some are indeed necessary. Digitisation? Yes if you must, no one wants to call for a delivery update. Cloud? Probably.
So Modern Data Stack then? Well, it comes with a way of working, an ethos around software best practices, some thoughts around who is responsible for what, and how to engage as an impactful product minded team.
Therein lies hope. Where Hadoop was a big data solution, Modern analytics is possibly a process improvement. Doing things a bit more coherently. A slightly easier way to do the same old.
But I worry that it is probably not enough, or not even the case.
The ease with which enterprise data modelling experts have co-opted the data modelling conversation on LinkedIn tells me that the more likely post-fact analysis is going to lead to something along the lines of:
MDS brought Enterprise capabilities to Startups, and not the other way around.That is a shuddering thought if there ever was one?
As mentioned, and regardless, Modern Data Stack has started simmering in the Enterprise, and is now well on its way. A new song with the same dance.
So what?
Back to the opening point on context.
Take a moment to understand the context behind opinions, product features, books, blogs, tweets and toots.
When consuming in this space it is critical to ensure the relevance of what you are reading, to you.
Is that (this) advice or content applicable to you?
More importantly, the person writing that (this) blog. What is their background? What are they saying. What have they not said?
Thanks to Simon Späti for insightful feedback on an early draft. Simon published a very insightful analysis on how enterprise data systems actually work. Also thanks to my team of editors for the proofreading.
20 Feb 2023 edit - Google is indeed an enterprise by the definition broadly described in this post
The tweet was inspired by a conversation with a company, who have tasked their newest team member with building a Data Warehouse so that the Engineering team could keep shipping.
The reason slack/twitter/Linkedin works at all is that we broadly presume everyone is of the following: data folks, largely all working in tech, scaling and not so much, generally all building web-apps, B2B and B2C, VC funded, ~Head of Data consulting, engineers too, building a modern Data Analytics practice, hate Data Mesh, jaded Jaffles, ~SQL, mostly ready to move into Product.
I could try find a link for each of those but as a reader I find the pressure to discover ALL of the context a bit overwhelming.
Data people do actually come from a pretty consistent set of contexts. Why is this? My guess, data people are the glue that is applied to deal with complexity. Filling the gaps in the matrix. People, varyingly hired to fix the complexity problem.
CIO? Frank Slootman on the enterprise CIO:
It used to be IT was king of the hill, it still is in some place. But now business is just as technical as IT. So their roles are shifting and you get a much more balanced environment between what the business makes decisions on and what IT is really in charge of, because IT doesn't really know how to apply technology to the business, but the business does. We see that balance changing.
And I have that conversation often, by the way, with CIOs who are your typical infrastructure guys, they manage for cost and risk and these kinds of things. But they're infrastructure people, they really are enablers, but they don't really know how technology impacts the business. The business does.
This was written before they started firing people.