If you have any ideas for improving this investigation, I’m all ears!

----------------------------

Since reading the OpenAI data stack post, I’ve suspected that dbt/SQL might get in the way of LLMs when looking at the data stack more holistically.

By data stack, I’m talking Modern Data Stack: ETL core app data into Snowflake/BigQuery, load other API data like Stripe in as well, do SQL joins to get answers (if unfamiliar then this post might not be that clear).

All the SQL + metadata might just be more human useful than LLM useful.

Or the system might have been better designed for when we didn’t have Coding Agents.

I’ve wondered about this a bit:

On a related note to dbt, a strongly coupled feeling, I’ve felt that your main app’s code is underutilized as a source of meaning and structure:

Then I see this point from Open AI, and I’m like, YES!

I’ve felt this acutely. Maintaining SQL files in dbt has just so much surface area, and so little logic!

But also reading the OpenAI post, I see they are still running most of their analytics logic in SQL!?

I really struggled to believe that the OpenAI data, at the fastest-growing, most well-funded supercompany in recent memory, is doing exactly what I would do.

They are running the Covid era MDS at the core of all the other stuff.

Same way, same tools, driven by the same FinOps need.

I don’t expect SQL to go anywhere, that is not what I’m getting at. I also don’t think dbt should go anywhere necessarily. Standards are set in times of disruption, and if dbt is the analytics standard then so be it.

But I wanted to scratch the itch. So I built a benchmark.

I present The Modern Data Benchmark (or MDS Gym?).

A small experiment comparing how LLM agents perform across different data architectures, given the same data and the same questions.

The pro-forma result:

Across 7 LLM models, warehouse+dbt had a 5% pass rate. App-unified architectures had 38-48%.

*the numbers in this post are directionally correct, like the MDS.

A quick history of how we got here

Before we dive into it, some context.

The Modern Data Stack came out of a specific organizational need.

1. Marketing needs to track ad spend.

2. Finance needs to reconcile Stripe revenue.

These are important but non-core functions, so they get staffed by analyst-operator types who can usually write SQL but not Python. Add Redshift/Snowflake and all roads lead to you SQL.

dbt emerged to give those SQL queries just enough software engineering discipline, version control, modularity, templating, without forcing anyone to leave SQL. It was a rational solution to a real constraint: these teams could not write very good code, and didn’t have a place to write it.

The gravity of that constraint pulled everything towards raw SQL against a data warehouse. No types, no abstractions, nothing but SQL. dbt came along to solve the obvious shortcomings (shitshow) of lots of SQL. Version Control+Jinja 🤯 and it was, I was working in enterprise data before, dbt was wild!

OpenAI’s data warehouse confirms the pattern

What surprised me was looking at OpenAI’s data warehouse. On the surface it looked sophisticated: context layers, embeddings, all the works. But at the core, two things stood out.

First, it was driven by FinOps. The example given was revenue reconciliation from Stripe. Even at frontier AI companies, FinOps drives the data warehouse.

Second, it uses SQL (and I guess dbt). The team likely had used dbt before, so they reached for it again. This isn’t a criticism, it’s how standards form. Not by systematic evaluation, but by repetition.

The question worth asking

dbt’s value was making SQL manageable for human analysts. But at scale there are now tens of thousands of lines of SQL that no human is ever going to read. The SQL is increasingly being consumed by agents.

If the consumer is an agent, “easiest for humans to read” stops being that important.

The relevant question becomes:

For an agent answering business questions, which representation of the system is the most legible, robust, and correct?

It all felt rather benchmarkable.

The split-brain problem

The modern data stack creates a structural split. Your app has your core business logic: users, statuses, transactions. Separately, you have a data warehouse that holds a lagging copy of that data, plus third-party data like Stripe that only exists in the warehouse.

To answer anything useful, you need to join app data onto Stripe data inside the warehouse, using SQL, with constrained logic. The “single source of truth” in the warehouse is never truly trustworthy. Your actual source of truth is the production database and someone else’s API, and the warehouse is always behind.

The crux: what if you brought the data closer to home? Shift left? Strongly typed? Asked Codex 5.3 to spar with Opus 4.6 on turbo mode? What would they do?

I guess they’d load Stripe data into a structure your app understands, with proper types and constraints, and they’d run analytics against the unified codebase..?

If revenue is a key business capability, and we’re no longer as code-constrained as we were, why not model it as a first-class concept?

(this has a million small holes, but stay with me)

The benchmark

Three sandbox environments, same data, same three analytical tasks: ARPU, churn rate, and LTV.

Small data, known correct answers. Size isn’t the test. We are looking at architecture.

In the app sandboxes, Stripe data is represented as internal data with types.

App Tables + Stripe Tables → Single typed context → Code → Metric

In the dbt sandbox, it follows the traditional pattern: third-party data loaded and joined via SQL.

App DB + Stripe → Replication → DuckDB raw tables → Staging SQL → Marts SQL → Metric

The model must discover the schema and produce executable code that returns the correct number. No hints, no hand-holding.

The key difference: in app architectures, the model has one typed context. In the warehouse, it must navigate staging models, column naming conventions, and SQL casting to arrive at the same answer.

How the evaluation works

Each run works like this:

1. Fresh sandbox. A clean copy of the sandbox template is created with the synthetic data loaded. No prior work carries over between tasks.

2. Agent loop. The model gets a system prompt describing the architecture and four tools: read_file, write_file, list_files, and done. It has up to 10 turns (API round-trips) to explore the codebase, discover the schema, write its solution, and signal completion. Temperature is set to 0.

3. No hints. The model is told what to compute (e.g., “ARPU for active users”) and given the function signature, but not how. It must figure out join keys (users.stripe_customer_id → invoices.customer_id), column names, and time anchoring on its own by reading files.

4. Execution app-typed: the TypeScript function is imported and called with the data arrays.

5. Execution app-drizzle: the async function runs against a pre-loaded SQLite database via Drizzle ORM.

6. Execution warehouse-dbt: the SQL is executed in DuckDB with raw tables created from JSON (staging/mart views are built from any SQL files the model wrote)

7. Scoring. Pass/fail is purely numeric: does the output match the expected value within tolerance? (±1 for integers like ARPU/LTV, ±0.001 for rates like churn). No partial credit, no style points. If the code crashes, it’s a fail. If it returns the wrong number, it’s a fail.

8. Flexible matching. The validator accepts naming variations (calculateARPU, computeArpu, getArpu, etc.) and searches multiple directories for SQL files, so models aren’t penalized for reasonable naming choices.

Expected values are computed from the same data by a reference implementation in the benchmark harness itself, not hand-coded, so they’re guaranteed consistent.

THE RESULTS

The ORM sandbox was the clear winner: Opus got a perfect 3/3 every single run, and cheaper models like Kimi and Grok matched it.

The warehouse-dbt column is almost entirely zeros. Only Opus managed a single pass, and even that was inconsistent across runs.

Looking closer at variance on multi-run stability (n=5 per Anthropic model)

• Opus: Zero variance.

• Sonnet: High variance (~0.9 std).

• Haiku: Fluctuates between 0-1 passes.

Mid-tier models are at the edge, and the architecture is what pushes them over or pulls them back.

Where dbt struggles

The failure modes:

• Schema mismatch: wrong column names (created_at vs usage_created_at, org_id vs organization_id). Staging conventions that the model has to guess.

• Type mismatch: interval math on VARCHAR timestamps without casting.

• File naming: incorrect output filenames or failure to write the metric model.

Even thorough models struggle with dbt

Adding in a measure of unique files read per task in warehouse-dbt:

• Haiku: 1.1 files (barely looks at the schema)

• Sonnet: 3.7 files (reads staging files but still gets column names wrong)

• Opus: 4.4 files (reads everything, still only 1/3 pass rate)

Even when the model does its homework, the warehouse architecture introduces enough indirection to trip it up. It’s not a laziness problem, the representation has too many seams.

In the app sandboxes, failures were simpler (wrong join key, missing function) and more recoverable in typed code.

What I take from this

This benchmark tests a narrow but important thing: can an agent read a schema, write executable logic, and return the correct metric? It doesn’t test full dbt workflows with Jinja, ref/source, or materializations. It tests the core analytical task that everything else is built to support.

A few observations (not conclusions, this is early and the sample is small):

1. The architecture matters more than the model. The same model that fails at dbt can succeed in a typed environment.

2. ORMs are surprisingly agent-friendly. Drizzle over SQLite was the strongest sandbox, even mid-tier models could navigate it. Typed schema + query builder + unified context seems to hit a sweet spot.

3. Indirection has a cost that compounds. Each layer of staging, naming convention, and type casting is a place where an agent can silently go wrong. Types and co-location seem to reduce that surface area.

What’s next

The current tasks (ARPU, churn, LTV) are intentionally simple, canonical SaaS metrics on synthetic data. The architecture signal is clear, but the questions need to get harder to be convincing.

A few directions:

1. What is a “fair” dbt project? After the initial results I started adding hints to the dbt sandbox to get some passing runs, things like cast annotations in staging models so the model doesn’t trip on DuckDB timestamp arithmetic. I added CAST(created_at AS TIMESTAMP) AS usage_created_at in the staging layer as it kept tripping up on that. It sort of helped: adding a single cast hint let Sonnet pass org_churn_rate where it previously crashed on a runtime error (

   details

   ), but it wasn’t that consistently helpful. It also felt like a slippery slope. How much documentation and scaffolding do you add before the dbt sandbox stops being representative of what an AI agent would setup. Remember this was all setup by Codex 5.2, I didn’t touch a thing! A real dbt project lives somewhere on this spectrum, and where exactly is an open question. But maybe that’s the point. You can keep adding scaffolding to SQL, cast hints, schema docs, Jinja templating, ref() pointers, semantic YAML, and each one closes a small piece of the gap. At some point you have to ask: is the SQL architecture, with all the scaffolding you need to make it work for agents, converging on the thing you’d build if you just started with types and a unified codebase?

2. Realistic drift. The current benchmark is static, the data is clean and complete. Real analytics is messier: late-arriving Stripe invoices, missing stripe_customer_id mappings, schema changes mid-pipeline. Adding sync delay scenarios would test whether the split-brain problem is as bad in practice as it is in theory.

3. Linting as agent feedback. Early experiments with TypeScript typecheck + SQLFluff showed that giving agents lint feedback and extra fix attempts improved ORM more than dbt scores, but the improvement might just be from extra turns, not the lint signal. SQLFluff style rules seem to add noise that distracts smaller models. A schema-only mode that only surfaces missing tables/columns could be a cleaner signal. I tried some things there, but nothing clear.

4. Information parity. A typed codebase inherently carries more structural information (types, constraints, relationships) than raw SQL with YAML docs. You could argue that’s confounding. I guess so? But that’s also the point: the architecture is the information density. Still, enriching the dbt sandbox with comprehensive YAML schema docs would test how much of the gap is “types help” vs “unified context helps.”

5. Turn-level analysis. Currently only tracking file-read counts. Understanding the step-by-step reasoning, where models go wrong, when they recover, would give sharper insight into why architecture matters.

Other interesting things:

• Semantic layer sandbox (Drizzle-Cube). The baseline benchmark included Drizzle-Cube but the architecture benchmark didn’t. Adding it would test whether pre-defined measures help or constrain agents. I’m hopeful this pushes Drizzle well beyond comparison!

• Does the “context layer” exist? There’s a lot of hand-waving right now about “context layers” and “context graphs.” When you boil these down, they often look like a data warehouse or semantic layer in new language. My position: if you can’t demonstrate a simple instance of a complex idea, it doesn’t meaningfully exist. Next step is to build sandboxes for the best-case context graph blogs and run them through the same benchmark.

• Harder queries.

  @matsonj’s platinum set

  from the BIRD text-to-SQL benchmark covering complex joins, CTEs, NULL handling, and conditional aggregation. Also KramaBench which tests full data pipelines, not just single queries, and from the benchmarks referenced by Sphinx including DABStep (real Adyen payments data). If agents struggle with simple ARPU, what happens with real analytical complexity?

• Costs. I tracked the costs, the results were somewhat interesting, it seemed like Opus was actually often cheaper as it took fewer laps to get the answer. I need to benchmark this more carefully.

  Pareto performance cost curve

  .

A request

I’m no longer that close to dbt. Things may have moved on. If you’re actively working in dbt and you look at the warehouse sandbox and think “that’s not how we’d set it up,” I genuinely want to hear that. Is this a realistic task? Is this a fair test? The benchmark is open source, you can set up the dbt sandbox the way you think it should be, and run the same evaluation. If a well-configured dbt project closes the gap, that’s a finding worth publishing too.

Caveats:

1. Small n, synthetic data, single-pass runs for some models, no full dbt compilation. This is directional, not definitive. Run it yourself, add harder tasks, prove me wrong.

2. Claude wrote this out from a voice-note I recorded.

Link to benchmark repo · Link to OpenAI data stack post

The analysis tracks three frameworks for mapping the cyclical nature of cultural phenomenon (one two three). These don’t map perfectly, and number three doesn’t fit the enterprise context. The idea: Something new catches on, it grows big, it loses its way, there is a fight / collapse / phase-change and then it stabilises. The subheadings are from John Cutler:

Anything helpful will eventually become commodified, industrialized, and watered down to the point of being unrecognizable. It happened with Agile, and is happening with strands of product management, DevOps, design, etc.

Phase 1: Precycle

MDS EMERGES FROM PRACTICE

People like solving problems that resonate with other people, open source vibes, this was the early MDS scene.

Before MDS, there were enterprise data vendors and mostly they weren’t very useful for fast moving companies, in the sense that you could only afford them if you budgeted in the $xx millions and you planned in half decades. Redshift precipitated a change and Snowflake accelerated it. Pay-per-query.

Redshift and Snowflake laid the soil that led to the Modern Data Stack. They created the easiest to buy large scale data storage that anyone had ever seen. The problem now became keeping track of the transformations necessary to deal with the vast data recycling centres that were suddenly viable to be created by smaller and smaller teams.

dbt, a relatively generic idea (SQL templating in python) became the foundation for a new movement to deal with the issue, right place, people, audience, right time.

I was instantly hooked. dbt took the lead among an array of relatively fragmented solutions all jockeying around the task at hand.

MDS NAME COINED / FIRST BLOGS

A brief google and it seems like dbt first used the term "Modern Data Stack” on January 29, 2018, (I couldn’t be bothered to dig any deeper, history in the form of a sea-shanty with a broken banjo).

The concept caught on. I was working as a Data Engineer doing Redshift transformations, dbt achieved this in a far better way. Before that, doing this stuff was super expensive, testing and version control was tricky, everything was slow.

Blogs came out describing the way. Here I generously quote myself describing the series of blog posts that introduced the new way

We, the desperate, listened closely. The message I heard: bring the best of software development to startup data analytics

I was likely referring to this blog post. It all sounds rather exciting in retrospect, and it was!

MDS MINDSET / MANIFESTO / PRINCIPLES / EVANGELISTS

The whole thing coalesced (yup) around the Community. dbt slack and Locally Optimistic slack were the absolute centre(s) of the data world for a fair amount of time.

When those inside the cutting-edge scenes band together to support, teach, and create with each other, their niche and experimental projects can become the new normal on top of which the next generation builds.

The vendor guidelines kept things civil in the dbt chat, and Locally Optimistic was generally smaller and less frantic. Dbt was often the target of staffing firms who would point their junior devs at the dbt slack and say “here is your technical support” and they would paste 200 line error logs into the main chat and say “what do??” and then disappear.

I think a real kicker was the Open Source hook. Everyone wants to work with open source software. Now data people could do that!

There were reams of Guides, here is mine. (filler content for Dataform, now acquired by Google). The writing wasn’t very good in hindsight, but I stand by my suggestions, move fast with simple tools. Here is another good guide. These ideas were evaluated on some or other believability and the good ideas were amplified.

Alongside this, Validation that the data trade was noble fanned the flames.

dbt ran their first conference in 2020. 2021 they ran their second, this time it got big.

Phase 2: Growth

MDS BIG WINS

Snowflake IPO late 2020 set things in motion, the biggest software IPO ever was just the beginning of the end for MDS. dbt raised a ton of money, so did Fivetran and most of the others.

EXPLOSION OF MDS PATTERNS / BEST PRACTICES

Locally Optimistic was always and remains an ardent supporter of delivering relatively hype-free and insightful best practices. Their community remains well moderated and insightful, with very friendly and well moderated non-vendor participants. Remains a top place to visit.

Community best practices and advice about starting a company community slack started to pop up. If you were VC backed, you need 1000+ people in your slack otherwise how would you get any product feedback. This worked well in my opinion, despite the snark. If I am legitimately interested in your product I’d love to speak to the engineer building it. Companies started adding the slack activity into their go-to-market strategy, ELT’ing my activity into their CDP for better PLG or whatever.

SMALL MDS CONSULTANCIES

An acquaintance (or quite a few actually) went full throttle on consulting and quickly employed 100s of people and made a ton of revenue with massive companies.2

dbt has a huge list of vendors. This was the most reliable way to make good money in the Modern Data Stack ecosystem, but the allure of building products pulled lots of people into building data tools instead.

YOU ARE NOT DOING MDS / CERTIFICATIONS TITLES

The need to teach analysts how to be Analytics Engineers was pretty clear, they needed git, python, jinja, Data models, star schemas, denormalization, Kimball.

A few courses came out doing this stuff, first was the Analytics Engineers Club. I think the data modelling skills described are super useful, and generally fell into something that Data Engineers didn’t do and Analysts didn’t do, and so they were just ignored.

This wasn’t explicitly gatekeeping, more just demand being met by an increase in supply. I was firmly of the opinion that hiring ops people was beneficial in this space as they could easily grasp the tech and generally had a better sense for the problems that were worth addressing.

MDS VENDORS FUND CONFERENCES

There were a few smaller conferences other than Coalesce (the conference), and Snowflake summits, Data Council seemed to be the most consistently worthwhile.

In 2021 the dbt Coalesce conference peaked:

This cultural energy reaches its crescendo during Coalesce, dbt Labs’ annual conference. The conference ostensibly takes place over a number of live-streamed talks, but its beating heart is on Slack. Every talk inspires a tidal wave of excitement, encouragement, and general good cheer. Every channel is the parents’ section at track meet: Ready to erupt when their kid crosses the finish line, and equally ready to hop the fence and pick them up if they fall. 

MDS ECOSYSTEM INFOGRAPHICS

The vast ecosystem now needed a map, ModernDataStack.XYZ maps out the components, and I think does an adequate job of categorising them.

Here is another map, with their criteria giving you an idea of the qualifying criteria. 3

MDS VENDOR COMPETITION HEATS UP!

Once the ball started rolling, there was a lot of money being poured into the space. New BI tools, new ELT, data quality, data reliability, observability, metrics, semantics, all started being picked over.

I wrote a blog comparing dbt and Dataform. Dataform was acquired as mentioned, and dbt took centre stage, centre diagram.

Two big themes were Reverse ETL and CDP:

Another somewhat-in-the-weeds, but fun to watch part of the landscape has been the tension between Reverse ETL (again, the process of taking data out of the warehouse and putting it back into SaaS and other applications) and Customer Data Platforms (products that aggregate customer data from multiple sources, run analytics on them like segmentation, and enable actions like marketing campaigns). 

Reverse ETL was always a tricky thing. I like the idea of reverse ETL, but ultimately it would often build on very weak foundations:4

The CDP space also sort of heated up, and had significant overlap with MDS. The MDS gave you the tools to construct your own solution, and design the specifics to suit your needs, whereas the CDP kinda gave you much deeper capabilities, but less control (for the customer analytics niche).

FIRST ENTERPRISE VENDOR MDS OFFERINGS

Enterprise oriented MDS style tools and even direct clones started cropping up. The awkwardly named Coalesce was the first early dbt alternative since Dataform (Coalesce is also the name of dbt’s conference). I don’t know much about it.

MDS GARTNER MAGIC QUADRANT

There isn’t a fully fledged Gartner Quadrant. The whole point of MDS was all the unbundled solutions. There was a post on their forum. I predict we will see a “new wave” once all the tools are more consolidated and Gartner figures out how to position all of this.

I guess there are likely enterprise equivalently branded MDS style solutions that have happened. The only funny one I could find within 5 seconds of Googling was IBM Gen z/X.

OFFERINGS TO SOLVE MDS PROBLEMS

I almost wrote a blog decrying the state of things when I was spammed by an ex digitisation evangelist crypto expert, now selling data-something-dot-AI. Selling magic and then supplying junior analysts with git and SQL just stank of an enterprise sales cycle. This was indicative of the beginning of the end.

DON'T DO MDS / ENTERPRISE MDS

Data mesh was the funniest part of the entire data hype cycle. Effectively a fully fledged framework to decentralise data ownership, it read like a Business School Blockchain Certification (apparently, I didn’t read it). The response from the MDS pure snark. Pedram had the last word here. I guess maybe you had to be there.

There was heated posturing on Linkedin about the failure of MDS to properly address the problem of data modelling. Data Vault, Anchor modelling. How about a full circle all the way back to one big table?

This was also an Enterprise / Startup culture clash, but probably mostly a mutual lack of context and incompatible cadence. A 100 person SF startup growing beyond terminal velocity vs a declining mid-tier bank in the Bavarian hinterland aren’t going to meet each other with much common language. I wrote something about that here:

Phase 3: Involution / Stagflation

The cracks started to show in a few ways: cost, complexity, BI tools, sprawl.

Cost:

Snowflake et-al being super expensive was the first ominous crack. When interrogated, massive cloud bills often weren’t attributable to any value. It was presumed when these at kick off that they would be both expensive and lead to value. The expensive part was duly stomached but then the value part was late to the party.

ELT, synonymous with MDS, came under cost pressure. People realised that the work was pretty predictable and so cheaper, better, faster options became abundant.

And then AWS, Salesforce and Snowflake all began to murmur about ‘zero-ETL’ which basically meant they will co-access or whatever the data that is in Salesforce from Snowflake. Basically ruining the Fivetran business model.

“What if we could eliminate ETL entirely? That would be a world we would all love. This is our vision, what we’re calling a zero ETL future.

dbt:

Dbt was really just a victim of success, when you push into the unknown you will find out, and dbt found out. Before dbt, teams kept their reliance on a data warehouse simple and un-collaborative. One engineer would generally be responsible, they would have de facto veto on data modelling changes. dbt democratised that:

Did we achieve more collaboration on an analytics code base? ✅

Did we achieve more leverage through reusable and modular code? ✅

Did we also buy more complexity, resulting in longer maintenance and debugging cycles? Unfortunately, also ✅ 🤓

Turns out the price of enabling people to build a more complex code base is… a more complex codebase, and everything that comes with that.

Solving one problem and in doing so creating another is the essence of progress. I give dbt a pass here. Nonetheless, things got very complex, and not just limited to dbt.

There were numerous hot takes pointing to the shortcomings of dbt, and a slew of dbt alternatives popped up, all “faster horses” in my mind. We need cars.5

dbt having raised a ton of money, had to do layoffs and quickly figure out a business model.

Complexity:

You could be lead to believe that you need one from each of the following “MDS categories”. However most teams generally limited themselves to a BI tool, a database, an ETL tool and dbt.

The issue was the abundance of overlapping options.

Team size:

The other awkward thing to address was the growth in MDS team size. I worked with a team that wanted to double their data team from 20 to 50. MDS became known to run on human middleware:

Cash is injected. This means more employees of all sorts are hired. More employees increases the demand for more reports and analytics. More demand means more human middleware is created in the Clueless layer when you adopt the Modern Data Stack paradigm of throwing everything into your cloud data warehouse of choice.

More Clueless human middleware creates more tables, tables, tables to many more reports and KPIs and metrics. They have to buy new products and hire new people to manage the complexity.

This criticism was semi-reasonable6, but honestly I think this is often just the cycle of technology. Something comes along, presents an opportunity to differentiate, it works for some, it doesn’t work for others. People need to be involved. How many people? Probably a few. Oops too many. OK less people.

A more primary issue with buying anything is knowing if you are ready for it. Does this company need a better data capability? Falling behind is a real risk, that compounds!

The Data Stack was sold broadly, for some it worked, quite often it didn’t.

BI tools:

Business Intelligence tools continued to underwhelm, primarily because of the split between traditional reporting and exploratory analytics.

The paradigm of “reporting” is in my mind a dead end. Like delivering a menu and then never taking an order, BI tools were informative instead of interactive.

Traditional BI just don’t move the needle in the same way that newer tools like Hex do. (Hex described this paradigm in deleted article, they now position themselves as a data tool that does reporting too). I use Hex daily. It is relatively cheap, it works very well and has sufficient depth to replace a fair chunk of MDS and technology infrastructure too.

To be fair to BI tools, they were the last mile delivery problem built on a relative house of cards, so were pretty much destined to be the pain cafe.

Sprawl

There was Vocal Criticism from product analytics people about the sprawling nature of MDS.

Product analytics is a mature side-car to the MDS, and the tooling built by Posthog et al is generally end-to-end integrated. From their perspective, the MDS approach led to poor outcomes:7

Those are a few of the criticisms that illustrate the point. Essentially the belleweather for the change in direction of the flock.

MDS IS DEAD POSTS

There were a few. This became a bit of a trope. Meta analysis of the trope is far more palatable. Hope you agree.

BACK TO BASICS MOVEMENT

There was always talk of a bundling. In essence the MDS was unbundling, and at some stage the tide would turn.

This was actually discussed at length in early 2022, where most everyone agreed that the unbundled approach was great for many things (experimentation, investing, entertainment, curiosity), but it wasn’t very productive.

MDS LATE ADOPTERS

There is still the MDSFest:

“A community-led celebration of ideas and perspectives on the modern data stack”.

Sounds great but I literally just came across it researching for this blog so I don’t actually know much about it

I don’t know where this video comes from but I think it demonstrates the idea that one person can pull together an entirely viable, semi-scalable data platform from the best of the open-source stuff, as was always intended. Silver lining.

EVANGELISTS MOURN STATE OF WAY

Some of the early evangelists mourned their loss. It did literally feel magic. You could achieve so much with so little. This was a reality.

A shrinking pie makes zero sum games more likely too. A shrinking pie means you have to work pretty hard just to stem the flow:

Everyone Wants a Piece of the Pie, Nobody Wants to Bake
…
if you don’t build things to solve a pain you’ve had in the hopes that it’ll solve someone else’s, if you don’t give away your hard work for free, then kindly, please, shut the fuck up.

Matt Turk knows all about all of this, having documented the data space for ages:

The MDS is now under pressure. In a world of tight budgets and rationalization, it is almost too obvious a target. It’s complex (as customers need to stitch everything together and deal with multiple vendors). It’s expensive (lots of copying and moving data; every vendor in the chain wants their revenue and margin; customers often need an in-house team of data engineers to make it all work, etc). And it is, arguably, elitist (as those are the most bleeding-edge, best-in-breed tools, serving the needs of the more sophisticated users with the more advanced use cases).

Overfunded startups, overcrowded teams, too many compute credits, too much badly structured SQL and lots of criticism.

The substance boiled down:

MDS was a series of relatively experimental tools strung together to demonstrate varyingly good levels of Product Market Fit, but not quite demonstrating an ideal operating model .

The result is a great target for more refined, more niche, bundled equivalents.

Phase 4: Postcycle

END OF MDS STATUS GAMES

There is less attention and so less opportunity for status games. Most people have some trusted source of information and they rely on it, less interested in the minutia and more worried with whatever their pressing problems are.

A large chunk of data practitioners are now product focussed, in role, in company, in orientation. This is sensible as product is probably one of the best maturation directions, as it was a primary consumer of data outputs. ex-Data know how far to trust the data systems and what lies they tell. The same applies to operations.

MDS CONSOLIDATES WITH FOCUS

The data systems are largely good enough and so the bottleneck becomes what to do with the data. Data engineering was the bottleneck. In January 2022 I gave the opinion that Data Engineering was no longer the primary bottleneck to delivering insights/value/whatever.

Now in August 2023, I will now say that the Data Function is no longer the primary bottleneck in delivering insights/value. A good data team can ingest, model, analyse and distribute insights pretty easily. The challenges are now more subtle:8

Track 1: “All I want is to know what's different”

Track 2: “The emotionally informed company”

Track 3: “The truth is out there The only thing stopping us from finding it is us”

Track 4: “Will we ever have clean data? Probably not, but maybe we can work with messy data”

Matt Turk describes where things go from here:

The convergence of streaming and batch processing is an evergreen, and important theme. So is the convergence of transactional (OLTP) and analytical (OLAP) workloads

This Analytics vs Operational thing is critical. All I will add is that all data problems stem from the fact that the blue and a pink blob are handled by different teams.

MDS LEFT US WITH MORE / PRACTITIONERS STILL CARE

The whole argument around MDS now mostly dusty, it can simply be described as a good idea that explored all the avenues and turned over all the stones.

Data technology was just a nightmare before, and a better solution is always necessary. The unbundling of the big systems into discrete elements was effective, and allowed people to experiment, learn, share and iterate on that cycle towards something that was very effective. Timo Dechau describes this:

The funny thing about evolution and potentially the one often missed out. Evolution is never linear. It branches out, explores, and creates massive amounts of variants. That is the beauty of it.

But it is also why there is never “the” next. But hundreds of next. And out of them, at some point, we will see a step changing the ways in such a good way that we could declare it as a new paradigm.

We are not there yet. But we can already see the branches, which is exciting.

He has another good line on of thinking that maybe Product and MDS will converge.

Product analytics on top of your events in your data warehouse. No more weird data loadings and enrichment (where most of them never worked). And mostly, no two setups for classic BI and product analytics use cases. We spent some more time with this approach later.

IN CLOSING

STILL REAL PROBLEMS TO BE SOLVED

The opportunities are there. My core issue with MDS was that data modelling remained a complete nightmare. As the SaaS systems that run a business got more complex, the effort to consolidate went up and the accuracy of the consolidation went down.

My take on data modelling:

1. There is an optimum level of data modelling done by software developers building apps and not just leaving it to the data team (Analytics vs Operational)

   

2. There is more work to be done exposing better data models via API from SaaS companies, and especially internally.

3. There absolutely has to be better (easier, pre-populated, more automated, less fragile, less complicated) data modelling techniques.

The immediate future for data modelling involves an important role in the next big thing

A puddle-deep dive into the world of enterprises and how they compare to startups.

Does it generalise?

A key thing I come back to with working it data, writing about it and especially reading about it, is the following: does this generalise? Does this apply to a general context?

We all have a specific set of experiences and sufficient time to learn from others. This gives each of us the perspective that forms our opinions on the world.

So occasionally, we will say something like1:

This tweet led to fascinating conversations with many people, experts and non-experts alike.

What is even more fascinating, is that the context that caused this statement to resonate so strongly is entirely presumed, or even unknown.2 The context, the specific experiences that led to the specific insight, is unknown.

This I would argue is a key issue with a few of the rifts in data discussions. Data mesh, modern data stack, bundling, data modelling, dbt, human middleware, contracts, purpose, gods, etcetera.3

How many of these rifts are due to a lack of generalised context?

Let’s have a look.

Who are we?

Data people do come from a pretty consistent set of contexts:4

The monolithic startup driven by a charismatic CEO. The meshy enterprise and its tangled mess of systems. The methodical R&D group with its tinkering innovators. The mutinous midsize company and its C-Suite fights over whose department initiatives are really driving growth.

Across all of these complexity busting teams, possibly the clearest dimension that I've noticed is the different, almost opposed operating models between the scrappy startup/scaleup mode and the sprawling enterprise mode when it comes to implementing data solutions and systems.

Here are a few key ways of describing characteristics that really distinguish:

• CTO vs CIO5 responsible for data

• Data needs to be useful vs data needs to be accurate

• Growing vs stable (dying)

• Emergent vs traditional

• Product market fit vs risk averse

This is worth discussing because we have long since reached the point where these conversations feel quite like worlds colliding

(the enterprise data-mesh car crash through the fence of startup analytics world),

and this was like two previously un-contacted tribes with each their own gods and deities coming together and not loving the look of each other. People just screaming right past each other. Here is me screaming:

My Background

I have always worked in and with smaller businesses, the median people number being ~150. I briefly worked as a solutions engineer for a company that nearly exclusively sold products to big enterprise companies, multinationals, banks.

This experience was not agreeable and shocked me in a few ways. Primarily, how incredibly complicated the enterprise customer operations can be, and how dysfunctionally the enterprise customers ran their technology projects.

Complicated is fine, complicated is not inherently complex, and just requires careful consideration, rather than rocket science, to figure out what to do. But in these older businesses, what to do is staggeringly complicated.

And so if you have worked in a startup, you may experience enterprise as static, as if nothing happens.

Well, things happen, just nothing changes,

At least not satisfyingly so, and certainly not quick enough for any feedback loops to develop, and absolutely not long enough for anyone to be around for long enough to be held accountable for their actions.

What ends up happening is pretty organic, often incredibly misguided, and very likely even counterproductive without realising it:

Not only do systems expand well beyond their original goals, but as they evolve they tend to oppose even their own original goals. … For example, incentive reward systems set up in business can have the effect of institutionalizing mediocrity. This leads to the following principle. Systems tend to oppose their own proper function.

Working in a company selling software was great, but selling to such established mega-corps was not my style of toasted cheese sandwich.

This blog has simmered as a comparison between Startup-land and "your typical lower tier enterprise". The result is a massive generalisation, highlighting the best of startups and comparing them to the worst of enterprise. I have a minor sliver of context, and I'm adding it to the mix. I'm not an enterprise/startup expert, but I do have experience, which means I have some context to share.

Many people in this space go their whole lives not looking under the hood of an enterprise, or startup, and this is for them.

So what about startups

Anyone reading this probably has an idea of startup land.

Startups have the benefit of a mostly clean slate, minimal baggage and an attitude of must take the time to get the tech systems right. Data is very often part of the strategic differentiation (guided or misguided as it may be).

Single data system, single tech, migrate quickly, do it now, cutting edge, sorry cancel the contract we turned that off.

Often wrong but always correcting, both the data stack and the company. Startups pivot on a cent(dime/penny), making extremely confident moves, with everyone in the team ready to have their role/responsibility severely disrupted, and be thrilled about it (unless you get fired).

Some great points I seem to have copied from a slack about what to expect at a startup if you’ve only worked in an enterprise:

You will ALWAYS be building the plane as you fly it. Take time to step back and really think about your foundation and the scale it needs to operate at.

My advice would be to try to be as ‘T’ shaped as possible. Don’t worry about being amazing at lots of things - just be good enough at lots of things and best-in-team at one thing in particular.

Just do things, permission is unaffordable overhead.

Prioritize aggressively because if you do the wrong things the company dies.

Hot on the trail of customer needs and wants, iterating through the early stages of getting product market fit, and then once the scaleup stage is reached, all hands to the optimisation engine.

Startups are relatively simple - everyone is aligned. If not then it is probably just a small enterprise.

Summary

• Uncompromised, as everything matters, and the business will probably fail

• Any single failure could be fatal, but only can be minimally mitigated

• Aligned incentives

• Fragile, like a newly lit fire with massive potential 🧨

Scale-ups?

Scale-ups are what happens when a startup does exceedingly well (product market fit or excessive funding) and transitions from chaos into chaos on a mad hiring spree. The metamorphosis is described here:

The growth of a company from startup to enterprise has a lot in common with the growth of a caterpillar into a butterfly. And those parallels are especially meaningful when you’re in the goopy, amorphous pupa phase.

It can be disorienting to go from a stage where the emphasis is on quick execution for immediate results to a stage where ideas take longer to incubate and impact is measured in years, not days.

Often enough a chaotic nightmare where no one knows what truly matters. The inherent value of this stage is being tested by the current lack of growth VC funding.

Pouring fuel on the fire now that you won’t smother it 🔥

So then what about enterprise

Enterprise is entirely different. With endless silos, centralising and decentralising efforts, complicated by mergers, locations, borders, timezones and subdivisions. The focus of data is on basic coherence, control, and standardisation.

I'm not talking about Apple here. FAANG doesn't have this issue, they just build solutions. CV buffering for engineers, they have other issues. They have issues relating to hiring people they don't need to prevent other FAANGs from having access to them, and other rumours about deliberately depressing profits.6

I am talking about your mid-tier banks, in forgotten regions. Companies that need technology to survive, but have a complicated relationship with it. Think COBOL stories. Think of General Electric post-merger with one of the 100's of companies in one of the very many sectors. Declining profits, fired CEOs, disillusioned staff. Failed ERP implementations. The long tail of companies that just exist on momentum and middle management careering.

At a true enterprise, one can expect many legacy tech stacks, each with a plan and migration timeline. Keep it reliable, supported, staffed and contracted, ideally a single vendor.

What this leads to is Zombie Projects, with incredibly disillusioned teams running them:

when a project has failed or has ballooned in size to the extent that it will never be completed, and everyone knows it. When that happens, and no one wants to face the facts, and so the project continues to move forward, then it becomes a BS project.

Individuals start to realise that the project has died but will remain funded, and the system supports itself. They realise that despite whether they work or not, nothing happens and nothing matters, other than the charade itself. A reminder that this really drains motivation.

That intro is probably a bit harsh, What does the enterprise do well?
Running incredibly complex, massively scaled businesses, often successfully, credit where it is due.

Hero Warship

Another recognisable enterprise pattern is hiring saviours.

Person X, accolade Y, education Z (MBA) is heralded as the big new hire with the big new plan, assigned a big new budget to (finally) achieve the big broad goal.

Hero comes off the back of recent success in the exact same situation at a competitor. They start with an investigation into the “business needs” through many committees, leading to a feature hit-list-box-ticking exercise. Eventually, only massive companies like Oracle or IBM are in the running.

Cynically, this project is entirely for the purpose of leveraging it into a new role at a new bank (Failing Up), years before the project has any chance of fruition.

Another version of this game: In walk the management consultants. Strategise the flavour of the month (Modern Data Mesh), sell the dream, plan the diagram:

optimise cut reduce expand single source modern truth AI prediction profit, drops mic

The contract staffing company arrives, waterfall chart in hand and by the time the paint has dried most of the original stakeholders have jumped ship as digitisation data AI blockchain cost management experts, and are spamming the success of the project before it has even begun!

What are you sinking about

I think of enterprise as a system that has built so many incredibly reliable systems to prevent the business from failing that they become rigid.

Incredibly strong. Incredibly inflexible. Entire functional areas begin to decay, while the rigidity remains.

Put people in charge of rules, meetings, and forms, and their first inkling will be “there should be more rules, meetings, and forms”.

The rigidity and strength prevent flexibility, and so change becomes increasingly unlikely. This eventually leads to a state of dysfunction. The structure remains, but the function gradually dissolves. Idiosyncrasies like change review boards outright inhibiting changes.

Broken systems, operating in an organic state of failure:

The system continues to function because it contains so many redundancies and because people can make it function, despite the presence of many flaws. After accident reviews nearly always note that the system has a history of prior ‘proto-accidents’ that nearly generated catastrophe.

Summary

• Compromised, primarily due to risk aversion

• Severe misalignment

• Hot embers, of a big fire 📛

Looking Ahead

As we have seen, the enterprise has largely taken note of the successes (and failings) of the organic brew of recent data tech.

My guess is that over the next few years, we will see this accelerate (they don’t read the salty tweets), but it will largely squish the pulp out of it in the outsourced and cross-matrix-managed divisional regional organisational chaos, and after decades of implementation, will abandon it to the pile of supported and business-critical but largely ignored technology.

Alongside we’ll see ever more of the offering of cargo-cult as a service from the consultancies forcing their way into the fray, with AI names and outsourced quasi-bundled solutions.

Often these kinds of trends are misguided, but some are indeed necessary. Digitisation? Yes if you must, no one wants to call for a delivery update. Cloud? Probably.

So Modern Data Stack then? Well, it comes with a way of working, an ethos around software best practices, some thoughts around who is responsible for what, and how to engage as an impactful product minded team.

Therein lies hope. Where Hadoop was a big data solution, Modern analytics is possibly a process improvement. Doing things a bit more coherently. A slightly easier way to do the same old.

But I worry that it is probably not enough, or not even the case.

The ease with which enterprise data modelling experts have co-opted the data modelling conversation on LinkedIn tells me that the more likely post-fact analysis is going to lead to something along the lines of:

MDS brought Enterprise capabilities to Startups, and not the other way around.

That is a shuddering thought if there ever was one?

As mentioned, and regardless, Modern Data Stack has started simmering in the Enterprise, and is now well on its way. A new song with the same dance.

So what?

Back to the opening point on context.

Take a moment to understand the context behind opinions, product features, books, blogs, tweets and toots.

When consuming in this space it is critical to ensure the relevance of what you are reading, to you.

Is that (this) advice or content applicable to you?

More importantly, the person writing that (this) blog. What is their background? What are they saying. What have they not said?

Thanks to Simon Späti for insightful feedback on an early draft. Simon published a very insightful analysis on how enterprise data systems actually work. Also thanks to my team of editors for the proofreading.

20 Feb 2023 edit - Google is indeed an enterprise by the definition broadly described in this post

Where am I?

Throughout history, this question has usually been resolved somewhat approximately, but always with a strong desire for better accuracy.

From the perspective of explorers deep in the fog of discovering new worlds, each incremental step in increasingly accurate methods of navigation was a new opportunity, much like an entrepreneur sees the incremental tech capabilities as a wonder of possibilities.

Let us pick up the thread of increasing accuracy at the very interesting inflection point of celestial navigation.

Using the angle of stars to the horizon, coupled with some maths and reference charts, one can approximate latitude, with fair accuracy.

Latitude (remember, steps on a La(t)dder) are the ones that go across the map. How far North or South.

Longitude (the down ones), however could only be very roughly estimated due to the lack of effective seagoing clocks and the process being more complicated.

For most of history, Marine Navigators could at best get to the latitude they knew their destination was on, and sail along along that latitude until they bumped into their destination (Land Ho!), or whatever was between them and the destination.

Often this was something hard and sharp.

Crossing the Atlantic Ocean between Europe and the Caribbean (see figure 1) was pretty attainable, with a great heuristic:

Sail South until the butter melts, and then follow the Sun.

The butter melting is the approximate Latitude that you are aiming for in the Caribbean. Following the Sun is heading West. When you spot land, you have arrived!

Beyond that, things get far trickier.

Rounding Cape Horn or Cape of Storms are both formidable challenges, even today. Without accurate East/West determination, you were left with far fewer heuristics and some mediocre workarounds:

As the Crow Flies – When lost or unsure of their position in coastal waters, ships would release a caged crow. The crow would fly straight towards the nearest land thus giving the vessel some sort of a navigational fix. 1

The race to map the world meant that enhancing location accuracy was as valuable then as it is now (maps, the new oil).

And so, a 1700s British government-sponsored technical arms race was established, matched only by the Data Stack sparring of 2021 in terms of strong narratives. The desired outcome - a better clock.

The marine chronometer (a much better clock) was conceived of and improved to such a degree that it enabled navigators to know where they were. 

The chronometer enabled better longitude determination. With an extreme emphasis on better (just a tiny snippet of the history of navigation, which may or may not be fascinating, I won’t impose).

The point, and why I find this a relevant thing to idle upon, is because this approximate location resolution process still exists. When you learn to navigate a modern sailboat, you are instructed to determine your location by taking 3 compass readings, resulting in a triangle, resolving to a fair chance that you are or at least were in that triangle.

One does this despite having a GPS, for obvious reasons to anyone who has ever relied on any highly available service - backup and validate.

And no, the numbers don’t match. Find peace therein.

The point is that within the context of “where am I?” there is always a certain uncertainty around how well all the data feeds are working. Quite literally in the rules of of the ocean, you need to continuously consult “all available means” to assure yourself that you know where you are. Granted the rule is for avoiding a collision, the point is that you may not presume to trust a single data point from a single system.

The link to the data

…is still coming, first another bit of history.

Sir Francis Chichester was another straightforward British navigator type with an entrepreneurial streak, having gone to New Zealand to set up a forestry startup with NZ Combinator. He later picked up an interest in flying, and went to the UK to buy a plane and fly it BACK to New Zealand over a few months - a new Tesla the modern equivalent I presume.

Through this long and dangerous trip (England to New Zealand in 1929 in the above single-seater float plane), he encountered a few setbacks in terms of knowing where he was.

Once in NZ, he decided to tackle crossing the Tasman sea, thought to be a bad idea at the time because navigation would be an issue. He needed to stop halfway across to refuel, and the tolerance for missing the refuelling stop at Lord Howe Island was near zero. If he wasn’t able to land there, he’d certainly be beyond rescue or recovery.

He was suddenly outrunning the metrics layer of a previous generation (neat), so he needed to develop his own:

The challenge of the Tasman remained and Chichester realised that he could reach Australia if he fitted [the plane] with floats to alight on the sea and refuel at Norfolk Island and Lord Howe Island.

The real problem now was to find these tiny spots in the sea. The only method of position-fixing available was to take sunshots with a sextant - not easy when you’re alone in the cramped vibrating cockpit - then laboriously work out the fix with pencil and paper.

He decided to use the principle of ‘off-course navigation’ i.e. deliberately aiming for a point to one side of the island. When this point was reached there would be no doubt which way to make a 90º turn for the final leg to the island. To reduce potential errors in calculation, Chichester worked out a series of examples based on his estimate of the sun’s position at the time of his expected arrival at critical points on his course.

The primary issue was around the speed at which he moved, but through a clever application of logic and tolerance for approximations, he managed (quite literally barely based on the reading of his book) to make it across.

He then went on to become the Chief Data Officer for the Royal Airforce flying school (training WW2 pilots), a prolific seller of maps, as well as a solo sailor (which is where he became famous).

The point here is that he was moving too fast for existing methods. The dynamic was changing too quickly for existing technologies. SQL Server SSIS was no longer sufficient. It may have been sufficient if you were doing milk runs in a steamship from Bristol to Londonderry (or a mid-tier bank with a CIO intent on minimal risk), but certainly not if you were trying to set a navigation record across the Tasman sea.

The Data Navigator

This is the link: Data + Navigation

Today, the problem of longitude has been solved to centimetre accuracy through satellite navigation2. However, the reason for the strong nautical-themed trivia, is that they demonstrate an excellent ability to manage and thrive with uncertainty.

Both Chichester and Cook knew the limitations of the BI tools of their time. They supplemented these with experience and intelligence to surmount what were pretty tough odds, to get to wherever they felt needed arriving.

For clarity, most navigators are indeed supporting roles to the Captain. The Navigator would use their combined technical skills, intuition and experience (collectively, wisdom) to give the Captain guidance on where and when. The Captain would then integrate all insights into their resource planning (a startup on the sea).

What is clear is that the Navigator didn’t rely on navigation aids as a crutch, but rather as means to better outcomes through a clear understanding of the weaknesses and an attempt at improving them.

So what is all of this leading to?

First, the Data Navigator is a useful analogy for startups and their data abuse.

When exploring, there is a benefit to be had from pushing into the unknown, and new methods are often required.

Second, bad data causes problems. Often it’s not the data, but rather the navigator.

Third, the navigator provides context which can be used as part of a higher purpose: alignment, motivation and maybe even consensus.

Three thoughts, three sections. Back to the sea, from whence we came.

1 - Exploration as a startup analogy

Navigation is well applied to the data team when the team is supporting the exploration of new worlds.

What the navigator does is supplement and confirm a mental model that someone has built about how the world works.

The map is not the territory

The idea maze maps nicely from historical exploration to startups - barely anything is known and there is a commercial framework based on discovery leading to reward. A startup is entirely lost, by definition, in the idea maze:

A good founder is capable of anticipating which turns lead to treasure and which lead to certain death. A bad founder is just running to the entrance of (say) the “movies/music/filesharing/P2P” maze …  without any sense for the history of the industry, the players in the maze, the casualties of the past, and the technologies that are likely to move walls and change assumptions.

The navigator has a theory about where they are and are going. This information complements the plan.

The startup has a plan of action and looks for confirmation or invalidation.

A navigator needs to understand their science in addition to the history and futures of their space. This becomes acutely more important when there is some form of race, as was typically the case with all historic navigation.

Innovation mixed in with luck and intuition

A great supplement to this is that the navigator doesn’t rely entirely on data. They use it to complement their intuition.

In a rapidly developing environment, the intuition delivered by a navigator gives an indication of which inputs can be trusted over others, what to prioritise and how to simplify the decision space.3

I think the Data Navigator concept echoes the purpose of data within a risky environment, where there is upside to getting it right.

2 - When it goes wrong

There are many documented situations where among other failings, huge ships just run aground.

He left her on autopilot, but strong currents overnight pushed the ship to the north and east and the chief officer altered her course towards the north. When Captain Rugiati awoke he saw that the Scilly Isles were unexpectedly off his port, not starboard bow4

The issue arises when the Captain receives information from the navigator that undermines the model. That number doesn’t look right. That island doesn’t look right.

Twitter bots, Substack daily views5. These issues are easily overlooked until they blow up.

It ain’t what you don’t know that gets you into trouble. It’s what you know for sure that just ain’t so.

— ?

Dealing with these is the actual skill of the navigator. Being sure of what they know, and balancing this with what they don’t.

Navigators provide the best supplement to decision making, trading off speed against accuracy, and relying on intuition, and overcoming ingrained preconceptions and bias.

Speed not haste

Accuracy trades off against speed almost directly.

Crucially, this needs to be seen in the appropriate context. Some data teams function to ensure accuracy. In that context, the Navigator mindset is likely misapplied, and it’s probably better to rely on an auditor/historian mindset6.

Startups require speed, due to competition. Accounting requires accuracy, due to compliance.

Speed and hurry should not be seen as excuses for bad data. As with navigation, so with data, there are pretty firm fundamentals that are easy to achieve, can be effectively relied upon and supplement measurement and improvement of the basics.7

"To err is human, but to persist [in error] is diabolical."

— Latinum Sayinum

As a navigation route starts to mature and grow in traffic, then the balance favours accuracy, to drive efficiency. Similar dynamic when a startup becomes an established scale-up.

Hierarchy

It should be noted that the Nautical Navigator operates in a strictly hierarchical environment. There are no committees at sea, the Captain has the final word, typically for good reason, though the shipping disaster earlier indicates how this becomes an issue, as described in Black Box Thinking:

When we are confronted with evidence that challenges our deeply held beliefs we are more likely to reframe the evidence than we are to alter our beliefs. We simply invent new reasons, new justifications, new explanations. Sometimes we ignore the evidence altogether.

Many maritime disasters, when deconstructed, relate to power dynamic issues, indicating mental inflexibility around what was true. Running your ship into a reef or ignoring a warning about a storm are great examples of ignoring the data.

This is because new insights are disruptive and hard to integrate. People are happier with consensus than they are with uncertainty, even when the uncertainty might save them.

Organisations think they want more insights and innovation.

They are deluding themselves.

Organisations suppress insights for reasons that are locked inside their corporate DNA: organisations prize predictability and they recoil from errors.

— Gary Klein

Granted that the above line was in the context of innovation, not avoiding disaster, but the same “don’t rock the boat” mentality underlies both aversions to innovation and failing to avert course.

My anecdote - If you are on a UK train and someone is in your seat, and has the same seat booked, then at least one of you is on the wrong train - an uncomfortable insight.

The hardest thing for the navigator to overcome is an unwanted insight.

3 - Purpose

To bring things closer to the point then. A data team ultimately brings a shared context to an organisation. We are here, and aiming there.

As we know, the shared context is abstract, flexible, and transient. All the data does is add a foundation to the abstraction of reality we use to support and enhance our shared context. The data tends to be the one part that is consistent, reliable and infallible.

This has an underlying seemingly heretical belief: a single source of truth is an abstraction.

Single source of truth is a fairytale, data teams help reconcile this untruth.

What it looks like is shared context, with firmly drawn lines, in pen, on paper, that indicate that the boundary of the territory is exactly HERE.

Trails

And so what the data team does is simplify, create some abstractions, and identify options, and trails:

Complete freedom is not what a trail offers. Quite the opposite; a trail is a tactful reduction of options.

― Robert Moor, On Trails: An Exploration

This is to simplify and create shorthands, narratives, and reasonable decisions.

However, the Data Navigator needs to keep this simplification in mind, always apply their experience and skill to avoid disaster, and ensure the drift from reality to map isn’t too severe:

The map maker, the surveyor, the compass reader, the ships crew, and everyone that had some part in the determination of the final pen to paper knows that there are many sources of uncertainty, each possibly extending the truth in their own limited capacity that may indeed lead to the pen on paper determination being slightly but disastrously off.

We had 424,000 daily active users yesterday.” The pessimetricist thinks — hopefully, he does not say this — “Actually, you had in excess of 424,000 HTTP requests from devices associated at least temporarily with unique user accounts registered in your internal systems over a 24-hour time period that survived a number of arbitrary assumptions in your data processing systems that passed muster six months ago but which haven’t been re-evaluated meaningfully since.

— Stephen Bailey

But pen to paper it is, and from that moment onwards, and until a better map is created, that is the single source of truth, the true story.8

Certainty Matters

If the way we are collecting data, storing data, transforming data, distributing data, consuming data and then sharing data each have between 0.1% and 1% chance of error, then that error gets accumulated, and in some cases amplified, apply a narrative and suddenly the truth could be anything.

Is the data correct? What is correct? What is?

This doesn’t land well with people on the receiving end of data systems and old maps. Accountants doing financial reconciliation on data warehouses, and someone who has smashed their fibreglass boat into an uncharted granite rock9.

When a data team gives an indication of uncertainty - this causes thoughts along the lines of YOU CANNOT BE SAYING WHAT YOU ARE SAYING.

The map, still, is not the territory. It is part of building consensus and alignment.

In the same boat

Context isn’t enough, one needs consensus:

Data professionals can build consensus as the company becomes more diverse. Data systems can establish methods for understanding the world even as it becomes more complex. [My emphasis]

— Stephen Bailey again! Perennial Truth Architectures

What that means, is that using context (meaning), the navigator and the team need to build from that shared context towards consensus (opinion or position reached by a group as a whole).

Without consensus, we get stuck in the wrong quadrant of a 2x2 matrix, where autonomy and indirection lead to the night watch sailing the boat in one direction and then the day watch on handover reversing course and backtracking. Consensus is king.10

Last Words

To wrap things up, the story here is as follows:

1. Help your team/company by navigating. Reconciling uncertainty is your special responsibility.

2. Hierarchy and inertia are cultural issues that flummox good intentions and amplify bad data. A purely technical orientation will only get the message so far, a voice and an opinion are necessary to succeed.

3. Build a shared context, and use it to reach consensus.

Navigation [Data] is easy.

If it wasn't, they wouldn't be able to teach it to Sailors [Business People].

— James Lawrence

This is a narrative for the near future of Data Engineering in startups, and I think makes some interesting points. I do think the post has avenues for expansion, especially counter-arguments in the context of enterprise tech. 

Hello to new subscribers, and a shoutout to anyone using the RSS feed.

Thanks to the LocallyOptimistic.com community for some lively discussions on an early draft of this post.

Intro

The core premise of this post is:

Most businesses' data engineering needs have been solved or will shortly be solved by managed services that 10 years ago would require endless and extensive self-built ETL pipelines, databases and tools.

For the exceeding majority of businesses, this means they can and should focus on building capacity for business logic, analysis and predictions instead of data engineering.

The minority of businesses that need streaming services / low latency batch data, will further push the boundary, using specialist Data Engineers.

The implications are that while Data Engineering is growing rapidly, so too are the forces that will undermine the need for Data Engineers, and the current under-supply of competent engineers will lead to an over-supply of junior engineers (this should ring a bell to the Web-dev then Full-stack then Data Science boot camps).

Let us break down the premise further, as that is a massive generalisation, and to the reader of this niche corner of opinion, may seem counterintuitive, inflammatory, and frankly, stupid. 

6 points, expanded into 6 sections. Let's go.

1. Majority

Keep in mind the context for the majority of businesses - technology is often an expensive misdirect when implemented badly. This is exceedingly true when the technology is not directly aligned with their competitive advantage. Majority here means all businesses investing in technology, not just the typecast "blitz scale tech business". All businesses should take advantage of data tooling, or they will in effect be flying blind relative to their peers (or get an advantage over their peers if they get it right).

2. Data engineering

Plumbing of the data - ETL, data warehouse, streaming, batch, orchestration, infra etc. Niche skills that are hard to hire for. 

Distinct from Software Engineering. Mostly not backend developers, more commonly generalists, occasionally highly skilled specialists. 

Data Engineering is quite contentiously named as is made clear later.

3. Business logic

This is covered elsewhere - but in the most abstract terms - businesses should hope that their engineers’ primary focus is on improving the ability to represent the businesses various states, and enhance the ability to interact and modify these states, and even predict outcomes to modifications.

Businesses should strive not to have people worrying about managing infrastructure, plumbing, ops etc over and above what is strictly necessary. Playing on the margin of this point is what the CTO does. 

4. Managed services

Think about Sysadmins of the mid-2000s, arcane knowledge that is now redundant in almost every business, due to AWS, then Heroku, now Vercel, Supabase etc flying up the stack. (Or hadoop specialists. Big Data DBA anyone?).

Same with Data Engineering. Tech abstraction as a service. Managed Services are arriving fast with the likes of Snowflake, Fivetran and the commodifying follow-ons. They’re aggressively chasing down the almighty dollar, undercutting margins, offering better cost structures, as well as a flurry of bundling mergers and consolidation.

5. The minority

Many businesses will still have an exceedingly strong need to increase their advantage through data engineering. Take High-Frequency Trading as an example. These businesses will progress the field, and the best data specialists will be needed in those spaces.

6. Implications

This one is clear, don’t get caught on the wrong side of any sea change. I would (do) argue that the ETL engineer skill-set is mostly going to be marginalised until :skull:.

In much the same way that the market demand for boot-camp Data Scientists is low, due in part to oversupply, better tooling and additionally a reorientation around the expectations of a Data Scientist, so too do I propose that Data Engineering demand dynamics will change. 

I'd like to hope that the rate of ETL code being written is in decline because most can rely on managed services or open-source ELT extractors. 

This point gets some pushback, discussed further in part 6.

But more generally, here is something about changes in the tide:

When the tide turns, there is a definite moment when the tide has indeed turned, but that change in direction becomes apparent to different boats at different times. This depends on context, location, keel depth and distance from both the equator and the moon (not to mention the sun). The gravitational pull has changed, but the water doesn’t start moving everywhere at the same time. 

This blog also, as it became clear through writing it, and quoting sources, agrees with a certain viewpoint on specialisation, the link shall possibly become obvious.

So, that is the intro, I’m going to stick with those 6 sections so that coherence abounds, and explain / expand the points in finer detail. 

1. Majority 

Nearly every company needs a data person. Any company that has ambitions to beat market returns on their investor’s capital and doesn't have someone in a broadly data-dedicated data role will certainly struggle to compete.

10-15 years ago an easy indication that a company wasn't keeping up was having no IT person, the modern equivalent is having no Data Person. 

But the premise is that that person no longer needs to be a Data Engineer.

Reference the sysadmin/dba type roles, which for 99.99% of businesses does not exist, because cloud providers hire those people and abstract their role into a service.

The thrust is that Data Engineering could go the way of the dba. Niche, specialised.

Who/What is the data person then?

Data engineering in the ETL/ELT sense has historically been complex, difficult, emergent, at times chaotic, and required niche software engineering skills.

Now, Extract and Load for most businesses using generic SaaS tools, is solved. Using the standard set of CRM, HR, Finance, and Ops tools, 80% of your ELT work is done for you at a standard, predictable price. 

Commoditised EL SaaS is ubiquitous, with the second wave (of EL providers) offering better services at more favourable terms than Fivetran, with multiple variants and mutations.

T for transform, with general best practices courtesy of dbt, is where the bulk of the  analytics work lies. Critically this is where the data person should start. 

They should be, to some degree, what is known as an Analytics Engineer, but possibly more usefully, not a specialist Data Engineer (nor a Data Scientist for that matter, but that bridge has been crossed).

They should be a purple data generalist:

The data world needs more purple people — generalists who can navigate both the business context and the modern data stack. Let's put aside skillset dichotomies, and learn to feel comfortable in the space between.

If you do need a Data Engineer, probably for some or other niche API that isn’t supported by your EL tool, then this is great work to outsource! (My day-to-day work: Supporting companies on their fringe data engineering needs when their internal team wants some extra capacity or capability).

But the first full time data hire needs to be obsessed with business impact.

2. Data Engineering

A tale of two types of data Engineers: Again with the generalisations! In my view and from what I've seen in the job market, there are two types of data engineers at the moment:

(1) Data Engineers: Software engineers, Data

Described as: Software engineering specialists, with data as the core specialisation, who can focus on the niche areas of data engineering and can work with complex real-time data systems.

Needed When: Only required in tech businesses, and only when software engineers cannot assist. This is not needed for 99% of businesses and these candidates know what they want to work on and have the agency to decide.

Characteristics:

• Tools-oriented

• Computer scientists / Very good software engineers

• Driven by curiosity

• Driven towards perfection of the craft

• Want the solutions to be elegant, optimised

• A specialised role for a specialised business problem

Currently and into the future hired to do the following (quote from a slack group from someone who may or may not want the shoutout):

When building out some data-focused applications, like, say, a streaming data enrichment layer that serves up some curated data real-time to other micro-services, we need software engineers and data engineers. Occasionally you’ll find unicorns who can do it all (we have a few of them), but the vast majority of software engineers aren’t experienced enough with data to also be able to solve complex, big-data, non-SQL problems as well as someone more specialised could.

(2) Data Engineers: Solutions oriented engineers, Data

Described as: Business optimisers. Data engineers that engineer data because it is the biggest blocker in the optimisation of a bigger picture issue, namely analytics as it relates to business improvement efforts. I love this post from erikbern.com:

You work with the recruiting team to define a profile for a generalist data role, that emphasizes core software skills, but with a generalist attitude and a deep empathy for business needs. For now, you remove all the mentions of artificial intelligence and machine learning from the job posting.

Needed when: Data engineering data extraction and centralisation is identified as the key issue in a long line of issues. The primary bottleneck in the optimisation process.

Characteristics:

• Goals oriented

• Background in an adjacent engineering field

• Driven by optimisation, the ultimate goal

• Utilitarian problem solvers, relied upon to get the job done

• Functionally broader skill set, maybe even new to the domain, and not (yet) experts in technology

Historically hired to do the following:

If you were on a “traditional data team” pre 2012, your first data hire was probably a data engineer. You needed this person to build your infrastructure: extract data from the Postgres database and SaaS tools that ran your business, transform that data, and then load it into your data warehouse.

Currently hired to:

Build data warehouse, pipelines, dimensional modelling, deploy analytics tools, string it together, but critically, to drive change in a business.

In short - Type 2 wants the solution to be cheaper, easier, faster, best fit, 80/20, is less intrinsically interested in the how and more interested in the impact on outcomes. 

In my opinion, this is basically now Analytics Engineers, and if you disagree with my take on this concept, speak to an Analytics Engineer who had the title Data Engineer, and ask them if they can relate. Similar experience to those Data Scientists who preferred Analytics Engineering.

Another way to think about these distinctions is (erikbern.com again):

I often think of people as (and this is an unfair crude generalization etc) roughly on a spectrum between tools-oriented and goal-oriented.

Memed as:

3. Business Logic

Engineers as optimisation specialists

My background is in industrial engineering, which is broadly a stats'y engineering field incubated in the optimisation of systems (typically factories).

Layouts, flows, bottlenecks, JIT, supply chain etc. Mostly a solved field in many regards (shout out to the bullwhip effect 🚛 🚚 🚛).

The broad optimisation process for most of those businesses looks something like

1. Collection of SaaS and ERP-like systems to track and account for things

2. Data engineering to extract the various states

3. Analytics on the states

4. Decisions to change the states

5. Track decisions in ERP (i.e. repeat)

When I left engineering school, ERP implementation was where the demand was, and large chunks of engineers ended up implementing/consulting/suggesting various guises of a(n) ERP / CRM / database / app / spreadsheet / chalkboard.

However, it became clear to me (with hindsight) that this was quickly becoming a commodity technology and skillset (i.e. outsource to contractors), and that Data Engineering was the real skill bottleneck.

Businesses were amassing large data sets but struggling to access them, let alone analyse them, and so having lucked into a DE role, I made this transition.

Optimising down the optimisation list

Data Engineering is no longer the bottleneck! This is a huge relief, because Data Engineering is not optimising, rather just a necessary lift and shift. It is purely an operational burden brought about by decisions made with siloed data as the tradeoff. 

Now surely 3. Analytics on the states is the biggest hurdle and opportunity. 

Analytics is currently a headache, which requires significant investment, and where I suggest the investment is made. There is much more value in time spent on the building of “Business Logic”. In this case, Analytics. 

Analytics extends far beyond data modelling and analysis, encompassing business processes, people processes, management and communication. 

Analytics also pushes back into software engineering, system designing and overall value chain analysis. 

4. Managed services

The Data engineering "type (2)" makeover

My day to day is where I form this opinion. I have done much less data engineering as it relates to Data Warehouse fine tuning, and ELT troubleshooting since the tools became so much easier. I do a lot more analytics and a lot more modelling. The problem has moved, onwards, up-system. The old bottleneck has largely been removed and solved. Optimised.

To quantify this stance, consider why  there is a literal tsunami of new spins on data products: metric-stores, reverse ETL, metadata, discovery, quality, etc etc. Great data from Benn Stancil:

In 2017, Y Combinator—an incubator of both startups and the Silicon Valley zeitgeist—funded 15 analytics, data engineering, and AI and ML companies. In 2021, they funded 100 (my emphasis)

These are viable partly because the EL bottleneck was eased, the storage got cheaper and dbt made the whole thing more manageable.

Suddenly the problem wasn't getting the data, it was using the data.

Typically the domain of the elite. The reason Airbnb, Linkedin etc have needed a data catalog for near decades is because they had the engineering clout to make it necessary.

The sudden simplification of this process has meant that the next, hitherto unknown bottleneck gets suddenly bashed into, and there is immense value to be gained by unlocking it.

Build it, will they come?

If offered, many businesses will jump at a SaaS subscription, rather than spending that money on hiring/expanding an engineering team.

The term engineering is derived from the Latin ingenium, meaning "cleverness" and ingeniare, meaning "to contrive, devise" wiki

When the data is easy to centralise, combine and analyse, engineers won't be needed to devise and contrive data combining solutions.

They can go and contrive and devise something else, that is complex, and that gives the company a competitive advantage.

Eventually, analytics engineering could face the same turn of the tide. When the tooling gets so good that the team is composed entirely of analysts and product people, and no contriving engineers.

In the same way that structural engineers are only required when building on quicksand, data engineers are only required when building upon a dataswamp. As the tooling gets better, so do the foundations stabilise.

On the margin

The companies I advise and work with often have much less need for Data Engineering at the outset.

However to clarify one point - when they do need Data Engineering, it is a requirement for specialisation. There is indeed more Data Engineering to be done, but this is increasingly specialised (this is a semi-deliberate contradiction to this entire post that I am OK with).

The companies need help with the edge-case, marginally viable solution, where something emerges, crucial to them, that falls through the cracks of the 80/20 SaaS solutions. The point is that these needs come later down the line. Not at the outset of a data project, but later, once the bulk of the crucial, impactful elements are working and generalist data practitioners have exhausted their options.

A caveat: the assemblage of the appropriate tools in the appropriate order to match business needs and maturity is a tricky problem indeed. Probably something that would benefit from the skills of a Data Engineer. More on that in section 6.

5. Minority

Data ENGINEERING isn't going anywhere.

I recently discussed this with someone from a quant hedge fund, and while they had a computer science background, they were "data" + "engineering" to a profound degree. They needed a real-time (real time real-time) data feed from all of the brokers, with extensive transformation across all of them. Multiple decision systems integrated with predictive models, and then reliably send orders back into that system, in near real-time.

This system literally was the business. Complex, differentiating. Building this was one of maybe two things that the company needed to execute to beat the competition.

Data engineering in certain contexts is necessary, but likely to be a specialisation increasingly of interest to the minority.

The above point alone isn't that contentious.

What is contentious is the WHEN.

Has the tied turned, is it still rising. Who is seeing the signs and who is missing them. Who is seeing evidence where there is none.

6. Implications and Evidence from the field

Hello?

2 more minutes, less hand waving I promise.

Implications for Engineers

This entire post makes the same point as this specialisation bombshell:

What is the right level of specialization? For data teams and anyone else.

It seems fair that, if tools didn't require so much knowledge to use (I'm looking at you, Kubernetes), then on the margin, the need for specialisation would be less.

The extension of this point is that because the data engineering toolset got so much better, the specialisation required is now less. Snowflake and BigQuery users agree. 

The implication for engineers whose work is now easier is the following:

Either you move in the direction of the new business problem.

Or you move to a new business that still has the old problem. 

Or you specialise further until you find another domain to play in, and wait for the tide to turn again.

Erik's blog above makes another point, which made me realise this is a mostly “deeply inspired” notion, so much so that I've used the tools/goals oriented concept in an earlier section.

I often think of people as (and this is an unfair crude generalisation etc) roughly on a spectrum between tools-oriented and goal-oriented. Some people have their favourite tools, and that's what they like to use. They make their whole career about honing a craft with those skills. Other people are more entrepreneurial, and don't care about what tools they use: they care about the ultimate goal.

This topic was quite contentious on Twitter. People made some very stern remarks about specialisation when Erik posted it initially, and I guess I'm not surprised. People are very likely going to fight against any concept that undermines their career domain.

However, this contentiousness further highlights the opportunity:

Contrarian ideas, when right, are "the valuable thing" from the Taleb and Zero to One books:

“What important truth do very few people agree with you on?”

Should this point be right, it will be proven right by (another) tool that reduces the need for specialisation and sells for ${LOTS} because it enables achieving Goal X (data-driven-whatnot) without hiring a team of 100 ludicrously demanding human specialists with endless needs.

Arguments, of which there are a few, against this, include that the startup ELT paradigm is a minority and that data engineering work is firmly entrenched in the structures of larger businesses, especially enterprises. The refinement I think is worth making, is that while this may be true, the hope is that it will become less so. Like the shift to the cloud, I would hope that what we describe as ELT now leads to us finding a better way of doing things, whatever it may end up being, that is as transformational for Data Teams as cloud computing was for Software Teams. (Noteworthy that “hybrid-cloud” has proven so popular with enterprise)

And a pushback to this: enterprises aren’t most businesses. Most businesses don’t have a large tech team, most businesses didn’t exist a decade ago. However most Data Engineers are not employed by most businesses, hence this disconnect. Most Data Engineers would disagree with this premise, but the point is that most businesses won’t need a Data Engineer. 

Looking at history, this happened before, take a look at Data Science as a field, maybe due for a renaissance in the guise of ML. “Data Science” was a crutch for companies not knowing what to do with their data. 

Implications for Businesses

The message from the communities and my experience is clear - Data Engineering as it once was is generally less of a challenge - but building a coherent “data platform” remains a chore. 

What is possibly the most complex part of “Data”, and what they really need help with is, what I suppose quite fairly is called Data Platform Engineering:

• EL tool can start costing inordinate amounts relative to the value gained. 

• X tool sunsetting Y feature 

• Adding a new business tool with an unsupported API that needs a singer tap built. This work typically is open-sourced, so eventually, there will be fewer needing singer taps (pray)

• Airflow proving to be a headache. According to Slack, 90% of airflow users are using managed services, so less specialisation in airflow will be needed (pray pray)

As an example of this, I’ve recently consulted on the best way to ELT some data from a few API sources unsupported by Fivetran, as well as Stitch/Airbyte. The decision complexity is quite high:

1. Is an orchestration tool such as Airflow/Prefect needed yet, and if so, which one?

   • If Airflow, then the AWS instance, the Astronomer version, or self-host?

   • Do we try at the outset to use Kubernetes? Is Airflow stable yet? It still feels overcomplicated.

   • If Prefect, will they as a new entrant be more reliable or still have teething issues?

   • What level of CI/CD for the tools? 

   • Would they benefit from Terraform?

2. Meltano, Airbyte or Singer extractor/tap spec?

   • Meltano [1] seems to be making excellent progress, but requires some minor hosting effort, and also requires an orchestrator. 

   • Airbyte seems to (seems to) be making more of a commitment to quality.

   • Both are wrangling with the ways of incentivising community maintainers.

This is just one “component” of the team’s ELT, not even the full picture of their Data Platform, and it is a subtly complex and consuming decision for those familiar. 

A great way to frame this is quasi-architectural DataOps flavoured generalist Data Guru role of the Data Platform Engineer (DPE):

DPE are thinking about what data exists, who should have which access, how to make it available for usage by people and tools, how to make it redundant (disaster recovery), how to enable discovery (catalog), etc   

Or another spin:

DPE just means that you are the Tech Lead of the Analytics Engineering.

While I don’t necessarily care for the DPE term over DE, I do think DPE aptly captures the key work that many Data Engineers now do, combining and ensuring cooperation between competing tools to build a coherent consumable data platform. 

More than anything, the developer experience for most of the necessary Data Platforms tools is just garbage. Airflow is a nightmare, GCP really a frightening pain, and AWS is just so much worse. The correct abstractions over all of this is a huge opportunity and the thing that the DPE needs to keep an eye on.  

[1] Worth your time to have a look at the Meltano SDK if you need to build an API extractor. Great team, developer experience and ambition. If you are a Data Engineer (either type), these open source projects are possibly the best intersection of your skills, interests and market demand. I set up the most lightweight way to run a Meltano ELT on AWS, using Terraform, and could use a review!

Closing

In closing, I broadly see the below chart as usefully inflammatory and marginally useful.

As Data Science gave way to Data Engineering enthusiasm, I'll say that Data Engineering enthusiasm possibly will have to give way to Analytics, currently called Analytics Engineering.

Following this will be the traditional Data Analyst role, in whatever new guise, which will make some resurgence.

However, the core Data Engineering skill-set, technological awareness and systems thinking, will remain vitally important, but perhaps not in the historical and existing notion of a Data Engineer. 

Appendix

Questions to ponder, hit the comments if you have some thoughts:

1. Will data science re-emerge now that the data wrangling tooling is getting so much better? What will this do to the hierarchy of data science? Maybe ML Engineer is a better candidate.

2. Where does MLops sit, it largely has felt disconnected from “Modern Data Stack”?

3. The enterprise dynamic is entirely different. Enterprise companies will need ETL engineers until the heat death of the sun, ̶a̶n̶d̶ ̶n̶o̶ ̶I̶ ̶d̶o̶n̶’̶t̶ ̶w̶a̶n̶t̶ ̶t̶o̶ ̶h̶e̶a̶r̶ ̶a̶b̶o̶u̶t̶ ̶i̶t̶.̶ and I’m somewhat curious how that plays out.

4. Is training Data Engineers a lost cause, along with training Data Scientists, Front-end devs?

5. Remember that 92% of startups disappear, but while we are stealing fun from tomorrow, we can satisfy ourselves knowing that someone will get it right, but for someone to be right someone else must be wrong.

Please consider subscribing for more on the subject of data systems thinking

What is group by 1

Who is Matt Arderne

The barrier to entry to working as a data analyst is reducing rapidly. If you are a combination of curious, technically astute, outcomes-oriented, driven, observant, people-oriented and a natural leader then the technology should not be a hindrance, as it is quite literally being made easier every single day.

Consider this as a primer, and reach out to me directly if you think you’d like to work in this space, especially if you are motivated and have any data experience.

Is it a good fit for me/you?

1. It is a great fit for people with good soft skills, strong intuition for business operations and a desire to make a difference in how a company operates. Are you interested in finding out the facts about what happened in the past and suggesting changes as to how they should operate in the future? To tell the CEO this? To back the decision and measure the changes? To possibly have been wrong? Good things to like the sound of.

2. There is Very Strong Demand in the startup space for these kinds of roles, so if you can convince the right person that you have what it takes then it is entirely possible to transition into this career, without any formal qualification. Typically good transitions are from engineering and technical operations in fast moving businesses.

3. With that goes the obvious implication that the current high demand may not be sustainable in the long term. 

4. The term Data Scientist has largely been merged into the analytics roles, as pure data science is very niche, and often misapplied. Typically companies need more data analytics than they do data science unless their core product is somehow related to data science, and even if they say it is, it often in reality very much is not.

5. I prefer working in or as a provider to startups and scaleups, as this environment is more dynamic. Larger organisations and enterprises have analytics functions, but they often have more functionally or departmentally specialised roles, which means less exposure to the business as a whole. As you become more senior, larger companies will provide a great growth opportunity. 

Data Analyst & Analytics Engineer stuff:

Necessary tech skills.

1. The Data Analyst / Data engineer concepts have been pushed together into a single role, Analytics Engineer, which broadly makes sense, and can be considered a role related to the Data Scientist. I explained this to some extent visually in this tweet. Further, this describes in a bit more detail what we do and don’t know about analytics engineering.

2. These roles are generally distinct from or work alongside Business Analyst and Product Manager roles.

3. If you can do some part-time education, this short course from one of the ex-dbt employees looks to be worthwhile considering. It is brand new, but they intend to teach the exact thing I usually look to hire for.

4. So, onto The Analytics Engineer skills/tools to know

    I’d give 60% focus on SQL, with maybe 10% each for Git, CLI, Python, BI tools.

   1. SQL

      1. Analytical queries like this tutorial, especially the “advanced” section, also this tutorial

      2. CTEs

      3. Window functions

      4. Focus on consistency, neatness, style

   2. Git basics

      1. Pull, branch, commit, merge

      2. Most people don’t need to know much more

   3. Command-line basics

      1. Navigation, creating, deleting, moving

   4. Python

      1. Python is a huge domain, super useful but harder to get to grips with, and also less useful unless you are in a specific niche that calls for it

      2. Jupyter notebooks

      3. Basic pandas for data manipulation

      4. Functions

      5. Virtual environments basics

   5. BI Tools

      1. BI tools are often prohibitively expensive, so the experience is often limited to one, and often the wrong one or an old one.

      2. Metabase is the defacto open source quick and simple option, it functions in a reasonably useful way. It is worth downloading and testing out. 

      3. Another shoutout to lightdash, a very simple BI tool that runs with dbt, and is familiar to users of Looker.

   6. Amendment: *Excel*

      1. This post strongly assumes Excel expertise, but some readers pointed out that this would be worth stating.

      2. Pivots, xlookup etc

      3. Highly recommend reading the post Doing Better With Excel

Analytics industry context:

Articles/books etc that I think worth having a look at to add some colour to the above.

1. I wrote some articles about what I think about technology, the data-as-a-utility-tool one is probably the only one worth reading.

2. Building a data team at a mid-stage startup: a short story

   1. This very neatly describes my jobs and career so far 

   2. A good representation of what the “data” industry looks like

3. The Modern Data Stack: Past, Present, and Future1

   1. dbt is the tool that I use for SQL transformations. It is probably the single most useful thing to learn in addition to the SQL, git, command line, Python list

   2. They are doing lots of thought leadership in the analytics space, with very good blog posts

4. Technology in Data Analytics

   1. Simplest, shortest overview of the most important tech tools, anything not on this list is possibly out of date or redundant, or used in a different context

5. The Analytics Setup Guidebook

   1. This is a deep dive into modern analytics, opinions are now pretty generic, but it is a BOOK

   2. Some interesting meta-analysis takes on the jobs in the industry here

   3. Possibly the thing that is  done most badly in the analytics space is data modelling, with the background here. This is probably what will be the hardest to attain, and near impossible to hire for, but something that is mostly trial and error anyway!

OK I learned / am learning those skills above

How do I get into analytics?

General advice being - go and start doing analytics in your current position, utilizing whichever data skills/tools/etc you have access to. Guaranteed there are data pain points you can solve (which requires you to have some business acumen and soft skills to identify & work on!). This will get you real-life experience which you can leverage into a full-time position within a couple years.

Thanks Nate Sooter, copied verbatim from your feedback on this post! Totally agree with this advice.

That is all I’ve got. Topical and relevant as of publishing. I’ll continue to add and amend, but for now, if you know of someone interested in Data, send them this. There is a world of nuance not covered here, the most interesting themes all covered in due course in a post here. Or not. 

As may be obvious from the long list of articles, there are lots of thoughts and opinions in this space. I have personally experienced the tension and growth of data analytics alongside data science and data engineering, (and also a frustrated relationship with software engineering). In very many contexts the data science/engineering/analytics terms are used overly interchangeably, and often end up meaning the exact same thing, but often not meaning anything similar at all, just to make it confusing, especially data engineering. See an upcoming post on Data Engineering: Backend Developer, or Data Analyst.2

And as I said, I’d love to hear from you if you are making this transition, or feel like understanding it in more detail. The above is just a primer! Get in touch to set up time directly, quickest via the dreaded: https://www.linkedin.com/in/m-ard/

Please consider subscribing for more on the subject of data systems thinking

What is group by 1

Who is Matt Arderne

Welcome to my third post, one I have wanted to write from the beginning. Getting these posts done isn’t easy, and the time between publishing is a commitment that I undertook rather lightly. Like most good ideas, this one is late, irrelevant, and likely only to be marginally useful. That said, here is a quick rundown on two of the “indicative-of-the-future” SQL tools in data analytics at the moment.

Dataform and dbt, Dbt and dataform

If neither dbt nor Dataform is familiar to you, stop and read my primer post on the modern data analytics stack. This post will still likely be a bit meta unless you are familiar with at least one of these tools. The below two-liner explanation from No frills data warehousing with dbt might be sufficient to get you through.

To use [dbt/Dataform], you only need to be familiar with SQL. The package relies on templating using [jinja/javascript] to enable nifty features like dependency graphs, macros or schema tests. Upon compilation, everything is translated into pure SQL and run on the database’s execution engine. It is quite fascinating how much you can do with such a minimalist tool.

Context

Because I find it interesting, here is my understanding of the history of these tools. Cmd+f to THE TLDR if the what why how and hearsay of technology isn’t your thing.

dbt was born from a common need. Fishtown Analytics, an analytics consulting team, solved their need for a better way to transform data inside a data warehouse. They built a way to manage SQL transformations, as an open-source tool way back in 2016 (epic to git time-travel).

They generously shared their experience through a transformative series of blog posts from their CEO Tristan Handy. He captured many struggling engineer/analysts attention with his “new way” of doing analytics in startups. We, the desperate, listened closely. The message I heard: bring the best of software development to startup data analytics

This was great, but what set this apart amongst the ever-growing set of open-source developer tools was, in my opinion, the “hype house” that was the early dbt slack community.

The Hype House is part of a millennia-old tradition of collaboration among those at the avant-garde of new forms of media, technology, and thought. Outsiders like me have always dismissed the novel as silly, faddish, or worse. When those inside the cutting-edge scenes band together to support, teach, and create with each other, their niche and experimental projects can become the new normal on top of which the next generation builds.

Source (I’ve waited a long time to paste this snippet. Scenius is quite a word, but the article is epic)

The building of this dbt tool was “in the open”, and a community was incubated alongside it. An enthusiastic community. An investible community. The new paradigm was incubated in the dbt slack channel, with contribution, opinion and criticism all weighed and measured in the passionate and growing community.

dbt was first a CLI tool. As is the trend with open-source, evolution is dynamic and open. Dataform was born when a front-end was created by a team of engineers who spotted the opportunity to bring dbt to even less technical analysts through a GUI. Later on, Dataform decided to migrate to a newly built dbt replacement backend for their frontend (also open-source). To match this development, or they had planned to anyway, dbt launched their own GUI SaaS tool (called Sinter, now dbt Cloud for the historians).

Personally, dbt led me to Dataform, and as a consulting data-engineer turned head-of-data, I was drawn by Dataform’s relative ease of getting started. I wanted a tool that I could stick in the hands of a Looker developer and have them hitting the same notes as I was hitting with dbt, with less friction, less cognitive burden, and less fiddle.

So, this post. We’ll briefly run through the most notable differences between dbt and Dataform, how/why to choose, and end with some thoughts on evolution.

Why me? I’ve been working with these tools for a while. I deployed dbt at a company and then migrated to Dataform when Dataform launched. I’ve deployed Dataform with another client and recently worked on a dbt project that used some of the dbt plugins. I now consider both on their relative merits when I make the assessments below.

If you’ve got this far but can go no further, these tools function similarly, with the TLDR worth a look.

The bulk of the value for me has been in having these tools as “ready to use” for data analysts who are strong with LookML, SQL and have exposure to data modelling concepts, but are less familiar with Airflow, CLI tools, Python environments, and git workflows (git undo everything). The singular terror that it is to do anything Python-related on Windows (as a Mac user) has been reason enough to default to avoid having to set someone up with the CLI, enabling git, managing Python dependencies, C++ redistributables, :shock:. Both tools allow developers to move in that direction if they choose, but offer easier onboarding via the SaaS version.

Thus, dbt/Dataform cloud is my primary point of consideration, because that has been my primary interest. Both are extensively used as CLI tools, which for many is the go-to. My focus has been in enabling teams that initially do not have the time/resources to upskill/maintain/support the tech, and so my considerations are within the context of the cloud offering. This may be contentious, but on average the analysts who come from Bizops or something of that nature seem to make for more rounded data analysts, and enabling them is key.

*complete aside, Denodo, a previous employer and data integration technology company, was born out of the need of their consulting team working with large multinationals. They too built a successful enterprise Data Virtualisation tool, which in a way achieves a similar outcome to dbt + Snowflake. In essence a powerful enterprise database multi-plug adaptor, and a SQL DAG builder.

dbt

dbt is an open-source tech success story. A dedicated team created the technology they wished they could pay for, built in the open, engaging the community and incubating a “new way” that enthusiasts began to feel very enthusiastic about.

dbt’s strengths lie in the powerful ecosystem they have developed. Illustrative of this are the integrations with tools such as Fivetran, Census, Databricks, that make raw dbt projects pretty mobile and feel like the beginnings of a standard for SQL transformation DAGs.

This ecosystem enables developers to build powerful open-source plugins, available to use and improve. These include useful macros, prebuilt transformations for specific sources, infrastructure management and even some interesting machine learning plugins. The community has built the tools they need on a common standard.

Because dbt is platform-agnostic it feels like an ecosystem rather than a tool alone. Any modern data warehouse is a viable engine to run it, and any modern data tool is likely considering how they can incorporate elements of it to take advantage of the momentum.

When considering dbt cloud, the online version it feels like an online IDE rather than a standalone tool, and according to the founder as of a 1 year ago they’re

in the very early days in improving the developer experience of writing dbt code. The dbt Cloud IDE is still in its infancy, and is only one of the many ways in which we ultimately believe that users will write dbt code that we want to facilitate. source

My take is that this rings true. The usability features are not quite there, a simple example is no autocomplete, which readily exists as a VSCode plugin. [TODO: Validate this is still true (please comment below)]

Dataform

Dataform built a great user interface and my initial impression was that it felt like an easy transition for someone from Looker to orientate and become productive quickly. The user interface has continuously been improved and the backlog of feature requests was quickly moved through, as the team was reactive to suggestions and rapidly added features that provided better prompts, insights and contextual information. Things like prompting about an unbuilt view being referenced, an effective autocomplete, and continuously compiling the SQL to raise errors like typos, glitches, invalid elements introduced prior to execution. These all added to the great user experience of developing.

My least favourite feature is that any templating is done in Javascript (instead of jinja in dbt). Ideally, I’d prefer something closer to Python than either, but jinja feels easier and more intuitive to a non .js type. 

[Ammendment] A friend’s opinion, below, which runs counter to mine on the jinja/js thing, and I agree with him, especially if you have a team who know Javascript! My counterpoint is that Looker analysts often find jinja easier to get started. Jinja does get very complicated if you try and do things like cohort funnel analysis.

Your main disadvantage of dataform I actually found to be a great advantage: templating is just javascript. Sometimes jinja starts feeling like a programming language (macros in macros etc.). But that makes a horrible developer experience. In dataform, you can just declare models in a regular old .js file, meaning you’re completely free to build whatever you want. Plus you have all the power and tooling of a mature programming language.

[End Ammendment]

Like dbt, Dataform has benefitted from community-maintained packages.

The most interesting thing about this story is that Dataform was acquired by Google. It seems likely that Dataform will be incorporated into Google’s Big Query data warehouse, I suppose as a “transformation” feature, or perhaps still standalone? This has interesting implications but it does mean that there will be some time getting Dataform integrated. It also means that as a tool it may have less of their historic great responsiveness to user suggestions and demands as they run through the enterprise compliance backlog. This does mean that it will be an easy first choice for Big Query customers to test out this paradigm of data transformation. Sadly, it also means that support for other data warehouses has been sunset.

Implications

The implications of the Google acquisition of Dataform are interesting to me, maybe given my unique experience with both tools, in that it pretty firmly plants dbt as the SQL data transformation standard. This has likely been the case regardless, based purely on momentum and head start that dbt has maintained. 

However, had Google maintained Dataform as an independent entity rather than incorporating it into Big Query as an exclusive feature or tool, it may have given Dataform the staging area and resources to properly pose a serious threat as an alternative. This would have been a preferable outcome for me personally. Dataform going exclusively Big Query suggests that the play is likely a more direct threat towards Snowflake’s dominance in the data warehouse space, as the feature race heats up.

From a more practical perspective, the TLDR comes into play. When heading along this road, the first decision you’re likely to make is Big Query or Snowflake (or even Redshift if you have lots of AWS credits… (those are ominous dot dots)) decision, and then if Big Query, deciding between dbt and Dataform. My preference is due to my familiarity with Snowflake, and as a practitioner and consultant, my bet is generally with the coverage and standard-setting dbt. 

TLDR

THE TLDR/

• Dataform was acquired by Google and is now exclusively GCP, and my guess is that it will be integrating directly into the Big Query platform. If you’re a startup in the GCP ecosystem, then that is compelling. If you like or are already using Snowflake, then Dataform is no longer an option.

• dbt has a huge ecosystem built around it, with real momentum benefitting from plugins, integrations and general compatibility across all the analytics tools.

• Both tools have a SaaS offering, effectively a cloud IDE + CICD. Great for getting connected and developing quickly. Both happily run a hybrid of cloud IDE and open-source core.

• The dbt community has led to many ecosystem “gap fillers” such as spectacles, lightdash, and technically even Dataform itself, all taking the dbt base and extending it or recombining it in different ways.

The modern ELT data stack that no one is going to second guess has Snowflake as the data warehouse, Fivetran doing the ExtractLoad and dbt doing the T for transforms. Note that Snowflake and Fivetran are doing the heavy-lifting, but don’t provide the magic. They are doing the quiet and dependable, probably get taken for granted (other than the price tag), but developing is where insight is captured, and developing is done with dbt.

/ THE TLDR

Evolution

If you jumped to the TLDR and then went further, this is likely where you’d want to stop. The rest is a quick self-indulgent attempt at hypotho-philosophising the evolutionary aspects of software building. Beware!

Watching the evolution of dbt and then Dataform quite closely and with entrenched interest, I have found the dynamics at play interesting, engaging, and commentable. What was also interesting is the entrenched alliances that quickly developed along with people’s preferences or opportunities. Sentiment generally went something along the lines of Dataform ripped the idea off dbt, which I feel is mostly wrong, as the idea isn’t novel, just very well executed and well-timed (IMO, comment below).

I made a meta-take comment on the sentiment in another wonderful slack channel, the comment has now been repurposed for this section, as rambly and raw as it is, I thought captured some of the point:

The tribalism vibe on this was always a weird one for me. us/them always felt counter-productive and tribal [1], but then perhaps that was inevitable. Early dbt felt very much like the beginnings of a tribe: lots of in-jokes, data-eology, new/old. Exciting! 

Dataform beginning public life as a dbt front-end ~early 2018 [2] felt pretty confirming.

It did two things for (us) users: 

1. affirmed the dbt value prop [3] 

2. made using the "new-paradigm" much more accessible to non-techy-data-devs [4] 

For reasons, Dataform built their own backend dbt replacement (Google software engineers, I don't know, but they probably do). This change positioned them at odds with one another, but besides that point, the 2 different tools enabled more users to adopt the new paradigm[5].

For me, as a user of dbt and then early dataform, best enabling this new paradigm was a combination of the two, and it was/is great! Collectively this is all a quantum leap for SQL analysts. I'm beyond thrilled that I don't have to use the tools I had to use pre-dbt. 

My post-ramble point: I think evolutionary pressure on technology is bloody brilliant for users. Competition is great, iteration should be encouraged. I'm looking forward to someone building an iteration on Looker to this same end.

The evolutionary pressure did two interesting things:

1. It affirmed the need for a SaaS tool for smaller and larger teams. dbt may never have needed to create a user interface, which would have frustrated my team building analytics efforts in smaller tech communities.

2. It created _an alternative_, which is the most important way to add creativity to the space. In real-time users expressed their preferences for different ways of achieving the same goal, in one great experiment.

Interesting parallels to this comparison can loosely be considered in other data tools such as Fivetran & Stitch (closed vs open-source), Snowflake vs Redshift (developer-friendly vs cheaper on paper) and even Tableau vs Looker (old way vs new way).

Both dbt and Dataform now have relatively independent niches, having benefited from the existence of the other. I'm personally glad for having met and got to know the Dataform team and enjoyed being tangentially involved in their growth. The product is great and has been a pleasure to use, and they are nice people who have put a lot of thought into solving my team’s problems.

I think dbt's success in raising lots of VC money is great, they have created many hours of additional output for each hour of development, and more broadly for an enhancement on the thinking in this space, also great people to deal with!

---comment footnotes---

[1] - Tribal ie The Robbers Cave Study

The Robbers Cave experiment studied how hostilities quickly developed between two groups of boys at a summer camp. The researchers were later able to reduce the tensions between the two groups by having them work towards shared goals. The Robbers Cave study helps to illustrate several key ideas in psychology, including realistic conflict theory, social identity theory, and the contact hypothesis. Source (disclaimer - I read about robbers cave this week)

[2] - dataform announcement date ~ May 2018

[3] - dbt on dataform

Hah! This is one of the implications of open-source! ... In my 3-year post, I explicitly called out this likelihood and embraced it

[4] - early dbt-cloud called Sinter, early 2017

[5] See superordinate goals conveniently linked in [1]

Closing

I enjoyed witnessing first-hand the evolution of "a new technology". But I think it probably fairer to call it a recombination, or a reorg. Bottoms up top-down shuffle.

The reshuffle rather than blinding innovation here is made clear by the fact that I could port the code I wrote in 2016 in Denodo’s VQL query language, designed in 2008, which was a large and involved SQL DAG, into something that dbt and Snowflake would happily use. Entirely independently from dbt, and I’d bet mutually unknown, a company created from a users perspective in essence the same way of working. I’m sure from an evolutionary perspective this divergent and now convergent evolution is an interesting quirk.

In that sense, nothing here is “new” enough to interest a computer scientist. The computer scientists I know have an allergic reaction to most of the analytics tooling. Messy and inefficient new ways of doing things, which then become the standards and that look like inflexion points after the fact, and I think for an analyst this change is in progress. Going back to the hype house concept:

Outsiders … always dismissed the novel as silly, faddish, or worse. When those inside the cutting-edge scenes band together to support, teach, and create with each other, their niche and experimental projects can become the new normal

The user is one of the many pressures that drive the evolutionary development of these tools but is best served by choice. Consolidation of technology into profit mode leaves the user’s needs as a low priority, whereas competitive pressure and alternatives put the users at the front. Current trends are about enabling a data analyst turned developer to create as much insight with as little engineering as possible, making it a great time to be an analyst. The users ultimately benefit when the technology scene is in inclusive, collaborative and open mode. Ultimately we are probably due for a period of consolidation mode, but perhaps more on that later.

Please comment if you have any feedback on any of this, I aim to improve with your help.

edit: Somehow the version on my personal backup blog got all the comments

* again interesting aside: Denodo was part of a different paradigm, with virtualisation being the word. Instead of ETL’ing your data into a data warehouse, you’d rather leave it where it lives and query it directly. Whether in a data warehouse, API, transaction database, or many others! A dream for a data architect faced with decades of legacy tech and a desperate need to unify access for analytics and as a data bus. Dremio is now continuing this trend, focussing more on the data lake concept.

Please consider subscribing for more on the subject of data systems thinking

What is group by 1

Who is Matt Arderne

In my first post, I justified an approach to achieve a scalable system for loading, storing, transforming and distributing data within an analytics context. 

In this post, we’ll be taking a look into my notebook on storing. Specifically, the things I’ve noted as useful when implementing Snowflake. These few notes, scripts and points of reference should save you some time and get you out onto the water sooner. 

Welcome to my pocket notebook, heading Snowflake Important Things - Jan 2020.

Why Snowflake

This isn’t paid content. Though with the flowery praise to come it should be (see contacts below). However, this post doesn't get too far into why Snowflake. Rather it explores how Snowflake. Nonetheless, we may need some justification.

Snowflake’s real value is the reduction of non-value-adding complexity for the user. Putting useful things in your path and keeping anything and everything operationally complex out of your way. Simple as that. If you’ve used PostgreSQL then this shouldn’t feel too foreign, minus index maintenance, table locks, performance issues and upgrades. Pretty standard SQL otherwise, and a few new concepts.

And you only pay for the capacity and performance you use.

That’s it really. 

It is just a SQL database. A very fast one, that handles loads of data, and has lots of usability features. It stores data in a columnar way (rather than rows), which means it is very fast. But you’ll still be writing SQL queries, in a mostly familiar pleasant SQL syntax. 

The primary alternative to Snowflake in this context is Google Bigquery. I’m no expert, but you’d struggle to go wrong with either. Snowflake offers a choice of AWS, Azure or GCP for your horsepower, so that might be reason enough for you to choose Snowflake. At some point, it should start to become clear that Snowflake is just a clever interface for storage and computation built on commodity cloud infrastructure. Very clever. S3 buckets + EC2 for anyone feeling like they’d rather DIY this part, or build a competitor. 

Last part of the intro fanfare: Snowflake is a Data Platform. This is made clear in their recent manoeuvring into the crowded, polluted sea of Data Marketplaces, and a peek into the BI world, with their very simple new Dashboards tool. However, the most platformy move here is a direct integration with Salesforce. More on this in the closing.

Context 

This post doesn’t get too far into the details of the doing, but rather points out things that are somewhat peculiar or unique to Snowflake. Things to be kept in mind when doing the initial deployment. 

The context also caters entirely towards doing your transforming tasks in a SQL transformation tool like Dataform or dbt. 

The structure of this post will loosely follow the order in which you’ll encounter and want to consider various new concepts and features as you implement Snowflake. 

We will start with an intro to a Snowflake deployment. We’ll then apply some structure to loading, after getting the security and costs watertight we will finally set sail with some interesting new features and capabilities. 

1. Deployment 

As of publishing this, you can sign up and get started with a free (no credit card), month-long trial, which gets you floating.

Once you’ve signed up, you’ll need a few things in place as part of the deployment. These include roles, users, databases and warehouses. 

New Concepts

The new concepts introduced here are warehouses and credits. 

/Warehouses

Essentially a warehouse is how you specify the power of compute that you use to run queries. This is interesting because you can assign a warehouse to a role. TRANSFORM roles can use a different warehouse to REPORT roles. This allows you to fine-tune your compute power and response time for various scenarios. Predictable power for TRANSFORM, snappy and responsive for REPORT to keep the end-users happy! 

Warehouses are NOT where you keep your data. Think of a warehouse like a sail that you hoist when the cold query winds blow from the East (or when the warm Summer trade-winds blow from the East depending on your preference).

Practically, a role is granted privileges to use a warehouse in much the same way a role is granted privileges to access a database. A warehouse also needs to be specified whenever a connection is made to Snowflake. 

grant all privileges on warehouse WAREHOUSE_REPORT 
to role ROLE_REPORT;

/Credits

You get billed based on your usage of credits.

Credits are consumed by storage and warehouses.

Every time* you start a warehouse, you pay per second in credits, and so credits are effectively your unit of currency. 

At the time of writing a credit is $2-$3, and negotiating that down when your annual contract value reaches ~$10k is the typical script. 

The outcome of this warehouse/credit scenario is you have a very granular cost breakdown of your query costs.

*Not every query starts a warehouse - see cached data section below.

Additional Notes:

• See a walkthrough of cost calculations, product tiers and implications here.

Permissions

This is the grant <PERMISSION> to <ROLE> part of the database deployment process. 

I like to follow either one of the following two deployment patterns: 

1. The Proof Of Concept (POC) keeps things as simple as possible, while still being stable and scalable. 

2. The Production option adds some additional structure on top of the POC. 

1. Proof of Concept 

This setup doesn't distinguish between PROD and DEV, and rather relies on branching features later on in the transformation, which is perfectly fine.

At the core are the 3 roles, with each only having the permissions necessary to function, without the ability to interfere with the other roles’ domains. 

• INGEST

  • Loads data

  • Can create schemas in RAW database

• TRANSFORM

  • Creates transformation scripts

  • Can read data in RAW

  • Can create schemas in ANALYTICS

• REPORT

  • Read-only access to ANALYTICS

This is shown in the relationship diagram below, where connections indicate permissions assigned. 

You’ll notice in the diagram that the USER_REPORT cannot access the RAW data, this is an entirely deliberate move towards ensuring that downstream tools cannot build a dependency on RAW data.

For further clarification on how all this works, I’ve created a starter kit for Snowflake, which creates the above diagram exactly, ready for a POC. If you’re considering a Snowflake implementation, it is well worth an hour to take a look. Pull requests welcome!

• https://github.com/mattarderne/snowflake-starter

2. Production

The following configuration takes the basics from the Proof Of Concept and enhances them to include a more robust separation between PROD and DEV. There is a duplication of all entities with _PROD with a _DEV version (_DEV not shown in this diagram for simplicity) and distinct role breakdown for accessing Databases. 

Additional Notes: 

• Snowflake case sensitivity is subtly different to PostgreSQL. 

  • Unquoted object identifiers are case-insensitive

  • “ANALYTICS” = ANALYTICS = analytics

• Create a user for every connecting system, and a user for every developer. This will enable you to track the source and cost of all queries. 

• If you already have a Snowflake database, you can visually analyse your setup with the snowflakeinspector.com, great for tracking poorly configured snowflake permissions that you may inherit.

• A very useful bit of code is the grant on future snippet, which allows you to grant all future tables in a schema with a certain permission. 

grant usage on future SCHEMAS in database RAW to role TRANSFORM

grant select on future TABLES in database RAW to role TRANSFORM

2. Extract and Load Nuance

/Loaders

If you are using Stitch, Fivetran or similar, you can target your data warehouse at this point. Assign the tool the appropriate role, warehouse, database and schema as specified in the deployment script (ROLE_INGEST, WAREHOUSE_INGEST, RAW). 

Stitch will create a schema based on the name you give to the job, so stick with something scalable. I like <loader>_<source> format, so you’ll start with something like STITCH_HUBSPOT. It’s key to note that this means you can later pop out the stitch part for a FIVETRAN_HUBSPOT or an ETL_HUBSPOT. 

/JSON

Managed ELT tools will load data as best as they can, typically as rows and columns, but often will insert your data as raw JSON into a single column. This is a good thing. It allows you to become familiar with the incredibly useful Snowflake JSON SQL syntax. 

If you write any custom ELT scripts, ensure when loading data to load all data as JSON variant type. This is the crux of ELT. Schemaless loading means your data lands without any notion of a schema, and so you can define the schema later on in one go in the transformation step. This can be seen as a big step, but it helps to be able to define ALL transformations in the transformation stage, and not have to go back to your Python scripts to add new fields.

Additional Notes:

• Start with a tutorial for handling JSON in Snowflake, just to get the basics. 

3. Secure the perimeter 

At this stage there is a risk of moving too fast, and that awkward speed wobble is avoided by taking stock and balancing the books.

The pre-retrospective things to attend to are Costs and Sensitive Data.

Costs

Snowflake is a powerful tool, and with the largest warehouse running into the thousands of dollars per hour, you want to do two things:

/Set a budget and limit

Determining what you are willing to spend in a month is a good start, and setting a policy to alert you at various increments of that amount will avoid a broadside attack from Finance. Setting the policy to disable future queries across specific warehouses or all of them is a good trip switch to ensure that you aren’t caught at sea.

/Get alerted 

Worse than running up a large bill (depending on who you ask) would be for your credit limit policy to come into play the moment you click run when demo’ing your fancy analytics to a client or stakeholder. 

For this reason, keeping close tabs on spikes in credit usage and becoming familiar with how and where your credits are going is very high on your new agenda. Remember this is SaaS, i.e. Operational Expense. All the costs lay ahead of you on this one.

SnowAlert is a tool that Snowflake maintains. I’ve adopted some of the queries as part of my suggested monitoring in the Snowflake-Starter repo. The queries look for spending spikes across the infrastructure and will return results only if they detect a spike. 

Last thing on cost management and this is more of an opinion.

Historically, database resources are specified against a budget for their max expected load. This left lots of performance headroom for the median query. One could view Snowflake costs with some equivalency to this performance headroom, in that a Snowflake query could run faster if you assign it a larger warehouse at increased cost. 

However there is a premium being paid for the flexibility, and so it benefits you to manage your fleet of warehouses carefully, lest they turn on you. Snowflake is an operational expense. This is a subtle shift. The crux is that every credit spent should “deliver value” in a somewhat meaningful way. 

Additional Notes:

• Snowflake caches results of queries, meaning that you won’t get charged for queries that hit the cache. This requires some nuance when modelling credit intensive processes like incremental updates. See this blog for a run-through.

• Snowflake charges lightly for access to metadata queries, this is because each time your transform tool runs, it queries the schema definition heavily. This was free, it now isn’t. The cost is negligible but it is worth noting what is going on. 

Sensitive Data

/Masking

Snowflake’s “Dynamic Data Masking” feature isn’t quite as dynamic as it sounds but is a welcome addition. You’ll create or replace masking policy EMAIL_MASK and attach that to a role. See this video for an explanation. This is a helpful addition to be able to define masks at an object level. This is a new (enterprise only) feature and works in conjunction or in addition to the standard masking features.

/Access Control

Enable a network policy that whitelists the IPs of Stitch, your BI tool, VPN etc.

Enable multi-factor authentication (MFA) with the Duo app. Duo is GREAT. It prompts for a password protected authorisation on your phone’s home screen. No excuses. All users assigned the ACCOUNTADMIN role should also be required to use MFA.

4. Setting Sail

Snowflake at this point, like setting sail, depends on where you want to go. In my previous post, I outlined what I’d do next, and it looks something like setting up a few data loading tools, writing transforms in Dataform and then distributing the results in an analytics tool. If you haven’t, please check it out.

I will not be overemphasising this section, but rather point out a few of the most interesting features that fall under analysing data. You could at this point treat Snowflake like you would a very tiny t2.tiny PostgreSQL instance, forget about it (other than the $) and continue. 

New features in themselves are not always so interesting, but what is interesting is what they enable when combined with existing features. As in technology, so in databases. 

/Swap With

alter database PROD swap with STAGE

Swaps all content and metadata between two specified tables, including any integrity constraints defined for the tables. Also swap all access control privilege grants. The two tables are essentially renamed in a single transaction. 

It also enables a Blue/Green deployment, which in simple terms means: Create a new database with your changes (STAGE), run tests on that, if they pass, swap it with PROD. If an hour later you realise you’ve deployed something terrible, swap it back. 

/Zero copy clone

create or replace table USERS_V2 clone USERS

Create an instant clone of Tables, Schemas, and Databases with zero cost (until you change the data). Great for testing, development and deployment.

/Time Travel

Combining the clone function, one can time travel to a table as it existed at a specified time (1 day back on the standard plan, 90 days on enterprise). The command below will recover the schema at the timestamp (wayward DROP perchance).

create schema TEST_RESTORE clone TEST at (timestamp=> to_timestampe(40*365*86400));

/External functions

Run a call to a REST API in your SQL. Great for those pesky ML functions. 

select zipcode_to_city_external_function(ZIPCODE)
from ADDRESS;

Closing Meta Industry Thoughts

Snowflake is building a platform, meaning they are building the one-stop-shop for your data needs. The notion of Data Loading is likely going to become more fringe. Snowflake has already moved in this direction with Salesforce.

Einstein Analytics Output Connector for Snowflake lets customers move their Salesforce data into the Snowflake data warehouse alongside data from other sources. Joint customers can consolidate all their Salesforce data in Snowflake. Automated data import keeps the Snowflake copy up to date.

This off-the-shelf analytics is a reasonable next step, perhaps in this case due to investment by Salesforce into Snowflake, but that aside, the data space is finding where lie its layers of abstraction, and this is shown in these industry moves.  

Snowflake is building a platform, doing it well, and charging you for it. Engineering time remains expensive, and so outsourcing this to Snowflake’s managed platform will be a welcome relief. However there are no free lunches, and Snowflake is building something bigger than a data warehouse. What this means is that if you take too much, you’ll be stuck with too much. 

Echoing Dremio, there is always a thought towards a modular data architecture “that’s built around an open cloud data lake* (e.g S3) instead of a proprietary data warehouse”. I generally agree with this premise. Snowflake is built on top of AWS or Azure or GCP, and so is (was) a thin layer on top of raw storage and compute. 

* More on data lakes here

Snowflake is marching towards the abstractions seen in Software Engineering, where every job is a feature for them to build. Snowflake has built Data Warehouse Engineer, it is building ETL Engineer and will likely build Data Engineer in some version soon. 

“It is not the ship so much as the skilful sailing that assures the prosperous voyage.” - George William Curtis

Please comment if you have any feedback on any of this, I aim to improve with your help.

Thanks to Dan Lee for reviewing and contributing to this post.

Please consider subscribing for more on the subject of data systems thinking

Subscribe now

What is group by 1

Who is Matt Arderne

Within the companies I have worked for and plan on working for, uncertainty is a common thread. Sales may continue to accelerate, funding should land next quarter, we hope to keep in touch. The uncertainty may be more concrete. We should change to a new CRM. We probably need to stop reporting in Excel. 

I’ve put together some opinions on what has worked for me in managing uncertainty when architecting data systems that need to cater for many parallel futures.

Two Buckets

Designing solutions for analytics systems can stylistically or abstractly be described as a problem of two buckets. Bucket one is full of the typical problems a business might have. The business usually then approaches “the data team” with problems such as help us define a metric / store the data / visualise the KPI / distribute the report.

In this simplistic utopia, bucket two, the Solutions Bucket, is typically filled with lots of products and opinions, like Snowflake / Big Query / my last company used Tableau / group by 1 etc.

 
select * from problems_bucket
   inner join (select * from solutions_bucket) 

The catch when architecting a solution is that you’re given only one scoop from the solution bucket, with the hope that it covers as many of the items in the problem bucket as possible. The solution bucket is usually very resource-intensive, expensive, time-consuming and gathers momentum once that scoop is moving.

The decision to re-scoop is not going to be taken lightly. The first scoop usually needs to be made under significant uncertainty and pressure. This is a time to be making bets that will serve you in many of your uncertain futures.

Travel Light

For this reason, I invoke the spirit of a prepper, where travelling light is as essential as being prepared. Enter the Swiss army knife.

My ideal scoop of the solution bucket, like a good utility tool, has a nice healthy mix of scalability, ease of use, and utilitarian functionality. Bare metal that stands the test of time and rests easily on the hip, ready for action!

More concretely, a lightweight data architecture describes modularity, where each component plays a specified part in the greater whole, without restricting the system. This enables upgrading, downgrading and replacing as necessary. 

With that in mind, I’ll be describing my opinion / experience / preference for a utilitarian data architecture.

Context

The context of this article skews heavily to the typical first-hired-one-person-data-team scenario and is generally applicable if that person is within a small business, a startup or a small team within a larger organisation. It can be extended to a data team within a larger organisation that is rethinking their architecture. 

New paradigms start from the ground up, and so it can safely be assumed that this paradigm will be what banks implement in 50 years while the rest of us use quantum computing to think the data into order.

The driver for this workflow arises from the need to centralise data across multiple systems, typically at the point where there are 3+ key business apps or systems. 

If you're the one called in to take over from the last guy who burnt the ETL (extract-transform-load) candle on both ends and now has a 1000 yard stare, then this might hit a nerve.

Ingesting, Storing, Transforming, Distributing. Four verbs for four (4) sections that describe what will be covered, and the order.

1. Ingesting

I generally subscribe to the opinion that engineers should avoid writing custom ETL code whenever practically possible, and rather use a managed SaaS ETL tool. This resembles the corkscrew of our Swiss army knife. Powerful and simple.

Managed ETL tools allow you to connect to your supported sources, point those at your data warehouse and have data flowing in a matter of minutes. You are paying for specialisation here. Post-implementation, the ETL specialist at the end of an intercom is worth their (initially) nominal fee.

If you cannot get your data into your data warehouse with a managed ETL, or you cannot strike the cost/benefit balance, then you’ll have to start building. This is a great time to think about the possibility of contracting the work to a specialist. They’ll bring the expertise, and over time you can consider internalising that skill as you see fit. Because the work is narrowly described and easily measured, this is a great piece of work for outsourcing. Budget for a maintenance contract, and keep an eye on those Managed ETL services as a replacement option over time.

Some additional thoughts:

• Ingesting raw data (JSON or tables) into your data warehouse is key. Don’t spend time at this point doing transforms in python, there isn’t time. ETL has been surpassed by ELT (extract-load-transform). This new paradigm is now established.

• A Google sheet is a data source. Time is of the essence and done is better than perfect. Data validation, spreadsheet protection and read-only permissions do a database maketh. Use this one sparingly, as word may get out.

2. Storing

Balancing the trifecta of scalability, cost and performance is key when picking the backbone of your system. Your data may start small, or large, or small with a risk of growing large. Stopping to change a tire in bear country is never a good look, and neither is a data warehouse migration.

Managed data warehouses balance the trifecta, with scalability from a team of 1 to 100(n), megabytes to terabytes+, cost starting near zero, and performance flexibility to suit your budget and need.

Your contract for a data warehouse should begin near $0 and go up from there. Start negotiating your unit costs down once the value of the contract becomes significant, or sooner. The point is that you can get started, prove value, and iron out the details down the line.

Snowflake is a good start, Big Query does wonders. Microsoft is probably up to something with Azure. Redshift is squarely in the migrated_from category. All can scale beyond your VC backer's wildest dreams.

This is the knife of your Swiss army knife, simply put, a knife needs to be sharp, a data warehouse needs to be powerful. The main attraction.

3. Transforming

Pliers apply leverage. A Swiss army knife doesn’t have pliers, which is why no one owns one, preferring a utility-tool. Loosely applying the same logic, the Transformation Layer has long been the missing link in the analytics stack, with various frustrating attempts at enabling elegant management of transformations. 

The broad goal here is to enable access to your data for your business users while abstracting away as much of the source system complexity as possible. The outcome is clean, documented, coherent, reliable, logical, self-explanatory and performant data that can be relied upon by the Distributing tools. This is the highest leverage point in your pipeline. Leverage that magnifies both gains and mistakes.

SQL is the language of analysis, and a collection of SQL scripts best describes the transformation of data landed RAW in your data warehouse to ultimately transformed and ready for ANALYTICS. The Analytics mentioned here is the schema that you expose to your Distributing tools.

Dataform is a tool that takes that simple concept and runs with it, making writing sophisticated transformations a delight for analysts. Simply explained, Dataform is a SQL editor that enables analysts to build complex transformations in a way that is maintainable and interpretable. Dataform is differentiated by three concepts from software engineering that are put in the hands of the analyst: 

1/ Continuous Deployment

A deployment of new code or changes to your transforms should be a thing that happens continuously, and without fear. This is achieved through automated schema tests, continuously deploying code, and data validity and quality tests. This is achieved through the assertions in Dataform, among other useful features.

2/ Version Control

If your job involves writing SQL code, and doesn't involve version control, then perhaps more than anything else, this article was written for you.

3/ Modularity

If your SQL queries typically run into the 100's or 1000's of lines, with sub-queries galore, then breaking that into individual reusable modular components will feel like our man on a rock below. Extend this with JavaScript and suddenly you will be able to truly express yourself.

4. Distributing

### TODO - setup a BI tool

Distribution of data. Commonly described as an Analytics Tool or BI tool aka The Last Mile delivery problem.

In a physical product, and an analytics project, the last mile of delivery is often both the most expensive and time-consuming part of the delivery mechanism. This is the point where the surface area expands massively, and the usage pattern permutations explode. Bluntly; the neatly organised cookie-cutter data pipeline gets punched in the face by the needs of the user.

The utility-tool analogy falls apart somewhat at this point, as arguably the pliers should be used here. Just like this utility-tool, it can get a bit confusing.

Broadly speaking the distribution problem gets broken into two categories. BI tools and Analytics Tools. The distinction is murky, like your requirements. Generally speaking, these tools are either:

1/ Good at solving the operational reporting problems of business: Metrics, KPIs, lots of users, lots of operational complexity (tools like Looker, Metabase).

2/ Good at solving the analysts’ problems: complicated questions, nuanced analysis, vague outcomes, forecasts, predictions (tools like Mode, Periscope, Jupyter Notebooks).

A rule of thumb is that you need a good few business users who are comfortable writing complicated SQL or Python before Option 2 will be feasible. This decision is largely based on the operational complexity and technical fluency of the stakeholders in this grand adventure, and generally Option 1 is more broadly applicable.

If you’ve done good work in your Transforming layer, then you can get away with a compromise here, and use a cheaper tool as a stop-gap, or use an array of tools, or allow the team to choose whatever suits them. Ultimately, you want to trend towards a single source of truth for KPI / Metric type numbers, and aim to automate their delivery.

My experience

I've honed in on my preferred data stack, described below. This stack is likely a feasible option for your goals if they are related to aligning your business on key metrics. Especially so if you have multiple SaaS or custom software systems floating around that drive these metrics. What you’ll end up with is something like the following diagram.

Ingesting/ As mentioned, I prefer to use a SaaS ELT tool like Stitch or Fivetran, as they reduce the need for ongoing maintenance where possible. Stitch is the cheaper option, and a great low-cost starting point, with the following useful additions:

• It has a great Import API that allows some simplification of ELT scripts if you do need to write them.

• It has a useful Google Sheets Integration, as well as the usual Postgres, Hubspot, Salesforce, Google/Facebook ads etc.

Storing/ The stack described orients towards BigQuery or Snowflake, with PostgreSQL also a feasible option. I prefer the scale / cost model of Snowflake.

• Snowflake scales up to enterprise but starts from $2/credit, so can be a very cost-effective bet with typical small loads running around 2-5 credits per day. This can get very expensive if you don’t manage it carefully with limits.

• PostgreSQL will require a migration in the future, so unless you are very cost sensitive, the cost / benefit generally leans in favour of Snowflake.

• I have a simple SQL script used to setup Snowflake ready to use for a POC, and I like to use these scripts to track Snowflake credit usage in combination with Dataform assertions.

Distributing/ This is where business users will interact with and judge the success of your system, so to spend your budget on the rest of the components but cut corners on the distribution tool is a bad idea. That said, BI tools can have expensive annual contracts.

• Metabase is a great open-source BI tool and should give you a good place to start. The cost jump is quite severe up to Looker / ChartIO, but so is the feature set.

• These tools are trickier to migrate from, and so it is reasonable to expect to be locked-in for the mid-term.

Transforming/ This may be premature depending on the level of sophistication of logical transformations required to answer your questions, but at some stage it will make sense to move your transforms to the data warehouse from the BI tool. 

• The best of breed at this stage is Dataform or dbt. These tools enable software development best practices (git, testing, documentation).

• There is relatively little involved in adding this from the start, and significant gains to be had if used to build a logical data model from the start.

• I have deployed Metabase successfully with https and nice scalability using these Docker scripts.

In future editions I’ll be diving into the above specifics, stay tuned.

Conclusion

Taking the time to properly implement a reasoned and scalable analytics infrastructure is an axe sharpening exercise with benefits that may compound massively over time. Second-order benefits to aim for include increasing the data proficiency of your team, enabling evidence-based decision making and most importantly, increasing alignment.

Most businesses follow similar patterns, and in survival as in business, preparation is key.

“Give me six hours to chop down a tree and I will spend the first four sharpening the axe.” - Abe

This was a guest post on the Dataform.co blog

Please consider subscribing for more on the subject of data systems thinking

Subscribe now

What is group by 1

Who is Matt Arderne