For better or worse, AI rapidly changes how we all work as analytics engineers.

The better:

• Small changes are quicker than ever.

• Boring tasks no longer make up 70% of our work.

• Complex operations can be done solo instead of with an entire team.

The bad:

• There is now an overwhelming amount of changes to review.

• Our data is more vulnerable now than ever.

• Changes can feel out of our control and difficult to trace.

I recently used Claude Code to help me fix a failing data quality test in our daily data pipeline. One of the fields had too many characters and couldn’t be processed by the data warehouse. While Claude was able to offer some solutions like only adding unique values to an array and unioning two arrays, none of these schema changes were backwards compatible.

Tools like Claude and Cursor make it easy to generate database and schema changes, but lack the proper insight that we need as analytics engineers to feel comfortable following these changes. While speed is great, we also need clear control, traceability, and rollback, which these agents don’t yet give us.

This is exactly why data governance frameworks exist.

However, even if a company does have one, it most likely hasn’t caught up to AI. Instead, teams are improvising and figuring it out as they go.

But this isn’t the right way to approach things.

You need to address the most common problems with AI and data governance before it’s too late and AI reaches mass adoption throughout your company.

In this article, I’m discussing the top 3 problems I see with AI and data governance and how you can fix them with the right tools and practices.

Next Thursday, I’m hosting a webinar with Liquibase on all things schema changes, AI-driven development, and data governance.

We will discuss what the shift to AI systems looks like across the modern data landscape and what it means for analytics engineers for database delivery, governance, and reliability.

You will leave with a strong understanding of:

• What controls need to be built into the schema change process

• Foundational data principles that are still important in this age of AI

• What the future of data governance looks like

• How analytics engineers will empower stakeholders in a new way

You can sign up to join this conversation for free here.

Problem 1: Your sensitive data is exposed to AI.

If you think your data is safe with AI tooling, it’s not.

AI accesses much more data than you intend it to. PII, financial records, health data, passwords, you name it.

Snowflake CLI and MCP are great examples of tools that can access all your data when not properly governed.

I recently downloaded Snowflake’s MCP to make backwards compatibility validation easier. I set it up with a specific service account and very limited permission on what it could access in order to protect any PII. Despite this, after a few prompts, it tried to brute force access to Snowflake using my local development permissions. Luckily, I had approvals turned on and was able to stop it.

With using these tools, most people expose them to data that they don’t intend to. What seems like simple context to the agent is actually exposed data.

Solution: The only real solution to this problem is closely auditing your data.

You need to make sure every person and tool that has access to your data warehouse has least-privileged access controls scoped specifically to AI tooling. This means everything that touches your warehouse has the minimum level of access needed to accomplish its goal.

For example, a product manager should only ever have read access on their business domain schema.

Your dbt orchestration job should only ever have write access on the analytics production database, not any raw data tables.

To protect your data even further, all PII should be masked at the schema layer with the exception of any values needed by the data team. Even if AI were to access a production schema that it shouldn’t, PII shouldn’t be able to be read by humans or tools.

I like to always ask myself, “if everything were to go wrong, what needs to be in place for our data to still be protected?”.

This means you need to consider situations like the one I mentioned above where the MCP brute forced its way to use my local development access. I need to access all this data, but how can I make it so even if AI does use my personal credentials, the data is safe?

Data masking and strict role separation are great ways to ensure this.

Problem 2: AI makes a lot of changes that feel impossible to trace and review.

If you’ve ever let an agent loose to handle database changes, you know the pain of it altering 15 different files and hundreds of lines of code. It feels exhausting to try to trace what it did and why, and then validate those changes.

These changes help improve our code but only if they are correct and can be made sense of. To make sense of a complex string of database changes, we need modern tools and processes that allow us to trace these changes.

But what if these changes can’t be traced at all?

There are many processes in the data world that still feel archaic, and database changes are one of them. Most were designed for a world where schema changes were rare and extremely manual (definitely no AI helpers).

This means no version control, review cycles, and rollback capabilities. AND, if something goes wrong, a full-team effort is required or a change is just impossible. Changes made by humans, let alone AI agents, can’t be traced.

In my last role, engineering would frequently change underlying schemas, breaking our data models. This involved a lot of panic and long Slack threads to try to fix what was broken. It was impossible to trace back the exact change that broke everything downstream and the issue would then land on the data team to figure out a work around.

Now, add an AI agent into this data engineering workflow that’s generating and pushing schema changes faster than ever, and the problems you could once ignore become louder.

Solution: Bring tools to the data stack that make development with AI more transparent and put control back in the hands of engineers.

Liquibase Secure does just this. It adds agent-safe governance to AI-assisted database change by enforcing policy checks, traceability, approvals, and rollback before changes reach production.

This means you no longer have to deal with the same frustrations that I did when downstream dbt models break due to improper schema changes. Instead of the responsibility falling on the data team to investigate breaking changes, they can easily be seen, understood, and then rolled back with the help from Liquibase Secure.

Instead of slowing AI down, this tool gives it the same guardrails that good engineering already demands everywhere else in the stack to then enable easier reviewing and traceability. This means less incidents in production, greater availability of data to stakeholders, and greater confidence in AI for data work.

Problem 3: You can’t audit what you can’t trace.

Compliance frameworks like the EU AI Act are beginning to require traceability at the schema level, not just the data level. This means that every change that AI makes needs to be version controlled and traceable.

Most teams have dbt lineage (model-to-model dependencies, column-level lineage, freshness checks) but dbt lineage starts at the source in your warehouse. If an AI agent silently renames a column or restructures a table, dbt has no visibility into that.

It tells you the story from the source forward, not what happened to the source.

Solution: You need to leverage schema-level lineage that captures the full who/what/when/why of every change.

Again, this goes a level deeper than the lineage in your dbt models and traces data back to the source. What dbt lineage assumes is stable, schema lineage proves actually is.

Tools like Liquibase Secure’s dbDoc surface this beneath the dbt layer, giving teams the complete picture: dbt tells you what your data looks like, schema lineage tells you how it got that way. Together they close the audit gap that compliance frameworks are increasingly targeting.

I know I would have loved to have this tool all of the times I was banging my head against a wall, trying to figure out how the data changed in the source. Unexpected schema changes have happened numerous times to me with Hubspot and Intercom, making it time-consuming and stressful to debug.

Schema-level lineage is also ideal because it removes engineering as the bottle neck for breaking changes at the source. If a dbt test starts breaking on a staging model or downstream, you can have full visibility past the dbt layer into the schema layer.

The future of schema changes

I can’t remember the changes I made to my data 3 days ago, let alone 3 weeks ago. As AI takes on more of the heavy technical lift, transparency and traceability becomes even more important. We need to be able to look back on all changes made to our databases and data models by AI.

A tool like Liquibase Secure brings data governance a level deeper further than the transformation layer into the schema layer. This allows us to take greater ownership of the work being done between engineering and analytics, empowering analytics engineers to own the entire data stack.

Governance tooling like Liquibase Secure is quietly setting the standard for how AI and data infrastructure will coexist. Teams that prioritize governance throughout the entire data pipeline, not just in the data warehouse or BI layer, are the ones that will inevitably pull ahead in the race of AI.

The ones that don’t will end up with a sloppy code base that can’t scale because AI is making changes that we can’t keep up with, with zero visibility. We are already beginning to see a wave of people who are struggling to debug their vibe-coded apps or are dealing with leaked data.

No way am I letting the Learn Analytics Engineering community be one of these people!

It seems quite naive now, looking back, but you don’t know what you don’t know. Unfortunately, these lessons are typically learned the hard way.

A few weeks in, I started a Snowflake trial to help improve my skills and experiment with my own portfolio data pipeline project. The data was filtered, and only a few hundred rows, so I figured my costs would be very low.

Well, I was wrong.

Snowflake hit me with a FAT invoice.

The invoice from Snowflake was for $1,571. This was my own personal project, not a project I was building using my 9-5’s Snowflake plan, which meant I owed this money. $1,571 is a lot of money for a 20-something just starting in her data career.

Little to say, I’ve never made these cost-consuming mistakes in my data warehouse ever again.

In today’s article, you’ll learn from the mistakes I’ve made over the years as an analytics engineer, so you don’t have to learn them the hard way.

Subscribe now

Today’s newsletter is sponsored by Greybeam.

The best way to support my work is to check out the cost savings you can get with Greybeam and click the links below.

Not every query needs a warehouse. 99% of BI queries are small, scanning less than 100GB. Running that on Snowflake is like taking a helicopter to go get groceries. Greybeam acts as an intelligent routing layer, matching every workload to its most efficient compute engine:

• Cost optimization: Only pay for the warehouse for the 1% that needs it

• Faster queries: They run 3x faster on the right engine

• Data portability. Manage your data as Apache Iceberg tables, readable by any engine

Stop spending blindly. Start saving on average 86% of your compute costs immediately. No migration. No code changes.

[ Try one month free ] - Book a demo

Mistake #1: Segregating warehouses by purpose and size.

When I designed my very first data warehouse, I thought it was best to keep resources separated by purpose. Data ingestion had its own set of resources. Ad-hoc analysis had its own set of resources. Orchestration had its own set of resources.

This meant that each of these processes had its own warehouse for compute.

While it seemed like a great idea to keep all of the workloads separate, it actually ended up costing us more money than it should have. Instead of warehouses being able to lend capacity to an intensive workload, some were left underutilized while others were forced to scale up.

Because each warehouse also has a minimum start-up cost, we paid this for each warehouse instead of only paying for one.

Instead of separating warehouses by function, create a few core right-sized warehouses that can run your workloads effectively, depending on what needs to be run and when. You can then monitor credit consumption by tag in QUERY_HISTORY and set alerts if a particular workload is burning more than expected.

Mistake #2: Not configuring auto-suspend.

My $1,500 Snowflake bill was caused by the auto-suspend periods of my warehouses being so high. And by so high, I mean 60 seconds. You wouldn’t think this is a long time to be running compute, but it is!

Auto-suspend periods define the amount of time your warehouse can go without any activity before being shut down. This means you are paying to keep it up and running only to avoid a cold start. So, if your auto-suspend is 10 minutes and a new query comes in every 8 minutes, you are paying for the warehouse to be active indefinitely.

It’s important to find the sweet spot where you aren’t continually paying to resume your warehouse when there are bursts of queries with short periods in between, but you also aren’t paying for something with no activity.

Needless to say, if you are running a small portfolio project, you can most likely set this to the lowest possible setting. 😉

Greybeam has an excellent deep dive article on all of these cost-saving hacks for Snowflake users.

Mistake #3: Building all models as incremental tables in dbt.

Just because you can, doesn’t mean you should. When I first started using dbt, I thought incremental models were the coolest feature (and still do!). I wanted to make every model an incremental model, because if I can optimize, why not!?

Well, there are repercussions when it comes to using incremental models. If you are dealing with a large volume of data, the pros greatly outweigh the cons. However, when you’re dealing with tables that otherwise take a few seconds to completely refresh, incremental builds aren’t worth the hassle.

As you know, data is messy. Incremental logic can often miss late-arriving data, not account for source records that get updated rather than just inserted, and use unreliable timestamp fields.

When you run into these types of issues, you need to fully refresh your incremental models. This ends up wasting your time with manual intervention and running up compute on large refreshes.

I always recommend building intermediate data models as views first and then materializing them as a table if needed. With mart models, start with full refresh tables and configure them to be incremental as performance calls for it. This will save you from compute costs but also unnecessary struggle.

Mistake #4: Exposing PII to the average user (especially in the age of AI).

If you’ve used any type of MCP recently, you’ve most likely exposed sensitive data. I hope that isn’t the case, but it’s becoming more and more common. We think our data is safe with these AI tools, but it’s not.

We need to protect our PII at all costs. This means that nobody should have access to PII in their data warehouse, even developers. Most of us don’t need health records, social security numbers, or phone numbers to model business metrics.

I haven’t touched Snowflake’s MCP outside of a very limited scope for this very reason. Our PII has yet to be masked, so I have yet to expose all of our data to these tools.

Before you start playing around with Snowflake MCP or a similar tool, be sure to apply data masking on PII fields in your data warehouse. You can apply data masking based on roles. If a role is any role other than admin, users should not be able to see sensitive customer data.

This will protect you from any accidental permission mixups, because God knows Claude will try to get access to as much data as it can!

I can say I’ve come a long way in my data warehousing skills from when I first became an analytics engineer years ago.

Whether you’ve been in the industry for a while or are just starting out, understanding how to save costs and improve pipeline efficiency are skills any employer will love.

Saying you saved your company $$$ thousands of dollars a month on compute costs is a HUGE flex! I honestly can’t think of anything else that’s more likely to get you a promotion or land a better role.

Have a great week!

Madison

I’ve been dedicating most of my days for the last few months to successfully working Claude Code into my analytics engineering workflow. For most of the process, it was fun, new, and exciting.

I felt like I was flying through my work at record speed. Claude was my coding companion who helped me solve anything and everything!

Don’t get me wrong, Claude is still helping me work faster and more efficiently.

However, now that the shininess has worn off, I’m starting to see it for who it really is.

Over-eager and profligate.

So what does this mean for AI and analytics engineering?

Share

Today’s newsletter is sponsored by Greybeam.

Thanks to them, this post is free for everyone to read! The best way to support my work is to check out the cost savings you can get with Greybeam and click the links below.

Your snowflake bill is bigger than it should be.

Most cost optimization advice stops at “set auto-suspend to 60 seconds.” This guide doesn’t. Greybeam’s free Snowflake Cost Optimization Guide walks you through 10 checks that can cut 30–60% of your costs with the exact SQL to run against your account today.

The greatest cloud savings emerge from costs never born. This guide helps you find the spend most teams overlook, fix it, and put guardrails in place so it stays that way.

Get the free guide · Get a demo

2 Major Problems with AI

I’ve said this before, but I really want to emphasize this now: human discernment is so important.

This week, I used Claude Code to make some basic changes to a few metrics. The changes involved adding an additional app mode for two very specific metrics. Instead of updating the three metrics that I needed, it decided to update all ecomm metrics.

I kept going back and forth with it, specifying that I only wanted it to update the two related metrics and touch nothing else. For some reason, Claude wasn’t understanding and kept overstepping.

Instead of just updating the metrics myself, I ended up wasting an hour trying to reason with it, only to still end up with the wrong outcome.

No matter how much context I gave it, and how specific I tried to be, we were running in circles. This is the perfect example of where you can land with AI when you’re not careful.

There are even rumors going around that Claude has gotten dumber in recent weeks…

I see two main problems:

• AI agents wanting to do too much, too fast, all at once

• AI agents burning costs and wasting resources

Subscribe now

Problem #1: Overstepping boundaries

AI agents are like overeager circus monkeys wanting to please their masters. They will do something just for the sake of doing it, like change metrics you didn’t ask to be changed.

This can quickly become overwhelming when you’re trying to complete a very small, specific piece of work, and it starts touching all areas of your code base. Not only does this make for bad data changes, but it also frustrates us as analytics engineers.

I’ve realized that I need to always utilize plan mode to get the result I want. It’s much easier to narrow the scope in a plan before AI makes the actual changes to your code. Plan mode allows you to reason back and forth and dump the context that you need in one brain dump before the work gets done. This then helps provide all the context upfront.

I used to think plan mode was just for bigger projects like complex data modeling, but I’ve found it necessary for smaller-scoped work like the “simple” metric change I mentioned above.

Not to mention, better scoping of your work also gives greater insight into what AI does and doesn’t have access to.

Today, I used my notebook validation skill to open a Snowflake notebook, and Claude tried to brute force a permissions override from the very clearly scoped service account to my personal user, despite very clear directions not to do this.

This is why I never auto-accept its actions and use plan mode before implementing any changes.

Problem #2: Increasing costs

With AI being shoved in our faces every second of every day, the easy thing to do is default to AI. Instead of adding a simple test myself, I ask AI to do it. Instead of writing the SQL myself, I ask AI to do it.

It can be easy to default to AI for every analytics engineering task, despite the true level of effort being quite small.

Well, AI can do it, so why would I ever do it myself!?

Because of costs, that’s why.

The costs of using AI for the small, very human-capable tasks are quite high.

It’s high on the environment.

It’s high on our mental health and cognitive abilities.

It’s high on our’s or our company’s wallets (whether they realize it or not yet).

How you use Claude Code affects how much you pay. Resuming a chat session, for example, has been shown to increase token usage my 15%. How you use it with data is even more risky!

If you are thoughtful about the rollout of your data warehouse with AI, you will be closely monitoring behaviors and costs. However, what happens when companies roll out widespread data access through Snowflake MCP via Claude Code? Claude will be able to run any gnarly query against your data warehouse without many guardrails. How will long-running and inefficient queries be managed?

It’s become more and more important to use tools and strategies to monitor and control your data warehouse costs with AI. Trust me, you don’t want a surprise bill from Snowflake. I’ve learned that lesson the hard way.

Rawdogging without AI

Frankly, once you learn how to harness AI to do a lot more, I think we all need to then scale it back to figure out what we can rawdog without AI.

I don’t know about you, but I don’t want all the skills I built years developing to disappear. The AI fatigue on the brain is real! I plan to utilize AI for complex planning, dynamic code changes, and documentation. This is where I see it make the biggest difference in my work.

I plan to keep my skills sharp by manually opening a GitHub PR every now and then and continuing to write one-off SQL queries.

As analytics engineers, we need to be honest with ourselves about what’s working and what’s not. Continue learning and developing AI in your workflows, but do so where it makes sense. Don’t overcomplicate the simple things.

In the meantime, start really thinking about governance and how these problems will scale when your greater organization gets access to your data for AI usage.

This is where the real difficult problems will start. They’re better solved on a small scale in your personal workflow first, before stakeholders start facing these same problems.

Have a great week!

Madison

This is what happened last week at work. My company had a hack week where we could solve any issue that’s been pressing us, with the help of AI, of course. The goal was to improve our AI skills and hopefully make something cool in the process.

Lately, my team has been talking about how difficult it is to validate AI code changes in the data world. Not only do you need to make sure your code performs as expected, but you also need to evaluate for backwards compatibility.

I’ve been leveraging Claude Code to do some painful refactoring in our dbt project, and while the changes it’s made seem great in theory, they end up changing the underlying data in unexpected ways. This then makes it difficult to evaluate why the underlying data changed, and if it’s ok to push the changes despite the changes.

In other words, we needed a way to validate the backwards compatibility of data changes using Claude Code.

Another analytics engineer on my team and I set out to solve this problem by creating an AI skill that automatically opens a GitHub PR and links to a validation notebook that it opens in Snowflake with context and queries needed to validate backwards compatibility.

In this post, I’m going to talk about this skill I built with a fellow analytics engineer to automatically open a Snowflake validation notebook upon AI code changes and my key findings along the way.

I’ll share learnings on:

• Principle of Least Privilege

• data governance for MCPs (users, roles, Snowflake security)

• using a CLI tool vs MCP

• iterating on and adjusting instructions in AI skills

• the final project and outcome and why it’s purpose is so important for analytics engineers in the age of AI

Principle of Least Privilege with AI and data

Read more

The truth is that it looks different every day.

We talk with stakeholders to get to the root of their needs, debug failing data pipelines, analyze produc…

Read more

Your company hasn’t dabbled much in AI yet, which means it hasn’t set up any guardrails for how you can use it. This is great for you because you can …

Read more

Our sales team has been looking for data on how well our product features correlate to different outcomes, such as revenue, return on ad spend (ROAS), and average order value. Their hypothesis stated that the greater the number of product reviews, the better the sales. I needed to provide the data to back this up.

I dumped all of my thoughts into Claude Code, asking it to summarize what I was looking for. I then used this to have it create me a step-by-step plan to perform this analysis using the models in my dbt project.

One by one, it provided me with a query that I ran in Snowflake, validating the results or giving it feedback on what didn’t work.

There were only a small number of things that went wrong, and when they did, they were easy fixes.

This goes to show the power of having a dbt project with foundational data models and well-documented code. Claude was able to problem-solve any issues by simply using the code that was already written.

In this article, I’ll walk you through the exact prompts I used to get an entire analytics case study in under an hour. I’ll also share the knitty-gritty details of why Claude was able to do this so well, including what makes a data model easy for AI to use and the most important types of documentation when using agents.

How to Prompt Claude for Data Analysis

Read more

I felt lost in my work as an analytics engineer. I was working towards building a data model that impacted how the business would measure revenue, but I felt like all the forces were against me.

The business’s processes were an unpredictable mess.

My coworker and I had completely different viewpoints on h…

Read more

Why?

AI is often wrong. It could have been trained on poor data. It could have been trained on outdated data. Despite what many people want to believe, it’s not t…

Read more

If they aren’t replacing AEs with AI, then who really can?

OpenAI recently posted an Analytics Engineer role on their Go-To-Market (GTM) team. Thi…

Read more

My employer recently began mandating AI training for its engineers because, not only does it want the company to be ahead of the curve, but it also wants its engineers to keep up with the times. In just a few years, if you don’t know how to use AI, you probably won’t get hired.

I don’t say this to cause fear, but more so to motivate you to start learning these tools now. The sooner you learn them, the more of a leg up you have over those who wait until it’s absolutely necessary.

I also want to mention- it’s never been an EASIER time to break into analytics engineering or data. AI can help accelerate your learning beyond what would be possible without it. The knowledge gap between those hungry to learn and those who have been in the field for 20 years has never been faster to traverse.

Now that I’m done my pep talk on WHY you need to be using AI, let’s get into testing Cursor. If you aren’t familiar, Cursor is an AI-powered IDE specifically for software engineers. It has an AI agent built directly into the IDE, giving it access to your entire code base and terminal.

While it’s superpower is software engineering, I wanted to test it out to see where it thrives and where it fails for data and analytics work, specifically within a dbt project.

With learning to use these tools comes learning WHAT to hand over to them and what to keep tight reins on yourself. AI agents still get a lot of things wrong, so it’s important that you take the time to understand their strengths and weaknesses so you don’t become entirely dependent on them but rather leverage them for their optimizations.

In this article, there are 5 major analytics engineering tasks I use to evaluate Cursor’s effectiveness for dbt, SQL, and data model design work.

• Test #1: Can it accurately make small code changes?

• Test #2: How well does it create templates and document data sources?

• Test #3: How well can it debug?

• Test #4: How well can it write and optimize SQL code within my dbt models?

• Test #5: Can it plan a data model design given a set of business requirements?

Let’s look at each of these in action and use the results to effectively measure the helpfulness of adding an AI IDE like Cursor to our analytics engineering workflow.

Read more

Most data teams have an ultimate goal of providing self-serve analytics, but never truly get there. Why is this? They often aren’t willing to go through the pain required to shift the current culture and state of mind.

When I joined the data team at Kit, we were in a vicious cycle of over-committing and rushing to deliver on stakeholders’ every wish. Instead of prioritizing and saying no, we would say yes to everything.

This led to a lot of one-off reporting queries and dashboards that couldn’t be reused. With every new request came a new data product that had to be built from scratch.

This is a natural pattern that many data teams without analytics engineers fall into. Data analysts don’t naturally build scalable data products, so everything becomes quick and dirty without much thought about the future. Tech debt continues to build in an effort to keep up with all the work coming in. Highly impactful projects get pushed to the back burner because there’s simply no extra time to work on them.

There comes a point in time when you have to put an end to this cycle. Because if you don’t, it will only continue until it implodes on itself.

To get to a spot where BI is truly self-service, a few things first need to happen:

• You’ve created a culture of ruthless prioritization where everything built is tied to a company KPI or goal.

• You focus on solving underlying problems rather than providing exactly what a stakeholder has asked for.

• You’ve created foundational data models that can answer many different business questions.

• You’ve educated your stakeholders on proper data and AI practices, differences between high and low quality data, and how to use your company data models.

What even is self-service BI?

Self-service is one of those terms that have been overly used in the data space. There was a time when everything had “self-serve” slapped in front of it. However, now with AI becoming baked into BI tools like Count, self-serve is finally becoming realistic.

Self-service means stakeholders have the data they need to explore and dig into their questions, without the help of the data team. It means they can feel confidant about the answers they are getting without needing to ask the data team to confirm if their answers are correct.

In short, self-service BI should be entirely disconnected from the data team apart from the initial work of getting the data available and ready to use within the BI tool.

Does your BI tool matter?

While I’ve always been skeptical that you need certain BI tools to enable self-service BI, I’ve used tools that make it nearly impossible to achieve this. After using Quicksight for nearly 2 years, I’ve realized that the tools available to you really do make a difference.

In today’s world, you absolutely need a BI tool with a high-quality, built-in AI agent to achieve self-service. Having an AI agent available to a stakeholder allows them to mimic 5+ years of experience diving into quality data and metrics that they otherwise wouldn’t have.

This means minimal data training for major impact.

Count’s AI agent, for example, allows stakeholders to follow its exact thought process, seeing what it’s looking at and thinking through to answer their question. This not only gives them the answer faster, but it allows them to learn from the agent and also question it when something in the process seems off.

It’s really important to work with AI agents that aren’t black boxes for this very reason. It puts discernment in the hands of the user still, empowering them throughout the process rather than giving over full control.

Of course, with this, I do want to emphasize the importance of having foundational data models that make this possible. AI agents like the one offered by Count have an easier time understanding data that is well-modeled compared to raw data. With raw data comes a higher likelihood of business logic being inferred and therefore incorrect.

Data modeling for AI agents

You can’t have self-service BI without well-built, scalable, high-quality data models. This is something no amount of AI capabilities will be able to replace.

Data models give your AI agents direction. They act as a map of business processes. Let’s look at two examples using the same data and same business questions. In the first example we are using the raw data with the AI agent and in the second we are using modeled data with the AI agent.

EX: Feeding AI agents raw data

I start by giving Count the raw data tables I want it to use to help me understand subscription plan changes. Because it doesn’t yet know the relationships between the tables, it explores the patterns between them, inferring their relationship.

It then begins to build its own set of core models. While these were actually pretty accurate, it created many different models at different grains rather than one core subscription plan changes model, which is all we really need.

This can be confusing for stakeholders to follow, as the models are at different grains, built for different use cases, and are not clearly documented with the business logic they are modeled after. Even if the agent is able to extract the correct insights, it makes it harder for stakeholders to validate.

Count’s agent gave me a pretty thorough description of the subscription changes that occurred, even detailing an important insight that the Explorer plan was the most churn-prone plan despite being one of the least used plans. Overall, it’s pretty impressive what it was able to do without data modeling, but imagine what it’s capable of with more direction!

EX: Giving AI agents modeled data

When you give an AI agent modeled data, you’ve already created the structure that it needs. You’ve cleaned the raw data, adjusted for any mistakes, and have applied the proper business logic.

You aren’t depending on AI to solve the hardest part of the data puzzle. Instead, you can hand over the data and give it the business context it needs to then make sense of the data.

Providing this well-documented data model also gives stakeholders direction in their attempts to make sense of the data. They don’t have to guess what is right or wrong, but are instead guided in the right direction.

When given modeled data, not only was Count able to work faster to develop insights and visuals (and aren’t they beautiful?), but it was able to dive deeper into my data with more confidence.

Other BI features for enabling self-service

In addition to an AI agent, stakeholders need space to drill down and explore data that isn’t in a data warehouse and doesn’t involve editing a dashboard. Business stakeholders shouldn’t have data warehouse access. They also shouldn’t be able to edit a dashboard.

With Quicksight, for example, stakeholders have no way to access the data used in dashboards unless they have access to the data warehouse. This created a huge hurdle for stakeholders that required the data team to constantly pull CSVs for them to then ingest into Excel. Or we had to do the “quick” analysis for them. Every. Single. Time.

Talk about a time suck!

Count has a collaborative canvas feature that allows stakeholders to work alongside AI agents, creating their own data resources entirely separate from those the data team has created. This allows stakeholders to build what they need while also leveraging what already exists, and collaborating with others in the meantime.

As much as I want to tell you a tool doesn’t matter, when it comes to self-service BI it does. There are things you will never be able to do unless you have the right tool. I couldn’t collaborate with stakeholders on a data drill down or even give stakeholders access to datasets that allowed them to explore metrics when I used Quicksight.

That being said, a good tool doesn’t allow you to skip the foundations of high quality data.

You still need to build scalable data models, or even the best BI tool will be prone to mistakes. You still need to make sure you are prioritizing the right work and saying no. If you don’t do this, even the best BI tool will be cluttered with useless dashboards and reports.

A solid, AI-enabled BI tool like Count can take you far when you use it the right way. It helps feed a cycle that allows you to put more of your effort towards foundational data models which then improve stakeholders’ self-exploration of metrics.

These efforts continue to compound, allowing everyone to focus on the big things that move company KPIs the most.

So no. Self-service analytics is not a myth. 😉 You just need foundational data models and a tool that allows you to make the most out of them.

Have a great week!

Madison

Some of these tasks it excels at, others it falls short. Planning a data model is one of those tasks where it seems to fall short. I find that…

Read more

In fact, I’ve learned all of the lessons I’m about to share with you the hard way. Nobody ever taught me these things when I started my career in analytics engineering, but I wish they had. Not …

Read more

I’ve learned a great deal about data modeling from my successes and failures over the past few years. I’ve had high highs of solving complex data quality issues and simplifying reporting work for data analysts on my team. I’ve also had low lows of completely bombing a project because I didn’t narrow down the scope or set clear expectations with stakeholders.

Today, I’m sharing my data modeling design process and the mistakes I’ve made along the way that have informed me of these lessons.

Every month, I’ll be sharing an in-depth, technical deep dive like this one for paid subscribers.

Before You Model: Discovery and Requirements

Read more

Both of these things are what you shouldn’t be doing.

If you didn’t read last week’s newsletter with Riki Miko, I recommend starting there. This discusses…

Read more

Whether you are new to analytics engineering or new to modern data stack tools, it’s easy to get caught up in vendor jargon, taking any slice of information they give you as fact.

That’s what happened to me when I first started in analytics engineering and was intr…

Read more

I fall somewhere in between, because the truth is- AI will change a lot about how we work as analytics engineers. It req…

Read more

If you’re interested in learning more about analytics engineering, The ABCs of Analytics Engineering ebook covers all of the major concepts, tools, and skills for succeeding as an analytics engineer.

Get more from Madison Mae in the Substack app

Available for iOS and Android

Get the app

Read more

What 2025 Taught Me

Read more