Big data analytics is the analysis of data too large, too fast or too varied for one conventional database, on systems that spread storage and processing across many machines. In practice, you collect data from many sources into a data lake or warehouse, process it in batches or as streams, and serve the results to dashboards and machine-learning models.
This guide explains what the term means in 2026, the architecture and tool families behind it, the four types of analytics with an example of each, what it costs, how to govern it, and when a small or mid-sized business is better off without it. If you are building analytics into a product of your own, read our guide to building data analytics software; for asking a database questions in plain language, see generative AI for databases.
What big data means in 2026
NIST's reference definition, in its Big Data Interoperability Framework, describes big data as extensive datasets, characterized mainly by volume, velocity, variety and variability, that need a scalable architecture to store, manipulate and analyze them efficiently. Volume is size. Velocity is the rate data arrives, such as card payments or sensor readings every second. Variety is the mix of forms: tables, JSON events, log lines, documents and images. Variability means those characteristics change, as when traffic spikes during a sale. Other "Vs" you will see, such as veracity (accuracy), NIST treats as concerns for the analysis rather than for the architecture.
The same document explains why the term has faded. A system counted as big data when the scale of the data forced its design toward parallel processing across many machines, and NIST calls that shift a one-time occurrence. Today the parallelism is built into the tools. Cloud warehouses spread each query across many machines without you running a cluster, and an analytical database such as DuckDB runs on anything from a small edge device to a server with terabytes of memory and more than a hundred cores. Microsoft's architecture guidance adds that the threshold differs by organization, from hundreds of gigabytes to hundreds of terabytes, and that the definition has moved from data size toward the value of the analysis. In 2026, big data analytics is mostly ordinary data engineering: the same pipelines, warehouses and models, sized to whatever volume you have.
Big data examples
Most data that earns the label comes from events. The Apache Flink documentation points out that card transactions, sensor measurements, machine logs and user interactions on a website or app are all produced as streams. Typical sources, and what makes each one hard:
| Source | Example | What makes it hard |
|---|---|---|
| Website and app clickstreams | Every page view, search and tap | Volume and velocity |
| Payment and order transactions | Card authorizations scored for fraud | Velocity: the answer is needed before the payment |
| Machine and vehicle sensors | Vibration or temperature readings every second | Velocity and volume |
| Server and application logs | Web server access logs, errors, traces | Volume |
| Text, images and audio | Support emails, reviews, call recordings, photos | Variety: no fixed schema |
| Operational databases | Orders, customers, stock levels | Variability, as systems and schemas change over time |
How big data analytics works
Whatever the tools, the architecture has four stages, with orchestration and governance running across all of them. Microsoft's reference architecture lists the same components: data sources, storage, batch processing, real-time ingestion, stream processing, an analytical data store, and analytics and reporting.
- Ingest. Pull data from source systems: scheduled batch loads from databases and SaaS APIs, change data capture from database logs, and event streams through a platform such as Kafka.
- Store. Land it in a data lake (files in object storage), a data warehouse (tables in an analytical database) or a lakehouse that combines the two.
- Process. Clean, join and aggregate it, in batch jobs on a schedule or continuously as a stream, with transformations written as tested code.
- Serve. Deliver results to BI dashboards and reports, to applications through data APIs, and to machine-learning models as training data and features.
Most platforms organize stored data in layers of increasing trust. Databricks calls this the medallion architecture: bronze for raw data as it arrived, silver for validated data and gold for enriched, business-ready tables. Keeping the raw layer lets you rebuild everything downstream when a transformation turns out to be wrong.
Data lake, data warehouse or lakehouse
| Data lake | Data warehouse | Lakehouse | |
|---|---|---|---|
| Stores | Files in open formats in low-cost object storage | Structured tables inside an analytical database | Open files in object storage, plus a table format |
| Schema | Applied when data is read | Applied when data is written | Enforced by the table format, and able to evolve |
| Good at | Keeping everything cheaply, for any engine | Fast, well-managed SQL for BI | Warehouse features on one open copy of the data |
| Watch for | Quality and governance pushed to whoever reads it | Cost, lock-in and weak support for ML workloads | Choosing and operating the format and engines yourself |
Before the lakehouse, the usual design had two tiers: a lake for raw data and a warehouse loaded from it for reporting. A 2021 CIDR paper by Databricks researchers reports that virtually all Fortune 500 companies they saw used it, and sets out its problems: two copies to keep consistent, an extra ETL step that can fail, and warehouse data that lags the lake. Their answer, the lakehouse, keeps data in low-cost object storage in an open file format such as Apache Parquet and adds a transactional metadata layer that records which files make up each version of a table. That layer gives plain files the features of a database table (ACID transactions, versioning, auditing), and it lets a SQL engine, a notebook and a model training job read the same data.
Batch and streaming
Batch processing works on a complete, bounded set of data, such as yesterday's orders. Stream processing handles unbounded data, event by event as it arrives. Flink's documentation draws exactly that line: a bounded stream has a start and an end and can be processed once all of it has arrived, which is batch processing; an unbounded stream never ends, so each event has to be handled promptly after it is ingested.
Batch is simpler to build and operate, and daily or hourly loads are enough for most reporting. Streaming earns its extra complexity when a decision cannot wait: blocking a suspicious card payment, updating stock during a sale, alerting on a machine that is overheating. Running both used to mean two code paths. The Lambda architecture keeps a batch layer (the cold path) beside a speed layer (the hot path), and Microsoft notes that the processing logic then lives in two places. The Kappa architecture sends all data through a single stream path instead, and Spark and Flink each run batch and streaming jobs on the same engine.
Big data tools: the main families
The tools fall into a few families, and most platforms use one from each. The table lists examples, not a ranking, and each description comes from the project's or vendor's own documentation.
| Family | What it does | Examples |
|---|---|---|
| Processing engines | Run transformations, SQL and machine learning in parallel | Apache Spark: a multi-language engine for data engineering, data science and machine learning on one machine or a cluster, in batches or real-time streams |
| Event streaming | Move events from sources to consumers, keep them and process them | Apache Kafka: publishes and subscribes to streams of events, stores them durably for as long as you choose, and processes them as they occur or later. Apache Flink: stateful processing over bounded and unbounded streams |
| Cloud data warehouses | SQL analytics, with storage and compute run by the vendor | BigQuery: fully managed and serverless, with storage and compute layers that scale independently. Snowflake: one central store, queried by independent compute clusters called virtual warehouses. Microsoft Fabric: a SaaS analytics platform on one logical lake, OneLake |
| Open table formats | Make files in object storage behave like tables | Apache Iceberg: a format for huge analytic tables that lets Spark, Trino, Flink and other engines work on the same tables at the same time. Delta Lake: ACID transactions and scalable metadata on S3, ADLS, GCS or HDFS, for batch and streaming alike |
| BI and data science | Turn tables into dashboards, reports, forecasts and models | BI tools such as Power BI or Looker; notebooks in Python or SQL |
The families overlap more every year. BigQuery reads Apache Iceberg, Delta and Apache Hudi tables, Fabric's warehouse separates compute from storage and keeps its tables in Delta Lake format, and Snowflake's virtual warehouses can run Spark workloads. The practical decision is therefore which warehouse or engine suits your team, with the data kept in a format more than one engine can read.
Data lakes began with Apache Hadoop, whose distributed file system (HDFS) and MapReduce spread storage and computation across clusters of computers. It is still maintained (release 3.5.0 arrived on April 2, 2026), but according to the CIDR paper, cloud object stores such as Amazon S3, Azure Data Lake Storage and Google Cloud Storage started replacing HDFS as the home of data lakes from 2015 onward.
The four types of big data analytics, with examples
Analytics answers four questions, each harder than the one before. Microsoft's training material uses the same ladder and adds a fifth rung, cognitive analytics, for AI over unstructured data such as reviews and support tickets. The examples below illustrate typical uses; they are not results from any client.
| Type | Question | Example (illustration) | Typical method |
|---|---|---|---|
| Descriptive | What happened? | Weekly sales by store, channel and product, built from point-of-sale and web orders | Aggregation, dashboards, scheduled reports |
| Diagnostic | Why did it happen? | Online checkout completions fell after a release; splitting by device, browser and payment method shows the drop is one payment option on one browser | Drill-down, segmentation, comparing periods |
| Predictive | What is likely to happen? | A four-week demand forecast per product and store, from sales history, promotions and public holidays | Time-series forecasting, machine-learning models |
| Prescriptive | What should we do? | Reorder quantities per warehouse that meet the forecast within lead times, shelf space and budget | Optimization, simulation, business rules |
Each type depends on the one before. A forecast trained on sales figures that two departments calculate differently will predict the wrong number with confidence, and an optimizer will then act on it. That is why the first two rungs deserve the effort: one definition for each metric, clean history, and pipelines that fail loudly when a source changes. How marketing teams use the predictive rung, from churn scores to marketing mix models, is covered in machine learning in marketing.
Note
Predictive and prescriptive analytics need history. A model can only learn patterns that are in its training data, so a seasonal forecast needs consistent data from past seasons. Start recording the events you will want to predict well before you plan to model them.
What big data analytics costs, and how costs grow
A cloud data platform has three meters: storage, compute and data transfer. They grow for different reasons: storage with how much you keep, compute with how often and how much you read, and transfer with how much leaves the cloud region.
Storage is billed per gigabyte per month and is the cheap part. On BigQuery's price list for the Iowa (us-central1) region, as of September 2026, active logical storage costs US$0.023 per GiB per month, falling to $0.016 once a table or partition has gone 90 days without changes, with the first 10 GiB free. A tebibyte of tables therefore costs about $23 a month to keep.
Compute is billed for the work queries do, so it grows with use rather than with data size. BigQuery's on-demand model charges for the bytes each query processes: $6.25 per TiB in the same region, after the first free TiB each month. The charge follows the columns a query selects and shrinks when tables are partitioned and clustered, while for a table that is not clustered, a LIMIT clause does not reduce the bytes read. Snowflake bills virtual warehouses per second while they run, with a 60-second minimum each time one starts, and each size up doubles the credits per hour.
Here is how that plays out, as illustrative arithmetic from those list prices rather than a quote. A dashboard that re-reads a whole 1 TiB table every 15 minutes, with nothing partitioned, clustered or cached, runs about 2,880 scans a month: roughly $18,000 of on-demand compute against $23 of storage. Partition the table by date so each refresh reads only new rows, select only the columns the chart needs, or pre-aggregate into a small summary table, and the same dashboard costs a small fraction of that.
Data transfer, or egress, is the meter people forget. Moving data into a cloud is usually free and moving it out is not: Google Cloud's network pricing charges nothing for inbound transfer but prices traffic that leaves a region, whether to another Google region or to the internet. Egress adds up when a BI tool in another cloud, an on-premises system or a partner pulls large extracts every day, or when data is replicated between regions. Keep compute in the same region as the data, send aggregates rather than raw tables, and check where your BI tool runs before you connect it.
The controls that keep the bill predictable:
- Cap each query and each day. BigQuery's maximum bytes billed setting makes a query fail, without charge, if it would read more than the limit, and custom quotas cap the data processed per day per project or per user.
- Suspend idle compute. Snowflake enables auto-suspend by default; keep it on, and size each warehouse for its workload rather than for the busiest hour of the year.
- Partition and cluster large tables on the columns queries filter by, usually a date.
- Watch the bill like a metric. Put budgets and alerts on the billing account and review the most expensive queries and dashboards every month.
Governance and privacy for big data
The larger and more shared a platform becomes, the harder it is to say what is in it. Governance keeps four questions answerable: what data exists, where it came from, who may see it and how long it is kept.
- Data catalog. An inventory of every dataset with its owner, description and classification, including which columns hold personal data. In Databricks, Unity Catalog enforces access control when a table is queried, tracks lineage as data is used and logs activity for auditing; in Fabric, the OneLake Catalog is where data is discovered and governed.
- Lineage. A record of which jobs read which datasets and wrote which others, so you can see what breaks when a source changes and find every copy of a record. OpenLineage is an open standard for collecting it, built on three entities: datasets, jobs and runs.
- Access control. Roles that decide who can read which tables, plus row-level filters and column masking for sensitive fields, enforced in the platform rather than in each dashboard.
- Retention and deletion. A rule for how long each dataset is kept, and a way to delete one person's records everywhere, raw layer included. Table formats help: Iceberg, for example, supports targeted deletes in SQL.
Privacy law applies to analytics as much as to the application that collected the data. The EU's General Data Protection Regulation requires personal data to be collected for specified purposes, limited to what is necessary (data minimisation) and kept in identifiable form no longer than needed (storage limitation). Article 25 asks for data protection by design, with measures such as pseudonymisation, and Article 35 requires a data protection impact assessment before systematic and extensive profiling that leads to decisions with significant effects on people. Infringing the basic principles can bring fines of up to €20 million or 4% of worldwide annual turnover, whichever is higher. Similar laws apply elsewhere: California's CCPA gives consumers the right to know what is collected about them, to have it deleted and to opt out of its sale or sharing, and Canada's PIPEDA rests on fair information principles that include limiting collection and limiting use, disclosure and retention.
Important
"Keep everything, storage is cheap" is a cost argument, not a legal one. A lake that holds raw personal data indefinitely conflicts with storage limitation and turns every deletion request into a search. Classify personal data when it lands, pseudonymize it where the analysis does not need to know who someone is, and set a retention period for each dataset.
When a small or mid-sized business does not need big data tools
Often the real problem is not volume. Reports are assembled by hand from spreadsheets, a monthly report takes days of copying and pasting, and two departments bring different revenue numbers to the same meeting. A cluster does not fix that. A well-modelled warehouse, tested pipelines and one good BI tool do.
A warehouse modelled as a star schema, with fact tables for events such as orders and dimension tables for customers, products and dates, is the approach Microsoft's Power BI guidance calls mature and widely adopted by relational data warehouses. PostgreSQL on its own replica, a small cloud warehouse or DuckDB, an in-process analytical database with no server to install or maintain, can run it. BigQuery's free tier alone covers the first TiB of queries and 10 GiB of storage each month.
| You probably do not need big data tools if | You probably do if |
|---|---|
| Your largest tables fit in one database and queries are fast once modelled | Queries stay slow on a well-modelled single server as data keeps growing |
| Daily or hourly refreshes are enough | Decisions depend on events within seconds, such as fraud checks |
| Your data is mostly tables from business systems | You analyze large volumes of logs, text, images or sensor data |
| One team uses the data, mostly for reports | Analysts, data scientists and applications need the same data through different engines |
| The problem is numbers that disagree | The problem is volume or speed, after the definitions are fixed |
Starting small is not a dead end. Keep transformations in version control and store data in open formats where you can, and a later move to a lakehouse becomes a change of engines, not a rewrite of your business logic.
If you are weighing this for your own business, our data platform service starts with a review of your sources, reports and current pipelines, ending in a target architecture and a first use case worth building. We build ingestion, a governed warehouse or lakehouse and a semantic layer, engineered with tests, lineage and access control, and the platform is chosen after we have seen your sources, your volumes and your team. If your current warehouse is sound, we add tests, definitions and monitoring to it rather than moving you.
How to start a big data analytics project
- Name the decisions. Write down the three to five decisions the data must support, and who makes each one.
- Inventory the sources behind those decisions, with an owner for each.
- Define each metric once, in writing, before building anything.
- Pick the smallest platform that meets the volume and freshness you actually need, keeping data in open formats.
- Build one pipeline end to end, with tests, lineage and access control from the first table.
- Reconcile and measure. Check the new numbers against the old reports, then track what each dashboard costs to run.
Related guides on pipelines, warehouses and analytics are collected under our data topic.
