What is data engineering and why does it matter?
Data engineering is the design and operation of the infrastructure that collects, cleans, combines and keeps data from different systems reliably ready for reporting or AI. Orders in the ERP, customer records in the CRM, sensor readings on the production line and targets kept in spreadsheets may each be correct on their own; but when a manager asks "which customers were profitable this quarter?", those sources have to be combined with the same definitions, at the same point in time and under the same rules.
Big data is the scale side of this work: when the volume, velocity or variety of data reaches the point that one database and manual transfers can no longer carry, distributed storage and processing are needed. Not every organisation has a "big data" problem, but almost every organisation has a data integration and data quality problem. From Ankara, Hatson Tech builds data pipelines, data warehouses and AI-ready data infrastructure for organisations across Türkiye.
What the service covers
- Data integration: Regular, traceable collection of data from ERP, CRM, accounting, production and web systems, through APIs, database connections or file transfer.
- ETL and ELT pipelines: Scheduled pipelines that extract, transform and load data, alert on failure and can be re-run.
- Data warehouse and data lake: Tables with consistent definitions for reporting, and cost-effective storage for raw data.
- Change data capture (CDC): Changes in the source system reach the target within minutes, where a nightly batch is not enough.
- Data quality and observability: Missing, duplicated or late data is caught by automatic rules and reported to its owner.
- Data preparation for AI: Labelled, versioned datasets with traceable lineage for model training, evaluation and RAG.
- Reporting and dashboards: Indicators for management, operations and sales tied to a single source of truth.
What is ETL, and how does it differ from ELT?
ETL (extract, transform, load) pulls data from the source, transforms it and then loads it into the target. In ELT, data is loaded into the target as it is and transformed inside the target system. Which one fits depends on data volume, the complexity of the transformations and the target platform:
| Situation | Better fit |
|---|---|
| Sensitive fields must be masked before reaching the target | ETL |
| The target can run large transformations cheaply | ELT |
| Raw data must be kept and reprocessed differently later | ELT |
| The source system is slow and only needed fields should be pulled | ETL |
| Changes must show up within minutes | Streaming with CDC |
| A daily report is enough and volume is moderate | A scheduled batch job |
What is a data warehouse, and how is its architecture built?
A data warehouse is a database where data from different sources is kept with shared definitions and history for reporting and analysis. Operational systems are designed to run day-to-day work; a data warehouse is designed to answer questions such as "compare with the same period last year" quickly and consistently. The architecture we build usually has three layers:
- Raw layer: Data from the source is stored unchanged, with load time and origin. This layer is what makes it possible to go back and reprocess when something goes wrong.
- Cleaned layer: Types are fixed, duplicates removed, and customer and product IDs matched across systems.
- Business layer: Definitions such as "net sales" or "active customer" are calculated in one place; reports and AI applications read from here.
This separation prevents the most common problem, the same indicator showing different values in two reports: the definition is written in one place and versioned.
Example: one sales view from three systems
Suppose a manufacturer keeps orders in its ERP, quotes in its CRM and returns in its support system. Every month-end, the sales team merges spreadsheets exported from the three systems by hand, which takes about two working days. Because customer names are written differently across systems, the same customer sometimes appears as two rows.
The pipeline pulls data from the three systems every night, merges customer IDs using the tax number and matching rules, and links returns to the relevant order. Unmatched records land on an automatic list and are resolved by their owner. The month-end report is no longer built by hand; it is ready on the dashboard in the morning, and the discussion moves from "which spreadsheet is right?" to "why is this customer losing margin?". The durations in this example are hypothetical and differ in every organisation.
Data quality and data preparation for AI
Most AI projects get stuck on data, not on the model. For a model or a RAG assistant to be reliable, these properties of the data must be designed in from the start:
- Quality rules: Required fields, value ranges, expected row counts and a latency limit; when a rule breaks, the pipeline stops and alerts.
- Lineage: The source and transformation behind a value in a report or an AI answer can be traced.
- Versioning: The dataset used for model training and evaluation is pinned, so results can be reproduced.
- Duplicate and leakage checks: The same record is prevented from landing in both the training and test sets.
We go into this in our article on data preparation for AI. For chunking and indexing document collections, see our articles on chunking at scale and vector databases. Prepared data is also the foundation for the RAG and semantic search solutions built on top of it.
Security, privacy and hosting
Data infrastructure is where personal data gets copied most. That is why which fields are masked, who can see which table and how long data is kept are decided at design time. We can build the infrastructure in the cloud, a private cloud or the organisation's own data centre; the choice affects cost, operating effort and the transfer rules under KVKK, Türkiye's data protection law, so it is discussed up front. For the framework, see our article on AI and data privacy.
Delivery process
- Discovery: Source systems, data owners, existing reports and the most disputed indicators are mapped.
- Definitions: The business definitions of critical indicators are written down and approved by their owners.
- First pipeline: The single most valuable flow, such as sales or stock, is built end to end and connected to a report.
- Quality and monitoring: Rules, alerts and lineage records are put in place.
- Expansion: New sources are added to the same architecture; AI use cases are built on this foundation.
What drives cost and timeline?
- The number of source systems and whether they offer API or database access.
- Data volume and freshness expectations: a nightly batch or minute-level streaming?
- The current quality of the data and the need to match IDs across systems.
- Hosting choice: cloud, private cloud or on-premise.
- Security, masking and retention requirements.
- How many reporting and AI use cases are targeted.
At the end of discovery these factors are itemised, so you can see which assumption each part of the quote rests on.