Data Science: Data Lifecycle
1. Definition
Data science is the interdisciplinary field that extracts knowledge and actionable insight from data by combining statistics and mathematics, computer science (programming, databases, machine learning) and domain knowledge. Its raw material is data in every form — structured (tables, transactions), semi-structured (JSON, logs, XML) and unstructured (text, images, audio, video, sensor streams). The data lifecycle is the set of stages through which data passes, from the moment it is created or captured until it is archived or deleted, and it is a loop: new data and feedback keep re-entering at the start.
2. The Data Lifecycle — Stage by Stage
┌────────────┐ ┌──────────┐ ┌───────────┐ ┌─────────────┐ ┌──────────┐
│ 1 COLLECT │──►│ 2 STORE │──►│ 3 CLEAN │──►│ 4 TRANSFORM │──►│ 5 EXPLORE│
└────────────┘ └──────────┘ └───────────┘ └─────────────┘ └────┬─────┘
▲ │
│ ┌─────▼─────┐
│ │ 6 MODEL │
│ └─────┬─────┘
│ │
┌──────┴──────────┐ ┌───────────────┐ ┌────────────────────────┐ │
│ 9 MONITOR, │◄──│ 8 DEPLOY & │◄──│ 7 INTERPRET & VISUALISE│◄───┘
│ MAINTAIN, │ │ DECIDE │ └────────────────────────┘
│ ARCHIVE │ └───────────────┘
└─────────────────┘ │
└──────── feedback, new data, retraining ────────┘
| Stage | What happens | Tools / notes |
|---|---|---|
| 1. Collection / acquisition | Data is captured from sources: databases and transactions, application logs, web/API data, IoT sensors, surveys, images, social media, open-government portals | APIs, web scraping, ETL jobs; data quality is decided here, not later |
| 2. Storage and management | Data is persisted and organised: data warehouse (structured, modelled), data lake (raw, any format), lakehouse, cloud object storage, DBMS | SQL/NoSQL, S3/Blob, partitioning, access control, backup, governance |
| 3. Cleaning / preparation | Fixing missing values, duplicates, outliers, inconsistent formats, wrong units, encoding errors, spelling variants | Commonly 60-80% of total project effort; "garbage in, garbage out" |
| 4. Transformation / integration | Standardising, joining multiple sources, aggregating, deriving features, normalising/scaling, encoding categories | ETL/ELT pipelines, SQL, pandas/Spark, dbt, feature stores |
| 5. Exploratory Data Analysis (EDA) | Understanding distributions, trends, correlations and anomalies before modelling; hypothesis generation | Summary statistics, histograms, scatter plots, box plots, correlation matrices |
| 6. Modelling and analysis | Applying statistics and ML: classification, regression, clustering, forecasting, A/B testing, causal analysis | scikit-learn, statsmodels, PyTorch/TensorFlow, R; train/validation/test discipline |
| 7. Interpretation and visualisation | Turning results into meaning: which feature matters, how confident, what it implies for the business; dashboards and narratives | Power BI, Tableau, Matplotlib, storytelling; check bias and confounding |
| 8. Deployment and decision | Insight or model is put to work: report to management, dashboard for operations, API/pipeline inside an app, automated rule or recommendation | CI/CD, model serving, human-in-the-loop for high-stakes calls |
| 9. Monitoring, maintenance, archive | Track data and model quality, detect drift, retrain, re-validate, secure and retain or delete data per policy, archive for compliance | Monitoring/alerting, MLOps, data governance, retention schedules |
3. Roles in a Data Team
Data engineer — builds pipelines, storage and infrastructure (the plumbing) · Data analyst — answers business questions with SQL, statistics and dashboards · Data scientist — builds models and runs experiments; strong statistics plus programming · ML engineer — takes models to production, scales and monitors them · BI developer — reporting and semantic layers · Data steward/architect — quality, definitions, lineage, governance · Domain expert — decides what actually matters and validates plausibility. A data scientist's toolkit: statistics, Python/R, SQL, ML, visualisation and communication — the last of which decides whether anything is adopted.
4. Data Quality — Dimensions to Remember
Accuracy (correct), completeness (nothing missing), consistency (same value everywhere), timeliness (fresh enough for the decision), validity (conforms to rules/formats), uniqueness (no duplicates), and provenance/lineage (where it came from and how it changed). Poor quality is the most common cause of failed data projects; governance, validation rules, source ownership and audits are the cure. Privacy and ethics matter throughout: consent, minimisation, purpose limitation, encryption, and compliance with the DPDP Act 2023 / GDPR.
5. Tools Landscape
Querying: SQL, pandas, Spark SQL · Programming: Python, R, Scala · Notebooks: Jupyter, Colab, Databricks · Big data: Hadoop, Spark, Kafka (streaming) · Pipelines: Airflow, dbt, NiFi · Visualisation/BI: Power BI, Tableau, Looker, Matplotlib/Seaborn/Plotly · ML: scikit-learn, XGBoost, PyTorch, TensorFlow, Hugging Face · Cloud/MLOps: AWS/GCP/Azure, MLflow, Docker/Kubernetes, feature stores · Governance: catalogues, lineage tools, access control.
6. Where the Lifecycle Shows Up
Retail/e-commerce: clickstream collection → warehouse → cleaning → basket analysis → recommendation model → deployed on the site → monitored for drift. Healthcare: device and EHR data → secure storage → de-identification and cleaning → diagnostics models → clinician review → audit and retention rules. Education (learning analytics): student answers and timestamps → topic-level mastery estimates → adaptive practice selection → dashboards for teachers → periodic re-estimation as students progress. Smart city: sensor telemetry → stream processing → anomaly detection → traffic/pollution actions → sensor health monitoring.
7. Exam Points
- Define data science and list the skills/fields it combines — 5 marks, almost certain.
- Draw and explain the data lifecycle stages — 7 marks, very frequent.
- Why is data cleaning said to consume most of the effort? Give examples of dirty data — 5 marks.
- Differentiate a data lake and a data warehouse — 3-5 marks.
- List the dimensions of data quality — 5 marks.
- Explain the roles of a data engineer, data analyst and data scientist — 5 marks.
- Name any four tools used at different stages of the lifecycle — 3 marks.