CATALOGUED PROJECT

SYSTEM / EVIDENCE / SOURCE

DQSentry

Data-quality platform that converts raw CSV exports into validated, scored, explainable datasets, issue histories, and reviewable reporting artifacts.

Source Live system

Reading repository metadata…

EVIDENCE LEDGER

Verifiable project claims before narrative.

README: data-quality pipeline

Documents Python validation, DuckDB staging, Pandas and PyArrow processing, YAML-driven rules, issue lifecycle tracking, reporting, and automation.

Live application

Recorded validation snapshot27 validation checks · 98.22 overall score · 5 failed checks

Synthetic demonstration snapshot used to exercise scoring and issue detection.

Issue explainability5/5 sampled issue previews include root-cause and recommended-fix guidance

Published scorecard and run history14 generated review artifacts

CASE STUDY

From problem to working system.

01 / PROBLEM

Make raw tabular exports reviewable by detecting quality problems, preserving issue history, and explaining what failed instead of returning only a pass-or-fail result.

02 / APPROACH

Built a Python pipeline around DuckDB, Pandas, PyArrow/Parquet, and YAML-configured validation, then added scoring, issue lifecycle tracking, a Streamlit upload surface, and generated reporting.

03 / IMPLEMENTATION

The decisions that turned the approach into a working system.

Implemented 27 automated validation checks with deterministic synthetic fixtures and regression expectations.
Tracked new, recurring, open, and not-seen issues while generating root-cause and recommended-fix guidance.
Published HTML, CSV, and Parquet evidence and added a configurable quality gate requiring a score of at least 90 with no failed severity-5-or-higher checks.

04 / OUTCOME

The recorded synthetic run scored 98.22, surfaced five failed checks, and produced root-cause plus recommended-fix guidance for all five sampled issue previews.

INTERACTIVE SYSTEM

DQSentry validation workspace

Run representative data-quality scenarios, inspect scoring and gate decisions, drill into issue previews, and review the artifacts produced by the documented DQSentry workflow.

In-browser portfolio simulation only. It models documented DQSentry scoring, gating, datasets, and issue concepts with representative fixtures; it does not upload files to or execute the Python, DuckDB, Great Expectations, or Streamlit pipeline.

Data-quality validation workspace

Run representative CSV-quality scenarios through the same high-level DQSentry flow: ingest, profile, validate and score, then publish review artifacts. The browser model uses DQSentry's documented score formula, quality-gate thresholds, dataset names, and issue-lifecycle vocabulary.

Validation checks

Recorded score

98.22

Failed checks

Review artifacts

The values above are repository-recorded project evidence. Check names and row values inside this embedded workspace are representative portfolio fixtures, not copied production or source data.

Quality score

98.22

Failed checks

5 / 27

Quality gate

Pass

Scoring trace

baseline − 100 × normalized penalty

1.78 penalty / 100.00 weight = 0.0178

Validation checks

Review failed checks first

27 total

Check inspector

User contact field completeness

SIM-USERS-EMAIL · users · severity 3

lifecycle: recurring

Root cause

A small group of representative user rows has an empty contact field after canonicalization.

Recommended fix

Backfill the source export when possible and quarantine unresolved rows before downstream outreach use.

Representative issue rows

USR-1042

user_id: USR-1042
email: (missing)
status: active

USR-1188

user_id: USR-1188
email: (missing)
status: active

Publishable outputs

Review artifacts

Cleansed dataset ZIP

Canonicalized tables ready for downstream inspection.

Issues CSV

Failed checks and issue lifecycle records.

Exceptions CSV

Rows excluded or requiring manual review.

Scorecard

Quality score, gate result, trends, and review context.

Recorded-shape representative fixture loaded. Changes stay inside this browser simulation.

Loading interactive workspace…

REPOSITORY DOCUMENTATION

README

Full repository documentation is available on demand for deeper technical inspection.

Expand full repository README

No README found for this repository.