Data platforms · AI engineering
Technical Lead Data Engineer

AI amplifies judgment. The engineer is the edge.

I lead enterprise data platforms and build AI workflows.

See shipped work How I work
Email Teams Power BI Web Fabric Tasks Databricks GitLab ROUX THE EDGE

The routine runs on its own. Architecture and release decisions stay with me.

443
AI sessions logged
1 864
Agent runs logged
1 019
Steps in longest run
day 172
Since 18 March 2026

A step is one AI reply or one tool call. Spending money, publishing and deleting stay approval-gated.

  • Cloudflare
  • Azure Databricks
  • Microsoft Azure
  • Claude
  • Claude Code
  • Power BI
  • Astro
  • GitLab
  • Microsoft Fabric
  • Resend
  • Unity Catalog
  • Playwright
  • VS Code
  • Python
Shipped

Built and running.

Reference design · Platform

The data platform I run

I'm responsible for an enterprise data platform, with a focus on architecture, reliable data pipelines and reporting. As a technical lead, I help the team make sound design decisions and maintain clear engineering standards.

  • Loading is configuration-driven: shared loaders do the work, so a new source is mostly a new row of configuration. The loader itself still has to be written well, once.
  • Changes reach production through a release path: test, merge request, main, then live, with checks on every merge.

Azure Databricks · Unity Catalog · Fabric · Power BI · GitLab CI

How I structure data platforms

Reference design · Layers

I structure data platforms so that source data, validation and business models have clear responsibilities. That makes quality issues easier to trace and changes easier to reason about.

  • Bronze keeps what arrived. Silver checks and refines it. Gold holds the shapes the business asks questions in.
  • Reports read from gold, never from raw, so a fix lands in one place.

Databricks · Delta Lake · Unity Catalog · Power BI

How I make ingestion reusable

Reference design · Loading

I use shared loading logic and configuration to reduce repeated code. Quality checks make failures visible before downstream work depends on the result.

  • Configuration says what to load and where. One well-written loader does the work for all of them.
  • A failed check stops the load and keeps the failing rows, so the fix can be tested against them.

Databricks Workflows · Delta Lake · Python

How I release changes

Reference design · Release

I want a release to be explainable and repeatable. Changes are checked, reviewed and verified where people actually use them.

  • Automated tests run before a person spends time reviewing. A red result stops there.
  • What gets deployed is the exact thing that passed. If any step fails, the change goes back to the start, never around.

GitLab CI · merge requests · automated tests

konradvanstaden.com, the homepage

An author's website I rebuilt

I replaced an ageing Joomla site with a static site and added online booking.

  • It has a homepage and one page for each of the author's 18 books.
  • Fonts, scripts and images come from its own domain. Social embeds load only when clicked.
  • The old email booking option stayed available until online booking was ready.
  • The scripted launch could be rolled back within five minutes. I checked the live site before calling it done.
Visit the live site ↗

Astro · Cloudflare Pages · Pages Functions · Resend

    How I work

    One half runs in shadow. The other returns to the light before it ships.

    The way I work is the product.

    1. AlignFrameSet the goal, permitted tools and data, and acceptance criteria.
    2. AlignReplayThe AI explains the plan; I correct it before work begins.
    3. ExecuteRunExecute the agreed plan within its approved limits.
    4. ValidateVerifyCheck the result against the acceptance criteria.
    5. ValidateShipApprove and release the checked result.

    What repeats becomes a safeguard.

    The machine's half

    What can run without me.

    The longest unattended run so far was 1 019 steps, building one of my own applications. Every run is logged and its result checked.

    Watch first, then trust

    I watch new automations until they are reliable. Spending money, publishing and deleting always need my approval.

    Bull in a china shop, then a diamond

    Build the rough version fast in a separate, safe environment. Then harden, test and document it before release.

    These examples come from my personal automations. I define the scope, check the result and approve its release. They sort my email, draft my support tickets and write my meeting notes.

    My half

    What still needs my judgment.

    If I cannot explain a change plainly, I do not ship it.

    Explain the plan first

    Before work starts, the AI explains the plan in plain English. I correct any misunderstanding first.

    Verify the real result

    A success message is not proof. I check what people will actually see or receive.

    Turn mistakes into safeguards

    When a mistake repeats, I turn it into a rule or an automated check. Serious risks get a safeguard immediately.

    Say hello

    Email Kosie

    Technical Lead Data Engineer · South Africa · LinkedIn

    How I structure data platformsSource files, databases and APIs land in Databricks and Delta Lake, where three layers each have one job: bronze preserves the raw data, silver validates and refines it, gold holds the business models. Power BI and analytics read from gold. An illustrative design based on the public medallion pattern.SOURCESFilesDatabasesAPIsexisting systemsDATABRICKS · DELTA LAKEBRONZEPreserve raw datakept as received, so any later step can be re-runSILVERValidate and refinequality rules run here, before anything depends on itGOLDBusiness modelswhat reports and people actually queryPower BIanalyticsread from goldILLUSTRATIVE DESIGN · PUBLIC LAKEHOUSE PATTERNone job per layer, so a problem is traceable to where it enteredHow I structure data platformsSource files, databases and APIs land in Databricks and Delta Lake, where three layers each have one job: bronze preserves the raw data, silver validates and refines it, gold holds the business models. Power BI and analytics read from gold. An illustrative design based on the public medallion pattern.SOURCESFiles, databases, APIswhatever the business already runs onDATABRICKS · DELTA LAKEBRONZEPreserve raw dataas received · replayableSILVERValidate and refinetyped · deduplicated · checkedGOLDBusiness modelsthe shape the questions takePower BI and analyticsread from gold, never from rawILLUSTRATIVE DESIGN · PUBLIC LAKEHOUSE PATTERN
    How I make ingestion reusableConfiguration names each load's source, destination and mode. A shared loader reads that configuration and the source data, then quality checks decide: a pass writes the destination tables, a fail goes to investigation and correction and back through the loader. An illustrative design based on the public metadata-driven pattern.FAILRE-RUNPASSCONFIGURATIONOne row per loadsource · destination · mode (full or changed rows) · scheduleINPUTSource datafile · table · APILOADERShared loaderone loader, many loadsCHECKQualitycounts · keysDestinationtables, on a passInvestigate and correctthe failing rows are kept, so the fix can be checked against themILLUSTRATIVE DESIGN · PUBLIC METADATA-DRIVEN PATTERNa new source is mostly a new row; the loader still has to be written well onceHow I make ingestion reusableConfiguration names each load's source, destination and mode. A shared loader reads that configuration and the source data, then quality checks decide: a pass writes the destination tables, a fail goes to investigation and correction and back through the loader. An illustrative design based on the public metadata-driven pattern.FAILRE-RUNCONFIGURATIONOne row per loadsource · destination · modeINPUTExample source dataa file, a table, an API pageLOADERShared loaderone implementation, many loadsCHECKQuality checkscounts · keys · types · freshnessDestination tableson a pass, and only on a passInvestigate and correctthen back through the loaderILLUSTRATIVE DESIGN · PUBLIC METADATA-DRIVEN PATTERN
    How I release changesA change starts as a GitLab merge request. Automated build and tests run first, then a person reviews and approves it, then the reviewed change is deployed, and finally the released result is verified where people use it. An illustrative design based on public pipeline concepts.IF IT FAILSCHANGEMerge requestGitLabTESTBuild and testautomatedREVIEWReviewa person approvesDEPLOYDeploythe passed buildVERIFYVerifyin real useExplainable and repeatablethe same artefact that passed ships; if any step fails, the change goes back to the start, never aroundILLUSTRATIVE DESIGN · PUBLIC PIPELINE CONCEPTSHow I release changesA change starts as a GitLab merge request. Automated build and tests run first, then a person reviews and approves it, then the reviewed change is deployed, and finally the released result is verified where people use it. An illustrative design based on public pipeline concepts.IF IT FAILSCHANGEMerge requestGitLabTESTBuild and testautomatedREVIEWReviewa person approvesDEPLOYDeploythe passed buildVERIFYVerifyin real useBack to the changenothing goes round the checksILLUSTRATIVE DESIGN · PUBLIC PIPELINE CONCEPTS
    How I structure a data platform, source to reportSources - systems of record, operational stores, files and web services - load a Databricks lakehouse whose bronze, silver and gold layers are driven by settings tables; SQL warehouses serve Power BI, a Fabric mirror and natural-language query; Unity Catalog governs the whole platform. An illustrative design based on public lakehouse patterns, not a picture of any one system. Secrets live in Key Vault with storage on ADLS Gen2, and GitLab CI deploys every change from test to production.JDBCJDBCHTTPSREADJDBCODBCRESTGOVERNSECRETSDEPLOYDATABRICKS LAKEHOUSEERP systemssystems of recordDatabasesoperational storesFiles & APIsflat files · web servicesETLWorkflows & pipelinesdriven by settings tables · one loader per layer · loads only what changedBRONZEBronzeraw · as receivedSILVERSilvervalidated · one versionGOLDGoldready for reportingSERVESQL warehousesone endpoint for reports · stops when idlePower BIdashboards · reportsFabric mirrorsynced copyAI/BI Genienatural languageUnity Catalogwho may read what · lineage from source to report · one place to manage itAzure Key Vault & ADLS Gen2secrets in a vault · no passwords in code · Delta storage in managed locationsGitLab CI/CDtest → merge request → main → production · checks run on every mergeTYPE KEYSource / consumerFocal surfacePrimary data pathPlatform-wide serviceOrchestratedILLUSTRATIVE DESIGN · PUBLIC LAKEHOUSE PATTERNHow I structure a data platform, source to reportSources - systems of record, operational stores, files and web services - load a Databricks lakehouse whose bronze, silver and gold layers are driven by settings tables; SQL warehouses serve Power BI, a Fabric mirror and natural-language query; Unity Catalog governs the whole platform. An illustrative design based on public lakehouse patterns, not a picture of any one system.GOVERNREADDATABRICKS LAKEHOUSESource systemssystems of record · databases · files and web servicesETLWorkflows & pipelinesdriven by settings tables · one loader per layerBRONZEBronzeraw · as receivedSILVERSilvervalidated · one versionGOLDGoldready for reportingSERVESQL warehousesone endpoint · stops when idleReportsPower BI · Fabric mirror · AI/BI GenieUnity Catalogwho may read what · lineage · one place to manage itILLUSTRATIVE DESIGN · PUBLIC LAKEHOUSE PATTERN