Blog

Build a portable, EU-compliant data platform and avoid vendor lock-in—discover our cloud-neutral stack in this deep-dive blog.

Can a European cloud provider like Ionos replace AWS or Azure? We test it—and find surprising advantages in cost, control, and independence.

Ingesting all data without quality checks leads to recurring issues. Prioritize data quality upfront to prevent downstream problems.

Boost data platform UX with a 5-step process:gather feedback, map user journeys, reduce friction, and continuously improve through iteration

Boost Kubernetes batch workload efficiency with smarter scheduling, autoscaling tweaks & spot instance handling.

Cut Azure logging costs: reduce log volume, use Basic tables via the new ingestion API, and try a custom Fluentbit plugin with Go.

Central DWHs hit scaling limits. Data products offer a modular, federated solution—flexible, reusable, and closer to business reality.

Build SAP CDC in Azure Data Factory with SAP views, but high IR costs. Kafka + Confluent offers a cheaper, scalable alternative.

Responsible AI depends on high-quality data engineering to ensure ethical, fair, and transparent AI systems.

Data leaders face pressure to balance AI hype with data landscape organization. Here’s how they stay focused, pragmatic, and strategic.

Medallion architecture limits self-service. Shift to data product thinking with input, private, and output data for agile, governed scaling.

The more people on a team, the more communication lines. Same goes for tools in your data stack, complexity scales fast

Data Product Portal integrates with AWS to manage data products, access, and tooling—enabling scalable, self-service data platforms.

Data demand grows, pressuring small teams. Shift to focused data product teams and use portals to stay efficient and avoid data siloes.

Data Product Portal links governance, access & tools for self-service data on AWS. Supports Terraform & API integration for automation.

Detect data changes efficiently by sorting and hashing DataFrames with Python—avoid re-running pipelines and reduce infrastructure costs.

Don’t give users a bag of tools—build a UI for your data platform to reduce complexity, boost adoption, and enable true self-service.

Integrate OIDC with the Data Product Portal for secure, user-specific access via SSO. Easy setup with AWS Cognito, Docker, or Helm.

Data Products are rising fast in 2024, focusing on user experience, collaboration, and governance—set to reach maturity within 2–3 years.

Clear team communication boosts data project success. Focus on root problems, structured discussions, and effective feedback to align better

Implement OAuth 2.0 Device Flow with AWS Cognito & FastAPI to enable secure logins for headless devices like CLIs and smart TVs.

The Data Product Portal is an open-source tool to build, manage & govern data products at scaleenabling clear access, lineage & self-service

Speed up AWS Lambda dev with a Makefile: build, deploy, test, and stream logs in one loop boost feedback cycles to just ~15 seconds.

The Data Product Portal is the missing link for scaling data democratization, beyond catalogs, it unifies access, governance & tooling.

Boost SQL generation with LLMs using prompt engineering, schema context, user feedback & RAG for accurate, business-aware queries.

In this publication, I showcase some Polars tricks and features.

How to leverage Snowflake’s support for interoperable open lakehouse technology — Iceberg — to save money.

5 key traits of successful data teams: ownership, business focus, software best practices, self-service, and company-wide strategy.

Query hierarchical data in Postgres using recursive CTEs. Navigate up/down trees, track depth, and aggregate—great for parent-child data.

Secure Snowflake SSO in browser-based VS Code using custom OAuth, CLI/API auth flow, and a dbt adapter for seamless cloud IDE integration.

Build a dedicated data-platform team to manage ingest,storage & tools, freeing business data teams to focus on creating value from insights.

Data teams succeed by shifting from tech-only focus to value delivery—combine product thinking, business goals & cross-functional roles.

A round-table discussion summary on imec’s approach to their data platform

Hot takes about my experience with cloud certifications

Discover the next evolution in data processing with DuckDB and Polars

Abandoned multipart uploads and expired delete markers: what are they, and why you must care about them thanks to bad AWS defaults.

We built a RAG chatbot with AWS Bedrock and GPT4 to answer questions about the Flemish government.

Building a data platform taught me: deleting code is vital, poor design has long-term costs, and dependency updates are never-ending.

Ensuring clean and reliable input is crucial for building robust services.

Snowflake to Trino dbt migration: watch out for type casting, SQL functions, NULL order, and window function quirks.

Data democratization is coming, but trust and governance are key. Start with pipeline observability: track runs, versions, and authors.

DuckDB on Azure: fsspec works for now, but native Azure extension is faster—especially with many small files. Full support is on the way.

This post describes two ways to speed up building your Docker images: caching build info remotely, using the link option when copying files

Exploring four methods to effectively manage and scale your data workflow dependencies with Apache Airflow.

Use Spark and Iceberg’s MERGE INTO syntax to efficiently store daily, incremental snapshots of a mutable source table.