DataStandIn

2026-10-26

Overview

DataStandIn is a synthetic data generator I founded and launched for testing apps and data pipelines. You give it a SQL schema (CREATE TABLE statements) and it generates related tables that look like production data, with every foreign key valid, even across deep chains of references. You control row counts, value semantics, distributions and skew, and you can inject controlled imperfections like nulls, typos and outliers. Runs are seeded, so they're repeatable, and the output can be exported as CSV, JSONL or Parquet, either as a zip download or directly to an S3 or GCS data lake.

Since no real customer records are involved, there is nothing to mask or re-identify.

The Approach

I built the whole thing alone, end to end:

  • An Angular frontend and a Python FastAPI backend
  • PostgreSQL on RDS (modeling, connecting, scaling), Redis on ElastiCache, and Celery workers for the generation jobs
  • Observability (logs, traces, metrics) on Grafana
  • AWS infrastructure managed by Terraform
  • CI/CD with GitHub workflows managing both the staging and production environments
  • And the operational side too, from designing the logo to producing the launch video and managing distribution

Engineering Challenges

  • Memory vs. slow IO Keeping everything in memory during generation makes running out of memory likely, while writing to files too often to save memory slows everything down because of the slow IO. The generation engine had to balance the two.

  • Referential integrity at scale Generating related tables isn't the hard part: sort them in topological order and generate the dependencies first. The challenge was carrying those dependencies forward without blowing up the memory footprint. Instead of keeping millions of primary keys in memory, or writing them to files and reading them back, the engine generates primary and foreign keys deterministically from an internal state, so every table that references them can regenerate them on demand. That included writing my own UUID deterministic generator, which is faster than giving Python's official uuid module a seed to deterministically produce a UUID.

  • Scaling without idle resources The system has to serve anything from tiny samples to massive datasets without keeping expensive resources idle. The engine estimates each job's workload (benchmarked and calibrated during development) and assigns it to one of two queues. The standard queue is consumed by an Auto Scaling Group of small instances that scales based on queue depth and estimated wait times. The large-jobs queue runs on ECS Fargate, with each task provisioned with exactly the resources that job needs instead of defaulting to the largest instance possible.

Testing

  • ~1,700 unit tests in the backend
  • ~500 unit tests in the frontend
  • ~350 integration tests
  • A couple dozen tests in the CI/CD pipeline

Website