Skip to content
Computese home
Menu
Healthcare08Data platform07Cloud

A serverless AWS data lake for healthcare centers

Healthcare centers keep and share their datasets in a data lake on AWS. We worked on it across DevOps, back end and data analysis: the serverless API and how it is secured, the catalog and search, and SQL straight over the lake.

Industry
Healthcare
Our work
DevOps, back end, data analysis
Cloud
AWS, serverless
Sign-in
Amazon Cognito, Active Directory as an option
Engagement
13 months, time and materials

Where it started.

Healthcare centers put their datasets into one data lake on AWS, so staff can find them and analysts can query them. It runs serverless: a web console and a CLI in front, an authenticated REST API, Lambda microservices behind it, and the data itself in Amazon S3. Our brief was to improve it end to end: how it is deployed and run, the API in front of it, and how its data is catalogued, searched and queried.

The shape of the work.

Drawn new for this page. No screens or data from the client's systems.

Staff use a web console and engineers a CLI. Both sign in with Amazon Cognito, optionally through Active Directory, and call a REST API on Amazon API Gateway, where a Lambda authorizer checks every call. Authorized calls reach AWS Lambda microservices, which work with the data: datasets in Amazon S3, metadata in Amazon DynamoDB, a search index in Amazon OpenSearch Service, tables in the AWS Glue Data Catalog and SQL through Amazon Athena. AWS IAM and Amazon CloudWatch cover every layer.

Fig. 1 The data lake, redrawn: who signs in, what checks each call, and where the data lives. Drawn new; no client systems shown.

How it ran.

  1. 01

    DevOps

    Deployment and operation of the serverless stack: API Gateway, Lambda, DynamoDB, OpenSearch and Amazon S3, an IAM role per service, and CloudWatch logs and events.

  2. 02

    Back end

    The REST API behind the console and the CLI: Cognito sign-in with Active Directory as an option, a Lambda authorizer on every call, and Lambda microservices behind each route.

  3. 03

    Data

    Datasets in Amazon S3, catalogued as tables by AWS Glue and queried with Amazon Athena, and their metadata in DynamoDB, indexed by Amazon OpenSearch Service for search.

A dataset’s path.

From a center’s upload to an analyst’s SQL query, across the services in the architecture above.

A dataset’s path: a center uploads it to Amazon S3 through the console or the CLI; its metadata is recorded in Amazon DynamoDB and indexed in Amazon OpenSearch Service; AWS Glue catalogues its files as tables; staff find it by searching in the console; and analysts query it with SQL in Amazon Athena. Every call passes the Lambda authorizer and is logged in Amazon CloudWatch.

Fig. 2 One dataset, from upload to a SQL query. Drawn new; no client data shown.

The stack, by layer.

Every product and practice in this work, named so an engineer can check it.

01Storage and catalog
  • Amazon S3
  • AWS Glue Data Catalog
  • AWS Glue crawlers
  • Amazon Athena
02Metadata and search
  • Amazon DynamoDB
  • Amazon OpenSearch Service (Elasticsearch)
03API and compute
  • Amazon API Gateway (REST)
  • AWS Lambda
  • Lambda authorizer
  • Microservices
04Identity and access
  • Amazon Cognito
  • Active Directory federation
  • AWS IAM
  • Least-privilege roles
05Operations
  • Amazon CloudWatch Logs
  • Amazon EventBridge (CloudWatch Events)
  • Serverless deployment
06Interfaces
  • Angular web console
  • Command-line interface

Constraints we worked within.

Health data is sensitive
Every call is authenticated and authorized before it reaches a service, and every service runs with only the permissions it needs.
No servers to run
Serverless throughout, so capacity follows use and there is no fleet of machines to patch.
Two ways in, one set of rules
A web console for staff and a CLI for engineers and automation, on the same API and the same permissions.
One place to look
API calls and service logs go to CloudWatch, so an incident starts from one trail.

What changed.

  • The lake runs serverless on AWS: no servers to patch, and capacity that follows use.
  • Every call is authenticated and authorized at the gateway before it reaches a service.
  • Datasets from every center sit in one catalog, searchable in the console and queryable with SQL.
  • One team carried DevOps, back end and data work for 13 months, under time and materials.

Questions this work answers.

How do you secure a data lake API on AWS?

Users sign in through Amazon Cognito, optionally federated with Active Directory. Every call to Amazon API Gateway passes a Lambda authorizer before it reaches a service, each service runs under a least-privilege IAM role, and activity goes to Amazon CloudWatch.

How do people find and query data in an AWS data lake?

Dataset metadata sits in Amazon DynamoDB and is indexed in Amazon OpenSearch Service, so the console can search it. AWS Glue crawls the files in Amazon S3 into tables, and Amazon Athena queries them with SQL, with no database servers to run.

The services behind this work

All work →

Start with a conversation.

Tell us what you run and what is getting in the way. You get a reply within 24 hours.

Talk to an engineer
Email
support@computese.com
Hours
Mon–Fri, 9:00–17:00 ET
Closed on statutory holidays
Office
110 Place d'Orléans Dr
Ottawa, ON K1C 2L9