# Building a Contact Tracing Platform
> Learn how to automate and build an application to trace exposures using Amazon's Cloud Development Kit (CDK)
- URL: https://blog.kochie.io/articles/02-contact-tracing
- Author: Robert Koch
- Published: 2021-08-23
- Updated: 2022-12-02
- Tags: software, maths, cdk
- Podcast: https://kochie.beehiiv.com/podcast/s/kochie_engineering/building_a_contact_tracing_platform
---
> TLDR: I made a contact tracing app that you can access
> [here on vercel](https://ct.vercel.app).
## Introduction
Infrastructure as code is the best way to manage the deployment and maintenance
of applications as well as the hardware required to operate them. Writing
configurations for applications as code allows for easy, repeatable, and
identical deployments across multiple environments. In this article I'm going to
describe how to use the AWS Cloud Development Kit (CDK) to create an application
and deploy it to an AWS environment.
AWS CDK is an open source framework for defining cloud application resources -
including compute, storage, and security resources. It's available in a number
of languages including TypeScript, Golang, .Net, and Java. The advantage to
using CDK is that you can develop the resources for your application in a
language your business already uses so developers don't need to learn a new
language to deploy and configure an application.
To show how easy it is to create an application with CDK I'g going to make
something topical. At the time of writing Melbourne is under it's 6th covid
related lockdown, one of the reasons given for the lockdown was that contact
tracing personnel were inundated with exposure sites, which needed to be
notified, which then needed their staff and customers to be isolated and tested,
while their whereabouts traced. This type of problem can be automated, so I
figured, why don't I try to build a contact tracing app?
GitHub repository: [kochie/contact-tracing](https://github.com/kochie/contact-tracing)
If you want to see the entire project or a particular snippet of the code I've
written for this demo you can find it on github.
## Architecture
### Backend
To start with, here is a basic architecture diagram of what the backend services
look like.
The backend architecture for this application is quite simple, the system is
based off of a serverless architecture so its cost is proportional to the amount
of use it receives. The API which will handle the contact tracing is managed
through Amazon AppSync - a managed graphql service. GraphQL is a language and
specification for data query and manipulation used most commonly with APIs.
Appsync is a great service because it gives front-end developers the ability to
query multiple databases, microservices, and APIs with a single endpoint. Since
there's only one endpoint it's really easy to update the API in realtime by just
adding more data sources. To create an API all you need to do is define a schema
and where to fetch the data from and you're mostly done. There are a few more
steps if you want to define authentication and more complicated data sources but
for the most part it's plug and play.
The check-in data for this application will be stored in Amazon DynamoDB. There
are a few reasons why I chose Dynamo, the first is that it's a fully managed
serverless product, so I don't have to worry about managing a database on a
server. The second is that it's fast, I mean _really_ fast. A standard query
takes about 5ms to complete, that's pretty fast (as we go on I'll talk about how
I'm caching a lot of the queries in the Dynamo Accelerator but this didn't speed
up my queries for the most part and I'll explain why later). The final reason is
that the type of data being used can be easily stored in dynamo as a key value
pair. Each check-in looks something like this example.
```json
{
"location_id": "1",
"user_id": "2",
"checkin_datetime": "2021-06-01T00:00:00+10:00"
}
```
Super nice and simple, and it's really easy to search for a check-in based on the
`time`, `user_id`, and `location_id`. DynamoDB queries need a _partition key_ to
search upon (a primary key analogue) and can be refined using a _sort key_, in
the application I'm using the `user_id` and `location_id` as partition keys
while the date will be the sort key. DynamoDB tables support a single partition
key, to use multiple partition keys in DynamoDB you need to use a
GlobalSecondaryIndex.
Authentication is vital for real world services. Having the ability to verify
users who will access sensitive data (check-in data is Personal Identifying
Information - PII) is absolutely needed. AppSync supports multiple simultaneous
authentication methods - including API keys, IAM user signatures, Amazon
Cognito, and external OpenID providers. For this demo I'm going to use Amazon
Cognito which is a managed authentication service that can store thousands of
user credentials securely in the cloud.
There are two components for the API in the `lib` directory, the _functions_
directory contains all the API code and the _graphql_ directory contains the
schemas.
#### What is CDK?
The AWS CDK is a framework that can be used to define and organise resources on
the AWS cloud. Any project in CDK is setup using "stacks" which are individual
deployments that resources are grouped by. In actual fact CDK stacks are
synthesized into Amazon CloudFormation templates and deployed using
CloudFormation.
CDK is available in a bunch of different languages but for this demo I'm going
to use TypeScript to define my CDK stack.
> I'm using CDK v2 is this demo because it's pretty close to being released
> soon. CDK v2 has a bunch of improvements over the first version which you can
> read about on the
> [aws blog](https://docs.aws.amazon.com/cdk/latest/guide/work-with-cdk-v2.html)
> but for this demo it won't matter too much which version you use.
The first thing to do is install the CDK CLI, now if you're using another
language like Go or Java the installation will be a bit different but there are
instructions for each language on the CDK website.
```bash
npm install -g aws-cdk@next
```
After this you can create a new CDK project in an empty directory based off the
app template as shown below.
```bash
cdk init app --language typescript
```
After this you should have a blank CDK project, woo! To verify you should have a
directory layout similar to the output below.
```none
ct_app
├── README.md
├── bin
│ └── myapp.ts
├── cdk.json
├── jest.config.js
├── lib
│ └── myapp-stack.ts
├── package-lock.json
├── package.json
├── test
│ └── myapp.test.ts
└── tsconfig.json
```
#### Defining the Schema
GraphQL is first and foremost a query language. It's used to describe the data
in an API, ask for specific data, and return predictable results. All queries
to a graphql API are sent to one http endpoint which supports POST requests. The
data sent in the post request defines the query/mutation and what information to
return.
For the graphql engine and clients to understand what queries are valid they
require a schema. Th graphql schema is a great tool for defining the scope of
the API, it has lots of advantages over traditional http endpoint schemas. The
first is that it's self documenting - since the API is defined by the schema
it's always exactly what queries the graphql engine is expecting. Secondly, any
graphql inspector like [GraphiQL](https://github.com/graphql/graphiql) can
automatically read the schema and lint your requests as you write them (once you
set up authentication with the server). Lastly since graphql is a strictly typed
language all the requests defined in the schema have types that can be used,
expanded, and checked against - meaning it's very hard to send an incorrect
query.
For this project there are two query files, the first is the `scalars.graphql`
file. This is used for linting purposes during development, the graphql
specification defines a few specific data types that all engines must support
(things like `String` and `Float`) but any engine implementation can have extra
built-in types, in fact you can specify your own in the schema. AppSync has
several built in types that we're going to use so, to help tools like vscode and
intellij understand these new types we can define another schema file that is in
the same directory as our main schema that won't get used by the CDK stack but
will be used by any tools that scan the directory.
For a complete list of AppSync built-in scalar types you can check out the
[documentation on the site](https://docs.aws.amazon.com/appsync/latest/devguide/scalars.html).
```graphql lineNumbers filename="scalars.graphql"
scalar AWSTimestamp
scalar AWSURL
scalar AWSDate
scalar AWSDateTime
directive @aws_api_key on FIELD_DEFINITION | INPUT_OBJECT | OBJECT | ENUM
directive @aws_cognito_user_pools on FIELD_DEFINITION | INPUT_OBJECT | OBJECT | ENUM_VALUE
```
```graphql lineNumbers filename="schema.graphql"
type CheckIn {
location_id: String
user_id: String
checkin_datetime: AWSDateTime
}
type Output {
items: [CheckIn]
nextToken: String
}
type LocationFlat {
location_id: String
latitude: String
longitude: String
}
type Node {
user_id: String
}
type Link {
source: String
target: String
time: String
location_id: String
latitude: String
longitude: String
}
type Flat {
nodes: [Node]
links: [Link]
locations: [LocationFlat]
}
type Query {
get_user_location_history(
user_id: String!
from: AWSDateTime
until: AWSDateTime
nextToken: String
limit: Int
): Output
get_location_attendees(
location_id: String!
from: AWSDateTime
until: AWSDateTime
nextToken: String
limit: Int
): Output
trace_exposure_flat(
user_id: String!
from: AWSDateTime
until: AWSDateTime
): Flat
}
type Mutation {
check_in(location_id: String!, user_id: String!): CheckIn
}
type Schema {
query: Query
mutation: Mutation
}
```
#### Writing the Functions
The backbone of an AppSync API are the data sources. The data sources map to a
query or mutation object and tell AppSync where to get or set the data for the
operation. At the time of writing there are 6 different data sources: DynamoDB,
Elasticsearch, Lambda, RDS, HTTP, and None (this is just an empty placeholder).
This API uses two different data sources. For simple key retrievals the API has a
DynamoDB resolver, this data source can query the table directly using a mapping
template and return the results. There are a few advantages to doing this, the
main one is that you don't need to write an extra lambda function in another
language to query the database.
The other resolver used is the lambda provider. AppSync can use Amazon Lambda
functions as a data source provider, using lambda to run functions allows your
API to run any arbitrary code needed to get a result for the API. Also since
lambda functions can now run docker images there isn't really any function that
your API can't do.
To set up a go project inside the CDK the only step you need to complete is
creating a go manifest file using `go mod`.
```bash
go mod init github.com/kochie/contact-tracing/lib/functions
```
The manifest file is used by the Go toolchain to track and download dependencies
for a program, it's also used in the build steps to validate the structure and
version of Go the program is compatible with. After making the manifest file you
can create a Go file in any sub-directory which is a great way to organise the
code you're going to write. Below is an example of how I've laid out the structure
of the contact tracing API functions.
```none
functions
├── common
│ └── utilities.go
├── contact_trace_flat
│ └── main.go
├── go.mod
├── go.sum
└── trace_exposure_over_time
└── main.go
```
##### A Quick Note on CDK V2 and Golang Lambda Functions
At the time of writing this article one of the major flaws in CDK v2 is that
there is no support for using languages other than JavaScript to write lambda
functions with and have them automatically bundled and deployed with the stack.
For the first version of CDK [Rafal Wilinski](https://github.com/RafalWilinski)
wrote a
[custom construct](https://github.com/RafalWilinski/aws-lambda-golang-cdk) that
will compile Golang lambda functions and bundle them with the stack however,
this construct doesn't work with v2 of the CDK. So to fix this I spent the
better part of a weekend a few weeks ago patching the code Rafal wrote and
making a version that is compatible with v2 of the CDK which you can find
[on GitHub](https://github.com/kochie/aws-lambda-golang-cdk-v2).
##### Common
A lot of the code to access the Dynamo table across functions is identical, so
I've refactored this code into its own separate file. The `utilities.go` file
has two public methods: `GetLocationVisitors` and `GetUserLocationHistory`. In
the snippet below I've only added the code required for `GetLocationVisitors` as
an example of how to build a function that can access Dynamo.
```go lineNumbers filename="utilities.go"
// NOTE: This is just an snippet of the entire file, to see the entire source goto
// https://github.com/kochie/contact-tracing/blob/master/lib/functions/common/utilities.go
package common
import (
"context"
"log"
"os"
"time"
"github.com/aws/aws-dax-go/dax"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue"
"github.com/aws/aws-sdk-go-v2/service/dynamodb"
"github.com/aws/aws-sdk-go-v2/service/dynamodb/types"
"github.com/aws/aws-xray-sdk-go/xray"
)
var client *dax.Dax
var tableName = os.Getenv("TABLE_NAME")
type CheckIn struct {
UserID string `dynamodbav:"user_id"`
LocationID string `dynamodbav:"location_id"`
CheckinDatetime time.Time `dynamodbav:"checkin_datetime"`
Latitude string `dynamodbav:"latitude"`
Longitude string `dynamodbav:"longitude"`
}
func init() {
daxEndpoint := os.Getenv("DAX_ENDPOINT")
cfg, err := config.LoadDefaultConfig(context.TODO())
if err != nil {
log.Fatal(err)
}
err = xray.Configure(xray.Config{
ServiceVersion: "1.2.3",
})
if err != nil {
log.Fatal(err)
}
xray.AppendMiddlewares(&cfg.APIOptions)
log.Println("Xray middleware applied")
daxCfg := dax.DefaultConfig()
daxCfg.HostPorts = []string{daxEndpoint}
daxCfg.Region = "ap-southeast-2"
daxCfg.RequestTimeout = 5 * time.Minute
client, err = dax.New(daxCfg)
if err != nil {
log.Fatal(err)
}
log.Println("Dynamo Accelerator configured")
log.Println("Ready to work.")
}
func GetLocationVisitors(locationId, from, until string, ctx context.Context) ([]*CheckIn, error) {
var expression string
expressionAttributeValues := map[string]types.AttributeValue{
":location_id": &types.AttributeValueMemberS{Value: locationId},
}
if from != "" && until != "" {
expression = `location_id = :location_id AND checkin_datetime BETWEEN :from AND :until`
expressionAttributeValues[":from"] = &types.AttributeValueMemberS{Value: from}
expressionAttributeValues[":until"] = &types.AttributeValueMemberS{Value: until}
} else if until != "" {
expression = `location_id = :location_id AND checkin_datetime <= :until`
expressionAttributeValues[":until"] = &types.AttributeValueMemberS{Value: until}
} else if from != "" {
expression = `location_id = :location_id AND checkin_datetime >= :from`
expressionAttributeValues[":from"] = &types.AttributeValueMemberS{Value: from}
} else {
expression = `location_id = :location_id`
}
paginator := dynamodb.NewQueryPaginator(client, &dynamodb.QueryInput{
TableName: aws.String(tableName),
KeyConditionExpression: aws.String(expression),
ExpressionAttributeValues: expressionAttributeValues,
})
locations := make([]*CheckIn, 0)
for paginator.HasMorePages() {
resp, err := paginator.NextPage(ctx)
if err != nil {
return nil, err
}
for _, item := range resp.Items {
checkin := CheckIn{}
err := attributevalue.UnmarshalMap(item, &checkin)
if err != nil {
return nil, err
}
locations = append(locations, &checkin)
}
}
return locations, nil
}
```
There are a few components from the above code I'd like the highlight before
moving on. The first is that the `CheckIn` struct is using `dynamodbav` tags to
define the relationship between struct properties and DynamoDB properties. Tags
are using ing Golang to provide additional context to properties in structs,
they're most commonly used for (un)marshalling JSON and XML data with structs
without the need for custom code to handle the process.
The second point is the code around `dax`. In this app I'm using DynamoDB
Accelerator (DAX) to cache the results of queries and speed up the API response
time. I'll go into more details about setting up and configuring DAX later but
for the time being wherever you see DAX code in the API you should know that you
can also use DynamoDB clients as well.
The final point to mention revolves around the pagination of results received
from DynamoDB. Any query that is made to DynamoDB will return a maximum of
[1MB per transaction](https://www.alexdebrie.com/posts/dynamodb-limits/#page-size-limit-for-query--scan),
to collect all the results from a query you need to use the paginator object.
The `NewQueryPaginator` will handle the work of collecting multiple pages of
data and is a drop in replacement for the standard `Query` method.
##### Trace Exposure Function
With the Dynamo table access separated into a different file the contact tracing
code can import the calls when needed using the `import` statement. The
structure of the code is fairly straightforward so I won't go into too much
detail; the flowchart below describes how the algorithm works.
```go lineNumbers filename="contact_trace_flat.go"
package main
import (
"context"
"time"
"github.com/aws/aws-lambda-go/lambda"
"github.com/kochie/contact-tracing/lib/functions/common"
)
type User struct {
UserId string `json:"user_id"`
From string `json:"-"`
Contacts []*Location `json:"contacts"`
}
type Location struct {
Time time.Time `json:"time"`
LocationId string `json:"location_id"`
Visitors []*User `json:"visitors"`
}
func HandleRequest(ctx context.Context, event interface{}) (*User, error) {
eventData := event.(map[string]interface{})
arguments := eventData["arguments"].(map[string]interface{})
userId := arguments["user_id"].(string)
from := ""
if _, ok := arguments["from"].(string); ok {
from = arguments["from"].(string)
}
until := ""
if _, ok := arguments["until"].(string); ok {
until = arguments["until"].(string)
}
seenUsers := make(map[string]bool)
seenLocations := make(map[string]bool)
rootUser := User{UserId: userId, From: from, Contacts: make([]*Location, 0)}
stack := []*User{&rootUser}
for len(stack) > 0 {
user := stack[0]
from = user.From
stack = stack[1:]
if _, ok := seenUsers[user.UserId]; ok {
continue
}
seenUsers[user.UserId] = true
checkins, err := common.GetUserLocationHistory(user.UserId, from, until, ctx)
if err != nil {
return nil, err
}
for _, checkin := range checkins {
locationID := checkin.LocationID
if _, ok := seenLocations[locationID]; ok {
continue
}
seenLocations[locationID] = true
f := checkin.CheckinDatetime.Add(-time.Hour).Format(time.RFC3339)
u := checkin.CheckinDatetime.Add(time.Hour).Format(time.RFC3339)
visitors, err := common.GetLocationVisitors(locationID, f, u, ctx)
if err != nil {
return nil, err
}
users := make([]*User, 0)
for _, visitor := range visitors {
if _, ok := seenUsers[visitor.UserID]; ok {
continue
}
u := User{visitor.UserID, checkin.CheckinDatetime.Format(time.RFC3339), make([]*Location, 0)}
stack = append(stack, &u)
users = append(users, &u)
}
user.Contacts = append(user.Contacts, &Location{
checkin.CheckinDatetime,
locationID,
users,
})
}
}
return &rootUser, nil
}
func main() {
lambda.Start(HandleRequest)
}
```
At this point I'd like to mention that the Lambda resolver that I've setup is
known more specifically as a **Direct Lambda Resolver**. The distinction is
because when using a Lambda resolver you can optionally include a mapping
template that can transform the API input before the Lambda sees it. To
understand more about how Direct Lambda Resolvers work you can
[read this blog about them](https://aws.amazon.com/blogs/mobile/appsync-direct-lambda/).
Because there is no mapping template for the function you need to take extra
care in parsing and validating the input to the function. Because Lambda doesn't
have any information around how the input event is structured it is defined
using an empty interface - which is the Go analogue of an **any** object.
Because of this you have to test and type assert any object that should be
inside the input.
#### Building the Stack
Now that the function code has been defined we can move onto building the stack
definition. Inside of the `lib` directory there should be a file similar to
`contact-tracing-stack.ts` this file contains the stack and resources that will
be added to the app. Inside the file there should be a single class which will
be the stack definition.
```typescript lineNumbers
export class ContactTracingStack extends Stack {
constructor(scope: Construct, id: string, props?: StackProps) {
super(scope, id, props)
// Define resources here
}
}
```
From here all the resources will be defined within the scope of the class
constructor[^1].
##### VPC
Virtual Private Clouds (VPCs) are an abstraction that lets you logically isolate
a virtual network. It's best practice to assign lambdas to a VPC and as we'll
see in [Using Dynamo DAX](#using-dynamo-dax) it's required to set up a DAX
server. VPCs are split into subnets which can be connected to the internet
(private vs public) and given unique names and subnet masks.
```typescript lineNumbers
const vpc = new ec2.Vpc(this, 'contacts-vpc', {
cidr: Vpc.DEFAULT_CIDR_RANGE,
subnetConfiguration: [
{
cidrMask: 24,
name: 'default',
subnetType: ec2.SubnetType.PUBLIC,
},
{
cidrMask: 24,
name: 'contact-tracing',
subnetType: ec2.SubnetType.PRIVATE,
},
],
})
```
##### DynamoDB Table
There are two resources that need to be created for the Dynamo database. The
first is the table itself, and the second is a global secondary index. The
Dynamo table will allow querying using a single partition key and optional sort
key for this table; the location ID will be the partition key and the check in
date will be the sort key. This will mean that queries the table based by
location will be trivial, but it won't be possible to run a query based on the
user. The way to solve this using Dynamo is to create another index. There are
two types of secondary indexes in Dynamo, Global Secondary Indexes allow for
another partition key while Local Secondary Indexes provide another sort key.
Since we're going to be searching for a user and sorting the results based on
the check in date we only need a Global Secondary Index.
```typescript lineNumbers
const contact_table = new dynamo.Table(this, 'contact-tracing-table', {
tableName: 'checkins',
encryption: dynamo.TableEncryption.AWS_MANAGED,
partitionKey: {
name: 'location_id',
type: dynamo.AttributeType.STRING,
},
sortKey: {
name: 'checkin_datetime',
type: dynamo.AttributeType.STRING,
},
billingMode: BillingMode.PAY_PER_REQUEST,
})
contact_table.addGlobalSecondaryIndex({
indexName: 'index_by_user',
partitionKey: {
name: 'user_id',
type: dynamo.AttributeType.STRING,
},
sortKey: {
name: 'checkin_datetime',
type: dynamo.AttributeType.STRING,
},
})
```
##### Using Dynamo Dax
Over the course of this project I found that some of the larger requests were
taking up to 16 seconds to run. In terms of making a responsive website that's
incredibly slow. As I investigated the issue I found that over 99% of the
execution time was being used writing for DynamoDB queries. Now like I said at
the top of this article Dynamo is really fast, queries usually take less than
5ms to run, but when you're creating a contact tracer that has to cross
reference thousands, potentially millions of people and locations those 5ms add
up. You can see the latency in, each segment represents a query operation; as
you can see the number of queries rapidly increases.
To speed up these request I decided to use DynamoDB Accelerator (DAX) to cache
the results of any Dynamo queries. DAX is an in-memory cache that can deliver up
to a 10 times performance improvement for DynamoDB queries. It does this by
creating a cluster in your VPC that can communicate with any application
connected in that VPC. DAX is a drop in replacement for DynamoDB - meaning that
if you write code to work with DynamoDB all you need to do is replace the
DynamoDB client with the DAX client and it will work right away, there's no need
to change any of the application logic to support DAX API calls.
My initial thought was that most of the queries that would be executed would be
repeats of previous queries - this assumption however, was wrong.
``
So as I'll mention earlier in the article I'm using Golang to write the lambda
functions, to use the AWS API in Golang you should use the official SDKs. The
latest version of the SDK `aws-sdk-v2` has been in active public development for
a few years and was classified generally available in January of this year.
Now the new SDK has a bunch of useful features that make it much easier to use
which I've [mentioned in the highlights](#common) (you can find a full list of
improvements
[on the SDK website](https://aws.github.io/aws-sdk-go-v2/docs/migrating/)), but
for the time being I wrote most of the function code before adding support for
DAX.
Now the new SDK does not have out of the box support for DAX, due to the
architecture of how DAX works it would require a major rewrite[^2]. The original
SDK doesn't have support either, but there is
[another SDK](https://github.com/aws/aws-dax-go) built for DAX that has support
for v1 of the Golang SDK.
Now here in lies the problem, because v2 of the Golang SDK is a complete
rewrite, this library is incompatible with the new SDK version. Now you'd think
that there would be support for the new SDK considering both of these repos are
official AWS repositories but alas.
So I took it upon myself to try to add support for the new version which you can
see [on GitHub](https://github.com/kochie/aws-dax-go). So great sounds like
problem solved right? Well no because the DAX protocol is pretty confusing, to
get the speed they're promising the DAX team isn't using HTTP to talk to the DAX
cluster, they're rolling their own custom protocol using TCP. So all I've done
is change the input and output structs of the DAX SDK to support the structs
from v2 of the AWS Golang SDK. Is this an elegant solution? No. Is this a good
solution? eh. But it is a solution.
``
So anyway if you want to use DAX with this application (or any other
`aws-sdk-v2` code) you need to add this snippet to the bottom of your `go.mod`
file to let the toolkit know that you want to use a fork of the original
repository.
```go.mod
replace (
github.com/aws/aws-dax-go => github.com/kochie/aws-dax-go master
)
```
After all this work I tested the API calls and success! Kind of; well the first
request is still slow but any identical subsequent requests are much faster
(less than 1ms). The only problem with this is that most of the requests are not
going to be identical so the speedup isn't really worth the effort. I've
documented what I've done for the sake of completeness but I don't recommend
using DAX for a problem like this.
```typescript lineNumbers
const daxRole = new iam.Role(this, 'dax-role', {
assumedBy: new iam.ServicePrincipal('dax.amazonaws.com'),
inlinePolicies: {
dynamo_access: new iam.PolicyDocument({
statements: [
new iam.PolicyStatement({
actions: ['dynamodb:*'],
effect: iam.Effect.ALLOW,
resources: [
contact_table.tableArn,
`${contact_table.tableArn}/index/index_by_user`,
],
}),
],
}),
},
})
const securityGroup = new ec2.SecurityGroup(this, 'security-group-vpc', {
vpc,
})
securityGroup.addIngressRule(ec2.Peer.anyIpv4(), ec2.Port.tcp(8111), 'DAX')
const daxSubnetGroup = new dax.CfnSubnetGroup(this, 'dax-subnet-group', {
subnetGroupName: 'contact-tracing',
subnetIds: vpc.privateSubnets.map((subnet) => subnet.subnetId),
})
const cache = new dax.CfnCluster(this, 'dax-contacts', {
clusterName: 'contacts',
description: 'DAX cache for contact tracing',
iamRoleArn: daxRole.roleArn,
nodeType: 'dax.t3.small',
replicationFactor: 1,
securityGroupIds: [securityGroup.securityGroupId],
sseSpecification: {
sseEnabled: true,
},
subnetGroupName: daxSubnetGroup.subnetGroupName,
})
cache.addDependsOn(daxSubnetGroup)
```
##### Cognito User Pool and Client
Next to set up Cognito; there are two resources that need to be created. The
first is the Cognito User Pool, this resource will hold all the information about
users who access the API - mainly their email and password. The second component
is the Cognito App Client, this resource is used as an endpoint for web services
to talk to Cognito, it specifies what authentication methods are valid.
```typescript lineNumbers
const user_pool = new cognito.UserPool(this, 'ct-user-pool', {
userPoolName: 'contact-tracing-pool',
signInCaseSensitive: false,
selfSignUpEnabled: true,
signInAliases: {
email: true,
},
autoVerify: {
email: true,
},
standardAttributes: {
email: { required: true },
},
passwordPolicy: {
requireSymbols: false,
requireDigits: false,
requireLowercase: false,
requireUppercase: false,
},
})
const user_pool_client = new cognito.UserPoolClient(
this,
'ct-user-pool-client',
{
userPool: user_pool,
userPoolClientName: 'contact-tracing-client',
}
)
```
##### AppSync API Schema
Now with the dependencies out of the way we can now create the AppSync API. The
first part is to define a new AppSync API resource and the schema the resource
will use.
```typescript lineNumbers
const api = new appsync.CfnGraphQLApi(this, 'ct-api', {
authenticationType: 'AMAZON_COGNITO_USER_POOLS',
name: 'contact-tracing-api',
userPoolConfig: {
userPoolId: user_pool.userPoolId,
awsRegion: this.region,
defaultAction: 'ALLOW',
},
logConfig: {
// This arn is premade if you already have an appsync api.
// If it's not there you can make it using CloudWatch.
cloudWatchLogsRoleArn:
'arn:aws:iam::457234467265:role/service-role/appsync-graphqlapi-logs-ap-southeast-2',
excludeVerboseContent: false,
fieldLogLevel: 'ALL',
},
})
const api_schema = new appsync.CfnGraphQLSchema(this, 'ct-api-schema', {
apiId: api.attrApiId,
definition: readFileSync(
join(__dirname, 'graphql/schema.graphql')
).toString(),
})
```
This will create an AppSync API that will use the attached schema we created
before. But at the moment none of the data values in the schema are connected to
a resolver. To do this we'll need to create some data sources and resolvers.
##### AppSync Data Sources and Resolvers
The Data Sources and Resolvers are a critical piece of the API infrastructure,
they connect AppSync to all the storage and compute resources behind your API so
making sure they're correctly defined is crucial.
The first Data Source we're going to make is for a direct DynamoDB connection.
To make this we're going to give AppSync full access to our DynamoDB to read and
write to our table. We're then going to point AppSync to the DynamoDB table we
created before.
```typescript lineNumbers
const table_role = new iam.Role(this, 'ItemsDynamoDBRole', {
assumedBy: new iam.ServicePrincipal('appsync.amazonaws.com'),
})
table_role.addManagedPolicy(
iam.ManagedPolicy.fromAwsManagedPolicyName('AmazonDynamoDBFullAccess')
)
const dataSource = new appsync.CfnDataSource(this, 'ItemsDataSource', {
apiId: api.attrApiId,
name: 'ItemsDynamoDataSource',
type: 'AMAZON_DYNAMODB',
dynamoDbConfig: {
tableName: contact_table.tableName,
awsRegion: this.region,
},
serviceRoleArn: table_role.roleArn,
})
```
Now we need to make a resolver for our Data Source. This resolver will be linked
to the `get_user_location_history` and `get_location_attendees` API queries
which will be used by the API to get results for specifics users and locations
on the pages
and [https://ct.vercel.app](https://ct.vercel.app/). Direct DynamoDB resolvers
are useful when you don't need to use much business logic to return API data.
The resolver only needs a mapping template that can convert the input into a
DynamoDB transaction. The result is then converted into a JSON compatible string
and returned to AppSync.
```typescript lineNumbers
const getUserLocationHistoryResolver = new appsync.CfnResolver(
this,
'getUserLocationHistoryQueryResolver',
{
apiId: api.attrApiId,
typeName: 'Query',
fieldName: 'get_user_location_history',
dataSourceName: dataSource.name,
requestMappingTemplate: `{
"version": "2018-05-29",
"operation": "Query",
"query": {
"expression": "user_id = :userId \
#if( $ctx.args.from && $ctx.args.until )
AND checkin_datetime BETWEEN :from AND :until",
#elseif( $ctx.args.from )
AND checkin_datetime >= :from",
#elseif( $ctx.args.until )
AND checkin_datetime <= :until",
#else
",
#end
"expressionValues": {
":userId": $util.dynamodb.toDynamoDBJson($ctx.args.user_id),
#if( $ctx.args.from )
":from": $util.dynamodb.toDynamoDBJson($ctx.args.from),
#end
#if( $ctx.args.until )
":until": $util.dynamodb.toDynamoDBJson($ctx.args.until),
#end
}
},
"index": "index_by_user",
"limit": $util.defaultIfNull($ctx.args.limit, 20),
"nextToken": $util.toJson($util.defaultIfNullOrBlank($ctx.args.nextToken, null))
}`,
responseMappingTemplate: `{
"items": $util.toJson($ctx.result.items),
"nextToken": $util.toJson($util.defaultIfNullOrBlank($ctx.result.nextToken, null))
}`,
}
)
getUserLocationHistoryResolver.addDependsOn(api_schema)
const getLocationAttendeesResolver = new appsync.CfnResolver(
this,
'getLocationAttendeesQueryResolver',
{
apiId: api.attrApiId,
typeName: 'Query',
fieldName: 'get_location_attendees',
dataSourceName: dataSource.name,
requestMappingTemplate: `{
"version": "2018-05-29",
"operation": "Query",
"query": {
"expression": "location_id = :locationId \
#if( $ctx.args.from && $ctx.args.until )
AND checkin_datetime BETWEEN :from AND :until",
#elseif( $ctx.args.from )
AND checkin_datetime >= :from",
#elseif( $ctx.args.until )
AND checkin_datetime <= :until",
#else
",
#end
"expressionValues": {
":locationId": $util.dynamodb.toDynamoDBJson($ctx.args.location_id),
#if( $ctx.args.from )
":from": $util.dynamodb.toDynamoDBJson($ctx.args.from),
#end
#if( $ctx.args.until )
":until": $util.dynamodb.toDynamoDBJson($ctx.args.until),
#end
}
},
"limit": $util.defaultIfNull($ctx.args.limit, 20),
"nextToken": $util.toJson($util.defaultIfNullOrBlank($ctx.args.nextToken, null))
}`,
responseMappingTemplate: `{
"items": $util.toJson($ctx.result.items),
"nextToken": $util.toJson($util.defaultIfNullOrBlank($ctx.result.nextToken, null))
}`,
}
)
getLocationAttendeesResolver.addDependsOn(api_schema)
```
The mapping template is written in a language called
[Apache Velocity](https://docs.aws.amazon.com/appsync/latest/devguide/resolver-mapping-template-reference-programming-guide.html),
you can find out more in the AWS documentatoin including a list of helper
functions for working with DynamoDB.
We're also going to use this data source to put new check ins into the table.
The process is exactly the same as a the above query but instead we'll use the
`PutItem` operation.
```typescript lineNumbers
const checkinResolver = new appsync.CfnResolver(
this,
'checkinMutationResolver',
{
apiId: api.attrApiId,
typeName: 'Mutation',
fieldName: 'check_in',
dataSourceName: dataSource.name,
requestMappingTemplate: `{
"version": "2018-05-29",
"operation": "PutItem",
"key": {
"location_id": $util.dynamodb.toDynamoDBJson($ctx.args.location_id),
"checkin_datetime": $util.dynamodb.toDynamoDBJson($util.time.nowISO8601())
},
"attributeValues": {
"user_id": $util.dynamodb.toDynamoDBJson($ctx.args.user_id)
}
}`,
responseMappingTemplate: `$util.toJson($ctx.result)`,
}
)
checkinResolver.addDependsOn(api_schema)
```
That's it. Now the `get_user_location_history`, `get_location_attendees`, and
`check_in` are connected to a data source. The final call we'll look at is
`trace_exposure_flat`. First we're going to create a Golang function using the
plugin that I mentioned in
[A Quick Note on CDK V2 and Golang Lambda Functions](#a-quick-note-on-cdk-v2-and-golang-lambda-functions)
and we're going to set some environment variable from resources we've already
created and create some security policies to grant access to resources.
```typescript lineNumbers
const contactTracingFlatFunc = new GolangFunction(
this,
'contact-tracing-flat-func',
{
entry: 'lib/functions/contact_trace_flat/main.go',
vpc: vpc, // created in the DynamoDB DAX section
vpcSubnets: {
subnetType: ec2.SubnetType.PRIVATE,
},
securityGroups: [securityGroup], // also created in the DAX section
environment: {
TABLE_NAME: contact_table.tableName,
DAX_ENDPOINT: cache.attrClusterDiscoveryEndpoint,
},
timeout: Duration.minutes(15),
memorySize: 1024,
initialPolicy: [
new iam.PolicyStatement({
actions: ['dynamodb:Query'],
effect: iam.Effect.ALLOW,
resources: [
contact_table.tableArn,
`${contact_table.tableArn}/index/index_by_user`,
],
}),
new iam.PolicyStatement({
actions: ['dax:*'],
effect: iam.Effect.ALLOW,
resources: ['*'],
}),
],
}
)
```
Next we're going to create a service role that AppSync can use to invoke the
Lambda function, as well as create a Lambda Data Source.
```typescript lineNumbers
const lambda_role = new iam.Role(this, 'LambdaRole', {
assumedBy: new iam.ServicePrincipal('appsync.amazonaws.com'),
inlinePolicies: {
lambda_access: new iam.PolicyDocument({
statements: [
new iam.PolicyStatement({
actions: ['lambda:InvokeFunction'],
effect: iam.Effect.ALLOW,
resources: [contactTracingFlatFunc.functionArn],
}),
],
}),
},
})
const traceExposureFlatLambdaDataSource = new appsync.CfnDataSource(
this,
'traceExposureFlatLambdaDataSource',
{
apiId: api.attrApiId,
name: 'TraceExposureFlatDataSource',
type: 'AWS_LAMBDA',
lambdaConfig: {
lambdaFunctionArn: contactTracingFlatFunc.functionArn,
},
serviceRoleArn: lambda_role.roleArn,
}
)
traceExposureFlatLambdaDataSource.addDependsOn(api_schema)
```
After we've created a data source we're going to connect it to a resolver like
we did before.
```typescript lineNumbers
const traceExposureFlatQueryResolver = new appsync.CfnResolver(
this,
'traceExposureFlatQueryResolver',
{
apiId: api.attrApiId,
typeName: 'Query',
fieldName: 'trace_exposure_flat',
dataSourceName: traceExposureFlatLambdaDataSource.name,
}
)
traceExposureFlatQueryResolver.addDependsOn(traceExposureFlatLambdaDataSource)
```
There you have it, now the lambda function we wrote before is connected to
AppSync and can be used for API calls. All that's left to do is deploy the
stack.
#### Deploying with CDK
Deploying with CDK is easy to do, there is a single command that will generate
the CloudFormation template and deploy it to the configured environment.
```bash
cdk deploy
```
One thing to mention here - A useful feature of CDK is the ability to use
CloudFormation Outputs to print attributes that are useful. For example the API
URL, Userpool Id, and AppClient Id are all values needed by the frontend which
can be hard coded into the environment.
```typescript lineNumbers
new CfnOutput(this, 'api-url', {
value: api.attrGraphQlUrl,
})
new CfnOutput(this, 'userpool-id', {
value: user_pool.userPoolId,
})
new CfnOutput(this, 'userpool-appclient-id', {
value: user_pool_client.userPoolClientId,
})
```
```bash
✅ ContactTracingStack
Outputs:
ContactTracingStack.apiurl = https://epmecp3h7naxbhpwebqxvdu2sq.appsync-api.ap-southeast-2.amazonaws.com/graphql
ContactTracingStack.userpoolappclientid = 6o5j6tu3veq99u6dqh187ut2dj
ContactTracingStack.userpoolid = ap-southeast-2_R3TWUkG9u
```
### Frontend
The frontend of the application is a React site using
[next.js](https://nextjs.org). Next is a great framework for developing websites
using React and can be hosted on any static web service. For this application
I'm using [vercel.com](https://vercel.com) as it's a free service that works
really well with next since they're both made by the same team. I do want to
point out that [Amazon Amplify](https://aws.amazon.com/amplify/) has great
support for NextJS as well but I'm using vercel as my preferred choice.
The first step is to create a next app, this can be done with the
`create-next-app` tool which is pretty similar to `create-react-app` if you're
farmiliar with that.
```bash
npx create-next-app frontend/
```
After running the setup command you should have a directory something like this.
```none
frontend
├── README.md
├── package-lock.json
├── package.json
├── pages
│ ├── _app.js
│ ├── api
│ │ └── hello.js
│ └── index.js
├── public
│ ├── favicon.ico
│ └── vercel.svg
└── styles
├── Home.module.css
└── globals.css
```
We now have a boilerplate NextJS project, running `npm run dev` will start up
the development server and you should see a starting page. We're not going to be
using this page or the API setup by next so you can delete the `pages/`
directory. We're going to make two files in the `src/pages` directory -
`_app.tsx` and `index.tsx`. They should look something like this.
```typescript lineNumbers filename="src/pages/_app.tsx"
import type { AppProps } from 'next/app'
function App({ Component, pageProps }: AppProps) {
return
}
export default App
```
```typescript lineNumbers filename="src/pages/index.tsx"
import React from 'react'
const Index = () => {
return
Hello, World!
}
export default index
```
After creating these pages if you run `npm run dev` next will warn you that you
don't have typescript or the right types installed. To fix this run the
following install script:
```bash
npm i --save-dev typescript @types/react
```
Now when you start the dev environment you should get a blank `Hello, World!`
page. Now we can set up the components of the app.
#### Setting up Authentication
When we set up the backend we created a Cognito userpool and client and
connected our userpool to appsync. We're now going to use our client to get an
authentication token that will work with AppSync, the flow should work something
like this:
1. User loads the website.
2. The website checks to see if the user is logged in.
3. If yes go to step 5.
4. Direct user to login or signup.
5. When user logged in ask Cognito for an authorization token to send with API
requests.
6. Cognito returns a token that can be used.
These steps are pretty straight forward but can be very tricky to do correctly.
Fortunately there is a prebuilt library for Cognito authentication that we can
use in the AWS Amplify package.
```bash
npm install --save-dev aws-amplify
```
Once the package is installed we can configure the `Auth` object to connect to
our Cognito client, for this we need the ID of the user pool and the client
which we had exported from our stack before. Let's save them for the time being
and add them to the code as environment variables.
```typescript{7-11} lineNumbers filename="src/pages/_app.tsx"
import React, { useEffect } from "react";
import { Auth } from "aws-amplify";
import type { AppProps } from 'next/app'
import "tailwindcss/tailwind.css";
Auth.configure({
userPoolId: process.env.NEXT_PUBLIC_USERPOOL_ID,
userPoolWebClientId: process.env.NEXT_PUBLIC_CLIENT_ID,
region: process.env.NEXT_PUBLIC_REGION,
});
function App({ Component, pageProps }: AppProps) {
return (
);
}
export default App;
```
##### Connecting Authenticator to React
To actually get an authentication token you need to build a login flow for the
app. Now there are two ways this can be done; you can either use the prebuilt
Amplify login screen or you can roll your own login. I've decided to build my
own login flow which I made for another project. If you want to see the relevant
pages they're available
[in the repo](https://github.com/kochie/contact-tracing/tree/master/frontend/src/pages).
```
src/pages/login.tsx
src/pages/signup.tsx
src/pages/confirm_signup.tsx
```
To make sure that the page shown to the user is always authenticated I've made a
simple hook to return the authentication state to the page. Be aware this is a
simple example of what should really be built into a React provider but it's
fine for example purposes.
```typescript lineNumbers filename="src/lib/authHook.ts"
import { Auth } from 'aws-amplify'
import { Dispatch, SetStateAction, useEffect, useState } from 'react'
export function useAuth(): [boolean, Dispatch>] {
const [isAuthenticated, setIsAuthenticated] = useState(false)
useEffect(() => {
;(async () => {
try {
const session = await Auth.currentSession()
console.log(session)
if (!!session) setIsAuthenticated(true)
} catch (err) {
console.error(err)
}
})()
}, [])
return [isAuthenticated, setIsAuthenticated]
}
```
#### Setting up Apollo
Now that our authentication is set up we can contact the AppSync API and run
queries/mutations.
Apollo is a GraphQL implementation that can be used to generate and create an
API similar to AppSync. For our use however, we're going to be using the open
source client library that Apollo provides to execute queries against our
endpoint. First thing we need to do is install Apollo.
```bash
npm install --save-dev @apollo/client
```
After Apollo is installed we need to configure it to work with a NextJS React
app and the authentication method we've built. There is an example in the
[NextJS repo](https://github.com/vercel/next.js/tree/canary/examples/with-apollo)
that explains how to connect Apollo. To connect with the Amplify auth library
all we need to do is fetch the token which can be seen in the `authLink`
context.
```typescript{22-23,30} lineNumbers filename="frontend/src/lib/apolloClient.ts"
import { useMemo } from 'react'
import {
ApolloClient,
createHttpLink,
from,
InMemoryCache,
} from '@apollo/client'
import { Auth } from 'aws-amplify'
import { setContext } from '@apollo/client/link/context'
import { onError } from '@apollo/client/link/error'
import { Output } from '../queries/common'
let apolloClient
const httpLink = createHttpLink({
uri: process.env.NEXT_PUBLIC_API_URL,
})
const authLink = setContext(async (request, { headers }) => {
// get the authentication token from local storage if it exists
let token
try {
const session = await Auth.currentSession()
token = session.getAccessToken().getJwtToken()
} catch {
console.log('NO TOKEN!!!!')
}
return {
headers: {
...headers,
Authorization: token ? token : '',
},
}
})
function createApolloClient() {
return new ApolloClient({
ssrMode: typeof window === 'undefined',
link: from([
onError((err) => {
console.log(err)
}),
authLink,
httpLink,
]),
cache,
name: 'react-web-client',
version: '1.3',
queryDeduplication: false,
defaultOptions: {
watchQuery: {
fetchPolicy: 'cache-first',
},
},
})
}
export function initializeApollo(initialState = null) {
const _apolloClient = apolloClient ?? createApolloClient()
// If your page has Next.js data fetching methods that use Apollo Client, the initial state
// gets hydrated here
if (initialState) {
// Get existing cache, loaded during client side data fetching
const existingCache = _apolloClient.extract()
// Restore the cache using the data passed from getStaticProps/getServerSideProps
// combined with the existing cached data
_apolloClient.cache.restore({ ...existingCache, ...initialState })
}
// For SSG and SSR always create a new Apollo Client
if (typeof window === 'undefined') return _apolloClient
// Create the Apollo Client once in the client
if (!apolloClient) apolloClient = _apolloClient
return _apolloClient
}
export function useApollo(initialState) {
return useMemo(() => initializeApollo(initialState), [initialState])
}
```
The `apolloClient` file creates a React hook that can be used by next as a
provider - a type of React component that _provides_ a context. A Context
provides a way to pass data through the component tree without having to pass
props down manually at every level [^3]. Now we can wrap our app in the
`` making our API available to any React component in the tree.
```typescript{15,18-20} lineNumbers
import React, { useEffect } from "react";
import type { AppProps } from 'next/app'
import { Auth } from "aws-amplify";
import { ApolloProvider } from "@apollo/client";
import { useApollo } from "../lib/apolloClient";
import "tailwindcss/tailwind.css";
Auth.configure({
userPoolId: process.env.NEXT_PUBLIC_USERPOOL_ID,
userPoolWebClientId: process.env.NEXT_PUBLIC_CLIENT_ID,
region: process.env.NEXT_PUBLIC_REGION,
});
function App({ Component, pageProps }: AppProps) {
const apolloClient = useApollo(pageProps.initialApolloState);
return (
);
}
export default App;
```
Now when we build the data components in our app we can automatically get data
from our API.
#### Getting Data From the API to our Page
Now that we have our Authentication and Data layer built let's test them by
displaying some content on our `index.tsx` page. We're going to make a simple
table display to show the data for a single user.
To start with we'll define the query we want to execute on the page, we'll call
it `get_user_location_history` and save it in a file can we import from later.
```typescript lineNumbers filename="src/lib/queries/get_user_location_history.ts"
import { gql } from '@apollo/client'
export const GET_USER_LOCATION_HISTORY = gql`
query GetUserLocationHistory(
$user_id: String!
$nextToken: String
$from: AWSDateTime
$until: AWSDateTime
) {
get_user_location_history(
user_id: $user_id
nextToken: $nextToken
from: $from
until: $until
) {
items {
user_id
checkin_datetime
location_id
}
nextToken
}
}
`
```
Then we can build a simple React page that can display the `UserTable` component
if the user is authenticated.
```typescript lineNumbers
import React from 'react'
import { GET_USER_LOCATION_HISTORY } from '../queries/get_user_location_history'
import { TopBar } from '../components/TopBar'
import { useAuth } from '../lib/authHook'
import NoAuth from '../components/NoAuth'
const Index = () => {
const [isAuth] = useAuth()
return (
{isAuth ? : }
)
}
```
The `UserTable` component is where all the business logic is found, we're going
to use the `useLazyQuery` hook to get the data from our API when a user presses
a button on the page.
```typescript lineNumbers
const UserTable = () => {
const [getUserLocationHistory, { loading, data, error, fetchMore }] =
useLazyQuery<{ get_user_location_history: Output }>(
GET_USER_LOCATION_HISTORY
)
const getMoreData = async () => {
await fetchMore({
variables: {
nextToken: data?.get_user_location_history?.nextToken || '',
},
})
}
const getData = (
userId: string,
fromDate: string,
fromTime: string,
untilDate: string,
untilTime: string
) => {
const variables = {
user_id: userId,
}
if (fromDate && fromDate.length > 0) {
const from = new Date(
`${fromDate}${fromTime ? 'T' : ''}${fromTime}`
).toJSON()
variables['from'] = from
}
if (untilDate && untilDate.length > 0) {
const until = new Date(
`${untilDate}${untilTime ? 'T' : ''}${untilTime}`
).toJSON()
variables['until'] = until
}
getUserLocationHistory({ variables })
}
return (
// honestly I did a bad job at refactoring and I
// didn't want to put 500 more lines of JSX in this
// blog so you can look at the entire file at
// https://github.com/kochie/contact-tracing/blob/master/frontend/src/pages/index.tsx
)
}
```
Now when you reload the page you should see something like this.
#### Building D3 components
So now that we have the data available to React we can now do something with it.
Since the user is on a site to view contact tracing data they probably want to
see the data requested. There are lots of different ways to view this data but
I'm going to explain how I made two of the layouts on the site in this article
otherwise this post would never get finished. We're going to make a Radial tree
and a Force-Directed tree.
There are lot's of ways to display data on a website, the earliest and most
mundane system is to simply use tables like we did before. But using tables to
display data can often obscure the more intricate connections and details in the
data. Having custom elements that can display the data in different ways to
emphasise and highlight specific facts and connections is a much more user
focused approach.
To help with this we're going to be using a library called D3 - a JavaScript
library for manipulating documents based on data. D3 allows you to bind
arbitrary data to the Document Object Model (DOM), and then apply data-driven
transformations to a document. This approach is incredibly powerful, allowing
developers to build anything within the scope of the DOM APIs.
Another advantage of D3 is the large set of utility modules that come included;
while building the different components we're going to use a fair few of them.
##### Radial Tree
The first component we're going to make is a radial tree, this is a type of tree
that has all the nodes sorted and aligned around a central root node. It's
really useful for understanding the depth of a tree and how sparse or dense it
is.
To start off we're going to make an empty NextJS page which will render an svg
component.
```typescript lineNumbers filename="frontend/src/pages/radial-tree.tsx"
import React, { useState, useRef } from 'react'
import { linkRadial, select, stratify, tree } from 'd3'
import { cloneDeep } from '@apollo/client/utilities'
const RadialTree = () => {
const svgRef = useRef(null)
const tooltip = useRef(null)
const [width, setWidth] = useState(800)
const [height, setHeight] = useState(400)
const resize = () => {
setWidth(window.innerWidth)
// the topbar is 64px
setHeight(window.innerHeight - 64)
}
useEffect(() => {
window.addEventListener('resize', resize)
resize()
return () => {
window.removeEventListener('resize', resize)
}
}, [])
return (
)
}
export default RadialTree
```
This should now render a blank page when you navigate to `/radial-tree`. Now
it's time to add our connection to the AppSync API using Apollo. The Apollo
client library for React manages both local and remote data with GraphQL.
To get the data from the server onto our page we're going to use the
`useLazyQuery` React hook. This method will take the query string and variables
and send these to the server.
First we'll define the query to execute. We can place this in a separate file
and export it into the component, this will be useful later as multiple
components need to use the same query. Plus it's also tidier.
```typescript lineNumbers filename="frontend/src/queries/trace_exposure_flat.ts"
import { gql } from '@apollo/client'
export const TRACE_EXPOSURE_FLAT = gql`
query TraceExposureFlat(
$user_id: String!
$from: AWSDateTime
$until: AWSDateTime
) {
trace_exposure_flat(user_id: $user_id, from: $from, until: $until) {
links {
source
target
location_id
time
}
nodes {
user_id
}
locations {
latitude
longitude
location_id
}
}
}
`
```
I'm using a custom component I made called `SearchBox` which just creates a form
and validates the inputs using `Formik`, you can read more about it in the repo,
for the time being assume it's a form and the `onSubmit` function runs when the
form is submitted.
```typescript{2,6-7,10,32-40,44-47} lineNumbers filename="frontend/src/pages/radial-tree.tsx"
import React, { useState, useRef } from 'react'
import { useLazyQuery } from '@apollo/client'
import { linkRadial, select, stratify, tree } from "d3";
import { cloneDeep } from "@apollo/client/utilities";
import { TRACE_EXPOSURE_FLAT } from '../queries/trace_exposure_flat'
import SearchBox from "../components/SearchBox";
const RadialTree = () => {
const [runQuery, { data, loading }] = useLazyQuery(TRACE_EXPOSURE_FLAT)
const svgRef = useRef(null)
const tooltip = useRef(null)
const [width, setWidth] = useState(800)
const [height, setHeight] = useState(400)
const resize = () => {
setWidth(window.innerWidth)
// the topbar is 64px
setHeight(window.innerHeight - 64)
}
useEffect(() => {
window.addEventListener('resize', resize)
resize()
return () => {
window.removeEventListener('resize', resize)
}
}, [])
const getData = (userId: string, from: string, until: string) => {
runQuery({
variables: {
user_id: userId,
from,
until,
},
});
};
return (
getData(userId, from, until)}
/>
)
}
export default RadialTree
```
So now when the user selects a date and user id the API will be called through
the React layer and return data to the view. Now all that needs to be done is
display it.
There are a lot of moving parts in modern React applications, let's just pause
for a moment and understand what we're doing and what the tools and frameworks
we're using are designed for.
First, React is a library for building user interfaces. In the traditional MVC
design pattern, React is the controller component, it's only job is to accept
inputs and convert commands for the model or view[^4].
Apollo is a framework for data and state management, it's designed to fetch
data, keep it in sync with the server and relay changes, it's the Model in the
MVC pattern.
The final part is the view, we have a list of data that needs to be displayed to
the user after they make a query, as eluded before to display the data we're
going to be using D3 which is a JavaScript library for creating interactive
visualizations in the browser. It's essentially a way of generating shapes and
charts using svg components and html canvases. But the way it does this is
amazing, there are so many tools and helper functions to build scales, shapes,
charts, lines, and colours that you can pretty much create anything you can
think of.
The advantage to D3 is the ability to create anything, the disadvantage is you
have the ability to create anything, there are no prebuilt components - you need
to know what you want to make.
To begin with, we're going to create another React hook that is dependent on the
size of the window, and the data. The hook is going to create an svg selection
using D3 that we can use to manipulate and edit the contents of the svg page.
```typescript lineNumbers
useEffect(() => {
if (!data) return
if (!svgRef.current) return
const svg = select(svgRef.current)
.attr('width', width)
.attr('height', height)
.append('g')
.attr('transform', `translate(${width}/2, ${height}/2)`)
return () => {
svg.selectAll('*').remove()
}
}, [width, height, data])
```
So now that we've got a selectable svg element we can put our data into it and
produce our visualisation. To do this we need to build a data structure that d3
understands and can parse to populate the svg nodes that it will create. There
are a few components that we need to create to do this. The first thing that we
need to do is clone the data received from apollo, this is because apollo data
is not extensible[^5]. We're also going to use the
[stratify](https://github.com/d3/d3-hierarchy#stratify) constructor to create an
operator to parse our API data into a d3 hierarchy, which is just a predefined
data structure that d3 tree functions can understand. Finally we need to create
a method that will create a tree from our data, the `tree` builder is exactly
what we need.
Combining all of these constructor elements together and adding them to our hook
gives the following code.
```typescript{11-25} lineNumbers
useEffect(() => {
if (!data) return
if (!svgRef.current) return
const svg = select(svgRef.current)
.attr('width', width)
.attr('height', height)
.append('g')
.attr('transform', `translate(${width}/2, ${height}/2)`)
const graph: Graph = cloneDeep(data.trace_exposure_flat)
const strat = stratify()
.id((d) => d.target)
.parentId((d) => d.source)
const rootUser = graph.nodes[0]
const treeBuilder = tree()
.size([2 * Math.PI, Math.min(height, width) / 2 - 10])
.separation((a, b) => (a.parent == b.parent ? 1 : 2) / a.depth);
const s = strat([
{ source: "", target: rootUser.user_id, location_id: "", time: "" },
...graph.links,
]);
const root = treeBuilder(s);
return () => {
svg.selectAll('*').remove()
}
}, [width, height, data])
```
Great, so now we have the methods and structures in place to add data to the
page, we're almost there. The last thing that needs to be done is to build the
tree visually using svg elements. In a tree there are two separate sections -
vertices, and edges.
Using the d3 `select` tool we can create a group of svg `path` elements to
represent the tree edges, the co-ordinates in the canvas are calculated using
the `tree` builder so we only need to call the `links` method to pull the data
for each element.
The vertices can be created in a similar fashion by building `circle` svg
elements and calling the `descendants` method to retrieve all the vertex data.
The entire svg generation can be seen in the below snippet, there is some code
that explains how the tree is wrapped into a circle as well.
```typescript{27-41,43-53} lineNumbers
useEffect(() => {
if (!data) return
if (!svgRef.current) return
const svg = select(svgRef.current)
.attr('width', width)
.attr('height', height)
.append('g')
.attr('transform', `translate(${width}/2, ${height}/2)`)
const graph: Graph = cloneDeep(data.trace_exposure_flat)
const strat = stratify()
.id((d) => d.target)
.parentId((d) => d.source)
const rootUser = graph.nodes[0]
const treeBuilder = tree()
.size([2 * Math.PI, Math.min(height, width) / 2 - 10])
.separation((a, b) => (a.parent == b.parent ? 1 : 2) / a.depth);
const s = strat([
{ source: "", target: rootUser.user_id, location_id: "", time: "" },
...graph.links,
]);
const root = treeBuilder(s);
svg
.append("g")
.attr("fill", "none")
.attr("stroke", "#555")
.attr("stroke-opacity", 0.4)
.attr("stroke-width", 1.5)
.selectAll("path")
.data(root.links())
.join("path")
.attr(
"d",
linkRadial()
.angle((d) => d.x)
.radius((d) => d.y)
)
svg
.append("g")
.selectAll("circle")
.data(root.descendants())
.join("circle")
.attr(
"transform",
(d) => `rotate(${(d.x * 180) / Math.PI - 90}) translate(${d.y},0)`
)
.attr("fill", (d) => (d.children ? "#555" : "#999"))
.attr("r", 2.5)
return () => {
svg.selectAll('*').remove()
}
}, [width, height, data])
```
After adding all these components if you refresh the page you should see
something like the tree below.
If everything is working you can now generate a tree that shows the contacts and
locations exposed to a person. But it's pretty hard to gleam any usable
information from this tree at the moment. We can fix this by adding some
contextual data using event listeners and add vertex and edge information to the
tree when a particular element is hovered over.
```typescript{4,28-29,46-72,85-96} lineNumbers
useEffect(() => {
if (!data) return
if (!svgRef.current) return
if (!tooltip.current) return;
const svg = select(svgRef.current)
.attr('width', width)
.attr('height', height)
.append('g')
.attr('transform', `translate(${width}/2, ${height}/2)`)
const graph: Graph = cloneDeep(data.trace_exposure_flat)
const strat = stratify()
.id((d) => d.target)
.parentId((d) => d.source)
const rootUser = graph.nodes[0]
const treeBuilder = tree()
.size([2 * Math.PI, Math.min(height, width) / 2 - 10])
.separation((a, b) => (a.parent == b.parent ? 1 : 2) / a.depth);
const s = strat([
{ source: "", target: rootUser.user_id, location_id: "", time: "" },
...graph.links,
]);
const root = treeBuilder(s);
const div = select(tooltip.current)
.style("opacity", 0);
svg
.append("g")
.attr("fill", "none")
.attr("stroke", "#555")
.attr("stroke-opacity", 0.4)
.attr("stroke-width", 1.5)
.selectAll("path")
.data(root.links())
.join("path")
.attr(
"d",
linkRadial()
.angle((d) => d.x)
.radius((d) => d.y)
)
.on("mouseover", function (d, i) {
select(this).transition().duration(100).attr("stroke-width", 3);
div.transition().duration(100).style("opacity", 1);
div
.html(() => {
const x = graph.links.find(
(link) =>
link.source === i.source.id && link.target === i.target.id
);
return (
"
"
);
})
.style("left", d.pageX + 10 + "px")
.style("top", d.pageY - 15 + "px");
})
.on("mouseout", function (d, i) {
select(this).transition().duration(200).attr("stroke-width", 1.5);
div.transition().duration(200).style("opacity", 0);
});
svg
.append("g")
.selectAll("circle")
.data(root.descendants())
.join("circle")
.attr(
"transform",
(d) => `rotate(${(d.x * 180) / Math.PI - 90}) translate(${d.y},0)`
)
.attr("fill", (d) => (d.children ? "#555" : "#999"))
.attr("r", 2.5)
.on("mouseover", function (d, i) {
select(this).transition().duration(100).attr("r", 5);
div.transition().duration(100).style("opacity", 1);
div
.html("User ID: " + i.id)
.style("left", d.pageX + 10 + "px")
.style("top", d.pageY - 15 + "px");
})
.on("mouseout", function (d, i) {
select(this).transition().duration(200).attr("r", 2.5);
div.transition().duration(200).style("opacity", 0);
});
return () => {
svg.selectAll('*').remove()
}
}, [width, height, data])
```
Now when you hover over a part of the tree you should see a handy tooltip.
##### Force-Directed Tree
Another visual we can create is the force-directed tree - a visualization that
uses a numerical integrator for simulating forces. This visualisation is really
good at showing clusters and relative sizes of tree groups. The setup and
implementation is identical to the radial tree we created before, the only
difference is the D3 render hook. In it we're going to create a simulation
object and link it to our tree structure.
```typescript lineNumbers
useEffect(() => {
if (!data) return
if (!svgRef.current) return
const graph: Graph = cloneDeep(data.trace_exposure_flat)
const strat = stratify()
.id((d) => d.target)
.parentId((d) => d.source)
const rootUser = graph.nodes[0]
const s = strat([
{ source: '', target: rootUser.user_id, location_id: '', time: '' },
...graph.links,
])
const svg = select(svgRef.current)
.attr('width', width)
.attr('height', height)
.attr('viewBox', [-width / 2, -height / 2, width, height].join(' '))
.append('g')
const dragSimulation = (simulation) => {
function dragstarted(event, d) {
if (!event.active) simulation.alphaTarget(0.3).restart()
d.fx = d.x
d.fy = d.y
}
function dragged(event, d) {
d.fx = event.x
d.fy = event.y
}
function dragended(event, d) {
if (!event.active) simulation.alphaTarget(0)
d.fx = null
d.fy = null
}
return drag()
.on('start', dragstarted)
.on('drag', dragged)
.on('end', dragended)
}
const root = hierarchy(s)
const links = root.links()
const nodes = root.descendants()
const simulation = forceSimulation(nodes)
.force(
'link',
forceLink(links)
.id((d) => d.id)
.distance(0)
.strength(1)
)
.force('charge', forceManyBody().strength(-60))
.force('x', forceX())
.force('y', forceY())
const locationMax = max(graph.links.map((l) => parseInt(l.location_id)))
const link = svg
.append('g')
.attr('stroke', '#999')
.attr('stroke-opacity', 0.7)
.attr('stroke-width', '2')
.selectAll('line')
.data(links)
.join('line')
.attr('stroke', (d) => {
const line = graph.links.find(
(l) => l.source === d.source.data.id && l.target === d.target.data.id
)
return interpolateSinebow(parseInt(line.location_id) / locationMax)
})
const node = svg
.append('g')
.attr('fill', '#fff')
.attr('stroke', '#000')
.attr('stroke-width', 1.5)
.selectAll('circle')
.data(nodes)
.join('circle')
.attr('fill', (d) => (d.children ? null : '#000'))
.attr('stroke', (d) => (d.children ? null : '#fff'))
.attr('r', 4.5)
.call(dragSimulation(simulation))
node.append('title').text((d) => 'UserID: ' + d.data.id)
simulation.on('tick', () => {
link
.attr('x1', (d) => d.source.x)
.attr('y1', (d) => d.source.y)
.attr('x2', (d) => d.target.x)
.attr('y2', (d) => d.target.y)
node.attr('cx', (d) => d.x).attr('cy', (d) => d.y)
})
return () => {
svg.selectAll('*').remove()
}
}, [data, height, width])
```
If you create a new file and copy the code from `radial-tree.tsx` and replace
the hook with the code seen above you should see something like this.
#### Mapping using MapBox
Visual components are great but sometimes you need to display data relative to
the real world. Maps are the best way of displaying geographical data. In this
component I'll show you how to overlay geographical data on a map.
We're going to make another page called `map.tsx` which will be our base. To
make the map we're going to use a library called
[mapbox](https://www.mapbox.com/). Mapbox is a great service that provides high
resolution maps with their SDK.
To set up the map we're going to import the JavaScript SDK and the map styles.
We're also going to create a div which will be the map container.
```typescript{3,9,34-35,46-50} lineNumbers filename="src/pages/map.tsx"
import React, { useCallback, useEffect, useRef, useState } from 'react'
import { TopBar } from '../components/TopBar'
import mapbox, { Popup } from 'mapbox-gl'
import { useLazyQuery } from '@apollo/client'
import { TRACE_EXPOSURE_FLAT } from '../queries/trace_exposure_flat'
import { cloneDeep } from '@apollo/client/utilities'
import SearchBox from '../components/SearchBox'
import 'mapbox-gl/dist/mapbox-gl.css'
interface Link {
location_id: string
time: string
source: string
target: string
latitude: string
longitude: string
}
interface Graph {
links: Link[]
locations: {
location_id: string
latitude: string
longitude: string
}[]
nodes: {
user_id: string
}[]
}
const Map = () => {
const [runQuery, { data, loading }] = useLazyQuery(TRACE_EXPOSURE_FLAT)
const [map, setMap] = useState()
const [markers, setMarkers] = useState([])
return (
<>
runQuery({ variables: { user_id: userId, from, until } })
}
/>
)
}
export default Map
```
After creating the page we need to initialise the map inside the container.
We're going to use the `useCallback` hook to make a custom reference for our map
that can be updated when the page loads the SDK.
```typescript lineNumbers
const ref = useCallback((node: HTMLDivElement) => {
if (node === null) return
const m = new mapbox.Map({
container: node, // container id
style: 'mapbox://styles/mapbox/streets-v11', // style URL
center: [144.96332, -37.814], // starting position [lng, lat]
zoom: 8, // starting zoom
accessToken: process.env.NEXT_PUBLIC_MAPBOX_TOKEN,
})
m.addControl(
new mapbox.GeolocateControl({
positionOptions: {
enableHighAccuracy: true,
},
trackUserLocation: true,
})
)
setMap(m)
return () => {
m.remove()
}
}, [])
```
When the map is initialised we can being displaying data using a react hook
similar to how we displayed data in the D# components.
```typescript lineNumbers
useEffect(() => {
if (!data?.trace_exposure_flat) return
if (!map) return
const graph: Graph = cloneDeep(data.trace_exposure_flat)
const m = graph.locations.map((location) => {
const exposeTime =
graph.links.find((link) => link.location_id === location.location_id)
?.time || ''
// This only happens if the location was exposed but no new people were checkin in.
// For this example we'll leave blank
if (!exposeTime) return
return new mapbox.Marker()
.setPopup(
new Popup({ offset: 25 }).setHTML(`
Location Id: ${location.location_id}
Time Exposed: ${
exposeTime ? new Date(exposeTime).toLocaleString() : 'Not Exposed'
}
`)
)
.setLngLat([
parseFloat(location.longitude),
parseFloat(location.latitude),
])
.addTo(map)
})
setMarkers(m)
return () => {
markers.forEach((m) => (m ? m.remove() : null))
}
}, [data, map])
```
And there you have it! Now the map can load data and then display it on a page.
If you refresh the page you should see something like this map below.
## Final Thoughts
If you've made it this far, well done! This article turned out a lot longer than
I originally planned. I thought about splitting it into multiple articles but I
think the flow of a single document is easier to understand and consume.
This project was actually a lot more complicated then I first assumed it would
be, the fact that both Dynamo Dax and XRay didn't have support for Golang
`aws-sdk-v2` really slowed down my development. The D3 charts also took a long
time to make, but once they started to work it was okay to iterate and improve.
If you've found this article useful or can't understand a word I've said you can
yell at me on Twitter, my handle is [@kochie](https://twitter.com/kochie).
[^1]: So there are multiple imports for CDK that you'll need depending on which
resources you define. The main CDK import `aws-cdk-lib` contains all the
modules for individual services, to import specific services the best way is
to alias the import like so `aws_cognito as cognito`.
[^2]: Maybe another project?
[^3]: The definition provided by the
[React docs](https://reactjs.org/docs/context.html).
[^4]: Okay so not quite. In the documentation it says "React isn’t an MVC
framework. React is a library for building composable user interfaces." But
in context to how React is used in this application it's essentially the
controller.
[^5]: For more information about this little quirk of JavaScript take a look at
the
[docs on MDN](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Errors/Cant_define_property_object_not_extensible),
actually since these components aren't modifying the data this clone isn't
really needed, but it's still good practice to treat API data as immutable
objects.