I should preface this article by saying there are better solutions than this approach. I was working with the resources I had (a database wasn't available to me). That said, spinning up a database that would receive updates from the Lambda and send the data to the UI would have been significantly more cost effective than uploading the file to S3 then fetching/caching the S3 response with a hacky solution.

The time it took me to figure out this approach, plus fix the shortcomings of it (like needing to fetch the file as part of the build/deploy process) cost the client more than standing up a database for a few years.

The gameplan here:

  1. Create an endpoint for retreiving articles with mock data
  2. Create simple UI to display article data
  3. Refactor endpoint to fetch article data from S3
  4. Write script (lambda) to fetch data from Substack and upload data to S3

Article API

Basic UI

Fetching from S3

A Hacky Solution to Poll Data

I'm not in love with this solution because of a minor optimization hit this technique suffers from hitting the local NextJS API. In a perfect world (and this might be cleaned up later), we wouldn't make calls to the Next API -- we'd use the service directly. The problem we run into is that in dev mode, NextJS reloads pages on each request but does not reload the API code in the same manner.

Create a postinstall script to get data

Add a postinstall script to package.json:

  "scripts": {
    "dev": "next dev",
    "test": "next test",
    "build": "next build",
    "start": "next start",
    "postinstall": "./scripts/postinstall.sh"
  }

Create the shell script in ./scripts/postinstall.sh:

echo "Fetching API responses for deploy..."

curl -m 10 https://[your-s3-bucket].s3.us-east-2.amazonaws.com/articles.json -o ./services/data/articles.json

Don't forget to give the correct permissions to the shell script: chmod +x ./scripts/postinstall.sh

https://blog.christophervachon.com/how-to-make-git-ignore-future-changes-to-a-committed-file/

Ignoring a file in git after it's committed ->

git update-index --skip-worktree [path-to-file]

To track the file again...

git update-index --no-skip-worktree [path-to-file]

To see all files ignored...

git ls-files -v . | grep ^S

Creating a Lambda

https://betterprogramming.pub/cron-job-patterns-in-aws-126fbf54a276