Next.js and Redis: a better way to load data from external APIs

In modern web applications, efficiently managing large amounts of data from external APIs is crucial. Processing all the data at once can lead to performance bottlenecks and degrade the user experience. In this article, we explore how to leverage Redis in a Next.js project to create a seamless content feed, handling large data loads efficiently and ensuring a smooth user experience.

Redis is often used as a cache or message broker, but it can also be leveraged to manage and process data from external APIs efficiently. We will demonstrate this using a project where we created a content feed by fetching posts from an external API that returns between 50 and 100 posts at a time, parsing and sorting them, and then storing them in a Redis queue for efficient processing.

Prerequisites

This guide assumes you have a basic understanding of Next.js and know how to set up a project. If you are unfamiliar with Next.js, please refer to the official Next.js documentation for getting started. Additionally, we will be using Docker to run Redis, but you can have Redis installed locally or use any other preferred method.

Setting Up the Project

1. Running Redis with Docker

To efficiently handle and process data, we’ll use Redis, and we’ll run it using Docker. Docker allows us to create lightweight, isolated containers that can run different services seamlessly. Here’s the docker-compose.yaml file to set up the environment:

version: '3'

services:
  next-app:
    container_name: content-feed-next
    build:
      context: .
      dockerfile: Dockerfile
    # Set environment variables directly in the docker-compose file
    environment:
      CONTENT_API_URL: ${CONTENT_API_URL}
      NUMBER_OF_POSTS_IN_BATCH: ${NUMBER_OF_POSTS_IN_BATCH}
    # Load environment variables from the .env file
    env_file:
      - .env
    volumes:
      - ./src:/app/src
      - ./public:/app/public
    restart: always
    ports:
      - 3000:3000
    networks:
      - my_network

  redis:
    container_name: content-feed-redis
    image: redis:alpine
    restart: always
    ports:
      - 6379:6379
    networks:
      - my_network

networks:
  my_network:

In this setup, we have defined two services: next-app and redis. The next-app service builds and runs our Next.js application. It sets up environment variables directly within the Docker Compose file and also loads additional variables from a .env file. The application’s source code and public assets are mapped using volumes, ensuring that changes to the codebase are reflected immediately. The service is configured to restart automatically and expose port 3000, making it accessible via http://localhost:3000.

The redis service runs the Redis container using the lightweight Alpine image. It maps port 6379, the default port for Redis, and is set to restart automatically if it fails. Both services are connected via a custom Docker network named my_network, which allows them to communicate using their service names as hostnames.

2. Configuring Redis Connection

Next, we need to set up a connection to Redis within our Next.js application. We’ll use the ioredis library, a robust and full-featured Redis client for Node.js. Here’s how we configure the Redis client:

// src/utilities/posts-queue.ts
import Redis from 'ioredis';

export const redis = new Redis({
  host: 'redis',
});

In this configuration, we import ioredis and create a new Redis client instance. The hostname is set to redis, which corresponds to the service name defined in our Docker Compose file. This setup allows our Next.js application to communicate with the Redis instance running in the Docker container.

3. Defining the Data Types

To ensure type safety and structure the data we receive from the external API, we define the necessary data types. This helps us manage the data more effectively throughout our application. Here are the types we defined:

// src/utilities/data-fetcher.ts
export type APIPostType = {
  id: string;
  imageUri: string;
  title: string;
  body: string;
  author: {
    first: string;
    last: string;
  };
  publishDate: string;
};

export type PostType = {
  id: string;
  imageUri: string;
  title: string;
  body: string;
  author: string;
  publishDate: Date;
};

The APIPostType represents the structure of a post as received from the external API. In contrast, the PostType represents the structure of a post as it will be used within our application. Here, the author field is a single string combining the first and last names, and the publishDate is converted to a JavaScript Date object for easier manipulation and formatting.

By setting up these foundational elements, we are now prepared to efficiently fetch, process, and display large amounts of data from external APIs in our Next.js application.

Fetching and Processing Data

1. Fetching Data from the API

To retrieve data from the external API, we use Axios, a popular HTTP client for making requests. The goal is to fetch the data, parse it into our defined structure, and then enqueue it into Redis for efficient processing.

Here’s how we fetch and store the posts:

// src/utilities/data-fetcher.ts
import axios from 'axios';
import { enqueuePosts } from './posts-queue';

export const fetchAndStorePosts = async () => {
  const { data } = await axios.get(process.env.CONTENT_API_URL);
  const posts = parseAndSortPosts(data);
  await enqueuePosts(posts);
};

In this function, we make a GET request to the external API using the URL specified in the environment variable CONTENT_API_URL. The response data, which contains the posts, is then passed to the parseAndSortPosts function for processing. After parsing and sorting the posts, we enqueue them into Redis using the enqueuePosts function. This ensures that the data is readily available for efficient retrieval and processing.

2. Parsing and Sorting Data

Once we have the raw data from the API, we need to convert it into a structured format that our application can work with. We also sort the data based on the publish date to prioritize recent posts.

Here’s the code to parse and sort the posts:

// src/utilities/data-fetcher.ts
export const parsePostData = (data: APIPostType): PostType => ({
  id: data.id,
  imageUri: data.imageUri,
  title: data.title,
  body: data.body,
  author: `${data.author.first} ${data.author.last}`,
  publishDate: new Date(data.publishDate),
});

export const parseAndSortPosts = (data: { posts: APIPostType[] }): PostType[] => {
  return data.posts
    .map(parsePostData)
    .sort((a, b) => b.publishDate.getTime() - a.publishDate.getTime());
};

The parsePostData function takes a single post object from the API response and transforms it into our PostType structure. This includes combining the first and last name fields into a single author string and converting the publishDate string into a JavaScript Date object.

The parseAndSortPosts function processes an array of posts by mapping each item through the parsePostData function. It then sorts the parsed posts in descending order based on their publish date, ensuring that the most recent posts appear first.

By fetching, parsing, and sorting the data efficiently, we prepare the posts for smooth and prioritized loading in our application, leveraging Redis to handle large amounts of data without performance bottlenecks.

Storing Data in Redis

1. Enqueuing Posts

Once the posts are fetched and processed, they need to be stored efficiently for later retrieval. Redis, with its fast in-memory data store capabilities, provides an excellent solution for this. We store the processed posts in a Redis list. Here’s how we enqueue the posts:

// src/utilities/posts-queue.ts
import { PostType } from './data-fetcher';

const QUEUE_NAME = 'posts';

export async function enqueuePosts(posts: PostType[]): Promise<void> {
  for (const post of posts) {
    await redis.rpush(QUEUE_NAME, JSON.stringify(post));
  }
}

In this function, we iterate over each post and push it onto the Redis list named posts. The rpush command appends each post to the end of the list. The posts are serialized to JSON strings before being stored, ensuring that all data types are preserved correctly.

2. Dequeuing Posts

To process the posts without overwhelming our system, we dequeue them in manageable batches. This approach ensures that our application handles data efficiently and remains responsive. Here’s how we dequeue the posts:

// src/utilities/posts-queue.ts
export async function dequeuePosts(count: number): Promise<PostType[]> {
  const pipeline = redis.multi();
  pipeline.lrange(QUEUE_NAME, 0, count - 1);
  pipeline.ltrim(QUEUE_NAME, count, -1);
  const results = await pipeline.exec();
  if (!results) {
    return [];
  }
  const postsData = results[0][1] as string[];
  return postsData.map((data: string) => JSON.parse(data, dateReviver));
}

In this function, we use a Redis pipeline to execute multiple commands atomically. The lrange command retrieves a specified number of posts from the list, while the ltrim command removes those posts from the list. The retrieved posts are then deserialized from JSON strings back into JavaScript objects. This ensures that only the specified number of posts are processed at a time, preventing performance bottlenecks.

3. Correctly parsing Dates

Dates are a common data type in APIs and need to be handled correctly to ensure accurate processing and display. To manage this, we use a custom dateReviver function that converts date strings into JavaScript Date objects:

// src/utilities/format-date.ts
const dateJSONFormat = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/;

export function dateReviver(key: string, value: any) {
  if (typeof value === 'string' && dateJSONFormat.test(value)) {
    return new Date(value);
  }
  return value;
}

The dateReviver function checks if a value matches the ISO date format. If it does, the string is converted into a Date object. This ensures that date strings are correctly parsed and can be used reliably in date operations and formatting.

By enqueuing, dequeuing, and correctly parsing dates, we ensure that our application processes data efficiently and accurately, leveraging Redis for fast in-memory storage and retrieval. This approach optimizes the handling of large datasets, maintaining performance and responsiveness in our Next.js application.

Integrating with Next.js

1. Fetching Posts on the Server

To fetch a batch of posts from our Redis queue, we create a server action in Next.js. This server action interacts with our data-fetching utility to retrieve the posts:

// src/actions/get-posts.ts
'use server';
import { PostType, getBatchOfPosts } from '@/utilities/data-fetcher';

export const getPosts = async (): Promise<PostType[]> => {
  try {
    return await getBatchOfPosts();
  } catch (e) {
    console.error('Error fetching posts', e);
    throw new Error(`An error happened: ${e}`);
  }
};

In this function, getBatchOfPosts is called to retrieve a batch of posts from Redis. If an error occurs during this process, it is caught and logged, and an error message is thrown. This ensures that any issues with fetching posts are handled gracefully.

2. Displaying Posts on the Home Page

Next, we integrate the fetched posts into our Next.js application by passing them to the PostFeed component. This is done in the Home component, which is responsible for rendering the homepage:

// src/app/page.tsx
import { getPosts } from '@/actions/get-posts';
import Navbar from '@/components/navbar';
import PostFeed from '@/components/post-feed';

export default async function Home() {
  const initialPosts = await getPosts();

  return (
    <>
      <Navbar />
      <main className="flex min-h-screen flex-col items-center justify-between bg-gray-50 py-12 lg:py-24">
        <div className="mx-auto max-w-7xl px-4 sm:px-6 lg:px-8">
          <div className="mx-auto max-w-lg">
            <h2 className="text-3xl font-bold tracking-tight text-gray-900 sm:text-4xl">
              Hot Takes
            </h2>
            <p className="mt-2 text-lg leading-8 text-gray-600">
              Check out the latest posts from our community.
            </p>
            <PostFeed initialPosts={initialPosts} />
          </div>
        </div>
      </main>
    </>
  );
}

In this component, we first fetch the initial batch of posts using the getPosts action. These posts are then passed as a prop to the PostFeed component, which is responsible for rendering the list of posts. The Navbar component is also included for site navigation, and the main content is styled with Tailwind CSS and structured to provide a pleasant user experience.

3. Implementing the Post Feed

The PostFeed component is responsible for displaying the list of posts and loading more posts as the user scrolls down. It uses the Intersection Observer API to detect when the user has scrolled to the bottom of the list and then fetches additional posts:

// src/components/post-feed.tsx
'use client';
import { getPosts } from '@/actions/get-posts';
import { PostType } from '@/utilities/data-fetcher';
import { useEffect, useState } from 'react';
import { useInView } from 'react-intersection-observer';
import Post from './post';

type PostFeedProps = {
  initialPosts: PostType[];
};

export default function PostFeed({ initialPosts }: PostFeedProps) {
  const [posts, setPosts] = useState<PostType[]>(initialPosts);
  const { ref, inView } = useInView();

  const fetchMorePosts = async () => {
    const newPosts = await getPosts();
    setPosts((prevPosts) => [...prevPosts, ...newPosts]);
  };

  useEffect(() => {
    if (inView) {
      fetchMorePosts();
    }
  }, [inView]);

  return (
    <section className="mt-16 w-full grid-cols-3 space-y-10 lg:mt-20 lg:space-y-20">
      {posts.map((post) => (
        <Post key={post.id} {...post} />
      ))}
      <div ref={ref}>Loading...</div>
    </section>
  );
}

The PostFeed component initializes with the initialPosts passed from the Home component. It uses the useInView hook from the react-intersection-observer library to track when the user has scrolled to the bottom of the list. When the bottom of the list is in view, the fetchMorePosts function is triggered, which fetches additional posts and appends them to the current list.

By leveraging the Intersection Observer API, we can efficiently load more posts as needed, improving the user experience by preventing long initial load times and reducing unnecessary data fetching.

Conclusion

Using Redis to manage and process data from external APIs can significantly improve the performance and scalability of your application. By enqueuing data into a Redis list and processing it in manageable batches, you can avoid performance bottlenecks and ensure a smooth user experience. This approach can be applied to various scenarios where external APIs return large amounts of data that need to be processed efficiently.

In this Next.js project, Redis provides a robust and flexible solution for managing intermediate data, making it an excellent choice for such use cases. By leveraging Redis’s capabilities, you can build applications that are both efficient and scalable.

Reference

For the complete source code and detailed implementation, please visit the GitHub repository.