Moving Push Notifications to FCM Topics: The Cost Is Keeping Subscriptions Right

I worked on the push notifications for an app’s backend, on Node.js, TypeScript, and Prisma. Staff create a notification in an admin panel and choose who gets it: one user, a group, everyone, or an uploaded CSV list.

Sending to one user is simple. The hard part was “everyone”: look up all the device tokens, then send to them. So I moved groups and “everyone” to FCM topics. The send got much simpler, and most of the work moved somewhere else.

Topics move the work, they don’t remove it

With a topic, devices subscribe to a name like group_12, and one message to that name reaches every subscriber. FCM handles the fan-out.

But now “who is in this group” lives in my database and “who subscribes to this topic” lives in FCM. They’re two copies of the same fact, and everything else in this post is about keeping them consistent.

The result was a mix, not a clean switch:

• Groups and everyone, on FCM devices: one topic per group, plus one for everyone.
• One user or a CSV list: device tokens.
• Devices that don’t use FCM: a different push vendor, by token. So “everyone” is actually a topic send plus a token send.

I tried one-time topics for CSV lists, then removed them

My first design for CSV lists was a temporary topic: subscribe everyone on the list to temp_xxx, send once, mark it used. It let a list and a group share one code path.

It lasted about six weeks. It had its own tables, and I removed all of it in a later refactor. My commit message says the separate infrastructure was no longer needed. On the latest branch I checked, CSV pushes go by token.

It also had a bug I should have caught. The scheduled cleanup only marked temporary topics inactive in the database. It never unsubscribed the devices from FCM, so it couldn’t free up what it claimed to clean.

Keeping subscriptions in step

I added a table recording each token’s subscription to each topic, with its status, last confirmation time, error, and retry information. FCM gives you no convenient way to ask “what is this token subscribed to?”, and without that table I couldn’t answer “why didn’t this user get it?”.

Group changes were only the beginning. Over the following months, the bugs that showed up were all the same kind of bug, a case where the two copies could drift:

• A user turns push off, and their devices shouldn’t subscribe to anything until they turn it back on.
• Turning it on or off has to sync the subscriptions, and a failed unsubscribe needs a retry.
• Deleting a member has to clean up their topics.
• Repeated device registrations can overlap, so I serialized registration and queued member topic updates.

Rate limits, and what I actually built

The limits were configuration, using only 80% of each to leave headroom:

const defaultLimits = {
  maxQPS: 3000,
  maxBatchSize: 1000,
  maxTopicsPerDevice: 2000,
  concurrentBatches: 3,
  rateLimitBuffer: 0.8,
};

The name “batch” flatters what happened next. FCM’s API accepts many tokens in one call, but my code still subscribed one token at a time inside each batch, with three batches running at once and a delay between rounds:

async function subscribeBatches(requests: SubscribeRequest[], limits = defaultLimits) {
  const size = Math.floor(limits.maxBatchSize * limits.rateLimitBuffer);
  const qps = Math.floor(limits.maxQPS * limits.rateLimitBuffer);
  const delayMs = Math.ceil(1000 / (qps / size));
  const results: SubscribeResult[] = [];

  for (const window of chunk(requests, size * limits.concurrentBatches)) {
    const batchResults = await Promise.all(
      chunk(window, size).map(async (batch) => {
        const batchResult: SubscribeResult[] = [];
        for (const request of batch) {
          batchResult.push(await subscribeOne(request)); // one token per call
        }
        return batchResult;
      }),
    );
    results.push(...batchResults.flat());
    await sleep(delayMs);
  }
  return results;
}

That gives per-token results and simple failure records, but it spends far more calls than necessary.

At that stage, large subscription jobs also ran in a queue that was a plain array inside the process, so they wouldn’t survive a restart. Sending was different: it went to a separate push service with a BullMQ queue on Redis, with retries. Two kinds of queue, with very different guarantees, and it’s easy to talk about them as if they were one.

Splitting topics by language

The app has English and Chinese, and a notification needs the right copy for each user. A single group_12 topic can only carry one message, so I split topics by language: all_en, all_zh, group_12_en, group_12_zh. A bilingual notification becomes one send per language, for each group.

Existing devices were already subscribed to the old names, so this needed a migration. I wrote a script for it, a rollback script, and a confirmation step that has to be typed before it runs on production.

About three months later, a problem showed up: devices could hold both languages’ topics at once, because registration only ever added subscriptions. So a language change never removed the old ones.

The fix was to stop thinking about changes. At registration, compute what this device should be subscribed to, compare it with the database, and remove the difference:

async function reconcileDeviceTopics(deviceToken: string, userId: string, language: Lang) {
  const memberships = await db.groupMember.findMany({ where: { userId } });
  const desired = new Set([
    allTopic(language),
    ...memberships.map((m) => groupTopic(m.groupId, language)),
  ]);

  const existing = await db.topicSubscription.findMany({
    where: { deviceToken, status: { not: "UNSUBSCRIBED" } },
  });

  for (const { topicName } of existing) {
    if (!desired.has(topicName)) await unsubscribe(deviceToken, topicName);
  }
  await Promise.all([...desired].map((topicName) => subscribe(deviceToken, topicName)));
}

This is the idea I’d keep from the whole project. It doesn’t care how the device got into its current state. It isn’t fully solved, though. My follow-up plan still lists per-token locking so overlapping registrations can’t overwrite each other, and moving language from the user to the device, since one person’s two phones can use different languages.

“The push didn’t arrive”

On Android, a preview push sometimes never showed up, while the final push sent seconds later did. So I set up a reproduction:

1. Take an Android test device offline.
2. Send the preview, then the final push 5 seconds later.
3. Bring the device back online about 60 seconds later.

The push service accepted both, FCM accepted both, and the device showed only the final one. That supports the idea that FCM replaced the pending message while the device was offline. It’s an inference, since I can’t see inside FCM. My report also says what it doesn’t explain: a language mix-up we’d seen earlier was a separate issue.

The useful habit was splitting “didn’t arrive” into segments: did we send it, did FCM accept it, did the device receive it. And it’s a good reminder that subscribing a device to a topic doesn’t promise that device will show every message.

What I took from it

Sending was never the hard part. Each bug here came from the same place: two systems holding the same fact, with code that updated them on events. What finally held up was recomputing the desired state and correcting the difference.