Using a substring check for unique IDs in a large text file is a

CameronWizard Advanced 3h ago Updated Jul 27, 2026 448 views 1 likes 2 min read

The script, reply_comments.py, runs in an automated pipeline twice a day. It pulls comments, filters out the ones I've already replied to, and checks a local Markdown file (drafts/comment_replies.md) to see if I've already started a draft. Each draft is marked with a header like ## 3b908, where 3b908 is the comment's id_code.

To keep the code "simple," I originally used a basic substring check:

def pending():
 try:
 drafted = open(DRAFTS, encoding="utf-8").read()
 except FileNotFoundError:
 drafted = ""
 out = []
 for a in api(f"/articles?username={ME}&per_page=100"):
 if not a["comments_count"]:
 continue
 for c in api(f"/comments?a_id={a['id']}"):
 if c["user"]["username"] == ME or replied_by_me(c):
 continue
 if c["id_code"] in drafted:
 continue
 out.append({...})

The flaw is that c["id_code"] in drafted doesn't check if the ID is a header; it checks if those characters exist anywhere in the file. Since my drafts include quotes from users and my own prose, it's only a matter of time before a random string of characters matches a comment ID.

I caught this when I realized a date in one of my drafts—specifically "2026"—perfectly mimics the format of a DEV.to id_code (short alphanumeric strings). If a new comment arrived with the ID 2026, my pipeline would see that string in my draft file and silently skip it. No error, no log, just a comment that never gets a reply.

I proved this with a quick test:

import re
text = open("drafts/comment_replies.md", encoding="utf-8").read()
headers = set(re.findall(r"^## (\S+)", text, re.M))

fake_new_code = "2026"
print(fake_new_code in headers) # False — not a header
print(fake_new_code in text)    # True — exists in prose

The irony is that I had already written the correct logic in a different part of the same script for an audit() function. To fix this AI workflow and prevent silent data loss, I switched to using regex to isolate actual headers:

def audit():
 ...
 drafted_codes = set(re.findall(r"^## (\S+)", drafted_text, re.M))

By anchoring the search to the start of the line (^## ) and using a set, the lookup is both precise and faster. It's a reminder that "it works in testing" isn't the same as "it's logically sound," especially when dealing with alphanumeric IDs in unstructured text.

pythonmcpdebuggingWorkflowAI Implementation

All Replies (3)

L
LazyBot Intermediate 11h ago
Might be worth adding a regex check if those IDs ever change format.
0 Reply
F
Finn47 Novice 11h ago
does the file get too big for this to stay fast? wondering about load times.
0 Reply
J
JordanSurfer Intermediate 11h ago
Did this for a log parser once. Works fine until the file hits a few MBs.
0 Reply

Write a Reply

Markdown supported