Is substring check for unique IDs in large text files efficient?
The script, reply_comments.py, runs in an automated pipeline twice a day. It pulls comments, filters out the ones I've already replied to, and checks a local Markdown file (drafts/comment_replies.md) to see if I've already started a draft. Each draft is marked with a header like ## 3b908, where 3b908 is the comment's id_code.
To keep the code "simple," I originally used a basic substring check:
def pending():
try:
drafted = open(DRAFTS, encoding="utf-8").read()
except FileNotFoundError:
drafted = ""
out = []
for a in api(f"/articles?username={ME}&per_page=100"):
if not a["comments_count"]:
continue
for c in api(f"/comments?a_id={a['id']}"):
if c["user"]["username"] == ME or replied_by_me(c):
continue
if c["id_code"] in drafted:
continue
out.append({...})
The flaw is that c["id_code"] in drafted doesn't check if the ID is a header; it checks if those characters exist anywhere in the file. Since my drafts include quotes from users and my own prose, it's only a matter of time before a random string of characters matches a comment ID.
I caught this when I realized a date in one of my drafts—specifically "2026"—perfectly mimics the format of a DEV.to id_code (short alphanumeric strings). If a new comment arrived with the ID 2026, my pipeline would see that string in my draft file and silently skip it. No error, no log, just a comment that never gets a reply.
I proved this with a quick test:
import re
text = open("drafts/comment_replies.md", encoding="utf-8").read()
headers = set(re.findall(r"^## (\S+)", text, re.M))
fake_new_code = "2026"
print(fake_new_code in headers) # False — not a header
print(fake_new_code in text) # True — exists in prose
The irony is that I had already written the correct logic in a different part of the same script for an audit() function. To fix this AI workflow and prevent silent data loss, I switched to using regex to isolate actual headers:
def audit():
...
drafted_codes = set(re.findall(r"^## (\S+)", drafted_text, re.M))
By anchoring the search to the start of the line (^## ) and using a set, the lookup is both precise and faster. It's a reminder that "it works in testing" isn't the same as "it's logically sound," especially when dealing with alphanumeric IDs in unstructured text.
All Replies (3)
Worried about performance on huge files. Does the load time spike after a certain gigabyte limit?
Ran into a wall with this on a log parser once the file hit a few MBs.
Adding a regex check would be a lifesaver if those ID formats shift. Which library handles that fastest?