from Hacker News

Ask HN: Regex on a File or Stream

by buzzdenver on 3/6/24, 9:04 PM with 6 comments

I just ran into this seemingly not that hard issue of trying to match a multi-line regex against a 3Gb text file. What is the right tool for this? grep and perl failed running into PCRE limits.

by jepler on 3/7/24, 2:44 AM
Maybe some other PCRE-compatible implementation offers streaming. For instance, https://www.intel.com/content/www/us/en/developer/articles/t... says it has this feature, but of course given who it's from it may be tied to a single brand of CPU.
github seems to be https://github.com/intel/hyperscan
by zaktoo2 on 3/6/24, 9:32 PM
Could you paste the regex portion of it please? Possibly some efficiencies to be gained there. You could also split the file into smaller chunks and then check the boundaries of the chunks.
by cvalka on 3/7/24, 3:36 AM
https://github.com/VirusTotal/yara
by burntsushi on 3/7/24, 2:53 AM
ripgrep should be able to handle it with the -U/--multiline flag.