#TIL you can pass variables to an #awk script with the option -v. This is useful, for example, when you want to include the file name in the output:
```
find . -type f -iname '*.csv' -exec awk -F, -v filename={} '{print filename, $2}' {} \;
```
Even though seemingly awkward at first glance, #awk is definitely one of the most versatile and useful tools on #linux.
Behold the epic tale of Janet's #PEG #module, where the author heroically excludes regular expressions like they're yesterday's news. 
Marvel at the labyrinth of #parsing magic that claims to be more readable, but only if you have a PhD in arcane text processing. 
https://bakpakin.com/writing/how-janets-peg-works.html #Janet #readability #textprocessing #regex #HackerNews #ngated
Back when I first wrote text processing code in the 90s on my Amiga 1200, I always used the ¤ symbol as a placeholder character for splitting and replacing to exclude things I wanted skipped without affecting character count. It was available on the Norwegian keyboard, and practically never used in text.
Recently I discovered that Unicode has two "Not a character" symbols perfect for the same usage: \uFFFE and \uFFFF.
They can be really useful!
2. Immediately after the split, replace U+FFFF with newline, but keep both versions of the line, and pass the one with the U+FFFF to the text paragraph parser. Everything else (like headings) gets the cleaned one.
3. After paragraph lines with a single break between them (belonging to the same paragraph) have been processed, THEN I replace the U+FFFF characters there.
It seems to work, but it took me like 3-4 hours to crack. 
4/4
I tried using the alternative line and paragraph separators from Unicode, but splitlines accepts them too. Then I discovered these Unicode characters:
U+FFFE <noncharacter-FFFE> not a character.
U+FFFF <noncharacter-FFFF> not a character.
The solution, then was:
1. Replace all occurrences of [br] with or without a trailing newline, using regex pattern "(?i)(?<!\\)(\[br\]\n?)", with a U+FFFF character.
3/4
This works fine in principle, but it is incredibly hard to figure out exactly when to make the replacement.
For instance, if I do it too early, the parser will split on the breaks as I use splitlines() early on. If I do it too late, I get double line breaks some places.
2/4
I've been struggling with solving an issue with my text editor project. The editor is plain text and uses a blank line to separate paragraphs.
The editor has an option to preserve or not preserve single line breaks inside paragraphs when generating the output.
However, some users want to not preserve them, but still want to be able to add hard breaks sometimes. So I've been trying out using [br] as a hard break shortcode.
1/4
I started working on a Python class to write MS Office Word documents from already tokenized formatted text. It took me 5 hours to get a working version that can handle most of the formatting I need.
I have already done this with the Open Document format. It took me significantly longer, but I do steal some code from that code for DocX.
That said, DocX is actually easier to generate the XML for it turns out.
Discovered a neat new tool last week: https://github.com/wr7/refold
It's similar to `fmt` and `fold` except that it automatically handles prefixes. Vim/Neovim `gq` can do this out of the box but fails (for me at least) when multiple prefixes are present, such as a Markdown block-quote inside Rust comments. E.g.
```
// > Some quoted text
// > to reflow.
```
`refold` handles this.
