from Hacker News

Boring Problems Need Attention Too

by simplegeek on 6/16/21, 12:14 PM with 40 comments

• by version_five on 6/17/21, 3:52 PM

  What this highlights most to me is the false precision people have come to expect. This is a real problem I've seen as an engineer / physicist working with business people. Things that I would consider "the same" because they are well withing the noise of one another get all kinds of "hold on now, I added up the numbers in the columns and got 22.1 but the columns total says 22" from people that don't understand rounding, significant figures, experimental error, etc. I know it's important to hide this from people because they end up getting distracted by it (or to occasionally let slip so someone can feel like they have done a thorough review because they found a column didnt add up to the rounded total). But I think it's more important overall to educate people about what kind of variation is important and what isn't, vs. getting hung up on the definition of a word.

• by 6gvONxR4sf7o on 6/17/21, 4:33 PM

  I'm sad that the author didn't do a manual word count. If it's only 1660 words or so, it would surely take less time than it took to write this post. It would probably make the author confront the difficulty in precisely defining word count.

  It turns out that a lot of things that initially seem trivial to precisely define aren't actually that precisely defined, like the length of the california coastline. This is, in my mind--and as a complete tangent--a great argument for wide programming and math education. When you're forced to be so goddamned precise all the time, it's very clear when an idea isn't fully defined.

• by SethTro on 6/17/21, 6:03 PM

  I diffed down to understand why Open Office (1663 consistent with Microsoft Office) differs from Google Docs (1655).

  "ago--never": G docs 1 word vs OpenOffice 2 words.

  "tiger-lillies--what": G docs 1 word vs OpenOffice 2 words (IDK what this should "really" be)

  "Wanting?--Water": GD 3 words vs OO 2 words

  In this case the disagreement springs, exclusively, from if the docs engine believes that double hyphens make a compound word (and potentially handling punctuation in the middle of such a compound word)

• by thehappypm on 6/17/21, 3:00 PM

  I mean.. are numbers words? Is a floating ! or - a word? Are hyphenated words 2 or 1? How about when a word is broken over two lines with a hyphen? Is ... a word? Is an unrecognized string a word? Does a figure caption count towards a word count? How about chapter titles? Or page numbers?

• by PaulHoule on 6/17/21, 2:45 PM

  There is no such thing as a "word".

  Look at the waveform of speech and you will see long silent gaps inside "words" as well as there frequently being no gap between "words".

  There are phrases like "Skinny Puppy" that can do the same job as a word, there are also structures smaller than words that people smush together to make words. The two even work together:

     missile
     anti-missile missile
     anti-anti-missile missile missile
  

  If you see "words" as the molecules of text there will always be an asymptote you can't overcome because segmenting text into words will sometimes introduce errors that you might not be able to recover from.

• by Pet_Ant on 6/17/21, 5:05 PM

  I remember during my PhD, a casual discussion on that figuring out the end of a sentence on it's own is surprisingly more complex then you'd think:

  "I work at the F.B.I. I like it there." "I work at the F.B.I., I like it there." "I work at the F.B.I. I like it there!"

  It's not as simple as counting periods.

  So that counting words can have corner cases is definitely understandable. Is "&" a word? It is literally just 'e' + 't' superimposed and "et" is definitely a word.

• by LarryMade2 on 6/17/21, 2:57 PM

  So they just leave us hanging there, no analysis, no reasoning of the results...?

• by asciimov on 6/17/21, 4:44 PM

  I'd rather see an explanation behind the issue rather than a complaint about people not working on boring stuff because a bunch of text editors can't agree on a word count.

  I remember taking a typing class some 25 years ago and being told a that a word count is typically every 5 characters. That way someone doesn't pad out their word count by using lots of small words.

• by jakub_g on 6/17/21, 5:57 PM

  IMO the issue here is that: everything that is not carefully specified as a precise algorithm, will be implemented differently.

  This is why each browser used to parse HTML differently.

  This is why you'd have compat or even security issues because some software used \r\n for newlines splitting while other used \n.

  Luckily the browser vendors formed WHATWG which created pretty precise specs which are maybe convoluted but at least everyone parses HTML in the same way, and each browser pretends to be every other browser for compatibility.

  2021 is really great for web compat, maybe not all browsers implement every API, but existing APIs are accompanied by very thorough test suites (Web Platform Tests): https://github.com/web-platform-tests/wpt

  Live results from nightly builds: https://wpt.fyi/results/?label=experimental&label=master&ali...

  Having said that I don't see vendors aligning on definition on word count any soon due to corporate inertia, lack of incentives and lack of "Word editors consortium" (or is there any?)

  A truly good function for word count might be pretty complex, and perhaps different for every language.

• by dcolkitt on 6/17/21, 6:15 PM

  I think a complement to this article is the essay "Reality has a Surprising Amount of Detail"[1]. The point is that things which superficially seem easy often have a lot of hidden complexity when you drill down to all the low-level details and corner cases.

  [1]http://johnsalvatier.org/blog/2017/reality-has-a-surprising-...

• by AbrahamParangi on 6/17/21, 4:43 PM

  The words of someone who's never tried to write a parser that operates on natural language.

  "I've seen things, you people wouldn't believe"

• by Zababa on 6/17/21, 6:59 PM

  There's a 2% variation between the lowest and the biggest. Is there a need to know a precise word count, where 2% of margin of error is not acceptable? If there's not, there's no problem. The author doesn't attempt to define what a word is, so maybe all of these are correct for their own definition of word? Maybe the definition of what counts as a word is the boring problem that needs a solution, but it doesn't look boring or easy to me. You can't ask people to count correctly something that you haven't even defined.

  Also, believing that complex problems are easy is not something only programmers do. I work currently a lot with Excel automation, and most people have no idea of what can be automated easily and what can't. I have some people coming that ask for automating a task they've never done manually and don't really know how to do precisely. I think that's the same mechanism of "overabstraction" that leads to people to say "WET instead of DRY" (Write Everything Twice instead of Don't Repeat Yourself).

• by spoonjim on 6/17/21, 7:23 PM

  The "boring problem" that always needs more attention than it gets is performance. Plenty of companies ship features that only reduce performance by 10,20,50 milliseconds but it all adds up and applications end up feeling very sluggish. Apple is one of the few where I feel like the perf doesn't regress over time (likely because the features don't really expand much either).

• by lbriner on 6/17/21, 3:57 PM

  Also boring businesses need attention. Loads of people would love to run a gym but you could make 10x more by selling toilet paper or paper clips.

• by bilater on 6/17/21, 5:57 PM

  The variance is probably due to a difference in what counts as a word + ambiguity around white spaces etc in each text editor. So it's not that the word counting problem is unsolved it's a problem of different companies adopting different definitions.

• by sgtnoodle on 6/17/21, 5:57 PM

  At some FAANG companies, fixing bugs and other "boring" projects don't get you promoted.

  I can tell when there's a perf/promo cycle coming up at Google because all the core apps on my phone change their UIs and get buggier and slower.

• by juancn on 6/17/21, 6:14 PM

  Counting words is easy, defining what a word is, is not.