from Hacker News

The Waiting Time Paradox, Or, Why Is My Bus Always Late?

by dragly on 10/28/18, 3:32 PM with 95 comments

  • by herodotus on 10/28/18, 5:14 PM

    Nice article. It reminds me of my year living in London, and taking the bus everyday to Imperial College from West End Lane in West Hampstead. There was a stop on both sides of the road - one for the outbound bus, and one for the inbound (the bus went from central London to a terminus and then returned mostly on the same route). Now we did not use schedules - way too inaccurate at rush hour, and the busses there were pretty frequent anyway. But we did expect an even chance of the inbound bus arriving before an outbound one did. My daughter and I became convinced after a while that this was not happening, so we invented a game (which we called "The Game of Life".) When our bus (inbound) arrived first, we added 1 to our score. We subtracted 1 for every outbound bus that passed before ours arrived (there were often more than 1). We realized that the result would be slightly skewed to the negative, but we expected the outcome to be close to 0 over time. Of course it was not. Anyway we extended the game to many statistical situations. For example, you go to the checkout line at the supermarket, and there are N people in front of you. When you get to the front of the line, you count the people behind you - call that M. If M is bigger than N, you scored life points. If it is smaller, you lost some. So you add M-N to your running score, and you get an idea of how lucky you are in life. However, I never followed up with any real analysis, so I enjoyed this article.

  • by ikken on 10/28/18, 7:48 PM

    This reminds me the bet in the bitcoin community [1]. If on average bitcoin blocks are produced every 10 minutes, and you learn that 5 minutes ago someone found a block, what is the average time you will wait for the next block? It turns out it's 10 minutes, not 5 minutes as you would intuitively think. (it's a memoryless process, so average expected time till block is always the same - 10 minutes - no matter how many blocks were recently found).

    In other words, when you're waiting for bitcoin transaction to be confirmed and go to check how long ago the most recent block was produced, in order to estimate how soon the next one will come - you're doing it wrong. Even if previous block was found 9 minutes ago, you're average waiting time for the next block is still 10 minutes.

    [1]. https://www.reddit.com/r/btc/comments/7rs8ko/dr_craig_s_wrig...

  • by agumonkey on 10/28/18, 10:25 PM

    Slightly related, my ghost town had few buses and sparses. I could never rely on printed hours. If I got there 10 min earlier to be sure, I'd still never be sure I'd wait 20 min for nothing because it was 11 min early. Of course half the time if I decide to walk to the next town where buses are many, I'd see all my town buses (both ways) pass me <yell-at-cloud.png>

    I think it made me completely careless about time, I would just go between stops and take the first one, go with the flow. By experience I'd know the range it would take for me to reach big places around the area.

    I had a friend who was completely foreign to this mode of thinking, she was very dilligent and fully trusting (although she mostly used trains so a lot less divergence).

    It reminds me of kid studies about intelligence / wealth ratios. When you're environment is random, you think random. When it's predictable you planify.

  • by twtw on 10/28/18, 5:59 PM

    > a Poisson process is a memoryless process that assumes the probability of an arrival is entirely independent of the time since the previous arrival. In reality, a well-run bus system will have schedules deliberately structured to avoid this kind of behavior: buses don't begin their routes at random times throughout the day, but rather begin their routes on a schedule chosen to best serve the transit-riding public.

    I've never really understood any example involving a poisson process. They always seem to involve bus arrivals or light bulbs burning out, and I can't understand why the memory less property would ever make any sense for these.

    Even if the bus system was poorly run, why would it make sense to assume that the expected value of time to arrival doesn't change based on how long you've been waiting?

    What is an actual phenomenon that is well modeled by a poisson process?

  • by taeric on 10/28/18, 5:14 PM

    Highly recommend reading this to any folks that are just sitting the discussions.

    The simulations were worth the article on their own. The real world analysis was a great bonus.

    Anecdotally, i was expecting confirmation bias to be the main culprit. Pleasantly surprised to seei was wrong.

  • by edoo on 10/28/18, 5:21 PM

    Hah great analysis. One factor with bus' is the schedule is likely planned to minimize early arrivals at the risk of being late more often. Usually when a bus is early it has to sit and wait until its departure time. A late running bus can be more efficient, and if kept until departure time might not ever get a chance to average down the bursts of lateness.

  • by stephengillie on 10/28/18, 4:39 PM

    OneBusAway is surprisingly accurate, at least in my experience. Google Maps has very good transit support too.

    One reason buses are late is because a bus must travel a circuit. Cars provide linear transportation, so the delay can only happen in the direction of your travel. Since buses run a circuit, they are impacted by delays in the direction opposite of your travel as well.

    Your bus might be late because the return route has traffic or other delays. Or maybe a drunk or drug user got in a fight with the driver and the police were needed. Or someone in a wheelchair had a problem getting onto the lift.

  • by gwern on 10/28/18, 5:11 PM

    The memorylessness of the Poisson process makes the statistical aspect a bit trivial. But here's an interesting variant: how should you update your beliefs while waiting if there is a certain probability that the bus won't come at all? "The Ups and the Downs of the Hope Function in a Fruitless Search", Falk et al 1994: https://www.gwern.net/docs/statistics/bayes/1994-falk

  • by jobigoud on 10/29/18, 1:28 PM

    I've encountered the inspection paradox in debates about factory farming and people talking past each other points.

    If you take the average farm, chances are that it's doing humane farming. But if you take the average animal, it has an overwhelming chance of being in an industrial farm.

  • by stornetn on 10/28/18, 11:13 PM

    Reminds me of a similar article that measured a similar kind of question about the wait times for NYC subways conditional on how long you've been waiting (https://erikbern.com/2016/04/04/nyc-subway-math.html). I think it's a pretty safe bet that people who like this post will like this article as well.

  • by ChrisFoster on 10/28/18, 10:36 PM

    It strikes me that even with a perfectly regular starting schedule, buses might clump together in time because the schedule is probably dynamically unstable. To explain, picking up passengers from a stop costs time and a long time between buses implies a high probability that passengers will be waiting at a given stop. This further adding to the delay and shortens the time to the next bus in the schedule.

    I'm sure drivers try to actively manage this, but if they didn't I suspect the system would naturally evolve toward pairs of buses leapfrogging each other on long routes.

  • by MaxBarraclough on 10/29/18, 5:13 PM

  • by varlock on 10/29/18, 4:47 PM

    Can't believe no one has yet mentioned the PASTA theorem - Poisson Arrivals See Time Averages (https://en.wikipedia.org/wiki/Arrival_theorem#Theorem_for_ar...). It is one of the theorems I remember the most from my Queuing Theory classes at the university!

  • by nakedrobot2 on 10/28/18, 9:22 PM

    In prague, the trams all run on time - within 2 minutes or less of the posted time. So I think this article is incorrect for this particular context.

  • by PascLeRasc on 10/29/18, 4:22 AM

    Is the inspection paradox what would happen if you surveyed everyone on how many siblings they had, and every sibling double-counted N-1 times (where N is the number of siblings in their family), inflating the resulting "average number of siblings", or is that something different?

  • by akane on 10/29/18, 1:16 AM

    On a related note, arrival time predictions can be biased early to prevent people from missing buses, which also increases the perception of lateness.

    https://nextbus.cubic.com/FAQs

  • by amai on 10/29/18, 12:13 PM

    Is there a "evil" distribution which maximises the waiting time? Or is the Poisson distribution already the theoretical "evil" maximum that a public transport provider can achieve?

  • by kuu on 10/29/18, 8:05 AM

    A bit off-topic: How can you integrate a jupyter notebook in a blog post like this one? It looks really nice!

    Nice article, btw, interesting topic!

  • by ezoe on 10/28/18, 10:59 PM

    I hate the poisson distribution because it completely against the naive instincts of how random behaves.

  • by mayankkaizen on 10/28/18, 6:36 PM

    Nice article. Since I just started learning Stats, I wish I could find more of such notebooks.

    Any recommendations?

  • by nyc111 on 10/28/18, 3:39 PM

    Would not be easier to actually time the actual waiting times as he waited for the bus every day?

  • by usgroup on 10/28/18, 7:58 PM

    This is a straightforward consequence of modelling an arrival process as a Poisson distribution with a constant rate of arrival lambda...

    Go from arrival to cumulative arrivals to time of arrival to recurrence of arrival (next arrival). All are Poisson processes, including the recurrence process, which has a fixed expected value.

  • by torgian on 10/29/18, 5:45 AM

    I’m glad I live in east Asia. Busses and trains are almost never late

  • by graycat on 10/29/18, 1:37 AM

    It's all much easier than that:

    It's just the Poisson process, e.g., with a nice chapter in E. Cinlar, Introduction to Stochastic Processes.

    Buses come as arrivals. So bus arrivals are a stochastic arrival process where stochastic just means varying randomly over time where, really, the randomly doesn't mean anything, includes deterministic arrivals, that is, known exactly in advance, but also admits any case of unpredictability.

    Well, in short, if have a stochastic arrival process with stationary, independent increments, then the arrival process is a Poisson process and there is a number, usually denoted by lambda, so that the times between arrivals are independent, identically distributed random variables with exponential distribution with arrival parameter, the arrival rate, lambda. The stationary means that the probability distribution of the times between arrival does not change over time. The independent increments means that the time from one arrival to the next is independent of all the past history of arrivals.

    The exponential distribution has the property, easy to verify with simple calculus, that the conditional expectation of the arrival time given that the arrival time is already greater than some number is the same as the expected arrival time.

    So, net, if bus arrivals form a Poisson process, then the time until the next bus arrives is the same after waiting five minutes as not having waited at all.

    Cinlar's treatment is nice because it is qualitative, that is, has assumptions that can often be confirmed or believed just intuitively. And we might not believe that bus arrivals meed the assumptions.

    This subject can continue with, say, hazard curves for equipment failures and a lot more about Poisson processes.

    E.g., the sum of two independent Poisson processes, say, Red buses and Blue buses, assuming that they are Poisson processes, is also a Poisson process with arrival rate the sum of the Red and Blue arrival rates. If randomly throw away some arrivals, then what is left is also a Poisson process with arrival rate adjusted in the obvious way.

    In Feller's volume II is the renewal theorem that the sum of independent arrival processes, Poisson or not, with mild assumptions, converges to a Poisson process as the number of processes summed grows. So, if the users of a sufficiently busy Web site act independently with mild assumptions, then the Web site will see arrivals accurately as a Poisson process.

    The vanilla Poisson process is Geiger counter clicks.

    There is much more to the pure and applied math and applications of Poisson processes.

  • by nyc111 on 10/28/18, 6:07 PM

    > When waiting for a bus that comes on average every 10 minutes, your average waiting time will be 10 minutes.

    This is very ambiguous. Unless he gives a time frame the numbers do not make sense. Average in a week? Average in a year? This is not how it works in real life.

    And I cannot accept his premise. My experience tells me that, in New York, when I used to take a bus to work, sometimes the bus was coming as I was walking to the stop; sometimes I would wait a long time. Sometimes not very long. There was no observable bias.