from Hacker News

Successful machine learning models: lessons learned at Booking.com

by joker3 on 10/7/19, 4:01 PM with 87 comments

  • by Odenwaelder on 10/7/19, 8:03 PM

    "Content Overload: Accommodations have very rich content, e.g.descriptions, pictures, reviews and ratings."

    Laughed at that one. Booking.com is so full of dark patterns that I dread using it.

  • by anthony_doan on 10/7/19, 5:43 PM

    Experimental design is just a t-test? At least accord to that picture it seems that way. There are no ANOVA or interaction test?

    Do websites usually just use t-test only? Like adding one feature at a time?

  • by carapace on 10/7/19, 5:36 PM

    As an aside,

    > developing an organisational capability to design, build, and deploy successful machine learned models in user-facing contexts is, in my opinion, as fundamental to an organisation’s competitiveness

    You hear that, right? In 2019 already you have to have AI and do it well to be competitive. I just wanted to point out how cyberpunk that is.

  • by woliveirajr on 10/7/19, 5:16 PM

    > Model performance is not the same as business performance

    This is interesting. Sometimes some people from business side consider that AI is the solution to all problems (as if there was just one catch-them-all AI solution) and some academic people think that the top-performance model for some classification task is the must-go, and all they forget that the goal is to earn money.

  • by beefield on 10/8/19, 5:17 AM

    Nice to see that I am not only one hating booking.com with a passion.

    But what really amazes me is the market failure that hotels and other accomodation providers can't come up with a co-op booking site. I am sure there are issues that are difficult to solve from competition point of view, but are they really so difficult to solve that the rent seeking fees of current booking sites are justified?

  • by sandGorgon on 10/8/19, 4:19 AM

    >>Booking.com go to some lengths to minimise the latency introduced by models, including horizontally scaled distributed copies of models, a in-house developed custom linear prediction engine, favouring models with fewer parameters, batching requests, and pre-computation and/or caching.

    Any idea what these are ? especially the pre-computation/caching and batching. I'm not able to see what advantage does batching bring...or how you can really cache a prediction request

  • by djp518 on 10/7/19, 10:13 PM

    > Once deployed, beyond the immediate business benefit they often go on to become a foundation for further product development.

    This is one of the reasons I am a big believer in having a system to track model research and deployment lineage. (I personally use Domino Data Lab for this. I also work for Domino, but use it in my own modeling work and that of others I mentor.) No matter which system you use to track lineage, I've found it important to have a strict history of retraining, versioning, and experimentation. When models are used in downstream systems from the one they were originally intended, it becomes even more critical to able to explain and reproduce the 'research' that led up to deployment.

  • by orasis on 10/7/19, 10:11 PM

    I’m glad they highlighted inference latency. This is a big issue that I’ve started running into at scale.

  • by villux on 10/8/19, 8:07 AM

    Any other good resources on production level deep learning practices?