from Hacker News

Show HN: Faast.js – Serverless Batch Computing Made Simple

by achou on 5/2/19, 6:38 PM with 31 comments

• by achou on 5/2/19, 6:49 PM

  Hi everyone, faast.js is a library that allows you to use serverless to run batch processing jobs. It makes it super easy to run regular functions as serverless functions. This is one of my first open source projects and I'd be happy to answer any questions here.

• by gregmac on 5/2/19, 7:10 PM

  From what I can tell, it's the invocation model and deployment that is unique here?

  You invoke faast from your local machine (or build server, or cron job, whatever), and in turn it deploys some functions to a serverless platform and runs them, then tears them all down when complete. Eg, from the site, this code runs locally:

      import { faast } from "faastjs";
      import * as funcs from "./functions";
  
      (async () => {
          const m = await faast("aws", funcs);
          try {
              // m.functions.hello: string => Promise<string>
              const result = await m.functions.hello("world");
              console.log(result);
          } finally {
              await m.cleanup();
          }
      })();
  

  You wouldn't want to run this code on serverless, as you'd be paying for compute time of just waiting for all the other tasks to complete.

  It would be useful to see a discussion about how and where to host this entry code, may even a topic on "Running in production".

  It's definitely a neat idea because if you control the event that kicks everything off anyway (eg: "create monthly invoices" or "build daily reports") you can deploy the latest version of everything, run it and clean it up in essentially a single step.

  (Please correct me if I've misunderstood any of the details here!)

• by asadlionpk on 5/2/19, 8:14 PM

  This can be great for scrapping jobs!

  There are IP-based rate limiters on sites (linkedIn, facebook, etc), but each lambda has a new public IP so by using faast.js, I can stay under the radar.

  Plus you can essentially spawn a headless chrome (puppeteer) to do advanced stuff.

• by dongxu on 5/2/19, 6:51 PM

  Very interesting project, the problem with Serverless service provided by different public cloud vendors is that programming and API are not uniform. I think Faast.js is on the right path to creating a unified interface for different Serverless services.

• by BrandiATMuhkuh on 5/2/19, 8:05 PM

  Love what you did!

  We resently were exactly in a situation where we had to do heavy processing of ~4000 items each running between 1-10minutes. To speed the process up we ran it on lambda. That means our process went down from 10h++ on a single core computer to about 15min running it on 4000 lambdas.

  Your library would have saved us quite some work as it would take away a lot of Aws config, deploy, etc....

  Btw: I'm thinking of building a similar library for multi core/webworkers for node.js. currently a lot of boilerplate is required on node.js to make a loop run parallel on all cores.

• by mring33621 on 5/2/19, 8:12 PM

  This is neat, but would be more useful if it could deploy cloud functions made in language {x} and provide local js proxies for them.

• by sourc3 on 5/2/19, 8:33 PM

  This is very neat! Last year I had to essentially do this on GCP and relied on a very similar implementation. Everyone was surprised to see JS being used for data processing but it worked wonderfully.

  One thing I want to ask is the retries, how do you handle that currently? I ran into multiple cases where functions would fail for transient reasons.

• by heathermiller on 5/2/19, 10:15 PM

  Reminds me a bit of like 2019's version of RMI...

• by dead_mall on 5/3/19, 4:53 AM

  Looks interesting. The concept reminds me of RPyC