from Hacker News

Show HN: Steel.dev – An open-source browser API for AI agents and apps

by marclave on 11/26/24, 1:34 PM with 52 comments

  • by jedberg on 11/26/24, 4:22 PM

    Very interesting. I worry that if I use your cloud, and a lot of other people do, all of your IP addresses will get banned by all the big players. It will definitely be a fun cat and mouse game!

    Related story: Way back in the day, PayPal was just getting started, and decided eBay transactions would be their perfect customer. The only problem is that eBay didn't allow scraping. So they built an entire proxy infrastructure to go around eBay's rules and scrape them.

    It worked. It worked so well, eBay bought PayPal.

    The side effect of this is that I got control of the PayPal proxy infrastructure since I was on the security team for both eBay and PayPal after the acquisition.

    We used that proxy farm to scrape the rest of the web looking for fake eBay sites (because they would block traffic from eBay's IPs) and we had the guys who built it help us build proxy defense for eBay and PayPal.

    So this could work in your favor, if you manage to constantly scrape a large target who might want to buy you. :)

  • by huss97 on 11/26/24, 1:39 PM

    Hello Hacker News! We’re Nas and Huss, co-founders of steel.dev (http://steel.dev). Steel is an open-source browser API for AI agents and apps. We make it easy for AI devs to build browser automation into their products without getting flagged as a bot or worrying about browser infra.

    over the last year or so, we’ve built quite a few AI apps that interact with the web and noticed - a. it was magical when you could get an llm to use the web and it worked and b. our browser infra was the source of 80% of our development time. Maintaining our browser infrastructure became its own engineering challenge - keeping browser pools healthy, managing session states and cookies, rotating proxies, handling CAPTCHA solving, and ensuring clean process termination. We got really good at running browser infrastructure at scale, but maintaining it was still stealing time away from building our actual products. So we wanted to build the product we wish we had.

    Steel allows you to run any automation logic on our hosted instances of chromium. When you start a dedicated browser session you get stealth, proxies, and captcha solving out of the box. We do this by exposing websocket and http endpoints so you can connect to these instances with puppeteer, playwright, selenium(in beta), or raw CDP commands if you’re built like that.

    Behind the scenes, we host several browser instances and route incoming connection requests to one of these instances. Our core design principle was to allow for every session to have its own dedicated browser instance + resources (currently 2gb vram and 2gb vcpu) while still allowing for quick session creation/connection times. Our first thought was to have separate nodes running in a Kubernetes cluster, but the cost of hosting warm browser instances would be expensive (which would be reflected in the pricing), and the boot times would be too slow to handle the scale that some customers required. We got around this by deploying our browser instance image on a firecracker VM, taking advantage of the lightning-fast boot times and ability to share a root FS.

    Today, we’re open-sourcing the code for the steel browser instance, with plans to open-source the orchestration layer soon. With the open-source repo, you get backwards compatibility with our node/python SDKs, a lighter version of our session viewer, and most of the features that come with Steel Cloud. You can run this locally to test Steel out at an individual session level or one-click deploy to render/railway to run remotely.

    We're really happy we get to show this to you all, thank you for reading about it! Please let us know your thoughts and questions in the comments.

  • by orliesaurus on 12/2/24, 3:31 AM

    Is this very different from https://github.com/browserbase/stagehand

  • by capiki on 12/2/24, 5:37 AM

    Can you say more about your product plans? It looks like this is directly competing with Browserbase — how will you differentiate? Looking forward to seeing how the product and company grows

  • by DanielKehoe on 12/2/24, 6:16 AM

    Happy to see there's a way to get browser automation for AI without building infrastructure to support it. Yet I don't see examples of connecting an LLM to drive a web session, just examples of using Puppeteer or Playwright or Selenium to drive a web session. Presumably your user base knows how to write custom code for an interface between Claude or OpenAI API and Puppeteer/Playwright/Selenium. Sadly, I don't know how to do that. Would it be fair to expect your documentation to help? What would you suggest to get started?

    Is the interface between Steel, or Puppeteer/Playwright/Selenium, something that might be implemented in the new Anthropic Model Context Protocol, so there's less custom code required?

  • by cranberryturkey on 11/26/24, 1:38 PM

    I don't understand what problem you're solving that puppeteer doesn't already solve.

  • by gregpr07 on 12/3/24, 2:26 PM

    Nice!! We are working on a higher level library over at https://github.com/gregpr07/browser-use.

  • by potamic on 12/2/24, 2:08 PM

    Browser as a service I understand, but what is the "for AI agents" part?

  • by Oras on 11/26/24, 6:36 PM

    Looking interesting, will definitely give it a go.

    Btw, there is inconsistency between pricing page and pricing on docs.

    Pricing page for developers is $59 Pricing in docs for developers is $99

  • by benzguo on 12/2/24, 3:27 AM

    beautiful docs + api ref! what are you using? (cool that you're doing open-source browserbase also, excited to check this out)

  • by amelius on 12/2/24, 1:47 PM

    Is there any AI out there that can detect and fill in WiFi login pages typically found on hotel and company guest networks?

  • by meiraleal on 12/2/24, 12:07 PM

    Why not a browser extension?

  • by doritopope on 11/26/24, 4:37 PM

    will definitely check this out. i saw the pricing model on the site, what's the motivation around being open-source here if you're providing the infra free of charge?

  • by puppycodes on 12/2/24, 4:37 AM

    I cant find any information on models you support?

  • by tndibona on 12/2/24, 1:20 PM

    Can this defeat perimeterX used by Zillow?