from Hacker News

Launch HN: Release (YC W20) – Orchestrate AI Infrastructure and Applications

by tommy_mcclung on 8/7/24, 2:50 PM with 39 comments

Hello Hacker News! We’re Erik, Tommy, and David, the founders of Release (https://release.ai/). We launched on HN in 2020 (https://news.ycombinator.com/item?id=22486031) after leaving TrueCar, where we managed a 300 person development team. Our original focus was making staging environments easier with ephemeral environments, but along the way AI applications started to emerge as an important and critical component of distributed applications. As we talked to customers using our original product, we realized we had built the underlying platform needed to address the needs of orchestrating AI applications and infrastructure. So here we are and we’re excited to share Release.ai with HN.

Here’s a video showcasing the platform and demonstrating how to easily manage new data and changes using the RAG stack of your choice: https://www.youtube.com/watch?v=-OdWRxMX1iA

If you want to try release.ai out, we’re offering a sandbox account with limited free GPU cycles so you can play around and get a feel for Release.ai: https://release.ai. We suggest playing around with some of the RAG AI templates and adding custom workflows like in the demo video. The sandbox comes with 5 free compute hours on an Amazon g5.2xlarge instance (A10 with 24GB VRAM, 8vCPUs and 32GB). You will also get 16 GB and 4vCPUs for cpu workloads such as web servers. You will be able to run an inference engine plus things like an api server, etc.

After the sandbox expires, you can switch to our free plan, which requires a credit card and associating an AWS/GCP account with Release to manage the compute in your cloud account. The free account provides 100 free managed environment hours a month. If you never go over, you never pay us anything. If you do, our pricing is here: https://release.com/pricing.

For those that like to read more, here’s the deeper background.

It’s clear that open source AI and AI privacy are going to be big. Yes, many developers are going to choose SaaS offerings like OpenAI to build their AI applications, but as open source frameworks and models improve, we’re seeing a shift to open source running on cloud. Security and privacy is a top concern of companies leveraging these SaaS solutions, which forces them to look at running infrastructure themselves. That’s where we hope to come in: we’ve built Release.ai so all your data, models and infrastructure stay in your cloud account and open source frameworks are first class citizens.

Orchestration - Integrating AI applications into a software development workflow and orchestrating their lifecycle is a new and different challenge than traditional web application development. Release also makes it possible to manage and integrate your web and AI apps using a single application and methodology.

To make orchestrating AI applications easier, we built a workflow engine that can create the complex workflows that AI applications require. For example, you can automate the redeployment of an AI inference server easily when underlying data changes using webhooks and our workflow engine.

Cost and expertise - Managing and scaling the hardware required to run AI workloads is hard and can be incredibly expensive. Release.ai lets you manage GPU compute resources across multiple clouds with different instance/node groups for various jobs within a single admin interface. We use K8s under the covers to pull this off. With over 5 years of building and running K8s infrastructure our customers have told us this is how it should be done.

Getting started with AI frameworks is time consuming and requires some pretty in-depth expertise. We built out a library of AI templates (https://docs.release.com/release.ai/release.ai-templates) using our Application Template format (which is kind of a super docker-compose: https://docs.release.com/reference-documentation/application...) for common open source frameworks to make it easy to get started developing AI applications. Setting up and getting these frameworks running is a hassle, so we made it one click to launch and deploy.

We currently have over 20 templates including temples for RAG applications, fine tuning and useful tools like Juypter notebooks, Promptfoo, etc. We worked closely with Docker and Nvidia to support their frameworks: GenAI and Nvidia NEMO/Nims. We plan to launch community templates soon after launch. If you have suggestions for more templates we should support, please let us know in the comments.

We’re thrilled to share Release.ai with you and would love to get your feedback. We hope you’ll try it out, and please let us know what you think!

  • by JoeCortopassi on 8/7/24, 8:03 PM

    I've noticed that while a bunch of developers have played with LLM's for toy projects, few seem to have any actual experience taking it to prod in front of real users. I’ve personally had to do so for a few startups, and it's like trying to nail Jell-O to a tree. Every random thing you change, from prompts to models, yields massively different/unpredictable results.

    I think because of this, a bunch of companies/tools have tried to hop in this space and promised the world, but often times people are best served by just hitting OpenAI/GPT directly, and jiggling the results until they get what they want. If you're not comfortable doing that, there are even companies that do that for you, so you can just focus on the prompt itself.

    So that being said, help me understand why I should be adding this whole system/process to my workflow, versus just hitting OpenAI/Anthropic/Google directly?

  • by BurritoKing on 8/7/24, 6:19 PM

    This looks awesome, getting started with AI development is daunting and I really like how this focuses on integrating with a bunch of open source frameworks and then deploying them into your own cloud (I always prefer to run the infrastructure, it feels weird to rely on something that's a complete black box).

    The sandbox environment with free GPU hours is a cool way to try things out without a big commitment too. It's nice seeing a product that genuinely seems to address the practical challenges of AI deployment. Looking forward to seeing how the platform develops!

  • by bradhe on 8/7/24, 8:33 PM

    Super interesting you guys have been working on this since 2020 if I'm reading the post title correctly? Would love to know the iterations you've gone through.

  • by todd3834 on 8/7/24, 6:47 PM

    This is very cool! I love seeing tooling targeting inference. I feel like stable diffusion and LLAMA have to be the primary use cases for these types of services. DALL-E is super lacking and GPT does actually start to get pretty expensive once you are using it in production.

  • by michaelmior on 8/7/24, 7:09 PM

    This looks cool, but I'm a little confused about the pricing model. It sounds like I'm paying you for every hour my jobs are running on my own infrastructure if I'm reading it right. That seems like a really odd way to price things if true.

  • by the_pascal on 8/7/24, 3:37 PM

    How does this compare to managed offerings like Google Gemini and AWS Bedrock? Thanks in advance and congratulations on the new product!!

  • by mcsplat2 on 8/7/24, 6:02 PM

    How do you hook up data to an environment? And what data sources do you support (Snowflake/etc?)

  • by mchiang on 8/7/24, 5:49 PM

    This is cool. I'd like to give it a try. Press a button, and get GPU access to build apps on.

  • by tommy_mcclung on 8/7/24, 9:37 PM

    We might not have made it clear in the post how to signup for the sandbox. Just head to http://release.ai and click on "Start Free Trial".

  • by jakubmazanec on 8/7/24, 6:02 PM

    Why do you have such generic name? It will make searching so much harder.

  • by bluelightning2k on 8/7/24, 6:21 PM

    Seems pretty vague. Something to do with half self-hosting open source LLMs with a proprietary docker-like template thing.

    Am I on the right track?

  • by nextworddev on 8/8/24, 12:33 PM

    Pretty insane that these folks are YC 2020 and just now pivoted again. Shows how hard company building is.

  • by sidcool on 8/7/24, 4:42 PM

    Congrats on launching. Interesting pivot.

  • by jodotgiff on 8/8/24, 12:05 AM

    Great system, no complaints!!

  • by drawnwren on 8/7/24, 7:42 PM

    I'm pretty much exactly your target market. I run a kubernetes, docker, poetry devops hell at an ml startup and was curious how your product helped. You got about 2 minutes of my time. I scanned your website, I have no idea what you do or whether you fix my problem.

    Not trying to be negative, but I think there may be a 30 second pitch to be made to people like me that isn't made on your site.

  • by billclerico on 8/7/24, 5:21 PM

    congrats!!

  • by bijutoha on 8/8/24, 3:51 AM

    What has been the feedback from early users regarding the ease of transitioning from your original product focus to the current AI orchestration platform?