from Hacker News

SQLite-HTTP: A SQLite extension for making HTTP requests

by b_mc2 on 8/10/22, 7:43 PM with 21 comments

• by alexgarcia-xyz on 8/10/22, 7:58 PM

  Author here, happy to answer questions!

  Direct link to the project: https://github.com/asg017/sqlite-http

  Few other projects of interest:

  - sqlite-html: https://github.com/asg017/sqlite-html

  - sqlite-lines: https://github.com/asg017/sqlite-lines

  - various other extensions: https://github.com/nalgeon/sqlean

  And past HN threads:

  - https://news.ycombinator.com/item?id=32335295

  - https://news.ycombinator.com/item?id=32288165

• by xnx on 8/10/22, 9:09 PM

  Side note: I love the attention that SQLite is getting now. It's so refreshing after the dark ages of misapplied technology: microservices, Docker, NoSQL, and countless front-end frameworks.

• by emadehsan on 8/10/22, 8:40 PM

  On a side note, Observable is stupendous. Interactive Network Graphs, Charts, Maps, Animations. Not sure why we don't see it used enough.

  Mike Bostock himself is constantly adding latest work https://observablehq.com/@mbostock. Great learning resource.

• by simonw on 8/10/22, 8:03 PM

  I picked up some very neat new SQLite tricks from this post independent of the extension itself (which is very cool) - I didn't know about the "define" module which lets you create new table-valued functions that are defined as SQL queries themselves.

• by 1vuio0pswjnm7 on 8/10/22, 8:24 PM

  Would be interesting if this can do HTTP/1.1 pipelining. Need to take a closer look.

  What makes me hesitate to look closer is that I almost always am doing some text processing after the HTTP response and before the SQL commands. Due to resource constraints, I do not want to store HTML cruft.

  I mostly am doing

  HTTP response --> text processing --> SQLite3

  rather than

  HTTP response --> SQLite3

  However I also have a need for storing different combinations of HTTP request headers and values in a database. I currently use the UNIX filesystem as a database (one header per file, djb's envdir loads the headers into environment from a selected folder), but maybe I could use SQLite3. For unprocessed text, e.g., from pipelined DoH responses, I use tmux buffers as a temporary database. Then do something like

     HTTP responses --> tmux loadb /dev/stdin
     tmux saveb -b b1 /dev/stdout|textprocessingutility --> ip-map.txt (append unique, i.e., "add unique")
  

  The ip-map.txt file gets loaded into memory of a localhost forward proxy.

  Databases, such as NetBSD's db, djb's cdb, kdb+, or sqlite3 can help with the "append unique" step if the data gets big.

  Note: Any JSON I retrieve is "text", not binary data. Most HTTP responses are HTML, sometimes with embedded JSON. Pipelined DoH is binary but I still use drill to print the packets as text. (When I finally learn ldns I will stop using drill.)

• by dimgl on 8/10/22, 8:47 PM

  While this is really cool (and a feat of engineering no less), I'm always really concerned when someone suggests to make an HTTP call from their database server.

  Many years ago my boss told me to make a scheduled task in Windows to execute a SQL query to make an HTTP call and I asked him why we couldn't just use crontab/cURL? His response: "cURL? Like from the 90s?"

  Anyhow I didn't last very long. Got fired shortly thereafter.

• by hsbauauvhabzb on 8/10/22, 8:18 PM

  How to get hacked: 2022 edition.

  SSRF and lambdas directly in your db engine. What other antipatters could you possibly want?

• by jayvanguard on 8/10/22, 8:50 PM

  Just because you can do it, doesn't mean you should do it.

• by mayli on 8/10/22, 10:43 PM

  Soon: Sqlite-OS