from Hacker News

Google Dataset Search

by abraxaz on 5/6/21, 8:15 PM with 42 comments

  • by abraxaz on 5/6/21, 8:18 PM

    Information on how to annotate datasets: https://developers.google.com/search/docs/data-types/dataset

    > We can understand structured data in Web pages about datasets, using either schema.org Dataset markup, or equivalent structures represented in W3C's Data Catalog Vocabulary (DCAT) format. We also are exploring experimental support for structured data based on W3C CSVW, and expect to evolve and adapt our approach as best practices for dataset description emerge. For more information about our approach to dataset discovery, see Making it easier to discover datasets.

    For more info on those:

    - W3C's Data Catalog Vocabulary: https://www.w3.org/TR/vocab-dcat-3/

    - Schema.org dataset: https://schema.org/Dataset

    - CSVW Namespace Vocabulary Terms: https://www.w3.org/ns/csvw

    - Generating RDF from Tabular Data on the Web (examples on how to use CSVW): https://www.w3.org/TR/csv2rdf/

  • by chatmasta on 5/6/21, 11:01 PM

    This is a great resource. At Splitgraph, we index ~40k open data sets, and we make sure to include structured metadata for each one, so we show up in these results. (example [0])

    One cool aspect of this metadata is that it allows a dataset to have multiple sources. So if two sites index the same dataset, there is no duplicate content penalty like there might be with textual content. If you search for a dataset, it will include links to all its sources (whether canonical or otherwise).

    For most of the data we index at Splitgraph, the canonical source is an open government data portal powered by Socrata (e.g. data.cdc.gov). We noticed that Socrata powered a lot of portals, so we wrote a Socrata plugin for Splitgraph, along with a scraper to index the metadata. The plugin basically implements a Postgres FDW so that Splitgraph can translate from SQL to the upstream query language. In this case, the plugin translates to Socrata's bespoke API language. But for private deployments we also have plugins for Snowflake, Postgres, some SaaS services, etc.

    If you find some data on Google Dataset Search with Splitgraph listed as a source, please take a look! Our "Data Delivery Network" (DDN) is implemented on top of the Postgres wire protocol, so you can connect with any Postgres client (or use our web editor). All the Postgres query syntax is available to you; you can even JOIN across any of the other 40k+ datasets indexed at Splitgraph. That includes "live data" like Socrata portals, but also versioned snapshots of data called "data images." Here's an example of a point-in-time query across two snapshots (basically a diff) [1], and another query that joins across tables at data.cityofchicago.org and data.cambridgema.gov [2].

    [0] https://www.splitgraph.com/cdc-gov/distribution-of-covid19-d... – "View Source" to see the Schema.org metadata

    [1] https://bit.ly/3epvxcj

    [2] https://bit.ly/3f1ll8K

    (Sorry for the bit.ly links. The URL for our query editor includes the full SQL string, and I don't want to mess up HN formatting.)

  • by davcancas on 5/6/21, 8:57 PM

    This dataset search engine has been around for years! We created DataMarket (https://datamarket.es) inspired by this site (and Auren Hoffman's SafeGraph).

  • by Der_Einzige on 5/6/21, 9:53 PM

    Stop, you're making the barrier to entry too low! /s

    This is really really cool. Between this and Hugginfaces Dataset and models hubs, AI/ML is really getting easier to use.

  • by john-tells-all on 5/6/21, 11:34 PM

    Dataset with 9,000 annotated cat images! => https://datasetsearch.research.google.com/search?query=cat&d...

  • by uptime on 5/6/21, 10:25 PM

    I have a lot to read before I get excited but if the team is here: Can we get DCAT for sets that are otherwise only discoverable with OAI-PMH? Seems like a divide between govt and academic repos that hinders harvesting.

  • by damirkotoric on 5/8/21, 3:52 AM

    Shameless plug. I wrote a piece about The State of Open Data Portals https://uxdesign.cc/designing-open-data-portals-for-governme... where I predict that it'll take a Google to really provide a single searchable dataset portal for the whole world.

    Doesn't take a genius to predict, but there ya go! Governments are assembling datasets in a very fragmented way. It'll take a private company to provide one single website to explore and find all datasets from around the world, making it easier to look at holistic patterns that are happening around the world, or compare patterns between countries.

    Though, I would expect a much better UX from Google nowadays. This site has more in common with Google Scholar than Google Search.

    And ultimately I'd like to see them build something where people don't need to download datasets in order to make use of the data.

    I compare the state of open data to the state of mapping software before Google Maps. You needed to download map files and open them on special software that you open on your computer to make sense of the data. And then Google Maps came along and flipped that whole model. Open data needs the same leap forward in order for more people to make greater use of open data.

  • by igravious on 5/7/21, 8:47 AM

    Discussion from Jan 2020: https://news.ycombinator.com/item?id=22130874 | 32 comments

    Discussion from Sept 2018: https://news.ycombinator.com/item?id=17919297 | 76 comments

  • by plaidfuji on 5/7/21, 1:49 AM

    I’ve come across this a few different times over the years... always seems enticing and potentially useful, but I’ve never found a real use for it. I suppose it provides a library of well-prepped datasets to test ML models on? Anyone ever used this for any practical purpose beyond a sandbox-type use case?

  • by smhx on 5/7/21, 1:31 AM

    another good resource that's more specific to machine learning is https://paperswithcode.com/datasets

  • by ravila4 on 5/7/21, 4:53 PM

    The lab I work in has a project that helps annotate datasets with metadata and register their schemas: https://discovery.biothings.io/

    A common barrier to making FAIR datasets is that not all data lends itself to be schema.org compliant. The idea is that instead of enforcing one schema to rule them all, we allow people to make their own schemas by extending existing ones, and register them in an API to be easily discoverable.

  • by jeresuikkila on 5/7/21, 9:08 AM

    They should add a "I'm Feeling lucky" button

  • by a_square_peg on 5/6/21, 10:59 PM

    Also in case the team is here... the updated date for ERA5 back extension to 1950-1978 (Preliminary version - https://datasetsearch.research.google.com/search?query=ERA5%...) is incorrect as this was only released last year (2020) but is stated as 2011.

  • by antpls on 5/8/21, 12:35 PM

    No mention of https://frictionlessdata.io/ dataset metadata format which is also used by Kaggle

  • by fabcomm on 5/6/21, 9:51 PM

    This is every data scientists’ dream.

  • by paulcole on 5/7/21, 12:16 AM

    I was looking up dentistry data sets (my industry) and came across this:

    https://www.arcgis.com/home/item.html?id=9850793c688e4eebaab...

    Can anybody explain why this showed up in a dataset search and what exactly the data is?

  • by sigmonsays on 5/6/21, 10:54 PM

    Do they have a deprecation notice up already?

  • by apprenticer on 5/7/21, 3:51 AM

    The privacy problem should be considered

  • by lunatuna on 5/7/21, 1:20 AM

    I'm not a scientist, so not the most scholarly first lookup, but tried searching for penis data[0]. The first link sent me to a site that requires signup to use [1]. No fun. Won't use again.

    [0] - https://datasetsearch.research.google.com/search?query=penis... [1] - https://data.world/jemus42/world-penis-data