Nasqueron Datasources: Difference between revisions

From Nasqueron Agora
Line 8: Line 8:
It also can enrich data from Wikidata, to qualify ways and offer rich typography labels.
It also can enrich data from Wikidata, to qualify ways and offer rich typography labels.
During that operation, it can generate [https://www.wikidata.org/wiki/User:Dereckson/FANTOIR_maintenance_report a maintenance report].
During that operation, it can generate [https://www.wikidata.org/wiki/User:Dereckson/FANTOIR_maintenance_report a maintenance report].
<source>
$ fantoir-datasource help
Import FANTOIR database into PostgreSQL
Usage: fantoir-datasource.exe <COMMAND>
Commands:
  import    Import from FANTOIR file generated by the DGFIP
  promote  Promote an imported FANTOIR table as the current FANTOIR table to use
  wikidata  Query Wikidata SPARQL end-point to enrich FANTOIR information
  query    Query the imported FANTOIR table
  help      Print this message or the help of the given subcommand(s)
</source>


=== Helper components ===
=== Helper components ===

Revision as of 01:56, 15 January 2023

'Nasqueron Datasources is a collection of tools and libraries, mostly written in Rust, to fetch and enrich data.

Datasources

Datasources components

FANTOIR (fantoir-datasource)

fantoir-datasource is a CLI tool to import the FANTOIR database into PostgreSQL. It's built with pipeline as a code in mind, so we can run it when a new FANTOIR version is released.

It also can enrich data from Wikidata, to qualify ways and offer rich typography labels. During that operation, it can generate a maintenance report.

$ fantoir-datasource help

Import FANTOIR database into PostgreSQL

Usage: fantoir-datasource.exe <COMMAND>

Commands:
  import    Import from FANTOIR file generated by the DGFIP
  promote   Promote an imported FANTOIR table as the current FANTOIR table to use
  wikidata  Query Wikidata SPARQL end-point to enrich FANTOIR information
  query     Query the imported FANTOIR table
  help      Print this message or the help of the given subcommand(s)

Helper components

The datasources repository contains the following libraries:

  • The opendatasoft-explore-api crate allows to query the Opendatasoft Explore API from Rust code. This API software is for example used for data.economie.gouv.fr for open data.

A SPARQL client can also be extracted from fantoir-datasource to be published as a standalone library (and so be used in outside projects).

Dependencies

The following relations exist in our components:

  • fantoir-datasource uses opendatasoft-explore-api to check if a new version is available and get the URL

Contribute

The Nasqueron Databases project on DevCentral is used for any issue, code idea, bug or task.

Repository to clone: datasources

Contact: Dereckson is willing to help you if you need to discuss this project.

Licences

Source code is released under BSD-2-Clause license.

The datasets we provide tooling for and consume are and must be licensed under free culture licences, tailored for sui generis database rights:

  • LO (e.g. FANTOIR)
  • ODbL (e.g. OpenStreetMap)

The datasets we produce are licensed under:

  • when a copyleft applies, for example when a dataset is ODbL, ODbL
  • when no copyleft applies, because all sources allow redistribution under any license: CC-BY 4.0