Fantoir-datasource

The fantoir-datasource tool allows to import and manage FANTOIR datasource.

Database setup

Database schema for fantoir-datasource

The tool uses a PostgreSQL database to store FANTOIR, Wikidata and configuration data.

Recommended workflow

To update correctly the PostgreSQL database, it's recommended to follow those steps:

1. fetch operation to get the last FANTOIR file and extract it ;
2. import operation to read it and insert data into PostgreSQL, use with -c option so it takes care to create the table, specify a NEW table so we don't interrupt any service during update process ;
3. wikidata operation to import rich metadata from Wikidata table, use with -tc option to truncate/create table as needed ;
4. query operation to test the new imported dataset with a known value ;
5. promote operation to update foreign keys and config to point to that new table.

It's possible to create a maintenance report for Wikidata during that import, using fantoir-datasource wikidata -tc --maintenance-report.

Usage

Most commands require DATABASE_URL environment variable to interact with PostgreSQL. The fetch command can be used without it.

$ fantoir-datasource help

Import FANTOIR database into PostgreSQL

Usage: fantoir-datasource <COMMAND>

Commands:
  fetch     Fetch the last version of the FANTOIR file
  import    Import from FANTOIR file generated by the DGFIP
  promote   Promote an imported FANTOIR table as the current FANTOIR table to use
  wikidata  Query Wikidata SPARQL end-point to enrich FANTOIR information
  query     Query the imported FANTOIR table
  help      Print this message or the help of the given subcommand(s)

Fetch

Check through the data.economie.gouv.fr API the last version available, compare with FANTOIRmmyy file present in working directory (if any). Download it, unzip it if needed.

To force overwrite the file, use fantoir-datasource fetch --overwrite.

Exit code can be used to programmatically know what happened:

Exit codes

Code	Description
0	Fetch operation succeeded
2	Parsing arguments issue (not fetch specific), or unzip format issue (fetch specific)
12	FANTOIR file already exist and --overwrite not specified
16	Download issue, at filesystem or HTTP level, stderr will provide information for humans or store in a log
32	unzip not found, you need to install unzip to decompress the archive
n	Exit code from unzip command (man 1 unzip)
127	Error with the unzip process, without knowing the unzip exit code

If the file is extracted and all is fine, the script will print on stdout:

FANTOIR_FILE=FANTOIR1022