Data Upload

The LIDO Data Management System provides different modalities for data upload that should satisfy all data provider's requirement:

  1. Influx-DB API: more efficient and powerfull way to upload data.
  2. Influx-DB web interface: powerful manual upload that reuires some knoledge of Influx-DB.
  3. LIDO web interface: Data upload page for simplified manual upload.
  4. FTP server: Data file transfer to the FTP server using a predefined data file template .

Before start reading this documentation page, we suggest to read the page InfluxDB Introduction and Data organization to get familiar with the terminology and InfluDB system.

Influx-DB API

This modality use the direct data upload by the API directly by the sensor/gateway or by another server application (shell-script, python, R, etc.) using the POST request method. The complete list of libraries and clients that can connetc to InfluxDB-API is avalable here: http://lidodata.laimburg.it/load-data/sources. To setup the configuration of the sensor/application the following parameters are necessary:

  • api-url: http://lidodata.laimburg.it/api/v2/write
  • bucket name: is the name of the "database" where you want to store your data and where you a token with writing permission.
  • measurement name: is the "table" where you want to collect your datasets.
  • authorization token: is the secret key you received with the registration or that you created by yourself (with writing permission) in the web interface.

Remember that the time parameter must have the Unix time format.

Have a look to the complete documentation, read the Write data with the InfluxDB API page.

Influx-DB web interface

This modality provides of the InfluxDB functionalities for user that are familiar to InfluxDB. It is more powerfull than the LIDO web interface, but more complex.

Unix time format for the time field is mandatory like in the example below.

Load file using the GUI and selecting the desired bucket using this template:

measurements_name parameter1="value",parameter2=value, parameter3=value,parameter3=value,parameter4="value" Unix_time_value

Here an example:

station_1 id_sensore="lido1",temperature=11.0,humidity=86,velocita_vento=6.8,direzione_vento="sw" 1664955600
station_1 id_sensore="lido1",temperature=11.5,humidity=80,velocita_vento=6.1,direzione_vento="sw" 1664955900
station_1 id_sensore="lido1",temperature=11.6,humidity=81,velocita_vento=7.8,direzione_vento="w" 1664956200
station_1 id_sensore="lido1",temperature=12.0,humidity=76,velocita_vento=6.8,direzione_vento="sw" 1664956500
station_1 id_sensore="lido1",temperature=12.3,humidity=71,velocita_vento=6.5,direzione_vento="nw" 1664956800

LIDO web interface

The LIDO interface of the LIDO data management system provides a friendly interface to upload your .csv files into the DB. This is made for manual files upload with a specific format, mainly for users that are not familiar with Influx web interface or to uplaod data files downloaded manually from the sensor.

Here is the .csv file template with required information in the first two rows:

organization_name,bucket_name,measurement_name
time,parameter1,parameter2,parameter3,parameter4,parameter5
time_value,value1,value2,value3,value4,value5
time_value,value1,value2,value3,value4,value5
time_value,value1,value2,value3,value4,value5
time_value,value1,value2,value3,value4,value5
......
.....

The first row describes where you want to uplaod your data and the second row describes the data type for each parameter. Remember to always name the time parameter with the label "time", don't use other labels like "date" or "day" otherwise the upload will not succeed. Time format must be in UTC: DD.MM-YYYY HH:MM:SS like in the example below.

laimburg,test_upload,station_1
time,id_sensore,temperature,humidity,velocita_vento,direzione_vento
time,string,number,number,number,string
05.10.2022 09:40:00,lido1,11.0,86,6.8,sw
05.10.2022 09:45:00,lido1,11.5,80,6.1,sw
05.10.2022 09:50:00,lido1,11.6,81,7.8,w
05.10.2022 09:55:00,lido1,12.0,76,6.8,sw
05.10.2022 10:00:00,lido1,12.3,71,6.5,nw
......
....

Each row of the file must always have the same number and order of columns, while different files may have different number and order of columns for the same measurement. Pay attention to always use the same name for the parameters in different files in order to visualize properly the timeserie. Temperature/temperatura or humidity/hum/umidità will be stored as different parameters. Also verify the correspondance between the second and third line. Files must always have unique names to avoid mistakes. If a file with same name wil be loaded, the system will answer with an error message.

FTP server post

Using the File Transfer Protocol (FTP) server you can uplaod all your data files using the client that you prefer. You can easilly automatize the transfer of you files using a shell script in your data server. An automatic process will complete the uploading from LIDO FTP server to the InfluxDB if the file respect the provided template as described for Influx-DB web interface. Time format must be in UTC: DD.MM-YYYY HH:MM:SS like in the example below.

You can upload files to FTP server using desktop client like WinSCP or Filezilla. You can also send file using comand lind in linux or window server. Here an example using a linux serve comand line:

scp file_name.csv username@eomountdata.eurac.edu

The user name will be send to you in the organization registration together with the user password. Files must always have unique names to avoid mistakes.