April 5th, 2019
The automated data flow implementation is finished
The creation of a new central database repository and the automated data flow were two key technological tasks that had to be undertaken due to the large quantity of data to be handled (the EBP online portals are collecting c. 45 millions new records every year) and the necessity of accessing these data in near real time.
The central database repository has three main functions: storing and administering the data, handling the data provisions from different local online systems (c. 20 different ones) and generating the content of the EBP viewer and other future EBP products. To automatically handle the data provisions from the local online portals an API web service was developed.
This API service provides a secure channel to submit the data, which is transferred using an predefined JSON schema. The API validates the JSON and basic data consistency and returns the required warnings to the sender. Also runs more detailed data checkings in the background.
Thanks to the work of the IT specialist of nearly 20 different online portals and the EBP IT team the data flow implementation has already been finished and with excelent results.
Currently, 15 online systems and the equivalent of 99.2% of the whole EBP data is submitted automatically to the EBP data repository on a daily basis using the EBP API service or using an hybrid approach that implies the use of the local system own API services to retrieve the data and the EBP API service to format and upload the data. Only three online systems (0.8% of the whole EBP data) are still submitting data manually.
This is a particularly important accomplishement and notably exceeds the initial LIFE EBP target: to connect automatically 12 online systems or at least the equivalent of more than 80% of all the whole EBP data.
To see all this complex system at full work, simply check the LIVE EBP viewer and enjoy watching even the latest bird movements!
The LIVE EBP viewer would have not been possible without the implementation of a new central database repository and an automated data flow system. Now the new central repository receives more than 99% of the whole EBP data automatically and on a daily basis (the equivalent to c 120,000 records every day, c. 45 million records by year). Then, also on a daily basis, the repository generates the tables used by the data visualization platform CARTO to render the different map visualizations in the EBP viewer.
Eleven online systems and the equivalent of 83.1% of the whole EBP data is submitted automatically using the EBP API service.
Four online systems and the equivalent of 16.1% of the whole EBP data are connected using an hybrid approach that implies the use of the local system own API services to retrieve the data and the EBP API service to format and upload the data.
Three online systems and the equivalent of 0.8% of the whole EBP data is submitted manually at least once every month.
