About
The Innovative Data Analysis (IDA) programme was an MBIE-funded research project led by Manaaki Whenua – Landcare Research that ran for 4 years (2014–2018).
The goal of the IDA programme was to research and develop processes to integrate and harmonise high priority heterogeneous land resource and biodiversity datasets about New Zealand to support a step change in the quality of environmental reporting.
People who make decisions about the environment of New Zealand need accurate and reliable environmental data. With these data, they can make informed decisions about natural resource management, set environmental policy, and assess progress towards ecological sustainability and whether policy initiatives or environmental management approaches are effective over time.
Environmental reporting relies on using a range of indicators to measure and report on the overall health of the environment. Indicators are created by combining data with scientific knowledge about the aspect of the environment being reported on.
The IDA programme aimed to have impact by supporting central and regional government to report on the state of the New Zealand environment in a standardised, statistically robust and transparent way. The programme was aligned with key initiatives such as New Zealand State of the Environment (SOE) reporting, Environmental Monitoring and Reporting (EMaR), and the National Science Challenges.
IDA used next-generation data analysis and informatics techniques and worked with data custodians and end users to develop statistical indicators for three domains: soil health, land use, and species occupancy. The programme focused on extracting knowledge and value from existing environmental data sets rather than on the collection of new data. Its focus was also not research outcomes but rather technical and social infrastructure outcomes.
The IDA programme had three domain components: land use, soil quality, and species occupancy. For each of these, the key components of the programme included the following:
- Data federation: bringing together heterogeneous spatial data from multiple sources to produce a suite of higher value information products.
- Modelling indicators: developing indicators to respond to environmental reporting requirements.
- Visualisation and delivery: enabling use by central and regional government and other agencies, through existing portals (e.g. the LRIS Portal, Manaaki Whenua's national land information portal), via open standards-based web data services and tools to visualise data.
Underpinning these components, characterisations of provenance, quality, uncertainties, and data and information workflows were also explored to enable auditable processes.
The programme itself was organised around 3 research aims:
- Generating land-domain indicators from existing distributed data
- Generating biodiversity indicators from existing distributed data
- Supporting innovative data analysis for environmental reporting
Research aim 1 and 2 focused on establishing and formalising domain-specific knowledge for harmonising, cleaning, and validating data to overcome current barriers to use and on the analyses needed to establish reporting indicators. Research aim 3 involved developing tools and data infrastructure components to support the other two aims and to identify the challenges and key principles to creating an environmental data and analytical infrastructure for New Zealand to improve data access, sharing and re-use, as well as to support data-driven science discovery.
Given the breadth of data and knowledge required, many staff were involved in the programme over its 4-year duration. Members of the team also undertook significant engagement with international peers to collaborate and leverage from IDA-related international initiatives. Such interaction brought several benefits: accelerated technical development; opportunities to leverage other participants’ know-how; and the adoption of international standards, best practice, and new technologies to shorten the development process in the programme, and thus programme costs. In addition, through this activity we increased New Zealand’s international research and technical profile.
The IDA programme generated many outputs, including white papers, academic papers, and reports, as well as software, data, and online tools. Many of the outputs are accessible and/or downloadable from this site.
Since the end of the project, the IDA team has been working to ensure that other researchers and stakeholders are aware of the outcomes of the programme through presentations, workshops, and continued access to data and tools. Some of the outputs have been incorporated in Manaaki Whenua's internal research data infrastructure. The team has also been seeking new funding opportunities to build on and evolve the work undertaken in the IDA programme.