Earth in the Cloud

The project

Earth in the Cloud is a research project funded by POR FESR Lazio 2014 - 2020 (Action 1.2.1) that inolve two research organizations, the University of Rome Tor Vergata (through the Department of Civil Engineering and Computer Engineering) and Sapienza University of Rome (through the Department of Computer Science). The project aimed at developing a new technology which, through techniques based on mathematical models and artificial intelligence, has the objective of automating, optimizing and minimizing the costs of the process of development, production and operation of applications that process Eart Observation data on cloud computing platforms. By exploiting the computing resources offered by the various competitive cloud providers on the market, the technology acts as a broker for the users, selecting the most efficient and cost-effective resources, and automating the deployment and execution process of operational workflows. The ultimate goal is to reduce for companies the operating and management costs of applications that convert OT data into value-added products and services, offering them considerable advantages in terms of competitiveness, time-to-market and scalability, allowing them to support rapid growth of their business.

The technology aims at the following specific objectives:

1 Provide support to the user to describe their operational workflows
2 Automatically identify the most efficient machine types and amount of computing resources
3 Automate deployment operations of operational workflows on one or more selected cloud platforms
4 Automate the activation and execution of operational workflows
5 Automatically create and resize the resource pool on which each operational workflow runs
6 Automate the detection of changes to the types of computing resources available
7 Monitor the execution of operational workflows

The project ended on April 2023. The estimated cost and the admitted contribution for the project are € 148,724.10.

Organization of activities

The project activities were divided into the following 5 Work Packages (WPs):

- WP1: Activities coordination and administration, communication and valorisation of results

- WP2: Analysis of user requirements and identification of technological requirements, definition of pilot test cases

- WP3: Study and development of techniques based on AI and mathematical models to support the automation and optimization of operational workflows in the cloud environment

- WP4: Technology design and development of the system's operational components

- WP5: Final testing and evaluation of results

Activity schedule diagram:

Developments

The project has come to the end. After the completion of the activities of work packages 2, 3 and 4, the final testing phase relating to WP5 was completed. Below are the technical/scientific requirements and objectives achieved in the context of the activities of the various work packages.

As part of WP2, dedicated to identifying the requirements and functionality of the technology, the following macro targets have been identified:

1adopt a workflow representation based on DAG (Directed Acyclic Graphs)
2offer one or more frameworks that make it easier for the user to build DAGs
3automate all operations for deploying workflows on the cluster of cloud machines on which they must be executed
4automate parallel execution of a workflow on multiple input datasets
5automate the choice of machines for the composition of the cluster on which to run a workflow from one or more cloud providers
6automatically identify for each specific workflow and based on the characteristics of the specific input data set the types of machines and the quantity of machines to use
7choose the machines and automatically size the cluster of machines on which a workflow is executed based on the requirements in terms of execution times (deadline) desired by the user
8choose the machines and automatically size the cluster of machines on which a workflow is executed based on the requirements in terms of real costs for the user for the execution of the workflow li>
9automate access to the different datasets to be processed based on the workflow execution schedule and the time intervals of data interest
10monitor the execution of operational workflows and possible errors/exceptions in execution

WP3 has produced the techniques and tools to:

1the identification, among the multitude of machines and configurations offered by cloud providers, of the most efficient types of machines for executing the various workflows
1the prediction of the execution time of the tasks of the various workflows based on the type and quantity of machines used, and in particular according to the characteristics (metadata) of each specific data set to be elaborate
1the search for optimal solutions in terms of scheduling the tasks of the various workflows based on the QoS requirements and the timing desired by the user for the execution of each workflow.

As part of WP4, the implementations of the set of services were completed for:

1the management of multiple connections to cloud service platforms
2access and management of computing resource catalogues
3orchestration functions for multiple acquisition/release of resources
4 the APIs for accessing storage and managing machine images
5workflow management and processing features
6functions for deploying workflows on clusters of machines in the cloud
7monitoring functions for distributed workflow execution

The testing phase, in the contex of WP5, completed an overall evaluation study on the developed technology, including in particular the following targeted studies:

1a study for the evaluation and refinement of the workflows produced to conduct the final testing of the technology
2an evaluation study focused on the capabilities of the developed technology to identify optimal solutions in terms of workflow scheduling, based on QoS and cost requirements established by the user
3an experimental comparison to evaluate the aforementioned capabilities of the developed technology compared to state-of-the-art solutions for workflow scheduling in a cloud environments
4an evaluation study of the techniques for predicting the execution time of the tasks of the various workflows based on the type and quantity of machines used
5 an evaluation study of an innovative workflow developed within the project for the large-scale identification of areas that have been subject to fires

The scientific publications related to the project are currently being submitted.

If you are interested in the technology developed by the Earth in the Cloud project, plea contact us using the form at the bottom of the page.

Earth in the Cloud