This is the continuation of the previous article. For better understanding it is recommended to look through the earlier blog posts.
System architecture elaboration
Creating architecture of a system providing receiving and processing of large amount of data in real time scale requires a lot of investigations. One of the main reasons of this is an absence of standard methods or systems providing a deep and profound vision of a large collection of heterogeneous data on data sources, analysis and representation in sight of view multilayer geographic information system
A multilayer geographic information system should provide following criteria: choice of a technological platform of energy production and its economic efficiency evaluation; risks evaluation; evaluation of ecological consequences of transition to renewable energy; evaluation of opportunities of transition to intellectual distribution grids (so-called Smart Grids).
GIS key layers are:
Geoinformational;
Energy sources;
Energy consumers;
Energy distribution and transition system;
Resources;
Energy production technologies;
Energy storage technologies;
Ecologic situation and potential threats;
Economic evaluation;
Juridical evaluation;
Data protection
From the point of view of functionality the proposed system differs from existing ones by a spatial distribution level, big data volume, and system components working in asynchronous manner. For its representation and implementation a 3M paradigm has been proposed, that includes: multilayer representation, multilayer architecture, multiagent interaction. In sight of view of this paradigm it is suggested to construct and implement the system prototype providing multilayer representation of monitoring data and data obtained from other sources (with different scaling and extension levels). System architecture construction is also supposed to construct in a multilayer form in accordance with world standards and attitudes to software engineering of large systems. Components of the system working in an asynchronous manner are to be implemented with the help of multiagent approach that has already been successfully implemented in several projects including some projects on the micro levels that have been applied in wind monitoring.
The system should include several layers. Firstly, a layer establishing a communication between weather stations and weather sensors. Later, a communication layer, used to transmit data on the global server. After this, a layer of information system that is a foundation for data gathering and storage. Later, a functional layer implemented on the base of the previous one. And finally, a business layer that is to be used in high level and intellectual tasks implementation, such as: planning, predicting, data analysis and data interpretation. For the system prototype implementation several architecture components have been completed.
The system includes both hardware and software part. Whereas the hardware part is considered to be secondary the software component is the most essential.
The software component architecture is defined by the following requirements:
Data gathering from different sources and collecting into one single storage for later analysis
Web based GIS (geographic information system) to work with map
Ability to upload own datasets of data visualization with the help of different tools and of uploading own visualization tools
Providing basic data analysis tools and ability to implement own analysis algorithms and to save them after in the system
The requirements list is broad and requires a complicated construction during which several improvements and changes should be applied. That is a main reason why an iterative approach of fast prototyping has been selected for the projects. The models on the UML language have been constructed to provide the representation of main cases of system usage and components interaction. The main use cases are presented below
While analyzing the system requirements, as it has been mentioned above, the multilayer GIS concept has been applied. Generally, such class systems are multilayer by default, but in this case the layer means a concrete dataset. Due to the necessity of storing not only the data, but also the tools of visualization and analysis, in the project the definition of a layer has been expanded. Each layer is a dataset with integrated tools for later processing
As the layer concept has been expanded and become more complicated the following user activity sequence has been accepted in the system:
Dataset uploading (or choosing the installed one)
Instead of uploading a dataset it is possible to connect existing database or a text file (in CSV, JSON, or fixedlengthcolumn format)
The data becomes a foundation of a layer
In the layer the data analysis and visualization tools are created (at the current step in the form of interpreted code)
The layer is activated (the data analysis algorithm starts and the result is visualized)
From the point of view of data volume, consolidation of data is important as it is used to summarize large quantities of datasets. If the system is supposed to gather data regularly, the respective data is better and more efficient to be found in the form of spreadsheets encompassed into larger worksheets. Data consolidation process involves a certain computational power, and one of the most popular tools of choice is text or table editors. Data consolidation may be implemented automatically by means of the tools that are incorporated within the program or certain plug-ins.
All the described steps may be presented in the form of sequence diagram. The user interaction scheme is to be verified an improved during the construction process.
Another essential problem is the data consolidation method. As the current amount of data is relatively small the problem does not seem to be significant.
But still as the datasets are updated and new sources are found the problem becomes more serious. At present there are two suitable solutions:
Data consolidation at the application layer
Data consolidation at the database layer
The first solution is more flexible but it is necessary to develop a separate extension for each energy source. In this case the application is working with a definite data format that is common for all data sources. This approach is to be implemented after the real system start. In case of prototype the second solution is more suitable. In this case the data is unified in the database at the table layer. The application in turn is to take the data format into consideration for each data source. The data consolidation at the application layer is presented on the following figure.
To be continued.
Comments