This article explores problems related to designing and developing the system to predict renewable energy (or similar projects) - the outcome of the presented project is a web application (not a desktop app). Many things have to be considered, like the optimal technologies, team composition, budget, timetable. It might be helpful for you to read this article before you choose to develop a similar system, as otherwise, it can cost you a lot of time and money.

Library GDAL (developed by OSGeo) is more or less standard for processing geospatial data. However, although it implements almost all the tools needed for this purpose, it is notoriously known for many operational problems. Mainly, it is practically impossible to install it quickly (as it requires many system-level dependencies), API is obsolete (designed in the late nineties), and it is not difficult to find unexpected behaviours, etc. So it is not surprising that many developers actively started to seek different tools. This post presents some possibilities for the most common issues (mainly focused on Python users).

One of the common challenges when dealing with big multidimensional data sets is how effectively store them. There are many problems to deal with, but maybe the most important is choosing the proper dimension order. Although typically, the selection of the optimal dimensional order does not impact the size of the output file (if you use some compression situation that may differ), it has a profound effect on the response time when accessing stored data.

Many open-source tools can help with the prediction of renewable energy production. For example, the Python library PVLIB for computation of production solar of photovoltaic panels implements essential scientific methods. However, if it comes to predicting energy from wind turbines, there is no similar library - but the computation itself is sufficiently simple - so everyone can write their methods. There is also a question of how precise predictions are, which requires a probabilistic approach.

The famous library xarray is more or less standard in the data analyst branch. During the last few years, it has become prevalent. It has been deployed in many projects, often without any careful decision if it is needed or if there is any other way to treat the issue. One of the essential xarray functionality is the reading and writing of the NetCDF files. This is also a focus of this article where at the end, you should be capable of making the informed decision whether or not to use xarray.

In-memory database REDIS provides a simple interface for caching various data types. In addition, it supports simple manipulations with arrays and scalar values, including support for complex data constructs (like queues). What is, however not implemented, is the support of multi-dimensional objects. These objects are beneficial if you need to cache frequently used constants (or interim results) for computations (like matrices).