Library GDAL (developed by OSGeo) is more or less standard for processing geospatial data. However, although it implements almost all the tools needed for this purpose, it is notoriously known for many operational problems. Mainly, it is practically impossible to install it quickly (as it requires many system-level dependencies), API is obsolete (designed in the late nineties), and it is not difficult to find unexpected behaviours, etc. So it is not surprising that many developers actively started to seek different tools. This post presents some possibilities for the most common issues (mainly focused on Python users).

One of the typical challenges when exporting numerical computations is how to include the way the calculations were achieved. Typically, if results should be in an Excel sheet, you want to see a formula if you click on each cell. A naive method is using some Excel driver (like XlsxWriter) directly. But if you are dealing with a big Excel sheet (or multiple sheets in one file), this approach becomes quite inconvenient. Therefore, this article presents a library called Portable Spreadsheet - it can easily export defined formulas in Python to many formats, including Excel and JSON.

Virtual environments are a beneficial concept that makes your coding much more accessible. There are many tools in Python for creating and managing virtual environments. This article analyses the native virtual environment created using the venv command on Linux and Windows. Another simplified and more user-friendly approach is to use Anaconda software (using command conda) or using IDE like PyCharm directly.

Many open-source tools can help with the prediction of renewable energy production. For example, the Python library PVLIB for computation of production solar of photovoltaic panels implements essential scientific methods. However, if it comes to predicting energy from wind turbines, there is no similar library - but the computation itself is sufficiently simple - so everyone can write their methods. There is also a question of how precise predictions are, which requires a probabilistic approach.

One of the common challenges of on-demand large datasets processing is the time required for data processing. The naive approach computes operation results on the same component as serves results (and horizontally scale the system when overloaded). The more complex (and generally speaking the only correct) approach is to use some asynchronous task queue based on distributed message passing (message broker). Another option is to stream data continuously as they come.

Code profiling is one of the most important parts of the code optimisation process. Generally speaking, it is the program analysis where we are mainly interested in memory usage and time complexity of the program (or each of its parts). We can also be interested in how often we use some function (or how often we call it, what is the latency). There are a lot of principles and tools to perform these tasks. This article is mainly focused on stack tracing, line-by-line code analysis tools, and function call graphs.

The famous library xarray is more or less standard in the data analyst branch. During the last few years, it has become prevalent. It has been deployed in many projects, often without any careful decision if it is needed or if there is any other way to treat the issue. One of the essential xarray functionality is the reading and writing of the NetCDF files. This is also a focus of this article where at the end, you should be capable of making the informed decision whether or not to use xarray.