Chapter 6. Data Evaluation

As soon as Recording in the status box (see Section 3.2, “Global Control Elements”) is switched on, Meteo-40 records and stores measurement data. In order to check current data, the web interface offers the possibility to have a closed look at live data (see Section 6.2, “Live Data Page”). In the Data menu, measurement data can be downloaded (see Section 6.7, “Access to Source Data” and Section 4.2, “System Administration”) and statistics can be configured, which will be saved as CSV file.

This page displays the currently measured data. The data is presented in a table and in a line chart.

Click on Start to display currently measured values. Stop ends displaying data. The table shows the total amount of available data. In comparison, the line chart can be customized by choosing sensors and connected channels.

	Note
	Enable JavaScript in your web browser to display live data in a chart.

In the Data → Statistics menu CSV files with statistics are listed and can be downloaded. Meteo-40 shows a calendar with available statistics. Click on a month to highlight days with available CSV files. To navigate through the calendar, click on a month or use the arrows to move one month ahead or back. Click on a highlighted day to list all CSV files, which have been generated for this day.

To view the available CSV files for a whole month, click on Show month; for a complete year, click on Show year. You can jump between the years and months by clicking on a month resp. year in the calendar.

All CSV files for the selected period are listed in a table, the file name refers to the time interval covered. For each file, a link to download the corresponding signature, its size in bytes, the methods through which the file has been shared and a link to visualize the matching source data are available. To display the source data corresponding to a file, click on the arrow in the Source column.

Below the list of CSV files it is indicated how many statistics CSV files are currently visible, from the available files in the data logger, e.g., 1 of 53 files.

If CSV files have been uploaded to an AmmonitOR project, sent per Email or copied to a configured SCP/FTP server, it will be tagged by a checkmark in the corresponding box.

To manually download or delete one or more files, select them and Click Download selected or Delete selected respectively. For the download, the CSV files will be compressed in one ZIP file. If you want a certain file to be sent or copied again, you can also mark the selected files as not sent with the option Mark as not sent

	Important
	Deleting CSV statistic files does not delete the corresponding source data, statistic files can be generated again as long as the matching source data are still in the Source data storage.

To immediately generate statistic files form the available source data, click on Generate now. Meteo-40 will generate new files for the configured file interval, containing a line with the calculated statistics for each statistics interval. For the standard configuration of daily file interval and ten minutes statistics interval, there will be 144 lines.

If Generate partial files is selected, not only the files for a complete file interval will be generated but also a file with the residuary data until the moment of the generation. This setting is only recommended for testing porpouses.

Figure 6.1. Downloading Statistics

6.3.1. Configuring Statistics and CSV files

In order to configure statistics and CSV files, go to the Data → Statistics menu. You can define content and interval of the statistics.

Meteo-40 automatically suggests the standard statistics for each evaluation and channel. Additional statistics can be selected manually.

Click on Select statistics in the Configure statistics section to create the statistics, e.g., average, maximum, minimum, standard deviation, count, sum and median. Tick checkboxes to include evaluators / channels in the statistics. Remove checkboxes to exclude evaluators / channels. Additionally, Meteo-40 displays internal current and internal voltage in the CSV files.

Figure 6.2. Configuring the Statistics

The options are:

Select statistics

Select individual statistics for every evaluator and channel. See Figure 6.4, “Statistics Configuration Page”.

Statistics interval for files

Defines over which time interval statistics are calculated, generating a new line in the CSV file. All current data that has been collected since the last generated file is considered to generate new aggregate data. Several intervals are available from 1/4 s to 1 h.

File interval

Determines, how often a new aggregate data file in CSV format has to be created. Selecting Hourly, produces one file per hour that includes data from the last hour. Daily generates only one file per day, etc. Several intervals are available from every 5 minutes to weekly. The file interval is also applied to Gust data files.

Generate partial files

Generate CSV files up to the current moment, even if the file interval has not yet finished.

Statistics interval for SCADA, Switch manager and snapshots

Defines over which time interval statistics are calculated. The calculation occurs at the end of the interval and the data are available over SCADA, applied to Switch manager or sent/copied as a snapshot. All current data from that period is considered to generate new aggregate data. Several intervals are available from 1/4 s to 1 h.

SCADA Summary interval

Applied only to the Summary statistic of the SCADA Modbus server. Typically used to obtain a daily summary of sun status or precipitation measurements.

Secondary Files

Suplementary statistics files can be generated with a different interval for the statistics calculation. Secondary statistics can be configured here. See Section 6.4, “Secondary and Tertiary Statistics Files”.

Tertiary Files

A further suplementary statistics interval can be configured here. See Section 6.4, “Secondary and Tertiary Statistics Files”.

	Warning
	Be careful with setting the statistics interval and the file interval! Low values for the statistics interval lead to huge statistics files, which you might need to remove manually once in a while. Low values for the file interval lead to many small CSV files, which might be hard to handle.

For solar-related evaluations Meteo-40 estimates certain values in addition to the measured values, i.e., estimated sun status, estimated direct normal irradiation. In order to create the estimated values, the GPS location of the measurement station has to be entered in the System → Administration menu. By default the estimated values are not included in the CSV file. If required, select the values for the statistics.

Meteo-40 uses vectorial calculations for wind direction, i.e., vectorial average, direction maximum, direction minimum, and vectorial standard deviation (Yamartino method). Hence, the calculated minimum value for the wind direction can be higher than the maximum in case of wind from northern directions.

Equation 6.1. Calculation of wind direction average

d = atan2( sum(sin(θ)),sum(cos(θ)))

The equation for the average wind direction d is given by Equation 6.1, “Calculation of wind direction average”. If d is less than zero, 2π is added.

Figure 6.3. Vectorial Calculation for the Wind Direction

Figure 6.4. Statistics Configuration Page

Press Save to finish the configuration of statistical values.

6.3.2. Maintenance Options

Under Maintenace we provide some options for trouble-shooting in case of CSV files generation problems.

Automatic system storage clean-up tasks are performed on CECS start in order to preserve the correct operation of the system. If more than 20000 CSV files are present on startup, all CSV files which can be identified as shared (via AmmonitOR, Email, SCP/FTP or USB flash-drive copy) are automatically deleted. You can also force the deletion of all already shared files by pressing the Delete sent statistics button.

Previous to generating a new CSV file the state of the storage is checked as well. If the available storage was insufficient or the number of CSV files was greater than 20000, the oldest CSV files which have already been shared are deleted.

Important

If none of the listed sharing options is used (AmmonitOR, Email, SCP/FTP or USB flash-drive copy), the data logger will not remove any existing files. This can lead to not generating any new files in case of memory shortage or even to a system malfunction in case of very large amount of files (over 40000). For this use case, you can choose to allow the deletion of the oldest files in order to free space by activating the option Automatically delete oldest files if needed.

The Fix checked files button will inspect all existing files, search for known incidences and regenerate the files if needed. This option is only available if the data logger detects any suspicious files. CSV fiiles which have been replaced with an empty file as described in Delete sent statistics will be marked as suspicious.

Sometimes it is necessary, to have additional statistics on top of the standard. E.g. with a different statistics interval or a different selection of values. You can activate the supplementary generation of statistics files with two different configurations. The files will be listed in a similar table as the main statistic files (see Figure 6.5, “Secondary Statistics Page” and Figure 6.6, “Tertiary Statistics Page”). If the files are generated, they will be automatically sent per Email or AmmonitOR additionally to the main statistic files.

	Warning
	Beware, that having more CSV files is more work for the data logger, more time and energy is consumed fills the memory of the data logger faster more bandwidth for sending is used

6.4.1. Secondary Statistics Files

Select Generate additional files with secondary interval to activate the generation of the secondary files. You can configure a statistics interval different from the main statistics interval. To select the statistics which will be calculated and included in the secondary files, clik on Select statistics.

Figure 6.5. Secondary Statistics Page

6.4.2. Tertiary Statistics Files

Select Generate additional files with tertiary interval to activate the generation of the tertiary files. You can configure an statistics interval different from the main and the secondary statistics interval. To select the statistics which will be calculated and included in the secondary files, clik on Select statistics.

Figure 6.6. Tertiary Statistics Page

Detecting extreme wind events or gusts can be interesting to compare different geographical terrains or to estimate wind turbines load. Meteo-40 offers the possiblity to genere extra CSV files including the sensible measurement values with the highest available measurement rate. To generate gusts data files you must activate this option in menu Data → Gusts.

One gusts CSV file will be generated per day (or the configured file interval in Data → Statistics menu) including all wind gusts of that period. If no gust was registered during the period, an empty file is generated. Empty files will not be visible in the Gust data files calendar but will be listed beneath.

To be considered a gust, the selected trigger evaluation must exceed the configured threshold value a minimum number of consecutive repetitons ( minimum repetitions parameter) and not more than the selected maximum repetitions. The gust as well as the previous and posterior number of seconds specified by time overhang parameter are included in the gusts CSV file (see .

Figure 6.7. Gust Data Calendar

Not every existing measurement is related to gusts analysis. The relevant evaluations must be selected. Press the Select evaluations button to access the evaluations selection site and select which evaluations must be included in gusts CSV files.

Figure 6.8. Gust Evaluations Selection

	Important
	If the gusts data generation is activated in a running project, all existing source data will be analyzed and gusts CSV files will be generated. This process could take several hours.

Meteo-40 stores measurement data in standard Comma-Separated Value (CSV) format with appended information in ini file format. The CSV file is documented in RFC 4180 and well explained, e.g., in Wikipedia. Ini files are described, e.g., in Wikipedia as well. CSV files can easily be read by many programs and software tools, as is the ini file format.

	Note
	In almost all cases, the CSV files created by Meteo-40 are compressed using GNU gzip with the typical file extension .gz. gzip compressed files can be decompressed on all operating systems, see, e.g., Wikipedia for further details.

[Section 1]
key1=value
key2=value
[Section 2]
key1=value

6.6.2. CSV import in LibreOffice or OpenOffice

	Tip
	Use LibreOffice Calc or OpenOffice.org Calc. These are free, professional spreadsheet programs to process your CSV files.

The recommended settings to import the CSV file are shown in Figure 6.9, “LibreOffice CSV-file import settings”.

Figure 6.9. LibreOffice CSV-file import settings

6.6.3. CSV Import in Microsoft Excel 2010™

Depending on the version of Microsoft Excel™ the procedure can be slightly different. In general, we recommend using the import assistant of Microsoft Excel™.

	Note
	Save CSV files always in a directory. The described process in Microsoft Excel™ does not work, when files are opened via your email program.

1. Launch Microsoft Excel™ and open the Options window, which can be found in the File menu. Open the Advanced menu. In the Editing options section deselect Use system separators and use .(dot) as decimal separator and ,(comma) as Thousands separator. See figure Figure 6.10, “ Microsoft Excel 2010™ CSV Import (1)”.

   Figure 6.10.  Microsoft Excel 2010™ CSV Import (1)

   
   
   

2. After setting the options, click on the Data tab as shown in Figure 6.11, “ Microsoft Excel 2010™ CSV Import (2)”. In order to import the CSV file, click on From Text in the group Get External Data, locate and double-click the file you want to import. Follow the instructions in the Text Import Wizard.

   Figure 6.11.  Microsoft Excel 2010™ CSV Import (2)

   
   
   

3. In the first step of the wizard choose Delimited as Original data type and select Unicode (UTF-8) for the file origin. See Figure 6.12, “ Microsoft Excel 2010™ CSV Import (3)”.

   Figure 6.12.  Microsoft Excel 2010™ CSV Import (3)

   
   
   

4. In next step define Comma as delimiter and " as text qualifier as shown in Figure 6.13, “ Microsoft Excel 2010™ CSV Import (4)”.

   Figure 6.13.  Microsoft Excel 2010™ CSV Import (4)

   
   
   

5. In the last step of the wizard, select Text for the data format columns. See Figure 6.14, “ Microsoft Excel 2010™ CSV Import (5)”.

   Figure 6.14.  Microsoft Excel 2010™ CSV Import (5)

   
   
   

6. By clicking on Finish the file opens in your selected sheet. See Figure 6.15, “ Microsoft Excel 2010™ CSV Import (6)”.

   Figure 6.15.  Microsoft Excel 2010™ CSV Import (6)

   
   
   

7. In order to change the cell format of the date and time columns from text to date time format, highlight the column(s) and open Format Cells in the Numbers group on the Home tab. Select Custom and enter YYYY-MM-DD hh:mm:ss.000 as shown in Figure 6.16, “ Microsoft Excel 2010™ Format Cells”.

   Figure 6.16.  Microsoft Excel 2010™ Format Cells

   
   
   

Source data contains all electrically measured values neither interpreted nor with any statistical evaluation applied. If source data is available and the used sensors are known, it is possible to re-evaluate a complete measurement campaign.

	Important
	Source data is only recorded, when recording is switched on!

The source data storage is organized as a ring buffer, which means that the oldest data will be overwritten, when the storage is filled entirely. For compactness reasons data is saved in a binary format and cannot be read directly. However, you can view or download data.

To view or download source data, go to the Data → Source menu. Meteo-40 displays all months and related days, on which source data is available. As shown in Figure 6.17, “Screenshot of the Source Data Page”, select month, day and time period to display related source data in the table below.

Figure 6.17. Screenshot of the Source Data Page

The displayed source data can be downloaded as CSV file by clicking on the buttons below the table.

	Warning
	The volume of downloaded source data can be huge, e.g., a filled 2 GB memory can lead to a 10 GB CSV file.