Assignment 05#
Due date: 03.05.2023
This week’s assignment has to be returned in the form a jupyter notebook. Use jupyter notebooks for what they excel at: combining text, code, and figures. Use the markdown format to structure your assignments, and write text when asked to.
As explained in the video, I recommend to download this notebook in order to answer the questions directly in it.
Don’t forget the instructions!
Introduction: live meteorological data#
The institute website provides raw data from several stations around Innsbruck using a live feed at the following addresses:
https://acinn-data.uibk.ac.at/innsbruck/3 for the three days data
https://acinn-data.uibk.ac.at/innsbruck/7 for the seven days data
The datasets for other stations are available, per analogy:
two other stations are available but we’ll have a look at them next time
The data is shared by ACINN under a Creative Commons Attribution-ShareAlike 4.0 International License.
The data is provided in the json format, often used for web applications. Fortunately, this is very easy to read in python. To make your life easier, I have prepared a reading function below.
It is not important that you understand how it works exactly, but you should understand how to call the function as explained in the examples below.
import numpy as np
def acinn_meteo_data(station="innsbruck", ndays=3):
"""Parse live meteorological data from the ACINN servers.
Requires an internet connection!
Parameters
----------
station : str
one of "innsbruck", "obergurgl"
ndays : int
either 3 or 7 days
Returns
-------
(time, air_temp, wind_speed, wind_dir) :
the meteorological data (python lists)
Examples
--------
>>> time, air_temp, wind_speed, wind_dir = acinn_meteo_data()
>>> type(time)
<class 'numpy.ndarray'>
>>> len(time) > 0
True
"""
from urllib.request import Request, urlopen
from datetime import datetime, timedelta
import json
url = f'https://acinn-data.uibk.ac.at/{station}/{ndays}'
req = urlopen(Request(url)).read()
# Read the data
data = json.loads(req.decode('utf-8'))
# Convert the time
time = np.array([datetime(1970, 1, 1) + timedelta(milliseconds=ds) for ds in data['datumsec']])
# Read the data
wind_speed = np.array(data['ff'])
wind_dir = np.array(data['dd'])
air_temp = np.array(data['tl'])
# This is for missing data filtering - more on this in the next class
air_temp[air_temp < -99] = np.NaN
is_finite = np.isfinite(air_temp) & np.isfinite(wind_speed) & np.isfinite(wind_dir)
if np.any(~is_finite):
time = time[is_finite]
air_temp = air_temp[is_finite]
wind_speed = wind_speed[is_finite]
wind_dir = wind_dir[is_finite]
# Return the data
return time, air_temp, wind_speed, wind_dir
# Testing
import doctest
doctest.testmod()
**********************************************************************
File "__main__", line 22, in __main__.acinn_meteo_data
Failed example:
time, air_temp, wind_speed, wind_dir = acinn_meteo_data()
Exception raised:
Traceback (most recent call last):
File "/home/mowglie/.miniconda3/envs/py3/lib/python3.10/doctest.py", line 1350, in __run
exec(compile(example.source, filename, "single",
File "<doctest __main__.acinn_meteo_data[0]>", line 1, in <module>
time, air_temp, wind_speed, wind_dir = acinn_meteo_data()
File "/tmp/ipykernel_123760/3280285223.py", line 33, in acinn_meteo_data
req = urlopen(Request(url)).read()
File "/home/mowglie/.miniconda3/envs/py3/lib/python3.10/urllib/request.py", line 216, in urlopen
return opener.open(url, data, timeout)
File "/home/mowglie/.miniconda3/envs/py3/lib/python3.10/urllib/request.py", line 525, in open
response = meth(req, response)
File "/home/mowglie/.miniconda3/envs/py3/lib/python3.10/urllib/request.py", line 634, in http_response
response = self.parent.error(
File "/home/mowglie/.miniconda3/envs/py3/lib/python3.10/urllib/request.py", line 563, in error
return self._call_chain(*args)
File "/home/mowglie/.miniconda3/envs/py3/lib/python3.10/urllib/request.py", line 496, in _call_chain
result = func(*args)
File "/home/mowglie/.miniconda3/envs/py3/lib/python3.10/urllib/request.py", line 643, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 502: Bad Gateway
**********************************************************************
File "__main__", line 23, in __main__.acinn_meteo_data
Failed example:
type(time)
Exception raised:
Traceback (most recent call last):
File "/home/mowglie/.miniconda3/envs/py3/lib/python3.10/doctest.py", line 1350, in __run
exec(compile(example.source, filename, "single",
File "<doctest __main__.acinn_meteo_data[1]>", line 1, in <module>
type(time)
NameError: name 'time' is not defined
**********************************************************************
File "__main__", line 25, in __main__.acinn_meteo_data
Failed example:
len(time) > 0
Exception raised:
Traceback (most recent call last):
File "/home/mowglie/.miniconda3/envs/py3/lib/python3.10/doctest.py", line 1350, in __run
exec(compile(example.source, filename, "single",
File "<doctest __main__.acinn_meteo_data[2]>", line 1, in <module>
len(time) > 0
NameError: name 'time' is not defined
**********************************************************************
1 items had failures:
3 of 3 in __main__.acinn_meteo_data
***Test Failed*** 3 failures.
TestResults(failed=3, attempted=3)
The function can be called as follows:
time, air_temp, wind_speed, wind_dir = acinn_meteo_data(station='innsbruck', ndays=7)
---------------------------------------------------------------------------
HTTPError Traceback (most recent call last)
/tmp/ipykernel_123760/399944022.py in <cell line: 1>()
----> 1 time, air_temp, wind_speed, wind_dir = acinn_meteo_data(station='innsbruck', ndays=7)
/tmp/ipykernel_123760/3280285223.py in acinn_meteo_data(station, ndays)
31
32 url = f'https://acinn-data.uibk.ac.at/{station}/{ndays}'
---> 33 req = urlopen(Request(url)).read()
34 # Read the data
35 data = json.loads(req.decode('utf-8'))
~/.miniconda3/envs/py3/lib/python3.10/urllib/request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)
214 else:
215 opener = _opener
--> 216 return opener.open(url, data, timeout)
217
218 def install_opener(opener):
~/.miniconda3/envs/py3/lib/python3.10/urllib/request.py in open(self, fullurl, data, timeout)
523 for processor in self.process_response.get(protocol, []):
524 meth = getattr(processor, meth_name)
--> 525 response = meth(req, response)
526
527 return response
~/.miniconda3/envs/py3/lib/python3.10/urllib/request.py in http_response(self, request, response)
632 # request was successfully received, understood, and accepted.
633 if not (200 <= code < 300):
--> 634 response = self.parent.error(
635 'http', request, response, code, msg, hdrs)
636
~/.miniconda3/envs/py3/lib/python3.10/urllib/request.py in error(self, proto, *args)
561 if http_err:
562 args = (dict, 'default', 'http_error_default') + orig_args
--> 563 return self._call_chain(*args)
564
565 # XXX probably also want an abstract factory that knows when it makes
~/.miniconda3/envs/py3/lib/python3.10/urllib/request.py in _call_chain(self, chain, kind, meth_name, *args)
494 for handler in handlers:
495 func = getattr(handler, meth_name)
--> 496 result = func(*args)
497 if result is not None:
498 return result
~/.miniconda3/envs/py3/lib/python3.10/urllib/request.py in http_error_default(self, req, fp, code, msg, hdrs)
641 class HTTPDefaultErrorHandler(BaseHandler):
642 def http_error_default(self, req, fp, code, msg, hdrs):
--> 643 raise HTTPError(req.full_url, code, msg, hdrs, fp)
644
645 class HTTPRedirectHandler(BaseHandler):
HTTPError: HTTP Error 502: Bad Gateway
Exercise 1: data exploration#
Please answer the questions one by one in the cells below.
What is the dtype of the data stored in the air_temp, wind_speed, wind_dir arrays? And their size? And their shape?
# your answer here
What is the dtype of the data stored in the time array? Try to ask the following questions:
print(time.dtype)
print(type(time[0]))
# your answer here
What is the time resolution of the data? To answer this question, you can print the two first elements of the time array.
# your answer here
Exercise 2: plotting#
Create a graph of the temperature and wind in Innsbruck over the last 3 days. Start by noticing that matplotlib understands datetime objects very well, and create a graph which looks approximately like this: example image (note that your plot can be different with respect to dates, colors, etc. But be careful with the axis labels and legends).
# your answer here
Note: date formatting on axes is something which is very difficult to do automatically. If you feel like it you can try things like shown on the example below (optional).
# Define the date format
from matplotlib.dates import DateFormatter, HourLocator
ax.xaxis.set_major_locator(HourLocator(byhour=[0, 12]))
ax.xaxis.set_minor_locator(HourLocator())
ax.xaxis.set_major_formatter(DateFormatter("%b %d, %HH"))
fig.autofmt_xdate()
Now create a graph of the temperature at Innsbruck and Obergurgl over the last 7 days. The graph should look approximately like this: example image (note that your plot can be different with respect to dates, colors, etc. But be careful with the axis labels and legends).
# your answer here
Exercise 3: statistics#
Compute the average, maximum and minimum temperature in Innsbruck over the past 7 days.
# your answer here
Compute the average, maximum and minimum temperature in Innsbruck over the past 24H. For this, assume that the data has no missing timestamp and that the time resolution is constant (this is sometimes wrong but we don’t care here).
# your answer here
Compute the number of times that the 10m wind speed in Innsbruck was above 5 m/s over the past 7 days. Now convert this number to a percentage over the past 7 days.
# your answer here
Compute the number of times that the air temperature in Innsbruck was between 0°C and 10°C over the past 7 days. Now convert this number to a percentage over the past 7 days.
# your answer here
Bonus: average wind direction#
Look at the plot you made about wind direction. Can you think of what problem would occur if you try to compute the average of the wind direction?
Next week, we will address this problem together - so if you are bored you can start to think about how to solve it already ;-)