Weather Collector

Python package for collecting weather measurements.
Explore Usage topic »

Table of Contents

1. Features
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
4. Explanations
5. Restrictions
6. Appreciations
7. Contact

Features

• Fetching world's largest cities from third-party API.
• Fetching weather for provided cities from third-party API and store data into database.
• Easy to configurate cities at cities.json file. Only name is required.
• Making report for last and average temperature for every city.

Built With

Getting Started

Run docker compose.

Prerequisites

• Docker 20.10.21

1. Clone the repo.

   $ git clone [email protected]:MishaVyb/weather-collector.git

2. Define environment variables

     $ cd weather-collector
     $ nano prod.env

   debug = false
   open_weather_key = ...
   POSTGRES_USER = vybornyy
   POSTGRES_PASSWORD = vybornyy
   POSTGRES_DB = default

3. Build and run docker compose by predefined make command.

   $ make app

   WARNING!
   If database connection fails, try again in a few seconds. It could be because postress server is not running yet.

Run as python script.

Prerequisites

• python 3.10.4
• pip

Installation

1. Clone the repo.

   $ git clone [email protected]:MishaVyb/weather-collector.git

2. Activate virtual environment.

   $ cd weather-collector
   $ python3.10 -m venv venv
   $ source venv/bin/activate
   

3. Install requirements.

   (venv) $ pip install -r requirements.txt

4. Migrate database.

   (venv) $ alembic upgrade head

5. Define environment variables

     $ nano debug.env

   debug = true
   open_weather_key = ...

(back to top)

Usage

1. Init cities and run continuously collecting weather.

   $ python3 manage.py collect --initial

   For the beginning, collector looks for cites.json file where list of cities described.

   [
       {"name": "Moscow"},
       {"name": "Istanbul"}
   ]

   If that file does not exist, collector getting the most populated cities from GeoDB API.

   WARNING!
   Weather Collector do not guarantee that received cites is a real the most populated cities on the Earth at current moment. It's better to manually fill cities.json file.

   After that collector begin collecting weather every hour and store all data into database.

2. Change tracked cities.

   Describe cities at cites.json file and call for InitCities service.

   $ python3 manage.py init_cites

   By default cites will be appended to already handled ones. If you want to track only that new list of cities, use --override flag. It seting all existing cities at database not to be tracking for weather collecting anymore.

   $ python3 manage.py init_cites --override

   Or re-fetch cities. This line invokes InitCities with --override flag after fetching.

   $ python3 manage.py fetch_cites --override

   Or re-init and run collecting in one line.

   $ python3 manage.py collect --initial --override

3. Get weather report.

   $ python3 manage.py report
   $ python3 manage.py report --average
   $ python3 manage.py report --latest

4. More options.

   $ python3 manage.py --help

(back to top)

Explanation

1. Database Structure.

   CityModel
   Contains cities which weather collecting for.

   MeasurementModel
   For every weather measurement (API request) associated with the city.

   Open Weather API provides a lot of information about current city weather. Depending on location and current weather situation, some fields could appear some other could not. For that situation we decided to store all response root fields in separate tables.

   Why MainWeatherDataModel table?
   The basic reason for collecting weather is understanding how to cool down company's servers. Therefore, we parsing and store main field that contains current temperature. All other data storing as json at ExtraMeasurementDataModel for any future purposes.

   We may describe other tables to store all the data in relational (SQL) way later, if we will need it.

2. Services Structure.

   BaseService presents basic definition for all other services.
   DBSessionMixin for making operations with database.
   FetchServiceMixin for handling http requests.
   

Restrictions

1. Not async.
   When executing services, all http requests runs synchronously. It takes a lot of time and hold processing execution. It's better to make them in async way to reach more speed.

2. Cities names unique constraint
   When calling for InitCites service, all cities described at cities.json appending to database and there are now checking for repetitions. So database may contains several cities with the same name and location.

   To specify city explicitly, provide location coordinates or country code.

   [
     {"name": "..", "latitude": 31.1, "longitude": 121.4},
     {"name": "..", "countryCode": "BR"}
   ]
   

(back to top)

Appreciations

Great thanks for third-party API services used to handle this package.

• GeoDB - for presenting list of the most populated cities.
• Open Weather - for presenting cities location and their weather.

Contacts

Misha Vybornyy

(back to top)