In mathematics, a matrix is a rectangular array of numbers, symbols, or expressions, arranged in rows and columns. A matrix could be reduced as a submatrix of a matrix by deleting any collection of rows and/or columns.
There are a number of basic operations that can be applied to modify matrices:
A hash function is any function that can be used to map data of arbitrary size to data of fixed size. One use is a data structure called a hash table, widely used in computer software for rapid data lookup. Hash functions accelerate table or database lookup by detecting duplicated records in a large file.
In computer science, a binary tree is a tree data structure in which each node has at most two children, which are referred to as the left child and the right child.
In computer science, big O notation is used to classify algorithms according to how their running time or space requirements grow as the input size grows. In analytic number theory, big O notation is often used to express a bound on the difference between an arithmetical function and a better understood approximation.
Relational algebra is a family of algebras with a well-founded semantics used for modelling the data stored in relational databases, and defining queries on it.
The main application of relational algebra is providing a theoretical foundation for relational databases, particularly query languages for such databases, chief among which is SQL.
In SQL language, a natural junction between two tables will be done if :
- At least one column has the same name in both tables
- Theses two columns have the same data type
- CHAR (character)
- INT (integer)
- FLOAT (floating point numeric data)
- VARCHAR (long character chain)
SELECT <COLUMNS>
FROM <TABLE_1>
NATURAL JOIN <TABLE_2>
SELECT <COLUMNS>
FROM <TABLE_1>, <TABLE_2>
WHERE TABLE_1.ID = TABLE_2.ID
The INNER JOIN keyword selects records that have matching values in both tables.
SELECT column_name(s)
FROM table1
INNER JOIN table2 ON table1.column_name = table2.column_name;
The FULL OUTER JOIN keyword return all records when there is a match in either left (table1) or right (table2) table records.
SELECT column_name(s)
FROM table1
FULL OUTER JOIN table2 ON table1.column_name = table2.column_name;
The LEFT JOIN keyword returns all records from the left table (table1), and the matched records from the right table (table2). The result is NULL from the right side, if there is no match.
SELECT column_name(s)
FROM table1
LEFT JOIN table2 ON table1.column_name = table2.column_name;
The RIGHT JOIN keyword returns all records from the right table (table2), and the matched records from the left table (table1). The result is NULL from the left side, when there is no match.
SELECT column_name(s)
FROM table1
RIGHT JOIN table2 ON table1.column_name = table2.column_name;
It is impossible for a distributed data store to simultaneously provide more than two out of the following three guarantees:
- Every read receives the most recent write or an error.
- Every request receives a (non-error) response – without guarantee that it contains the most recent write.
- The system continues to operate despite an arbitrary number of messages being dropped (or delayed) by the network between nodes.
In other words, the CAP Theorem states that in the presence of a network partition, one has to choose between consistency and availability. Note that consistency as defined in the CAP Theorem is quite different from the consistency guaranteed in ACID database transactions.
Tabular data are opposed to relational data, like SQL database.
In tabular data, everything is arranged in columns and rows. Every row have the same number of column (except for missing value, which could be substituted by "N/A".
The first line of tabular data is most of the time a header, describing the content of each column.
The most used format of tabular data in data science is CSV_. Every column is surrounded by a character (a tabulation, a coma ..), delimiting this column from its two neighbours.
Entropy is a measure of uncertainty. High entropy means the data has high variance and thus contains a lot of information and/or noise.
For instance, a constant function where f(x) = 4 for all x has no entropy and is easily predictable, has little information, has no noise and can be succinctly represented . Similarly, f(x) = ~4 has some entropy while f(x) = random number is very high entropy due to noise.
A data frame is used for storing data tables. It is a list of vectors of equal length.
A series is a series of data points ordered.
Sharding is horizontal(row wise) database partitioning as opposed to vertical(column wise) partitioning which is Normalization
Why use Sharding?
-
Database systems with large data sets or high throughput applications can challenge the capacity of a single server.
-
Two methods to address the growth : Vertical Scaling and Horizontal Scaling
-
Vertical Scaling
- Involves increasing the capacity of a single server
- But due to technological and economical restrictions, a single machine may not be sufficient for the given workload.
-
Horizontal Scaling
- Involves dividing the dataset and load over multiple servers, adding additional servers to increase capacity as required
- While the overall speed or capacity of a single machine may not be high, each machine handles a subset of the overall workload, potentially providing better efficiency than a single high-speed high-capacity server.
- Idea is to use concepts of Distributed systems to achieve scale
- But it comes with same tradeoffs of increased complexity that comes hand in hand with distributed systems.
- Many Database systems provide Horizontal scaling via Sharding the datasets.
Online analytical processing, or OLAP, is an approach to answering multi-dimensional analytical (MDA) queries swiftly in computing.
OLAP is part of the broader category of business intelligence, which also encompasses relational database, report writing and data mining. Typical applications of OLAP include _business reporting for sales, marketing, management reporting, business process management (BPM), budgeting and forecasting, financial reporting and similar areas, with new applications coming up, such as agriculture.
The term OLAP was created as a slight modification of the traditional database term online transaction processing (OLTP).
-
Extract
- extracting the data from the multiple heterogenous source system(s)
- data validation to confirm whether the data pulled has the correct/expected values in a given domain
-
Transform
- extracted data is fed into a pipeline which applies multiple functions on top of data
- these functions intend to convert the data into the format which is accepted by the end system
- involves cleaning the data to remove noise, anamolies and redudant data
-
Load
- loads the transformed data into the end target
JSON is a language-independent data format. Example describing a person:
{ "firstName": "John", "lastName": "Smith", "isAlive": true, "age": 25, "address": { "streetAddress": "21 2nd Street", "city": "New York", "state": "NY", "postalCode": "10021-3100" }, "phoneNumbers": [ { "type": "home", "number": "212 555-1234" }, { "type": "office", "number": "646 555-4567" }, { "type": "mobile", "number": "123 456-7890" } ], "children": [], "spouse": null }
Extensible Markup Language (XML) is a markup language that defines a set of rules for encoding documents in a format that is both human-readable and machine-readable.
Bloodroot Sanguinaria canadensis 4 Mostly Shady $2.44 031599 Columbine Aquilegia canadensis 3 Mostly Shady $9.37 030699 Marsh Marigold Caltha palustris 4 Mostly Sunny $6.81 051799noSQL is oppsed to relationnal databases (stand for __N__ot __O__nly SQL). Data are not structured and there's no notion of keys between tables.
Any kind of data can be stored in a noSQL database (JSON, CSV, ...) whithout thinking about a complex relationnal scheme.
Commonly used noSQL stacks: Cassandra, MongoDB, Redis, Oracle noSQL ...
Reg ular ex pressions (regex) are commonly used in informatics.
It can be used in a wide range of possibilities :
- Text replacing
- Extract information in a text (email, phone number, etc)
- List files with the .txt extension ..
http://regexr.com/ is a good website for experimenting on Regex. Additionally, https://pythonium.net/regex is another regex tester for Python, with a built-in regex visualizer.
To use them in Python, just import:
import re
A Python Virtual Environment is an isolated space where you can work on your Python projects, separately from your system-installed Python. This is one of the most important tools that most Python developers use.
- Avoids dependency conflicts.
- Allows working on multiple projects with different dependencies.
- Keeps system Python clean and unmodified.
Virtual environments are created by executing the venv module.
On Linux:
python3 -m venv myenv
On Windows:
python -m venv myenv
This creates a folder named myenv, which contains the virtual environment. You can name the folder to anything you like.
On Linux:
source myenv/bin/activate
On Windows:
- Command Prompt (cmd):
myenv\Scripts\activate
- PowerShell:
myenv\Scripts\Activate.ps1
If you get a security error, run this command first:
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser
Now you can install required packages in this Virtual Environment using pip.
When you’re done, deactivate the virtual environment by running:
deactivate
To use the environment again, navigate to the project folder and use the same command that is used to activate the virtual environment.