You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# Architecture Overview - Machine Learning Services with Python
16
+
# Architecture overview for Machine Learning Services with Python
18
17
19
18
This topic provides an overview of how Python is integrated with SQL Server, including the security model, the components in the database engine that support external script execution, and new components that enable interoperability of Python with SQL Server. For details see the linked topics.
20
19
21
20
> [!IMPORTANT]
22
-
> Support for Python is available beginning with SQL Server 2017. This is a pre-release feature and subject to change.
21
+
> Support for Python is available beginning with SQL Server 2017 CTP 2.0. This is a pre-release feature and subject to change.
23
22
24
-
## Python Interoperability
23
+
## Python interoperability
25
24
26
-
SQL Server Machine Learning Services (In-Database) installs the Anaconda distribution of Python, and the Python 3.5 runtime and interpreter. This ensure near-complete compatibility with standard Python solutions. Python runs in a separate process from SQL Server, to ensure that database operations are not compromised.
25
+
SQL Server Machine Learning Services (In-Database) installs the Anaconda distribution of Python, and the Python 3.5 runtime and interpreter. This ensures near-complete compatibility with standard Python solutions. Python runs in a separate process from SQL Server, to guarantee that database operations are not compromised.
27
26
28
-
For more information about the interaction of SQL Server with Python, see [Python Interoperability](/python-interoperability.md)
27
+
For more information about the interaction of SQL Server with Python, see [Python interoperability](/python-interoperability.md)
29
28
30
-
## New Components that Support Python Integration
29
+
## Components that support Python integration
31
30
32
31
The extensibility framework introduced in SQL Server 2016 now supports execution of Python script, through the addition of new language-specific components. These components improve data exchange speed and compression, while providing a secure, high-performance platform for running external scripts.
33
32
@@ -41,12 +40,12 @@ All tasks are secured by Windows job objects or SQL Server security. Data is kep
41
40
42
41
For details, see [Security for Python](../../advanced-analytics/python/security-overview-sql-server-python-services.md)
43
42
44
-
## Resource Governance
43
+
## Resource governance
45
44
46
45
In SQL Server Enterprise Edition, you can use Resource Governor to manage and monitor resource use of external script operations, including R script and Python scripts.
47
46
48
47
For more information, see [Resource Governance for R](../../advanced-analytics/r/resource-governance-for-r-services.md).
49
48
50
-
## See Also
49
+
## Next steps
51
50
52
-
[Run Python using T-SQl](../tutorials/run-python-using-t-sql.md)
51
+
[Run Python using T-SQL](../tutorials/run-python-using-t-sql.md)
Copy file name to clipboardExpand all lines: docs/advanced-analytics/python/new-components-in-sql-server-to-support-python-integration.md
+28-20Lines changed: 28 additions & 20 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,5 +1,5 @@
1
1
---
2
-
title: "New Components for Python Integration | Microsoft Docs"
2
+
title: "Components for Python integration with SQL Server | Microsoft Docs"
3
3
ms.custom: ""
4
4
ms.date: "03/30/2017"
5
5
ms.prod: "sql-server-2016"
@@ -13,13 +13,13 @@ author: "jeannt"
13
13
ms.author: "jeannt"
14
14
manager: "jhubbard"
15
15
---
16
-
# New components in SQL Server to Support Python Integration
16
+
# Components in SQL Server to support Python integration
17
17
18
-
Beginning in SQL Server 2017, Machine Learning Services supports Python as an external language that can be executed from T-SQL, or executed remotely using SQL Server as the compute context.
18
+
Beginning in SQL Server 2017, Machine Learning Services supports Python as an external language that can be executed from T-SQL, or executed remotely using SQL Server as the compute context.
19
19
20
20
This topic describes the components in SQL Server 2017 that support extensibility in general and the Python language specifically.
21
21
22
-
## SQL Server Components and Providers
22
+
## SQL Server components and providers
23
23
24
24
To configure SQL Server 2017 to allow Python script execution is a multi-step process.
25
25
@@ -29,29 +29,34 @@ To configure SQL Server 2017 to allow Python script execution is a multi-step pr
29
29
30
30
Additional steps might be required to support remote script execution.
31
31
32
-
For more information, see [Set up Machine LEarnign Services](setup-python-machine-learning-services.md)
32
+
For more information, see [Set up Machine Learning Services](setup-python-machine-learning-services.md)
33
33
34
-
### **Launchpad**
34
+
### Launchpad
35
35
36
36
The SQL Server Trusted Launchpad is a service introduced in SQL Server 2016 that manages and executes external scripts, similar to the way that the full-text indexing and query service launches a separate host for processing full-text queries.
37
37
38
-
The Launchpad service will start only trusted launchers that are published by Microsoft, or that have been certified by Microsoft as meeting requirements for performance and resource management.
38
+
The Launchpad service can start only trusted launchers that are published by Microsoft, or that have been certified by Microsoft as meeting requirements for performance and resource management.
39
+
39
40
+ SQL Server 2016 supports R
40
41
+ SQL Server 2017 supports R and Python 3.5
41
42
42
43
The [!INCLUDE[rsql_launchpad_md](../../includes/rsql-launchpad-md.md)] service runs under its own user account.
43
44
45
+
> [!TIP]
46
+
> If you change the account that runs Launchpad, be sure to do so using SQL Server Configuration Manager, to ensure that changes are written to related files.
47
+
44
48
To execute tasks in a specific supported language, the Launchpad gets a secured worker account from the pool, and starts a satellite process to manage the external runtime:
49
+
45
50
+ RLauncher.dll for the R language
46
51
+ Pythonlauncher.dll for Python 3.5
47
52
48
53
Each satellite process inherits the user account of the Launchpad and uses that worker account for the duration of script execution. If the Python script uses parallel processes, they are created under the same, single worker account.
49
54
50
55
For more information about the security context of the Launchpad, see [Security](security-overview-sql-server-python-services.md).
51
56
52
-
### **BxlServer and SQL Satellite**
57
+
### BxlServer and SQL Satellite
53
58
54
-
If you run Process Explorer while a Python job is running, you might see one or multiple instances of BxlServer.
59
+
If you run [Process Explorer](https://technet.microsoft.com/sysinternals/processexplorer.aspx) while a Python job is running, you might see one or multiple instances of BxlServer.
55
60
56
61
**BxlServer** is an executable provided by Microsoft that manages communication between [!INCLUDE[ssNoVersion_md](../../includes/ssnoversion-md.md)] and Python (or R). It creates the Windows job objects that are used to contain external script sessions, provisions secure working folders for each external script job, and uses SQL Satellite to manage data transfer between the external runtime and [!INCLUDE[ssNoVersion_md](../../includes/ssnoversion-md.md)].
57
62
@@ -72,28 +77,29 @@ SQL Satellite uses a custom data format that is optimized for fast data transfer
72
77
73
78
The SQL Satellite can be monitored by using windows extended events (xEvents). For more information, see [Extended Events for R](../../advanced-analytics/r/extended-events-for-sql-server-r-services.md).
74
79
75
-
76
-
## Communication Channels Between Components
80
+
## Communication channels between components
77
81
78
82
+**TCP/IP**
83
+
79
84
By default, internal communications between [!INCLUDE[ssNoVersion_md](../../includes/ssnoversion-md.md)] and the SQL Satellite use TCP/IP.
80
85
81
86
+**Named Pipes**
82
87
83
88
Internal data transport between the BxlServer and [!INCLUDE[ssNoVersion_md](../../includes/ssnoversion-md.md)] through SQL Satellite uses a proprietary, compressed data format to enhance performance. Data is exchanged between Python and BxlServer in BXL format, using Named Pipes.
84
89
85
90
+**ODBC**
91
+
86
92
Communications between external data science clients and the [!INCLUDE[ssNoVersion_md](../../includes/ssnoversion-md.md)] instance use ODBC. The account that sends the script jobs to [!INCLUDE[ssNoVersion_md](../../includes/ssnoversion-md.md)] must have both permissions to connect to the instance and to run external scripts.
87
93
88
94
Additionally, depending on the task, the account might need these permissions:
95
+
89
96
+ Read data used by the job
90
97
+ Write data to tables: for example, when saving results to a table
91
98
+ Create database objects: for example, if saving external script as part of a new stored procedure.
92
99
93
100
When [!INCLUDE[ssNoVersion_md](../../includes/ssnoversion-md.md)] is used as the compute context for Python script executed from a remote client, and the Python executable must retrieve data from an external source, ODBC is used for writeback. [!INCLUDE[ssNoVersion_md](../../includes/ssnoversion-md.md)] will map the identity of the user issuing the remote command to the identity of the user on the current instance, and run the ODBC command using that user's credentials. The connection string needed to perform this ODBC call is obtained from the client code.
94
101
95
-
96
-
## Interaction of Components
102
+
## Interaction of components
97
103
98
104
The following diagrams depict the interaction of SQL Server components with the Python runtime in each of the supported scenarios: running script in-database, and remote execution from a Python terminal, using a SQL Server compute context.
99
105
@@ -111,27 +117,29 @@ After the script has been embedded in the stored procedure, any application that
111
117
4. BxlServer coordinates with the Python runtime to manage exchanges of data, and storage of working results.
112
118
5. SQL Satellite manages communications about related tasks and processes with [!INCLUDE[ssNoVersion_md](../../includes/ssnoversion-md.md)].
113
119
6. BxlServer uses SQL Satellite to communicate status and results to [!INCLUDE[ssNoVersion_md](../../includes/ssnoversion-md.md)].
114
-
7.[!INCLUDE[ssNoVersion_md](../../includes/ssnoversion-md.md)] gets results and closes related tasks and processes.
115
-
120
+
7.[!INCLUDE[ssNoVersion_md](../../includes/ssnoversion-md.md)] gets results and closes related tasks and processes.
116
121
117
122
### Python scripts executed from a remote client
118
123
119
124
You can run Python scripts from a remote computer, such as a laptop, and have them execute in the context of the SQl Server computer, if these conditions are met:
125
+
120
126
+ You design the scripts appropriately
121
-
+ The remote computer has installed the extensibility libraries that are used by Machine Learning Services.
127
+
+ The remote computer has installed the extensibility libraries that are used by Machine Learning Services
122
128
123
129
The following diagram summarizes the overall workflow when scripts are sent from a remote computer.
1. For function that are supported in RevoScalePy functions, the Python runtime calls a linking function, which in turn calls BxlServer.
128
-
2. BxlServer is included with machine Learning Services (In-Database) and runs in a separate process from the Python runtime.
133
+
1. For functions that are supported in **revoscalepy**, the Python runtime calls a linking function, which in turn calls BxlServer.
134
+
2. BxlServer is included with Machine Learning Services (In-Database) and runs in a separate process from the Python runtime.
129
135
3. BxlServer determines the connection target and initiates a connection using ODBC, passing credentials supplied as part of the connection string in the Python script.
130
136
4. BxlServer opens a connection to the [!INCLUDE[ssNoVersion_md](../../includes/ssnoversion-md.md)] instance.
131
-
5. When an external script runtime is called, the Launchpad service is invoked, which is turn starts the appropriate launcher: in this case, PythonLauncher.dll. Thereafter, processing of Python code is handled in a workflow similar to that when Python code is invoked from a stored procedure in T-SQL.
137
+
5. When an external script runtime is called, the Launchpad service is invoked, which in turn starts the appropriate launcher: in this case, PythonLauncher.dll. Thereafter, processing of Python code is handled in a workflow similar to that when Python code is invoked from a stored procedure in T-SQL.
132
138
6. PythonLauncher makes a call to the instance of the Python that is installed on the [!INCLUDE[ssNoVersion_md](../../includes/ssnoversion-md.md)] computer.
133
139
7. Results are returned to BxlServer.
134
140
8. SQL Satellite manages communication with [!INCLUDE[ssNoVersion_md](../../includes/ssnoversion-md.md)] and cleanup of related job objects.
135
141
9.[!INCLUDE[ssNoVersion_md](../../includes/ssnoversion-md.md)] passes results back to the client.
136
142
143
+
## Next steps
137
144
145
+
[Architecture overview for Python in SQL Server](architecture-overview-sql-server-python.md)
0 commit comments