forked from apache/flink
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[FLINK-2358] [dashboard] Add first stub of angular.js based dashboard.
Add README.md that outlines build instructions.
- Loading branch information
1 parent
e86f451
commit d59cebd
Showing
60 changed files
with
88,359 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,124 @@ | ||
| <!-- | ||
| Licensed to the Apache Software Foundation (ASF) under one | ||
| or more contributor license agreements. See the NOTICE file | ||
| distributed with this work for additional information | ||
| regarding copyright ownership. The ASF licenses this file | ||
| to you under the Apache License, Version 2.0 (the | ||
| "License"); you may not use this file except in compliance | ||
| with the License. You may obtain a copy of the License at | ||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
| Unless required by applicable law or agreed to in writing, | ||
| software distributed under the License is distributed on an | ||
| "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| KIND, either express or implied. See the License for the | ||
| specific language governing permissions and limitations | ||
| under the License. | ||
| --> | ||
|
|
||
| # Apache Flink Web Dashboard | ||
|
|
||
| The web dashboard is work in progress towards the new Flink runtime monitor. In particular, it will | ||
| provide the following missing features of the current web dashboard: | ||
|
|
||
| - Live progress monitoring (via life accumulators) | ||
| - A graph view of the program, as it is executed. | ||
| - A REST style API to access the status of individual jobs. | ||
| - A more modular design | ||
|
|
||
| The web dashboard can be activated by adding/uncommenting the config parameter | ||
| `jobmanager.new-web-frontend: true` in the `conf/fink-conf.yaml` file. | ||
| The dashboard listens at `http://localhost:8082`. | ||
|
|
||
| The new web dashboard is work in progress. It starts an additional HTTP server (by default at port 8082) | ||
| that serves the new web pages and additional background requests. It also relies on the old HTTP server | ||
| for some requests still. | ||
|
|
||
| **NOTE: Many values are placeholders still.** | ||
|
|
||
|
|
||
|
|
||
| ## Server Backend | ||
|
|
||
| The server side of the dashboard is implemented using [Netty](http://netty.io) with | ||
| [Netty Router](https://github.com/sinetja/netty-router) for REST paths. | ||
| The framework has very lightweight dependencies. | ||
|
|
||
| The code is regular Java code built via Maven. To add additional request handlers, follow the | ||
| example of the `org.apache.flink.runtime.webmonitor.handlers.JobSummaryHandler`. | ||
|
|
||
|
|
||
| ## Dashboard Frontend | ||
|
|
||
| The web dashboard is implemented using *angular.js*. The dashboard build infrastructure uses *node.js*. | ||
| The dashboard files are all pre-built, so one can try it out without building it. | ||
|
|
||
|
|
||
| ### Preparing the Build Environment | ||
|
|
||
| Depending on your version of Linux or MacOS, you may need to manually install *node.js* and *bower*. | ||
|
|
||
|
|
||
| #### Ubuntu Linux (12.04 and 14.04) | ||
|
|
||
| Install *node.js* via | ||
| ``` | ||
| sudo add-apt-repository ppa:chris-lea/node.js | ||
| sudo apt-get update | ||
| sudo apt-get -y install nodejs | ||
| ``` | ||
| Verify that the installed version is at least *2.11.3*, via `npm -version`. | ||
|
|
||
|
|
||
| Install *bower* via | ||
| ``` | ||
| sudo npm install -g bower | ||
| ``` | ||
| Verify that the installed version is at least *1.4.1*, via `bower -version`. | ||
|
|
||
|
|
||
| Install *gulp* via | ||
| ``` | ||
| sudo npm install -g gulp | ||
| ``` | ||
| Verify that the installed version is at least *3.9.0*, via `gulp -version`. | ||
|
|
||
|
|
||
| ### Building | ||
|
|
||
| The build process downloads all requires libraries via the *node.js* package management tool (*npm*) | ||
| and the *bower* dependency management tool. The final build tool is *gulp*. | ||
|
|
||
| ``` | ||
| cd flink-runtime-web/web-dashboard | ||
| npm install | ||
| bower install | ||
| gulp | ||
| ``` | ||
|
|
||
| The dashboard code is under `/app`. The result of the build process is under `/web`. | ||
|
|
||
| When building Flink with Maven (in particular the `flink-dist` project), the generated | ||
| files are copied into the build target, to the folder `resources/web-runtime-monitor`. | ||
|
|
||
|
|
||
| ### Developing | ||
|
|
||
| When developing the dashboard, every change needs to recompile the files and update the server: | ||
|
|
||
| ``` | ||
| cd flink-runtime-web/web-dashboard | ||
| gulp | ||
| cd ../../flink-dist | ||
| mvn -DskipTests clean package | ||
| ``` | ||
|
|
||
| To simplify continuous development, one can use a *standalone proxy server*, together with automatic | ||
| re-compilation: | ||
|
|
||
| 1. Edit the file `app/scripts/index.coffee`. Comment/uncomment the lines that define the `webServer`, `jobServer`, and `newServer` URLs. | ||
| 2. Re-compile the files via `gulp`. By calling `gulp watch`, the build-tool autocompiles future changes. | ||
| 3. Start the proxy server via `node server.js` | ||
| 4. Access teh dashboardat [`http://localhost:3000`](http://localhost:3000) | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| doctype html | ||
| html(lang='en') | ||
| head | ||
| meta(charset='utf-8') | ||
| meta(http-equiv='X-UA-Compatible', content='IE=edge') | ||
| meta(name='viewport', content='width=device-width, initial-scale=1') | ||
|
|
||
| title Apache Flink Web Dashboard | ||
|
|
||
| link(rel='stylesheet', href='css/vendor.css', type='text/css') | ||
| link(rel='stylesheet', href='css/index.css', type='text/css') | ||
|
|
||
| script(src="js/vendor.js") | ||
| script(src="js/index.js") | ||
|
|
||
| body(ng-app="flinkApp" ng-strict-di) | ||
| #sidebar(ng-class="{ 'sidebar-visible': sidebarVisible }") | ||
| nav.navbar.navbar-inverse.navbar-static-top | ||
| .navbar-header | ||
| a.navbar-brand(ui-sref="overview") | ||
| img.logo(alt="Apache Flink Dashboard" src="images/flink-logo.png") | ||
| a.navbar-brand.navbar-brand-text(ui-sref="overview") | ||
| | Apache Flink Dashboard | ||
|
|
||
| .navbar.navbar-sidebar | ||
| ul.nav | ||
| li | ||
| a(ui-sref="overview" ui-sref-active='active') | ||
| i.fa.fa-dashboard.fa-fw | ||
| | | ||
| | Overview | ||
| li | ||
| a(ui-sref="running-jobs" ui-sref-active='active') | ||
| i.fa.fa-tasks.fa-fw | ||
| | | ||
| | Running Jobs | ||
| li | ||
| a(ui-sref="completed-jobs" ui-sref-active='active') | ||
| i.fa.fa-server.fa-fw | ||
| | | ||
| | Completed Jobs | ||
|
|
||
| #content(ng-class="{ 'sidebar-visible': sidebarVisible }") | ||
| div(ui-view='main') |
30 changes: 30 additions & 0 deletions
30
flink-runtime-web/web-dashboard/app/partials/jobs/completed-jobs.jade
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| nav.navbar.navbar-default.navbar-fixed-top.navbar-main | ||
| #fold-button.btn.btn-default.navbar-btn.pull-left(ng-click='showSidebar()') | ||
| i.fa.fa-navicon | ||
|
|
||
| .navbar-title | ||
| | Completed Jobs | ||
|
|
||
| #content-inner | ||
| table.table.table-hover.table-clickable | ||
| thead | ||
| tr | ||
| th Start Time | ||
| th End Time | ||
| th Duration | ||
| th Job Name | ||
| th Job ID | ||
| th Tasks | ||
| th Status | ||
|
|
||
| tbody | ||
| tr(ng-repeat="job in jobs" ui-sref="single-job.plan({ jobid: job.jid })") | ||
| td {{job['start-time']}} | ||
| td {{job['end-time']}} | ||
| td {{job.duration}} | ||
| td {{job.name}} | ||
| td {{job.jid}} | ||
| td.label-group | ||
| bs-label(status="{{status}}" ng-repeat="(status, value) in job.operators") {{value}} | ||
| td | ||
| bs-label(status="{{job.state}}") {{job.state}} |
52 changes: 52 additions & 0 deletions
52
flink-runtime-web/web-dashboard/app/partials/jobs/job.exceptions.jade
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| pre | ||
| | org.apache.flink.client.program.ProgramInvocationException: The | ||
| | program execution failed: java.lang.Exception: The data preparation | ||
| | for task 'Reduce (SUM(2))' , caused an error: Error obtaining the | ||
| | sorted input: Thread 'SortMerger spilling thread' terminated due to an | ||
| | exception: The user-defined combiner failed in its 'open()' method. | ||
| | at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:472) | ||
| | at org.apache.flink.runtime.operators.RegularPactTask.invoke(RegularPactTask.java:360) | ||
| | at org.apache.flink.runtime.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:204) | ||
| | at java.lang.Thread.run(Thread.java:745) | ||
| | Caused by: java.lang.RuntimeException: Error obtaining the sorted | ||
| | input: Thread 'SortMerger spilling thread' terminated due to an | ||
| | exception: The user-defined combiner failed in its 'open()' method. | ||
| | at org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:607) | ||
| | at org.apache.flink.runtime.operators.RegularPactTask.getInput(RegularPactTask.java:1133) | ||
| | at org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduceDriver.java:94) | ||
| | at org.apache.flink.runtime.operators.RegularPactTask.run(RegularPactTask.java:466) | ||
| | ... 3 more | ||
| | Caused by: java.io.IOException: Thread 'SortMerger spilling thread' | ||
| | terminated due to an exception: The user-defined combiner failed in | ||
| | its 'open()' method. | ||
| | at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:785) | ||
| | Caused by: java.io.IOException: The user-defined combiner failed in | ||
| | its 'open()' method. | ||
| | at org.apache.flink.runtime.operators.sort.CombiningUnilateralSortMerger$CombiningSpillingThread.go(CombiningUnilateralSortMerger.java:264) | ||
| | at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:781) | ||
| | Caused by: java.lang.IllegalStateException: The runtime context has | ||
| | not been initialized. | ||
| | at org.apache.flink.api.common.functions.AbstractRichFunction.getRuntimeContext(AbstractRichFunction.java:49) | ||
| | at org.apache.flink.api.scala.operators.ScalaAggregateOperator$AggregatingUdf.open(ScalaAggregateOperator.java:261) | ||
| | at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:33) | ||
| | at org.apache.flink.runtime.operators.sort.CombiningUnilateralSortMerger$CombiningSpillingThread.go(CombiningUnilateralSortMerger.java:261) | ||
| | ... 1 more | ||
| | | ||
| | at org.apache.flink.client.program.Client.run(Client.java:345) | ||
| | at org.apache.flink.client.program.Client.run(Client.java:304) | ||
| | at org.apache.flink.client.program.Client.run(Client.java:298) | ||
| | at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:55) | ||
| | at org.apache.flink.api.scala.ExecutionEnvironment.execute(ExecutionEnvironment.scala:530) | ||
| | at com.gmp.MyJob$.main(MyJob.scala:33) | ||
| | at com.gmp.MyJob.main(MyJob.scala) | ||
| | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) | ||
| | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) | ||
| | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) | ||
| | at java.lang.reflect.Method.invoke(Method.java:483) | ||
| | at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437) | ||
| | at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353) | ||
| | at org.apache.flink.client.program.Client.run(Client.java:250) | ||
| | at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:374) | ||
| | at org.apache.flink.client.CliFrontend.run(CliFrontend.java:347) | ||
| | at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1088) | ||
| | at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1115) |
39 changes: 39 additions & 0 deletions
39
flink-runtime-web/web-dashboard/app/partials/jobs/job.jade
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,39 @@ | ||
| nav.navbar.navbar-default.navbar-fixed-top.navbar-main(ng-if="job") | ||
| #fold-button.btn.btn-default.navbar-btn.pull-left(ng-click='showSidebar()') | ||
| i.fa.fa-navicon | ||
|
|
||
| .navbar-title | ||
| indicator-primary(status="{{job.state}}") | ||
| | {{ job.name }} | ||
|
|
||
| .navbar-info.first.last.hidden-xs.hidden-sm | ||
| | {{ job.jid }} | ||
|
|
||
| .navbar-info.first.last | ||
| .label-group | ||
| bs-label(status="{{status}}" ng-repeat="(status, value) in job.operators") {{value}} | ||
|
|
||
| .navbar-info.first.last.hidden-xs.hidden-sm | ||
| | {{ job['start-time'] }} | ||
| | - | ||
| | {{ job['end-time'] }} | ||
|
|
||
| .navbar-info.last.first | ||
| | {{job.duration}} | ||
|
|
||
| nav.navbar.navbar-default.navbar-fixed-top.navbar-main-additional(ng-if="job") | ||
| ul.nav.nav-tabs | ||
| li(ui-sref-active='active') | ||
| a(ui-sref=".plan") Plan | ||
|
|
||
| li(ui-sref-active='active') | ||
| a(ui-sref=".statistics") Job Accumulators / Statistics | ||
|
|
||
| li(ui-sref-active='active') | ||
| a(ui-sref=".timeline") Timeline | ||
|
|
||
| li(ui-sref-active='active') | ||
| a(ui-sref=".exceptions") Exceptions | ||
|
|
||
| #content-inner.has-navbar-main-additional | ||
| div(ui-view="details") |
4 changes: 4 additions & 0 deletions
4
flink-runtime-web/web-dashboard/app/partials/jobs/job.plan.jade
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| .canvas-wrapper | ||
| div.main-canvas(job-plan, plan="plan", jobid="{{jobid}}") | ||
|
|
||
| div(ui-view="node") |
Oops, something went wrong.