wip jlpm package manager

squash without the embedded yarn-*.js

Fix clean:slate script and update docs

clean the yarn cache

Try with no concurrency

delete the lock file

Try mutex

try without prefer-offline

try with no options

Restore prefer-offline

Force npm on pack

wip yarn updates

Use yarn legacy and finish commands

Use a function for the app version

Code order

Add background task handling

add handling of abort callback

Update docs

Add option to skip watch build

Update docs

cleanup

Add the yarn lock update to the release workflow

cleanup

wip update tests

switch to 1.3.2 legacy to avoid cafile error

cleanup

wip updates

wip

wip

Cleanup and finish tests

fix handling of node_modules

wip

Suppress build warning if watching

cleanup

reinstate build command selection

Package integrity updates

Release [email protected]

Release [email protected]

Update the app versions for themes

cleanup

Clean up process behavior

Update lock files

Streamline watch modes

fix detection of sys vs app dir

remove jupyter widgets from lockfile

try disabling python tests

try with one test

try without finally

cleanup

cleanup

Add check for nodejs

Fix inclusion of yarnrc file

Do not override lockfile if the version has not changed

Use more explicit handling of lock file

Fix travis script

Fix rmtree handling on appveyor

Clean up handling of yarn.lock update

Fix handling of appveyor errors

add pytest-check-links to test deps

Add note about shasum [ci skip]

some prose fixes, fix up test extension

use binary for file read/write

Catch console errors in selenium test

fix test

Copy assets on emit and cleanup

fix syntax

Use jlpm in appveyor

fix webpack plugin

fix handling of tar directory

remove use of deprecated ioloop.instance

wip clean up subprocess handling

clean up and streamline jlpm command

Add proces.py

Clean up build handler and tests

Convert to class and add shutdown handling

cleanup

docs

Cleanup

fix jlpm and process return values

Fix ioloop lookup

Clean up handling of jlpmapp

use pty for rich output

cleanup

Run pdb on failure

cleanup

fixes

Fix dev watch mode

Fix settings tests

Fix dev watch mode on Windows

Update lock file

Add a process app

refactor

Debug travis test

Use pexpect and winpty

fix handling of popenspawn

Add type guard

docs

refactor the process helper

update yarn.lock

Fix test runners

Add python 2 handling

cleanup

debugging

Fix python 2 handling and add more debug out

more debug out

more debug out

simplified processes

wait for all of webpack output before starting

cleanup

docs

cleanup

Add windows specific requirements

always decode

wip

clean up decoding

wip

Fix encoding on posix

remove pexpect dependency

Handle other stdout encodings

cleanup and debug

debug out on appveyor

build handler cleanup

make builds silent

cleanup

switch back to popen

more travis debug

Try limiting test scope

Make starting a new session configurable

Do not start a new process group in process app

Fix argument passing

Try with no pty

Clean up handling of pty and starting a new session

More cleanup of pty handling

Prefer to use a pty

try travis with no pty

Make using a pty configurable and use it in when building

Use shell on windows

Try without starting session

Try with no pty but with new session

try only the build test on appveyor

Try with no actual build

Only capture stdout when watching

fix appveyor script

Clean up handling of subprocess args

add startup error handling

Reinstate travis test

Uniformly handle the command name

Clean up handling of node check

docs

try with just settings api

cleanup

more appveyor tests

fix appveyor script

Improve handling of process errors

Try not loading the server extension

Cleanup handling of process termination

reinstate test

clean up url resolver test

Clean up selenium check

Clean up selenium check

Clean up selenium check

Revert selenium changes

fix rendermime test

Add debug on each test

export a which command

import subprocess

Clean up the selenium test

make the first test a no-op

avoid copying mock packages

download miniconda to the home directory

more test debug

more test debug

more test debug

test cleanup

syntax fix

more test debug

Fix handling of mock packages

reinstate tests

clean up process app

docs

cleanup

remove debug out

fix handling of el

clean up appveyor

More explicit stop handling

forcible exit

Clean up

Clean up terminate function

do not wait for kernel ready on url resolver shutdown

debug python 2

cleanup

use threadpool for simple task runners

Fix import

Cleanup

docs

Switch to yarn workspaces

Use workspaces config

clean up release script

syntax fix

syntax fix

ci cleanup

add a quick check for buildutils

debug

cleanup

update ci

fix handling of build asset extraction

Use a checked in binary yarn

Add the binary yarn.js file

fix handling of task runner

cleanup

clean up handling of threads

Fix process cleanup

build the packages

wip refactor

clean up process classes

Add an explicit ioloop stop

Revert run-test for python 2 compat

Fix clean:slate to work on windows

add clean slate check to windows

Fix clean script

Do not run clean slate script on appveyor

Use npm for registry, cf lerna/lerna/688

Update yarn.lock

.gitattributes

@@ -0,0 +1 @@
+jupyterlab/yarn.js binary

.travis.yml

@@ -6,8 +6,7 @@ sudo: required
 addons:
   firefox: latest
 cache:
-  directories:
-    - $HOME/.npm
+  yarn: true
 env:
   matrix:
     - GROUP=tests

.yarnrc

@@ -0,0 +1,3 @@
+yarn-path "./jupyterlab/yarn.js"
+workspaces-experimental true
+registry "https://registry.npmjs.org"

CONTRIBUTING.md

@@ -26,9 +26,9 @@ All source code is written in [TypeScript](http://www.typescriptlang.org/Handboo
 ### Installing Node.js and npm
 
 Building the JupyterLab from its GitHub source code requires Node.js version
-6+ and Node's package manager, ``npm``.
+4+.
 
-If you use ``conda``, you can get them with:
+If you use ``conda``, you can get it with:
 
 ```bash
 conda install -c conda-forge nodejs
@@ -74,24 +74,30 @@ steps:
 git clone https://github.com/<your-github-username>/jupyterlab.git
 cd jupyterlab
 pip install -e .
-npm install
-npm run build  # Build the dev mode assets
+jlpm install
+jlpm run build  # Build the dev mode assets
 jupyter lab build  # Build the app dir assets
 jupyter serverextension enable --py jupyterlab
 ```
 
 Notes:
 
-* At times, it may be necessary to clean your local repo with the command `npm run clean:slate`.  This will clean the repository, and re-install and 
-rebuild.  If using a local install, you will need to re-run `pip install -e .`
+* The `jlpm` command is a JupyterLab-provided, locked version of the [yarn](https://yarnpkg.com/en/).  If you have `yarn` installed already, you can use
+the `yarn` command when developing, and it will use the local version of `yarn`
+ in `jupyterlab/yarn.js` when run in the repository or a built application
+ directory.
+
+* At times, it may be necessary to clean your local repo with the command `npm run clean:slate`.  This will clean the repository, and re-install and
+rebuild.  Note that we use `npm` in this one instance because the `jlpm`
+command may not be available at the time.
 
 * If `pip` gives a `VersionConflict` error, it usually means that the installed
 version of `jupyterlab_launcher` is out of date. Run `pip install --upgrade
 jupyterlab_launcher` to get the latest version.
 
 * To install JupyterLab in isolation for a single conda/virtual environment, you can add the `--sys-prefix` flag to the extension activation above; this will tie the installation to the `sys.prefix` location of your environment, without writing anything in your user-wide settings area (which are visible to all your envs):
 
-* You can run `npm run build:main:prod` to build more accurate sourcemaps that show the original
+* You can run `jlpm run build:main:prod` to build more accurate sourcemaps that show the original
   Typescript code when debugging. However, it takes a bit longer to build the sources, so is used only to build for production
   by default.
 
@@ -114,16 +120,16 @@ mode, the page will have a red stripe at the top to indicate it is an unreleased
 ### Build and run the tests
 
 ```bash
-npm run build:test
-npm test
+jlpm run build:test
+jplm test
 ```
 
 ### Build and run the stand-alone examples
 
 To install and build the examples in the `examples` directory:
 
 ```bash
-npm run build:examples
+jlpm run build:examples
 ```
 
 To run a specific example, change to the example's directory (i.e.
@@ -164,15 +170,16 @@ npm install --save jupyterlab
 ```bash
 git clone https://github.com/jupyterlab/jupyterlab.git
 cd jupyterlab
-npm install
-npm run build:packages
+pip install -e .
+jlpm install
+jlpm run build:packages
 ```
 
 **Rebuild**
 
 ```bash
-npm run clean
-npm run build:packages
+jlpm run clean
+jlpm run build:packages
 ```
 
 ## The Jupyter Server Extension
@@ -186,7 +193,7 @@ version 4.3 or later is installed.
 When you make a change to JupyterLab npm package source files, run:
 
 ```bash
-npm run build
+jlpm run build
 ```
 
 to build the changes and then refresh your browser to see the changes.
@@ -197,6 +204,9 @@ To have the system build after each source file change, run:
 jupyter lab --dev-mode --watch
 ```
 
+You can also run `jupyter lab --dev-mode --fast-watch` to skip
+the initial build if the assets are already built.  
+
 
 ## Build Utilities
 
@@ -205,22 +215,22 @@ To get a suggested version for a library use `npm run get:dependency foo`.
 To update the version of a library across the repo use `npm run update:dependency foo@^x.x`.
 To remove an unwanted dependency use `npm run remove:dependency foo`.
 
-The key utility is `npm run integrity`, which ensures the integrity of 
+The key utility is `npm run integrity`, which ensures the integrity of
 the packages in the repo. It will:
 
 - Ensure the core package version dependencies match everywhere.
 - Ensure imported packages match dependencies.
 - Ensure a consistent version of all packages.
 - Manage the meta package.
 
-The `packages/metapackage` package is used to build all of the TypeScript 
+The `packages/metapackage` package is used to build all of the TypeScript
 in the repository at once, instead of 50+ individual builds.
 
 The integrity script also allows you to automatically add a dependency for
 a package by importing from it in the TypeScript file, and then running:
 `npm run integrity && npm install` from the repo root.
 
-We also have scripts for creating and removing packages in `packages/`, 
+We also have scripts for creating and removing packages in `packages/`,
 `npm run create:package` and `npm run remove:package`.
 
 

MANIFEST.in

@@ -11,6 +11,8 @@ include setupbase.py
 
 include jupyterlab/package.app.json
 include jupyterlab/*.js
-include jupyterlab/.npmrc
+include jupyterlab/.yarnrc
+include jupyterlab/yarn.app.lock
+include jupyterlab/yarn.lock
 
 prune jupyterlab/tests

RELEASE.md

@@ -22,7 +22,7 @@ major release, to prevent consumers that are using the `^` semver
 requirement from getting a conflict.
 
 ```bash
-npm run publish
+jlpm run publish
 ```
 
 ### Publish the Python package
@@ -31,7 +31,7 @@ npm run publish
 - Prep the static assets for release:
 
 ```bash
-npm run build:static
+jlpm run build:static
 ```
 
 - Commit and tag and push the tag
@@ -117,7 +117,7 @@ shasum -a 256 dist/*.tar.gz
 - Run the following script, where the package is in `/packages/package-folder-name`:
 
 ```bash
-npm run patch:release package-folder-name
+jlpm run patch:release package-folder-name
 ```
 
 - Push the resulting commit and tag.

appveyor.yml

@@ -12,7 +12,7 @@ environment:
 
 # build cache to preserve files/folders between builds
 cache:
-  - '%APPDATA%\npm-cache'  # npm cache
+  - '%LOCALAPPDATA%\\Yarn' # yarn cache
 
 # scripts that run after cloning repository
 install:
@@ -30,24 +30,26 @@ install:
   - ps: Install-Product node $env:nodejs_version
   # Install newer Firefox
   - choco install firefox
-  # Make sure our packages are up to date.
-  - npm update
   # Ensure python and python scripts are from right version:
   - 'SET "PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%"'
   # Install our package:
   - 'pip install --upgrade -e ".[test]"'
-  - 'npm install'
-  - 'npm run build:main'
+  - 'jlpm versions'
+  - 'jlpm config current'
+  - 'jlpm cache list'
+  - 'jlpm install'
+  - 'jlpm run build:main'
   - 'jupyter lab build'
 
 build: off
 
 # to run your custom scripts instead of automatic tests
 test_script:
   # Run integrity first we we see the message.
-  - 'npm run integrity'
+  - 'jlpm run integrity'
   - 'py.test'
-  - 'npm run build'
-  - 'npm run build:test'
-  - 'npm test || npm test || npm test'
+  - 'git status'
+  - 'jlpm run build'
+  - 'jlpm run build:test'
+  - 'jlpm test || jlpm test || jlpm test'
   - 'python -m jupyterlab.selenium_check'

buildutils/src/add-sibling.ts

@@ -58,7 +58,7 @@ fs.writeFileSync(tsconfigPath, JSON.stringify(tsconfig, null, 2) + '\n');
 
 // Update the core jupyterlab build dependencies.
 try {
-  utils.run('npm run integrity');
+  utils.run('jlpm run integrity');
 } catch (e) {
   if (!process.env.TRAVIS_BRANCH) {
     console.error(e);

buildutils/src/build.ts

@@ -18,12 +18,6 @@ namespace Build {
    */
   export
   interface IEnsureOptions {
-    /**
-     * The input directory that contains the root package's assets.
-     * Defaults to the current directory.
-     */
-    input?: string;
-
     /**
      * The output directory where the build assets should reside.
      */
@@ -101,12 +95,12 @@ namespace Build {
    */
   export
   function ensureAssets(options: IEnsureOptions): void {
-    let { input, output, packageNames } = options;
-    input = input || '.';
+    let { output, packageNames } = options;
 
     packageNames.forEach(function(name) {
-      const packageDir = fs.realpathSync(path.join(input, 'node_modules', name));
-      const packageData = require(path.join(packageDir, 'package.json'));
+      const packageDataPath = require.resolve(path.join(name, 'package.json'));
+      const packageDir = path.dirname(packageDataPath);
+      const packageData = require(packageDataPath);
       const extension = normalizeExtension(packageData);
       const { schemaDir, themeDir } = extension;
 

buildutils/src/clean-packages.ts

@@ -11,8 +11,8 @@ import { readJSONFile } from './utils';
 
 // Get all of the packages.
 let basePath = path.resolve('.');
-let lernaConfig = readJSONFile(path.join(basePath, 'lerna.json'));
-let packageConfig = lernaConfig.packages;
+let baseConfig = readJSONFile(path.join(basePath, 'package.json'));
+let packageConfig = baseConfig.workspaces;
 let skipSource = process.argv.indexOf('packages') === -1;
 let skipExamples = process.argv.indexOf('examples') === -1;
 

buildutils/src/create-package.ts

@@ -37,5 +37,5 @@ inquirer.prompt(questions).then(answers => {
   data.name = name;
   data.description = description;
   utils.writePackageData(jsonPath, data);
-  utils.run('npm run integrity');
+  utils.run('jlpm run integrity');
 });

buildutils/src/ensure-repo.ts

@@ -178,7 +178,8 @@ function ensureJupyterlab(): string[] {
 /**
  * Ensure the repo integrity.
  */
-function ensureIntegrity(): void {
+export
+function ensureIntegrity(): boolean {
   let messages: { [key: string]: string[] } = {};
 
   // Pick up all the package versions.
@@ -188,6 +189,7 @@ function ensureIntegrity(): void {
     try {
       data = utils.readJSONFile(path.join(pkgPath, 'package.json'));
     } catch (e) {
+      console.error(e);
       return;
     }
 
@@ -242,15 +244,20 @@ function ensureIntegrity(): void {
   if (Object.keys(messages).length > 0) {
     console.log(JSON.stringify(messages, null, 2));
     if (process.env.TRAVIS_BRANCH || process.env.APPVEYOR) {
-      console.log('\n\nPlease run `npm run integrity` locally and commit the changes');
+      console.log('\n\nPlease run `jlpm run integrity` locally and commit the changes');
       process.exit(1);
     }
-    utils.run('npm install');
+    utils.run('jlpm install');
     console.log('\n\nPlease commit the changes by running:');
     console.log('git commit -a -m "Package integrity updates"');
-  } else {
-    console.log('Repo integrity verified!');
+    return false;
   }
+
+  console.log('Repo integrity verified!');
+  return true;
 }
 
-ensureIntegrity();
+
+if (require.main === module) {
+  ensureIntegrity();
+}

buildutils/src/make-release.ts

@@ -6,22 +6,32 @@
 import * as fs from 'fs-extra';
 import * as path from 'path';
 import * as utils from './utils';
+import { ensureIntegrity } from './ensure-repo';
 
-// Ensure package integrity.
-utils.run('npm run integrity');
+// Start from a clean slate.
+utils.run('npm run clean:slate');
+
+// Ensure integrity.
+if (!ensureIntegrity()) {
+    process.exit(1);
+}
+
+// Build the packages.
+utils.run('npm run build:packages');
 
 // Change to the jupyterlab dir.
 process.chdir(path.join('.', 'jupyterlab'));
 
 // Run a production build with Typescript source maps.
-utils.run('npm run build:prod');
+utils.run('jlpm run build:prod');
 
 // Update the package.app.json file.
 let data = utils.readJSONFile('./package.json');
 data['scripts']['build'] = 'webpack';
 data['scripts']['watch'] = 'webpack --watch';
 data['scripts']['build:prod'] = "webpack --define process.env.NODE_ENV=\"'production'\"";
 data['jupyterlab']['outputDir'] = '..';
+data['jupyterlab']['staticDir'] = '../static';
 data['jupyterlab']['linkedPackages'] = {};
 utils.writePackageData('./package.app.json', data);
 

buildutils/src/patch-release.ts

@@ -27,16 +27,16 @@ if (!fs.existsSync(packagePath)) {
 
 // Perform the patch operations.
 console.log('Patching', target, '...');
-utils.run('npm run build:packages');
-utils.run('npm version patch', { cwd: packagePath });
-utils.run('npm publish', { cwd: packagePath});
+utils.run('jlpm run build:packages');
+utils.run('jlpm version patch', { cwd: packagePath });
+utils.run('jlpm publish', { cwd: packagePath});
 
 // Extract the new package info.
 let data = utils.readJSONFile(path.join(packagePath, 'package.json'));
 let name = data.name;
 let version = data.version;
 
-utils.run('npm run integrity');
+utils.run('jlpm run integrity');
 utils.run('git commit -a -m "Release ' + name + '@' + version + '"');
 utils.run('git tag ' + name + '@' + version);