Skip to content

GitLab

Commit df6145a2 authored by Yuge Zhang's avatar Yuge Zhang
Browse files

Merge branch 'master' of https://github.com/microsoft/nni into dev-retiarii

parents 0f0c6288 f8424a9f
Show whitespace changes
Inline Side-by-side
Showing with 0 additions and 939 deletions
archive-del/README.Makefile.md deleted 100644 → 0
View file @ 0f0c6288
# Makefile and Installation Setup
NNI uses GNU make for building and installing.
The `Makefile` offers standard targets `build`, `install`, and `uninstall`, as well as alternative installation targets for different setup:
* `easy-install`: target for non-expert users, which handles everything automatically;
* `dev-easy-install`: target for developer users, which handles everything automatically;
* `install`: target for NNI normal users, which installs NNI by copying files;
* `dev-install`: target for NNI contributors, which installs NNI as symlinks instead of copying files;
* `pip-install`: target in favor of `setup.py`;
The targets will be detailed later.
## Dependencies
NNI requires at least Node.js, Yarn, and pip to build, while TypeScript is also recommended.
NNI requires Node.js, and all dependency libraries to run.
Required Node.js libraries (including TypeScript) can be installed by Yarn, and required Python libraries can be installed by setuptools or PIP.
For NNI *users*, `make install-dependencies` can be used to install Node.js and Yarn.
This will install Node.js to NNI's installation directory, and install Yarn to `/tmp/nni-yarn`.
This target requires wget to work.
For NNI *developers*, it is recommended to install Node.js and Yarn manually.
See their official sites for installation guide.
## Building NNI
Simply run `make` when dependencies are ready.
## Installation
### Directory Hierarchy
The main parts of NNI project consist of two Node.js modules (`nni_manager`, `webui`) and two Python packages (`nni`, `nnictl`).
By default the Node.js modules are installed to `/usr/share/nni` for all users or installed to `~/.local/nni` for current user.
The Python packages are installed with setuptools and therefore the location depends on Python configuration.
When install as non-priviledged user and virtualenv is not detected, `--user` flag will be used.
In addition, `nnictl` offers a bash completion scripts, which will be installed to `/usr/share/bash-completion/completions` or `~/.bash_completion.d`.
In some configuration, NNI will also install Node.js to `/usr/share/nni`.
All directories mentioned above are configurable. See next section for details.
### Configuration
The `Makefile` uses environment variables to override default settings.
Available variables are listed below:
| Name | Description | Default for normal user | Default for root |
|--------------------|---------------------------------------------------------|-----------------------------------|-------------------------------------------------|
| `BIN_PATH` | Path for executables | `~/.local/bin` | `/usr/bin` |
| `INSTALL_PREFIX` | Path for Node.js modules (a suffix `nni` will be added) | `~/.local` | `/usr/share` |
| `BASH_COMP_SCRIPT` | Path of bash completion script | `~/.bash_completion.d/nnictl` | `/usr/share/bash-completion/completions/nnictl` |
| `PIP_MODE` | Arguments for `python3 setup.py install` | `--user` if `VIRTUAL_ENV` not set | (empty) |
| `NODE_PATH` | Path to install Node.js runtime | `$INSTALL_PREFIX/nni/node` | `$INSTALL_PREFIX/nni/node` |
| `YARN_PATH` | Path to install Yarn | `/tmp/nni-yarn` | `/tmp/nni-yarn` |
| `NODE` | Node.js command | see source file | see source file |
| `YARN` | Yarn command | see source file | see source file |
Note that these variables will influence installation destination as well as generated `nnictl` and `nnimanager` scripts.
If the path to copy files is different from where they will run (e.g. when creating a distro package), please generate `nnictl` and `nnimanager` manually.
### Targets
The workflow of each installation targets is listed below:
| Target | Workflow |
|--------------------------|----------------------------------------------------------------------------|
| `easy-install` | Install dependencies, build, install NNI, and edit `~/.bashrc` |
| `dev-easy-install` | Install dependencies, build, install NNI as symlinks, and edit `~/.bashrc` |
| `install` | Install Python packages, Node.js modules, NNI scripts, and examples |
| `dev-install` | Install Python and Node.js modules as symlinks, then install scripts |
| `pip-install` | Install dependencies, build, install NNI excluding Python packages |
## TODO
* `clean` target
* `test` target
* `lint` target
* Test cases for each target
* Review variables
archive-del/README.Makefile_zh_CN.md deleted 100644 → 0
View file @ 0f0c6288
# Makefile 文件和安装配置
NNI 使用 GNU Make 来生成和安装。
`Makefile` 提供标准的目标 `生成`、`安装` 和 `卸载`, 以及不同设置的安装对象：
* `easy-install`: 针对非专家用户，自动安装所有内容；
* `dev-easy-install`: 针对专家用户，自动安装所有内容；
* `install`: 针对 NNI 普通用户，通过复制文件来安装 NNI;
* `dev-install`: 针对 NNI 开发人员，通过创建符号链接而不是复制文件来安装 NNI;
* `pip-install`: 针对使用 `setup.py` 安装的情况;
下文会有更详细的介绍。
## 依赖项
NNI 依赖于 Node.js, Yarn, 和 PIP 来生成，推荐安装 TypeScript。
运行 NNI 需要 Node.js 以及依赖库。 Node.js 依赖库 (包括 TypeScript) 可以通过 Yarn 来安装， Python 依赖库可以通过 setuptools 或者 PIP 来安装。
NNI *普通用户*可以用 `make install-dependencies` 来安装 Node.js 和 Yarn。 Node.js 会被安装到 NNI 的安装目录，Yarn 会被安装到 `/tmp/nni-yarn`。 安装过程需要 wget。
NNI *开发人员*推荐手工安装 Node.js 和 Yarn。 可浏览相应的官方文档了解安装过程。
## 生成 NNI
当依赖项安装好后，运行 `make` 即可。
## 安装
### 目录层次
NNI 项目主要由两个 Node.js 模块 (`nni_manager`, `webui`) 以及两个 Python 包 (`nni`, `nnictl`) 所组成。
默认情况下，Node.js 模块为所有用户安装在 `/usr/share/nni` 目录下，也可以只为当前用户安装在 `~/.local/nni` 目录下。
Python 包使用 setuptools 安装，所以安装路径依赖于 Python 配置。 如果以非管理员身份安装 ，并且没有虚拟环境的时候，要加上 `--user` 参数。
此外，`nnictl` 提供一个自动完成脚本，会被安装在 `/usr/share/bash-completion/completions` 或 `~/.bash_completion.d` 目录下。
在某些配置情况下，NNI 也会将 Node.js 安装到 `/usr/share/nni` 目录下。
以上所有目录都是可配置的。 可参考下一章节。
### 配置
`Makefile` 中可以用环境变量来替换默认设置。
支持的变量如下：
| 名称 | 说明 | 普通用户下的默认值 | root 下的默认值 |
| ------------------ | ------------------------------ | ---------------------------------- | ----------------------------------------------- |
| `BIN_PATH` | 执行文件路径 | `~/.local/bin` | `/usr/bin` |
| `INSTALL_PREFIX` | Node.js 模块的路径 (最后会加上 `nni`) | `~/.local` | `/usr/share` |
| `BASH_COMP_SCRIPT` | Bash 自动完成脚本的路径 | `~/.bash_completion.d/nnictl` | `/usr/share/bash-completion/completions/nnictl` |
| `PIP_MODE` | `python3 setup.py install` 的参数 | 如果 `VIRTUAL_ENV` 没有设置，会加上 `--user` | (无) |
| `NODE_PATH` | Node.js 运行时的路径 | `$INSTALL_PREFIX/nni/node` | `$INSTALL_PREFIX/nni/node` |
| `YARN_PATH` | Yarn 的安装路径 | `/tmp/nni-yarn` | `/tmp/nni-yarn` |
| `NODE` | Node.js 命令 | 参考源代码 | 参考源代码 |
| `YARN` | Yarn 命令 | 参考源代码 | 参考源代码 |
注意，这些变量不仅会影响安装路径，也会影响生成的 `nnictl` 脚本。 如果复制文件的路径和运行时的不一样（例如，创建发行版本包时），需要手工编辑 `nnictl` 和 `nnimanager`。
### 目标
安装目标的流程如下：
| 目标 | 流程 |
| ------------------ | ------------------------------------------ |
| `easy-install` | 安装依赖项，生成，安装 NNI，并编辑 `~/.bashrc` |
| `dev-easy-install` | 安装依赖项，生成，将 NNI 作为符号链接来安装，并编辑 `~/.bashrc` |
| `install` | 安装 Python 包，Node.js 模块，NNI 脚本和示例 |
| `dev-install` | 将 Python 和 Node.js 模块作为符号链接安装，然后安装 scripts |
| `pip-install` | 安装依赖项，生成，安装 NNI，但不安装 Python 包 |
## TODO
* `clean` 目标
* `test` 目标
* `lint` 目标
* 每个目标的测试用例
* 评审变量
\ No newline at end of file
archive-del/_config.yml deleted 100644 → 0
View file @ 0f0c6288
theme: jekyll-theme-cayman
\ No newline at end of file
archive-del/deployment/pypi/.gitignore deleted 100644 → 0
View file @ 0f0c6288
nni/
nni-yarn/
dist/
build/
*.egg-info/
nni-yarn.tar.gz
node-*.tar.xz
node-*/
archive-del/deployment/pypi/Makefile deleted 100644 → 0
View file @ 0f0c6288
CWD := $(PWD)/
UNAME_S := $(shell uname -s)
ifeq ($(UNAME_S), Linux)
OS_SPEC := linux
WHEEL_SPEC := manylinux1_x86_64
NODE_URL := https://nodejs.org/dist/v10.22.1/node-v10.22.1-linux-x64.tar.xz
else ifeq ($(UNAME_S), Darwin)
OS_SPEC := darwin
WHEEL_SPEC := macosx_10_9_x86_64
NODE_URL := https://nodejs.org/dist/v10.22.1/node-v10.22.1-darwin-x64.tar.xz
else
$(error platform $(UNAME_S) not supported)
endif
TIME_STAMP = $(shell date -u "+%y%m%d%H%M")
NNI_VERSION_VALUE = $(shell git describe --tags --abbrev=0)
NNI_VERSION_VALUE := $(NNI_VERSION_VALUE:v%=%)
# To include time stamp in version value, run:
# make version_ts=true build
ifeq ($(version_ts), true)
NNI_VERSION_VALUE := $(NNI_VERSION_VALUE).$(TIME_STAMP)
endif
NNI_VERSION_TEMPLATE = 999.0.0-developing
NNI_YARN_TARBALL ?= $(CWD)nni-yarn.tar.gz
NNI_YARN_FOLDER ?= $(CWD)nni-yarn
NNI_YARN := PATH=$(CWD)node-$(OS_SPEC)-x64/bin:$${PATH} $(NNI_YARN_FOLDER)/bin/yarn
.PHONY: build
build:
# Building version $(NNI_VERSION_VALUE)
python3 -m pip install --user --upgrade setuptools wheel
wget -q $(NODE_URL) -O $(CWD)node-$(OS_SPEC)-x64.tar.xz
rm -rf $(CWD)node-$(OS_SPEC)-x64
mkdir $(CWD)node-$(OS_SPEC)-x64
tar xf $(CWD)node-$(OS_SPEC)-x64.tar.xz -C node-$(OS_SPEC)-x64 --strip-components 1
wget -q https://github.com/yarnpkg/yarn/releases/download/v1.22.5/yarn-v1.22.5.tar.gz -O $(NNI_YARN_TARBALL)
rm -rf $(NNI_YARN_FOLDER)
mkdir $(NNI_YARN_FOLDER)
tar -xf $(NNI_YARN_TARBALL) -C $(NNI_YARN_FOLDER) --strip-components 1
cd $(CWD)../../src/nni_manager && $(NNI_YARN) && $(NNI_YARN) build
cd $(CWD)../../src/webui && $(NNI_YARN) && $(NNI_YARN) build
cd $(CWD)../../src/nasui && $(NNI_YARN) && $(NNI_YARN) build
rm -rf $(CWD)nni
cp -r $(CWD)../../src/nni_manager/dist $(CWD)nni
cp -r $(CWD)../../src/nni_manager/config $(CWD)nni
cp -r $(CWD)../../src/webui/build $(CWD)nni/static
mkdir -p $(CWD)nni/nasui/build
cp -r $(CWD)../../src/nasui/build/. $(CWD)nni/nasui/build
cp $(CWD)../../src/nasui/server.js $(CWD)nni/nasui
cp $(CWD)../../src/nni_manager/package.json $(CWD)nni
sed -ie 's/$(NNI_VERSION_TEMPLATE)/$(NNI_VERSION_VALUE)/' $(CWD)nni/package.json
cd $(CWD)nni && $(NNI_YARN) --prod
sed -ie 's/$(NNI_VERSION_TEMPLATE)/$(NNI_VERSION_VALUE)/' $(CWD)../../src/sdk/pynni/nni/__init__.py
cd $(CWD) && sed -ie 's/$(NNI_VERSION_TEMPLATE)/$(NNI_VERSION_VALUE)/' setup.py && python3 setup.py bdist_wheel -p $(WHEEL_SPEC)
cd $(CWD)
.PHONY: upload
upload:
python3 -m pip install --user --upgrade twine
python3 -m twine upload dist/*
.PHONY: clean
clean:
rm -rf $(CWD)build
rm -rf $(CWD)dist
rm -rf $(CWD)nni
rm -rf $(CWD)nni.egg-info
rm -rf $(CWD)node-$(OS_SPEC)-x64
archive-del/deployment/pypi/clean.ps1 deleted 100644 → 0
View file @ 0f0c6288
$CWD = $PWD
$OS_SPEC = "windows"
Remove-Item $CWD\build -Recurse -Force
Remove-Item $CWD\dist -Recurse -Force
Remove-Item $CWD\nni -Recurse -Force
Remove-Item $CWD\nni.egg-info -Recurse -Force
Remove-Item $CWD\node-$OS_SPEC -Recurse -Force
\ No newline at end of file
archive-del/deployment/pypi/install.ps1 deleted 100644 → 0
View file @ 0f0c6288
param([int]$version_os, [bool]$version_ts=$false)
[System.Net.ServicePointManager]::DefaultConnectionLimit = 100
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$CWD = $PWD
$OS_SPEC = "windows"
if($version_os -eq 64){
$OS_VERSION = 'win64'
$WHEEL_SPEC = 'win_amd64'
$NODE_URL = 'https://nodejs.org/download/release/v10.22.1/node-v10.22.1-win-x64.zip'
}
else{
$OS_VERSION = 'win32'
$WHEEL_SPEC = 'win32'
$NODE_URL = 'https://nodejs.org/download/release/v10.22.1/node-v10.22.1-win-x86.zip'
}
$TIME_STAMP = date -u "+%y%m%d%H%M"
$NNI_VERSION_VALUE = git describe --tags --abbrev=0
$NNI_VERSION_VALUE = $NNI_VERSION_VALUE.substring(1)
# To include time stamp in version value, run:
# make version_ts=true build
if($version_ts){
$NNI_VERSION_VALUE = "$NNI_VERSION_VALUE.$TIME_STAMP"
}
$NNI_VERSION_TEMPLATE = "999.0.0-developing"
python -m pip install --upgrade setuptools wheel
$NNI_NODE_ZIP = "$CWD\node-$OS_SPEC.zip"
$NNI_NODE_FOLDER = "$CWD\node-$OS_SPEC"
$unzipNodeDir = "node-v*"
(New-Object Net.WebClient).DownloadFile($NODE_URL, $NNI_NODE_ZIP)
if(Test-Path $NNI_NODE_FOLDER){
Remove-Item $NNI_NODE_FOLDER -Recurse -Force
}
Expand-Archive $NNI_NODE_ZIP -DestinationPath $CWD
$unzipNodeDir = Get-ChildItem "$CWD\$unzipNodeDir"
Rename-Item $unzipNodeDir $NNI_NODE_FOLDER
$env:PATH = $NNI_NODE_FOLDER+';'+$env:PATH
cd $CWD\..\..\src\nni_manager
yarn
yarn build
Copy-Item config -Destination .\dist\ -Recurse -Force
cd $CWD\..\..\src\webui
yarn
yarn build
cd $CWD\..\..\src\nasui
yarn
yarn build
if(Test-Path $CWD\nni){
Remove-Item $CWD\nni -r -fo
}
Copy-Item $CWD\..\..\src\nni_manager\dist $CWD\nni -Recurse
Copy-Item $CWD\..\..\src\webui\build $CWD\nni\static -Recurse
Copy-Item $CWD\..\..\src\nasui\build $CWD\nni\nasui -Recurse
Copy-Item $CWD\..\..\src\nasui\server.js $CWD\nni\nasui -Recurse
Copy-Item $CWD\..\..\src\nni_manager\package.json $CWD\nni
(Get-Content $CWD\nni\package.json).replace($NNI_VERSION_TEMPLATE, $NNI_VERSION_VALUE) | Set-Content $CWD\nni\package.json
cd $CWD\nni
yarn --prod
cd $CWD\..\..\src\sdk\pynni\nni
(Get-Content __init__.py).replace($NNI_VERSION_TEMPLATE, $NNI_VERSION_VALUE) | Set-Content __init__.py
cd $CWD
(Get-Content setup.py).replace($NNI_VERSION_TEMPLATE, $NNI_VERSION_VALUE) | Set-Content setup.py
python setup.py bdist_wheel -p $WHEEL_SPEC
archive-del/deployment/pypi/setup.py deleted 100644 → 0
View file @ 0f0c6288
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT license.
import setuptools
import platform
from os import walk, path
os_type = platform.system()
if os_type == 'Linux':
os_name = 'POSIX :: Linux'
elif os_type == 'Darwin':
os_name = 'MacOS'
elif os_type == 'Windows':
os_name = 'Microsoft :: Windows'
else:
raise NotImplementedError('current platform {} not supported'.format(os_type))
data_files = [('bin', ['node-{}-x64/bin/node'.format(os_type.lower())])]
if os_type == 'Windows':
data_files = [('.\Scripts', ['node-{}/node.exe'.format(os_type.lower())])]
for (dirpath, dirnames, filenames) in walk('./nni'):
files = [path.normpath(path.join(dirpath, filename)) for filename in filenames]
data_files.append((path.normpath(dirpath), files))
with open('../../README.md', 'r', encoding="utf-8") as fh:
long_description = fh.read()
setuptools.setup(
name = 'nni',
version = '999.0.0-developing',
author = 'Microsoft NNI team',
author_email = 'nni@microsoft.com',
description = 'Neural Network Intelligence package',
long_description = long_description,
long_description_content_type = 'text/markdown',
license = 'MIT',
url = 'https://github.com/Microsoft/nni',
packages = setuptools.find_packages('../../tools') \
+ setuptools.find_packages('../../src/sdk/pynni', exclude=['tests']) \
+ setuptools.find_packages('../../src/sdk/pycli'),
package_dir = {
'nni_annotation': '../../tools/nni_annotation',
'nni_cmd': '../../tools/nni_cmd',
'nni_trial_tool': '../../tools/nni_trial_tool',
'nni_gpu_tool': '../../tools/nni_gpu_tool',
'nni': '../../src/sdk/pynni/nni',
'nnicli': '../../src/sdk/pycli/nnicli'
},
package_data = {'nni': ['**/requirements.txt']},
python_requires = '>=3.5',
install_requires = [
'schema',
'ruamel.yaml',
'psutil',
'requests',
'responses',
'astor',
'PythonWebHDFS',
'hyperopt==0.1.2',
'json_tricks',
'netifaces',
'numpy',
'scipy',
'coverage',
'colorama',
'scikit-learn>=0.23.2',
'pkginfo',
'websockets'
],
classifiers = [
'Programming Language :: Python :: 3',
'License :: OSI Approved :: MIT License',
'Operating System :: ' + os_name
],
data_files = data_files,
entry_points = {
'console_scripts' : [
'nnictl = nni_cmd.nnictl:parse_args'
]
}
)
archive-del/deployment/pypi/upload.ps1 deleted 100644 → 0
View file @ 0f0c6288
param([bool]$test=$true)
python -m pip install --user --upgrade twine
if($test){
python -m twine upload --repository-url https://test.pypi.org/legacy/ dist/*
}
else{
python -m twine upload dist/*
}
\ No newline at end of file
archive-del/install.ps1 deleted 100644 → 0
View file @ 0f0c6288
param ([Switch] $Development)
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$install_node = $true
$install_yarn = $true
if ([Environment]::Is64BitOperatingSystem) {
$OS_VERSION = 'win64'
$nodeUrl = "https://nodejs.org/download/release/v10.22.1/node-v10.22.1-win-x64.zip"
}
else {
$OS_VERSION = 'win32'
$nodeUrl = "https://nodejs.org/download/release/v10.22.1/node-v10.22.1-win-x86.zip"
}
# nodejs
$yarnUrl = "https://github.com/yarnpkg/yarn/releases/download/v1.22.5/yarn-v1.22.5.tar.gz"
$unzipNodeDir = "node-v*"
$unzipYarnDir = "yarn-v*"
$NNI_DEPENDENCY_FOLDER = [System.IO.Path]::GetTempPath() + $env:USERNAME
$WHICH_PYTHON = where.exe python
if ($WHICH_PYTHON -eq $null) {
throw "Can not find python"
}
else {
$pyVersion = & python -V 2>&1
$pyVersion = ([string]$pyVersion).substring(7, 3)
if ([double]$pyVersion -lt 3.5) {
throw "python version should >= 3.5"
}
}
$WHICH_PIP = where.exe pip
if ($WHICH_PIP -eq $null) {
throw "Can not find pip"
}
$env:PYTHONIOENCODING = "UTF-8"
if ($env:VIRTUAL_ENV) {
$NNI_PYTHON3 = $env:VIRTUAL_ENV + "\Scripts"
$NNI_PKG_FOLDER = $env:VIRTUAL_ENV + "\nni"
$NNI_PYTHON_SCRIPTS = $NNI_PYTHON3
}
else {
$NNI_PYTHON3 = $(python -c 'import site; from pathlib import Path; print(Path(site.getsitepackages()[0]))')
$NNI_PKG_FOLDER = $NNI_PYTHON3 + "\nni"
$NNI_PYTHON_SCRIPTS = $NNI_PYTHON3 + "\Scripts"
}
$PIP_INSTALL = """$NNI_PYTHON3\python"" -m pip install "
if (!(Test-Path $NNI_DEPENDENCY_FOLDER)) {
$null = New-Item $NNI_DEPENDENCY_FOLDER -ItemType Directory
}
$NNI_NODE_ZIP = $NNI_DEPENDENCY_FOLDER + "\nni-node.zip"
$NNI_NODE_FOLDER = $NNI_DEPENDENCY_FOLDER + "\nni-node"
$NNI_YARN_TARBALL = $NNI_DEPENDENCY_FOLDER + "\nni-yarn.tar.gz"
$NNI_YARN_FOLDER = $NNI_DEPENDENCY_FOLDER + "\nni-yarn"
$NNI_YARN = $NNI_YARN_FOLDER + "\bin\yarn"
## Version number
$NNI_VERSION_VALUE = $(git describe --tags)
$NNI_VERSION_TEMPLATE = "999.0.0-developing"
if (!(Test-Path $NNI_NODE_ZIP)) {
Write-Host "Downloading Node..."
(New-Object Net.WebClient).DownloadFile($nodeUrl, $NNI_NODE_ZIP)
}
if (!(Test-Path $NNI_YARN_TARBALL)) {
Write-Host "Downloading Yarn..."
(New-Object Net.WebClient).DownloadFile($yarnUrl, $NNI_YARN_TARBALL)
}
$NNI_YARN_TARBALL = $NNI_YARN_TARBALL -split '\\' -join '\\'
$NNI_DEPENDENCY_FOLDER = $NNI_DEPENDENCY_FOLDER -split '\\' -join '\\'
$SCRIPT_PATH = $NNI_DEPENDENCY_FOLDER + '\extract.py'
$SCRIPT = "import tarfile",
("tar = tarfile.open(""{0}"")" -f $NNI_YARN_TARBALL),
("tar.extractall(""{0}"")" -f $NNI_DEPENDENCY_FOLDER),
"tar.close()"
[System.IO.File]::WriteAllLines($SCRIPT_PATH, $SCRIPT)
Add-Type -AssemblyName System.IO.Compression.FileSystem
function Unzip {
param([string]$zipfile, [string]$outpath)
[System.IO.Compression.ZipFile]::ExtractToDirectory($zipfile, $outpath)
}
if ($install_node) {
### nodejs install
if (!(Test-Path $NNI_NODE_FOLDER)) {
Unzip $NNI_NODE_ZIP $NNI_DEPENDENCY_FOLDER
$unzipNodeDir = Get-ChildItem "$NNI_DEPENDENCY_FOLDER\$unzipNodeDir"
Rename-Item $unzipNodeDir "nni-node"
}
Copy-Item "$NNI_NODE_FOLDER\node.exe" $NNI_PYTHON_SCRIPTS -Recurse -Force
}
if ($install_yarn) {
### yarn install
if (!(Test-Path $NNI_YARN_FOLDER)) {
cmd /C """$NNI_PYTHON3\python""" $SCRIPT_PATH
$unzipYarnDir = Get-ChildItem "$NNI_DEPENDENCY_FOLDER\$unzipYarnDir"
Rename-Item $unzipYarnDir "nni-yarn"
}
}
## install-python-modules:
### Installing Python SDK
(Get-Content setup.py).replace($NNI_VERSION_TEMPLATE, $NNI_VERSION_VALUE) | Set-Content setup.py
if ($Development) {
$PYTHON_BUILD = "build"
# To compat with file and links.
cmd /c if exist "$PYTHON_BUILD" rmdir /s /q $PYTHON_BUILD
$null = New-Item $PYTHON_BUILD -ItemType Directory
$null = New-Item -ItemType Junction -Path "$($PYTHON_BUILD)\nni" -Target "src\sdk\pynni\nni"
$null = New-Item -ItemType Junction -Path "$($PYTHON_BUILD)\nnicli" -Target "src\sdk\pycli\nnicli"
$null = New-Item -ItemType Junction -Path "$($PYTHON_BUILD)\nni_annotation" -Target "tools\nni_annotation"
$null = New-Item -ItemType Junction -Path "$($PYTHON_BUILD)\nni_cmd" -Target "tools\nni_cmd"
$null = New-Item -ItemType Junction -Path "$($PYTHON_BUILD)\nni_trial_tool" -Target "tools\nni_trial_tool"
$null = New-Item -ItemType Junction -Path "$($PYTHON_BUILD)\nni_gpu_tool" -Target "tools\nni_gpu_tool"
Copy-Item setup.py $PYTHON_BUILD
Copy-Item README.md $PYTHON_BUILD
Push-Location build
#update folders in setup file
(Get-Content setup.py).replace("src/sdk/pynni/", "") | Set-Content setup.py
(Get-Content setup.py).replace("src/sdk/pycli/", "") | Set-Content setup.py
(Get-Content setup.py).replace("src/sdk/pynni", ".") | Set-Content setup.py
(Get-Content setup.py).replace("tools/", "") | Set-Content setup.py
# install current folder.
cmd /c $PIP_INSTALL -e .
Pop-Location
}
else {
cmd /c $PIP_INSTALL .
}
# Building NNI Manager
$env:PATH = $NNI_PYTHON_SCRIPTS + ';' + $env:PATH
cd src\nni_manager
cmd /c $NNI_YARN
cmd /c $NNI_YARN build
Copy-Item config -Destination .\dist\ -Recurse -Force
# Building WebUI
# office-ui-fabric-react need longer time. the 180000 is in ms, mean 180 seconds, longer than default 30 seconds.
cd ..\webui
cmd /c $NNI_YARN --network-timeout 180000
cmd /c $NNI_YARN build
# Building NasUI
cd ..\nasui
cmd /c $NNI_YARN --network-timeout 180000
cmd /c $NNI_YARN build
cd ..\..
## install-node-modules
# it needs to remove the whole folder for following copy.
cmd /c if exist "$NNI_PKG_FOLDER" rmdir /s /q $NNI_PKG_FOLDER
$NNI_PKG_FOLDER_STATIC = $NNI_PKG_FOLDER + "\static"
$NASUI_PKG_FOLDER = $NNI_PKG_FOLDER + "\nasui"
cmd /c if exist "src\nni_manager\dist\node_modules" rmdir /s /q src\nni_manager\dist\node_modules
cmd /c if exist "src\nni_manager\dist\static" rmdir /s /q src\nni_manager\dist\static
cmd /c if exist "src\nni_manager\dist\nasui" rmdir /s /q src\nni_manager\dist\nasui
if ($Development) {
$null = New-Item -ItemType Junction -Path $NNI_PKG_FOLDER -Target "src\nni_manager\dist"
$null = New-Item -ItemType Junction -Path "$($NNI_PKG_FOLDER)\node_modules" -Target "src\nni_manager\node_modules"
$null = New-Item -ItemType Junction -Path $NNI_PKG_FOLDER_STATIC -Target "src\webui\build"
$null = New-Item -ItemType Junction -Path $NASUI_PKG_FOLDER -Target "src\nasui\build"
}
else {
Copy-Item "src\nni_manager\dist" $NNI_PKG_FOLDER -Recurse
Copy-Item "src\webui\build" $NNI_PKG_FOLDER_STATIC -Recurse
Copy-Item "src\nasui\build" $NASUI_PKG_FOLDER -Recurse
Copy-Item "src\nni_manager\package.json" $NNI_PKG_FOLDER
$PKG_JSON = $NNI_PKG_FOLDER + "\package.json"
(Get-Content $PKG_JSON).replace($NNI_VERSION_TEMPLATE, $NNI_VERSION_VALUE) | Set-Content $PKG_JSON
cmd /c $NNI_YARN --prod --cwd $NNI_PKG_FOLDER
}
Copy-Item "src\nasui\server.js" $NASUI_PKG_FOLDER
archive-del/install.sh deleted 100644 → 0
View file @ 0f0c6288
#!/bin/bash
make easy-install
archive-del/sdk/pycli/setup.py deleted 100644 → 0
View file @ 0f0c6288
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT license.
import setuptools
setuptools.setup(
name='nnicli',
version='999.0.0-developing',
packages=setuptools.find_packages(),
python_requires='>=3.6',
install_requires=[
'requests'
],
author='Microsoft NNI Team',
author_email='nni@microsoft.com',
description='nnicli for Neural Network Intelligence project',
license='MIT',
url='https://github.com/Microsoft/nni',
)
archive-del/sdk/pynni/.gitignore deleted 100644 → 0
View file @ 0f0c6288
dist/
build/
*.egg-info/
\ No newline at end of file
archive-del/sdk/pynni/pylintrc deleted 100644 → 0
View file @ 0f0c6288
# This pylintrc file is a little more strick than the one in root of code directory
# SDK source MUST pass lint rules in top level directory, and SHOULD pass rules here
[SETTINGS]
max-line-length=140
disable =
missing-docstring,
invalid-name, # C0103
no-member, # E1101: sometimes pylint cannot detect members correctly due to a bug
c-extension-no-member, # I1101
no-self-use, # R0201: many functions in this SDK are designed for override
duplicate-code, # R0801
too-many-instance-attributes, # R0902
too-few-public-methods, # R0903
too-many-public-methods, # R0904
too-many-return-statements, # R0911
too-many-branches, # R0912
too-many-arguments, # R0913
too-many-locals, # R0914
too-many-statements, # R0915
too-many-nested-blocks, # R1702
no-else-return, # R1705
chained-comparison, # R1716
no-else-raise, # R1720
protected-access, # W0212: underscore variables may be protected by whole SDK instead of single module
arguments-differ, # W0221: pylint cannot handle *args and **kwargs
super-init-not-called, # W0231: some interface classes do not expect users to call init
useless-super-delegation, # W0235: derived init may have different docstring
global-statement, # W0603: globals are useful to hide SDK internal states from user
unused-argument, # W0613: many functions in this SDK are designed for override
broad-except, # W0703: the SDK commonly catch exceptions to report error
fixme, # W0511
ignore-patterns=test.*.py
# List of members which are set dynamically and missed by pylint inference
generated-members=numpy.*,torch.*
archive-del/sdk/pynni/requirements.txt deleted 100644 → 0
View file @ 0f0c6288
# core
json_tricks
websockets
# hyperopt tuner
numpy
scipy
hyperopt==0.1.2
# metis tuner
scikit-learn>=0.23.2
archive-del/sdk/pynni/setup.py deleted 100644 → 0
View file @ 0f0c6288
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT license.
import os
import setuptools
def read(fname):
return open(os.path.join(os.path.dirname(__file__), fname), encoding='utf-8').read()
setuptools.setup(
name = 'nni-sdk',
version = '999.0.0-developing',
packages = setuptools.find_packages(exclude=['tests']),
python_requires = '>=3.6',
install_requires = [
'hyperopt==0.1.2',
'json_tricks',
'numpy',
'scipy',
'coverage'
],
package_data = {'nni': ['**/requirements.txt']},
test_suite = 'tests',
author = 'Microsoft NNI Team',
author_email = 'nni@microsoft.com',
description = 'Python SDK for Neural Network Intelligence project',
license = 'MIT',
url = 'https://github.com/Microsoft/nni',
long_description = read('README.md')
)
archive-del/sdk/pynni/ut.sh deleted 100644 → 0
View file @ 0f0c6288
#!/bin/bash
coverage run setup.py test
coverage html
archive-del/tools/bash-completion deleted 100644 → 0
View file @ 0f0c6288
# list of commands/arguments
__nnictl_cmds="create resume view update stop trial experiment platform webui config log package tensorboard top ss_gen --version"
__nnictl_create_cmds="--config --port --debug --foreground"
__nnictl_resume_cmds="--port --debug --foreground"
__nnictl_view_cmds="--port"
__nnictl_update_cmds="searchspace concurrency duration trialnum"
__nnictl_update_searchspace_cmds="--filename"
__nnictl_update_concurrency_cmds="--value"
__nnictl_update_duration_cmds="--value"
__nnictl_update_trialnum_cmds="--value"
__nnictl_stop_cmds="--port --all"
__nnictl_trial_cmds="ls kill"
__nnictl_trial_ls_cmds="--head --tail"
__nnictl_trial_kill_cmds="--trial_id"
__nnictl_experiment_cmds="show status list delete export import save load"
__nnictl_experiment_list_cmds="--all"
__nnictl_experiment_delete_cmds="--all"
__nnictl_experiment_export_cmds="--filename --type --intermediate"
__nnictl_experiment_import_cmds="--filename"
__nnictl_experiment_save_cmds="--path --saveCodeDir"
__nnictl_experiment_load_cmds="--path --codeDir --logDir"
__nnictl_platform_cmds="clean"
__nnictl_platform_clean_cmds="--config"
__nnictl_webui_cmds="url"
__nnictl_config_cmds="show"
__nnictl_log_cmds="stdout stderr trial"
__nnictl_log_stdout_cmds="--tail --head --path"
__nnictl_log_stderr_cmds="--tail --head --path"
__nnictl_log_trial_cmds="--trial_id"
__nnictl_package_cmds="install show list uninstall"
__nnictl_package_install_cmds="--name"
__nnictl_package_list_cmds="--all"
__nnictl_tensorboard_cmds="start stop"
__nnictl_tensorboard_start_cmds="--trial_id --port"
__nnictl_top_cmds="--time"
__nnictl_ss_gen_cmds="--trial_command --trial_dir --file"
# list of commands that accept an experiment ID as second argument
__nnictl_2nd_expid_cmds=" resume view stop top "
# list of commands that accept an experiment ID as third argument
__nnictl_3rd_expid_cmds=" update trial experiment webui config log tensorboard "
# remove already set arguments from candidates
__nnictl_remain_args()
{
local ret=${!1} # ret = $__nnictl_xxx_cmds
# prevent that "--trial_id" changes to "--_id" in this situation: "nnictl trial kill --trial_id"
if [[ ${ret} != "--trial_id" ]]; then
for arg in "${COMP_WORDS[@]::${#COMP_WORDS[@]}-1}"; do
local ret=${ret/$arg/} # remove it from $ret
done
fi
echo $ret
}
# complete files with specific extension
__nnictl_complete_extension()
{
COMPREPLY=($(compgen -f -X "!*.$1" -- ${COMP_WORDS[-1]}))
if [[ -z "${COMPREPLY[*]}" ]]; then
# if there is no matching file here, search in sub-directories
COMPREPLY=($(compgen -d -S "/" -- ${COMP_WORDS[-1]}))
compopt -o nospace
fi
}
_nnictl()
{
local cur=${COMP_WORDS[-1]}
local last=${COMP_WORDS[-2]}
if [[ ${#COMP_WORDS[@]} -eq 2 ]]; then
# completing first argument from __nnictl_cmds
COMPREPLY=($(compgen -W "$__nnictl_cmds" -- "${COMP_WORDS[1]}"))
elif [[ ${#COMP_WORDS[@]} -eq 3 ]]; then
# completing second argument from __nnictl_${FirstArg}_cmds
local args=__nnictl_${COMP_WORDS[1]}_cmds
COMPREPLY=($(compgen -W "${!args}" -- "${COMP_WORDS[2]}"))
# add experiment IDs to candidates if desired
if [[ $__nnictl_2nd_expid_cmds =~ " ${COMP_WORDS[1]} " ]]; then
local experiments=$(ls ~/nni-experiments 2>/dev/null)
COMPREPLY+=($(compgen -W "$experiments" -- $cur))
fi
elif [[ $last != -* || $last == --debug || $last == --foreground || $last == --intermediate || $last == --all ]]; then
# last argument does not starts with "-", so this one is likely to be "--xxx"
local args=__nnictl_${COMP_WORDS[1]}_${COMP_WORDS[2]}_cmds
if [[ $args =~ "-" || $__nnictl_2nd_expid_cmds =~ " ${COMP_WORDS[1]} " ]]; then
# the second argument starts with "-" or is an experiment id, use __nnictl_${FirstArg}_cmds
local args=__nnictl_${COMP_WORDS[1]}_cmds
fi
# remove already set arguments from candidates
local remain_args=$(__nnictl_remain_args ${args})
COMPREPLY=($(compgen -W "$remain_args" -- $cur))
# if this is 3rd arguments, try adding experiment IDs to candidates
if [[ ${#COMP_WORDS[@]} -eq 4 ]]; then
if [[ $__nnictl_3rd_expid_cmds =~ " ${COMP_WORDS[1]} " && ${COMP_WORDS[2]} != "list" ]]; then
local experiments=$(ls ~/nni-experiments 2>/dev/null)
COMPREPLY+=($(compgen -W "$experiments" -- $cur))
fi
fi
elif [[ ${COMP_WORDS[2]} == "export" ]]; then
# "export" command is somewhat unique
if [[ " --type -t " =~ " $last " ]]; then
COMPREPLY=($(compgen -W "json csv" -- $cur))
elif [[ " --filename -f " =~ " $last " ]]; then
# try to detect whether complete CSV file or JSON file
[[ "$COMP_LINE" =~ "csv" ]] && local export_csv=1
[[ "$COMP_LINE" =~ "json" ]] && local export_json=1
if [[ -n $export_csv && -z $export_json ]]; then
local ext="csv" # CSV only
elif [[ -n $export_json && -z $export_csv ]]; then
local ext="json" # JSON only
else
local ext="@(csv|json)" # both
fi
__nnictl_complete_extension "$ext"
fi
elif [[ " --trial_id -T " =~ " $last " ]]; then
# complete trial ID
if [[ -e ${HOME}/nni-experiments/${COMP_WORDS[2]} ]]; then
local trials=$(ls -d ~/nni-experiments/${COMP_WORDS[2]}/trials/* 2>/dev/null | grep -o '[^/]*$')
elif [[ -e "${HOME}/nni-experiments/${COMP_WORDS[3]}" ]]; then
local trials=$(ls -d ~/nni-experiments/${COMP_WORDS[3]}/trials/* 2>/dev/null | grep -o '[^/]*$')
else
local trials=$(ls -d ~/nni-experiments/*/trials/* 2>/dev/null | grep -o '[^/]*$')
fi
COMPREPLY=($(compgen -W "$trials" -- $cur))
elif [[ " --config -c " =~ " $last " ]]; then
# complete YAML file
__nnictl_complete_extension "@(yml|yaml)"
elif [[ " --filename -f " =~ " $last " ]]; then
# complete JSON file
__nnictl_complete_extension "json"
fi
}
if [[ ${BASH_VERSINFO[0]} -le 4 && ${BASH_VERSINFO[1]} -le 4 ]]; then
complete -F _nnictl nnictl
else
complete -o nosort -F _nnictl nnictl
fi
archive-del/tools/setup.py deleted 100644 → 0
View file @ 0f0c6288
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT license.
import setuptools
setuptools.setup(
name = 'nni-tool',
version = '999.0.0-developing',
packages = setuptools.find_packages(exclude=['*test*']),
python_requires = '>=3.6',
install_requires = [
'requests',
'responses',
'ruamel.yaml',
'psutil',
'astor',
'schema',
'PythonWebHDFS',
'colorama',
'netifaces',
'websockets'
],
author = 'Microsoft NNI Team',
author_email = 'nni@microsoft.com',
description = 'NNI control for Neural Network Intelligence project',
license = 'MIT',
url = 'https://github.com/Microsoft/nni',
entry_points = {
'console_scripts' : [
'nnictl = nni_cmd.nnictl:parse_args'
]
}
)
archive-del/uninstall.ps1 deleted 100644 → 0
View file @ 0f0c6288
$NNI_DEPENDENCY_FOLDER = [System.IO.Path]::GetTempPath()+$env:USERNAME
$env:PYTHONIOENCODING = "UTF-8"
if($env:VIRTUAL_ENV){
$NNI_PYTHON3 = $env:VIRTUAL_ENV + "\Scripts"
$NNI_PKG_FOLDER = $env:VIRTUAL_ENV + "\nni"
cmd /c if exist "$NNI_PYTHON3\node.exe" del "$NNI_PYTHON3\node.exe"
}
else{
$NNI_PYTHON3 = $(python -c 'import site; from pathlib import Path; print(Path(site.getsitepackages()[0]))')
$NNI_PKG_FOLDER = $NNI_PYTHON3 + "\nni"
cmd /c if exist "$NNI_PYTHON3\Scripts\node.exe" del "$NNI_PYTHON3\Scripts\node.exe"
}
$PIP_UNINSTALL = """$NNI_PYTHON3\python"" -m pip uninstall -y "
$NNI_NODE_FOLDER = $NNI_DEPENDENCY_FOLDER+"\nni-node"
$NNI_YARN_FOLDER = $NNI_DEPENDENCY_FOLDER+"\nni-yarn"
# uninstall
cmd /c if exist "$NNI_PKG_FOLDER" rmdir /s /q $NNI_PKG_FOLDER
cmd /c $PIP_UNINSTALL "nni"
# clean up
cmd /c if exist "build" rmdir /s /q "build"
cmd /c if exist "src\nni_manager\dist" rmdir /s /q "src\nni_manager\dist"
cmd /c if exist "src\nni_manager\node_modules" rmdir /s /q "src\nni_manager\node_modules"
cmd /c if exist "src\webui\build" rmdir /s /q "src\webui\build"
cmd /c if exist "src\webui\node_modules" rmdir /s /q "src\webui\node_modules"
cmd /c if exist "src\nasui\build" rmdir /s /q "src\nasui\build"
cmd /c if exist "src\nasui\node_modules" rmdir /s /q "src\nasui\node_modules"
cmd /c if exist "$NNI_YARN_FOLDER" rmdir /s /q $NNI_YARN_FOLDER
cmd /c if exist "$NNI_NODE_FOLDER" rmdir /s /q $NNI_NODE_FOLDER
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment