Commit a32ffa95 authored by qianyj's avatar qianyj
Browse files

update TensorFlow2x test method

parent e286da17
worker = []
port_number =40000
filePath='./nodefile'
with open(filePath, 'r') as f:
nodename = f.read().splitlines()
worker_nodes = nodename
num_index=flags_obj.task_index
for node in worker_nodes:
for index in range(4):
worker_sockets = ":".join([node, str(port_number + index )])
worker.append(worker_sockets)
os.environ['TF_CONFIG'] = json.dumps({
'cluster': {
'worker': worker
},
'task': {'type': 'worker', 'index': num_index}
})
---
name: "[Official Model] Bug Report"
about: Use this template for reporting a bug for the “official” directory
labels: type:bug,models:official
---
# Prerequisites
Please answer the following questions for yourself before submitting an issue.
- [ ] I am using the latest TensorFlow Model Garden release and TensorFlow 2.
- [ ] I am reporting the issue to the correct repository. (Model Garden official or research directory)
- [ ] I checked to make sure that this issue has not been filed already.
## 1. The entire URL of the file you are using
https://github.com/tensorflow/models/tree/master/official/...
## 2. Describe the bug
A clear and concise description of what the bug is.
## 3. Steps to reproduce
Steps to reproduce the behavior.
## 4. Expected behavior
A clear and concise description of what you expected to happen.
## 5. Additional context
Include any logs that would be helpful to diagnose the problem.
## 6. System information
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
- Mobile device name if the issue happens on a mobile device:
- TensorFlow installed from (source or binary):
- TensorFlow version (use command below):
- Python version:
- Bazel version (if compiling from source):
- GCC/Compiler version (if compiling from source):
- CUDA/cuDNN version:
- GPU model and memory:
<!--
Collect system information using our environment capture script.
https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh
You can also obtain the TensorFlow version with:
1. TensorFlow 1.0
`python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"`
2. TensorFlow 2.0
`python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"`
-->
---
name: "[Official Model] Documentation Issue"
about: Use this template for reporting a documentation issue for the “official” directory
labels: type:docs,models:official
---
# Prerequisites
Please answer the following question for yourself before submitting an issue.
- [ ] I checked to make sure that this issue has not been filed already.
## 1. The entire URL of the documentation with the issue
https://github.com/tensorflow/models/tree/master/official/...
## 2. Describe the issue
A clear and concise description of what needs to be changed.
---
name: "[Official Model] Feature request"
about: Use this template for raising a feature request for the “official” directory
labels: type:feature,models:official
---
# Prerequisites
Please answer the following question for yourself before submitting an issue.
- [ ] I checked to make sure that this feature has not been requested already.
## 1. The entire URL of the file you are using
https://github.com/tensorflow/models/tree/master/official/...
## 2. Describe the feature you request
A clear and concise description of what you want to happen.
## 3. Additional context
Add any other context about the feature request here.
## 4. Are you willing to contribute it? (Yes or No)
---
name: "[Research Model] Bug Report"
about: Use this template for reporting a bug for the “research” directory
labels: type:bug,models:research
---
# Prerequisites
Please answer the following questions for yourself before submitting an issue.
- [ ] I am using the latest TensorFlow Model Garden release and TensorFlow 2.
- [ ] I am reporting the issue to the correct repository. (Model Garden official or research directory)
- [ ] I checked to make sure that this issue has not already been filed.
## 1. The entire URL of the file you are using
https://github.com/tensorflow/models/tree/master/research/...
## 2. Describe the bug
A clear and concise description of what the bug is.
## 3. Steps to reproduce
Steps to reproduce the behavior.
## 4. Expected behavior
A clear and concise description of what you expected to happen.
## 5. Additional context
Include any logs that would be helpful to diagnose the problem.
## 6. System information
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
- Mobile device name if the issue happens on a mobile device:
- TensorFlow installed from (source or binary):
- TensorFlow version (use command below):
- Python version:
- Bazel version (if compiling from source):
- GCC/Compiler version (if compiling from source):
- CUDA/cuDNN version:
- GPU model and memory:
<!--
Collect system information using our environment capture script.
https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh
You can also obtain the TensorFlow version with:
1. TensorFlow 1.0
`python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"`
2. TensorFlow 2.0
`python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"`
-->
---
name: "[Research Model] Documentation Issue"
about: Use this template for reporting a documentation issue for the “research” directory
labels: type:docs,models:research
---
# Prerequisites
Please answer the following question for yourself before submitting an issue.
- [ ] I checked to make sure that this issue has not been filed already.
## 1. The entire URL of the documentation with the issue
https://github.com/tensorflow/models/tree/master/research/...
## 2. Describe the issue
A clear and concise description of what needs to be changed.
---
name: "[Research Model] Feature Request"
about: Use this template for raising a feature request for the “research” directory
labels: type:feature,models:research
---
# Prerequisites
Please answer the following question for yourself before submitting an issue.
- [ ] I checked to make sure that this feature has not been requested already.
## 1. The entire URL of the file you are using
https://github.com/tensorflow/models/tree/master/research/...
## 2. Describe the feature you request
A clear and concise description of what you want to happen.
## 3. Additional context
Add any other context about the feature request here.
## 4. Are you willing to contribute it? (Yes or No)
---
name: Questions and Help
about: Use this template for Questions and Help.
labels: type:support
---
<!--
As per our GitHub Policy (https://github.com/tensorflow/models/blob/master/ISSUES.md), we only address code bugs, documentation issues, and feature requests on GitHub.
We will automatically close questions and help related issues.
Please go to Stack Overflow (http://stackoverflow.com/questions/tagged/tensorflow-model-garden) for questions and help.
-->
# Description
> :memo: Please include a summary of the change.
>
> * Please also include relevant motivation and context.
> * List any dependencies that are required for this change.
## Type of change
For a new feature or function, please create an issue first to discuss it
with us before submitting a pull request.
Note: Please delete options that are not relevant.
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] Documentation update
- [ ] TensorFlow 2 migration
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
- [ ] A new research paper code implementation
- [ ] Other (Specify)
## Tests
> :memo: Please describe the tests that you ran to verify your changes.
>
> * Provide instructions so we can reproduce.
> * Please also list any relevant details for your test configuration.
**Test Configuration**:
## Checklist
- [ ] I have signed the [Contributor License Agreement](https://github.com/tensorflow/models/wiki/Contributor-License-Agreements).
- [ ] I have read [guidelines for pull request](https://github.com/tensorflow/models/wiki/Submitting-a-pull-request).
- [ ] My code follows the [coding guidelines](https://github.com/tensorflow/models/wiki/Coding-guidelines).
- [ ] I have performed a self [code review](https://github.com/tensorflow/models/wiki/Code-review) of my own code.
- [ ] I have commented my code, particularly in hard-to-understand areas.
- [ ] I have made corresponding changes to the documentation.
- [ ] My changes generate no new warnings.
- [ ] I have added tests that prove my fix is effective or that my feature works.
> :memo: A README.md template for releasing a paper code implementation to a GitHub repository.
>
> * Template version: 1.0.2020.170
> * Please modify sections depending on needs.
# Model name, Paper title, or Project Name
> :memo: Add a badge for the ArXiv identifier of your paper (arXiv:YYMM.NNNNN)
[![Paper](http://img.shields.io/badge/Paper-arXiv.YYMM.NNNNN-B3181B?logo=arXiv)](https://arxiv.org/abs/...)
This repository is the official or unofficial implementation of the following paper.
* Paper title: [Paper Title](https://arxiv.org/abs/YYMM.NNNNN)
## Description
> :memo: Provide description of the model.
>
> * Provide brief information of the algorithms used.
> * Provide links for demos, blog posts, etc.
## History
> :memo: Provide a changelog.
## Authors or Maintainers
> :memo: Provide maintainer information.
* Full name ([@GitHub username](https://github.com/username))
* Full name ([@GitHub username](https://github.com/username))
## Table of Contents
> :memo: Provide a table of contents to help readers navigate a lengthy README document.
## Requirements
[![TensorFlow 2.1](https://img.shields.io/badge/TensorFlow-2.1-FF6F00?logo=tensorflow)](https://github.com/tensorflow/tensorflow/releases/tag/v2.1.0)
[![Python 3.6](https://img.shields.io/badge/Python-3.6-3776AB)](https://www.python.org/downloads/release/python-360/)
> :memo: Provide details of the software required.
>
> * Add a `requirements.txt` file to the root directory for installing the necessary dependencies.
> * Describe how to install requirements using pip.
> * Alternatively, create INSTALL.md.
To install requirements:
```setup
pip install -r requirements.txt
```
## Results
[![TensorFlow Hub](https://img.shields.io/badge/TF%20Hub-Models-FF6F00?logo=tensorflow)](https://tfhub.dev/...)
> :memo: Provide a table with results. (e.g., accuracy, latency)
>
> * Provide links to the pre-trained models (checkpoint, SavedModel files).
> * Publish TensorFlow SavedModel files on TensorFlow Hub (tfhub.dev) if possible.
> * Add links to [TensorBoard.dev](https://tensorboard.dev/) for visualizing metrics.
>
> An example table for image classification results
>
> ### Image Classification
>
> | Model name | Download | Top 1 Accuracy | Top 5 Accuracy |
> |------------|----------|----------------|----------------|
> | Model name | [Checkpoint](https://drive.google.com/...), [SavedModel](https://tfhub.dev/...) | xx% | xx% |
## Dataset
> :memo: Provide information of the dataset used.
## Training
> :memo: Provide training information.
>
> * Provide details for preprocessing, hyperparameters, random seeds, and environment.
> * Provide a command line example for training.
Please run this command line for training.
```shell
python3 ...
```
## Evaluation
> :memo: Provide an evaluation script with details of how to reproduce results.
>
> * Describe data preprocessing / postprocessing steps.
> * Provide a command line example for evaluation.
Please run this command line for evaluation.
```shell
python3 ...
```
## References
> :memo: Provide links to references.
## License
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
> :memo: Place your license text in a file named LICENSE in the root of the repository.
>
> * Include information about your license.
> * Reference: [Adding a license to a repository](https://help.github.com/en/github/building-a-strong-community/adding-a-license-to-a-repository)
This project is licensed under the terms of the **Apache License 2.0**.
## Citation
> :memo: Make your repository citable.
>
> * Reference: [Making Your Code Citable](https://guides.github.com/activities/citable-code/)
If you want to cite this repository in your research paper, please use the following information.
# Copyright 2019 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ============================================================================
#
# THIS IS A GENERATED DOCKERFILE.
#
# This file was assembled from multiple pieces, whose use is documented
# throughout. Please refer to the TensorFlow dockerfiles documentation
# for more information.
# A list of assignees
assignees:
- saikumarchalla
#!/bin/bash
# Copyright 2016 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ==============================================================================
#
# Pylint wrapper extracted from main TensorFlow, sharing same exceptions.
# Specify --incremental to only check files touched since last commit on master,
# otherwise will recursively check current directory (full repo takes long!).
set -euo pipefail
# Download latest configs from main TensorFlow repo.
wget -q -O /tmp/pylintrc https://raw.githubusercontent.com/tensorflow/tensorflow/master/tensorflow/tools/ci_build/pylintrc
SCRIPT_DIR=/tmp
num_cpus() {
# Get the number of CPUs
if [[ -f /proc/cpuinfo ]]; then
N_CPUS=$(grep -c ^processor /proc/cpuinfo)
else
# Fallback method
N_CPUS=`getconf _NPROCESSORS_ONLN`
fi
if [[ -z ${N_CPUS} ]]; then
die "ERROR: Unable to determine the number of CPUs"
fi
echo ${N_CPUS}
}
get_changed_files_in_last_non_merge_git_commit() {
git diff --name-only $(git merge-base master $(git branch --show-current))
}
# List Python files changed in the last non-merge git commit that still exist,
# i.e., not removed.
# Usage: get_py_files_to_check [--incremental]
get_py_files_to_check() {
if [[ "$1" == "--incremental" ]]; then
CHANGED_PY_FILES=$(get_changed_files_in_last_non_merge_git_commit | \
grep '.*\.py$')
# Do not include files removed in the last non-merge commit.
PY_FILES=""
for PY_FILE in ${CHANGED_PY_FILES}; do
if [[ -f "${PY_FILE}" ]]; then
PY_FILES="${PY_FILES} ${PY_FILE}"
fi
done
echo "${PY_FILES}"
else
find . -name '*.py'
fi
}
do_pylint() {
if [[ $# == 1 ]] && [[ "$1" == "--incremental" ]]; then
PYTHON_SRC_FILES=$(get_py_files_to_check --incremental)
if [[ -z "${PYTHON_SRC_FILES}" ]]; then
echo "do_pylint will NOT run due to --incremental flag and due to the "\
"absence of Python code changes in the last commit."
return 0
fi
elif [[ $# != 0 ]]; then
echo "Invalid syntax for invoking do_pylint"
echo "Usage: do_pylint [--incremental]"
return 1
else
PYTHON_SRC_FILES=$(get_py_files_to_check)
fi
# Something happened. TF no longer has Python code if this branch is taken
if [[ -z ${PYTHON_SRC_FILES} ]]; then
echo "do_pylint found no Python files to check. Returning."
return 0
fi
# Now that we know we have to do work, check if `pylint` is installed
PYLINT_BIN="python3.8 -m pylint"
echo ""
echo "check whether pylint is available or not."
echo ""
${PYLINT_BIN} --version
if [[ $? -eq 0 ]]
then
echo ""
echo "pylint available, proceeding with pylint sanity check."
echo ""
else
echo ""
echo "pylint not available."
echo ""
return 1
fi
# Configure pylint using the following file
PYLINTRC_FILE="${SCRIPT_DIR}/pylintrc"
if [[ ! -f "${PYLINTRC_FILE}" ]]; then
die "ERROR: Cannot find pylint rc file at ${PYLINTRC_FILE}"
fi
# Run pylint in parallel, after some disk setup
NUM_SRC_FILES=$(echo ${PYTHON_SRC_FILES} | wc -w)
NUM_CPUS=$(num_cpus)
echo "Running pylint on ${NUM_SRC_FILES} files with ${NUM_CPUS} "\
"parallel jobs..."
echo ""
PYLINT_START_TIME=$(date +'%s')
OUTPUT_FILE="$(mktemp)_pylint_output.log"
ERRORS_FILE="$(mktemp)_pylint_errors.log"
rm -rf ${OUTPUT_FILE}
rm -rf ${ERRORS_FILE}
set +e
# When running, filter to only contain the error code lines. Removes module
# header, removes lines of context that show up from some lines.
# Also, don't redirect stderr as this would hide pylint fatal errors.
${PYLINT_BIN} --rcfile="${PYLINTRC_FILE}" --output-format=parseable \
--jobs=${NUM_CPUS} ${PYTHON_SRC_FILES} | grep '\[[CEFW]' > ${OUTPUT_FILE}
PYLINT_END_TIME=$(date +'%s')
echo ""
echo "pylint took $((PYLINT_END_TIME - PYLINT_START_TIME)) s"
echo ""
# Report only what we care about
# Ref https://pylint.readthedocs.io/en/latest/technical_reference/features.html
# E: all errors
# W0311 bad-indentation
# W0312 mixed-indentation
# C0330 bad-continuation
# C0301 line-too-long
# C0326 bad-whitespace
# W0611 unused-import
# W0622 redefined-builtin
grep -E '(\[E|\[W0311|\[W0312|\[C0330|\[C0301|\[C0326|\[W0611|\[W0622)' ${OUTPUT_FILE} > ${ERRORS_FILE}
# Determine counts of errors
N_FORBID_ERRORS=$(wc -l ${ERRORS_FILE} | cut -d' ' -f1)
set -e
# Now, print the errors we should fix
echo ""
if [[ ${N_FORBID_ERRORS} != 0 ]]; then
echo "Found ${N_FORBID_ERRORS} pylint errors:"
cat ${ERRORS_FILE}
fi
echo ""
if [[ ${N_FORBID_ERRORS} != 0 ]]; then
echo "FAIL: Found ${N_FORBID_ERRORS} errors"
return 1
else
echo "PASS: Found no errors"
fi
}
do_pylint "$@"
# Copyright 2019 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ============================================================================
#
# THIS IS A GENERATED DOCKERFILE.
#
# This file was assembled from multiple pieces, whose use is documented
# throughout. Please refer to the TensorFlow dockerfiles documentation
# for more information.
# Number of days of inactivity before an Issue or Pull Request becomes stale
daysUntilStale: 7
# Number of days of inactivity before a stale Issue or Pull Request is closed
daysUntilClose: 7
# Only issues or pull requests with all of these labels are checked if stale. Defaults to `[]` (disabled)
onlyLabels:
- stat:awaiting response
# Comment to post when marking as stale. Set to `false` to disable
markComment: >
This issue has been automatically marked as stale because it has not had
recent activity. It will be closed if no further activity occurs. Thank you.
# Comment to post when removing the stale label. Set to `false` to disable
unmarkComment: false
closeComment: >
Closing as stale. Please reopen if you'd like to work on this further.
limitPerRun: 30
# Limit to only `issues` or `pulls`
only: issues
name: CI
on: pull_request
jobs:
pylint:
runs-on: ubuntu-latest
steps:
- name: Set up Python 3.8
uses: actions/setup-python@v2
with:
python-version: 3.8
- name: Install pylint 2.4.4
run: |
python -m pip install --upgrade pip
pip install pylint==2.4.4
- name: Checkout code
uses: actions/checkout@v2
with:
ref: ${{ github.event.pull_request.head.sha }}
fetch-depth: 0
- name: Fetch master for diff
run: git fetch origin master:master
- name: Run pylint script
run: bash ./.github/scripts/pylint.sh --incremental
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
env/
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
*.egg-info/
.installed.cfg
*.egg
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*,cover
.hypothesis/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
target/
# IPython Notebook
.ipynb_checkpoints
# pyenv
.python-version
# mypy
.mypy_cache
# celery beat schedule file
celerybeat-schedule
# dotenv
.env
# virtualenv
venv/
ENV/
# Spyder project settings
.spyderproject
# Rope project settings
.ropeproject
# PyCharm
.idea/
# For mac
.DS_Store
# This is the official list of authors for copyright purposes.
# This file is distinct from the CONTRIBUTORS files.
# See the latter for an explanation.
# Names should be added to this file as:
# Name or Organization <email address>
# The email address is not required for organizations.
Google Inc.
David Dao <daviddao@broad.mit.edu>
* @tensorflow/tf-model-garden-team
/official/ @rachellj218 @saberkun @jaeyounkim
/official/nlp/ @saberkun @lehougoogle @rachellj218 @jaeyounkim
/official/recommendation/ranking/ @gagika
/official/vision/ @xianzhidu @yeqingli @arashwan @saberkun @rachellj218 @jaeyounkim
/official/vision/beta/projects/assemblenet/ @mryoo @yeqingli
/official/vision/beta/projects/deepmac_maskrcnn/ @vighneshbirodkar
/official/vision/beta/projects/movinet/ @hyperparticle @yuanliangzhe @yeqingli
/official/vision/beta/projects/simclr/ @luotigerlsx @chentingpc @saxenasaurabh
/official/vision/beta/projects/video_ssl/ @richardaecn @yeqingli
/research/adversarial_text/ @rsepassi @a-dai
/research/attention_ocr/ @xavigibert
/research/audioset/ @plakal @dpwe
/research/autoaugment/ @barretzoph
/research/cognitive_planning/ @s-gupta
/research/cvt_text/ @clarkkev @lmthang
/research/deep_speech/ @yhliang2018
/research/deeplab/ @aquariusjay @yknzhu
/research/delf/ @andrefaraujo
/research/efficient-hrl/ @ofirnachum
/research/lfads/ @jazcollins @sussillo
/research/lstm_object_detection/ @yinxiaoli @yongzhe2160
/research/marco/ @vincentvanhoucke
/research/object_detection/ @jch1 @tombstone @pkulzc
/research/pcl_rl/ @ofirnachum
/research/rebar/ @gjtucker
/research/seq_flow_lite/ @thunderfyc
/research/slim/ @sguada @marksandler2
/research/vid2depth/ @rezama
# How to contribute
![Contributors](https://img.shields.io/github/contributors/tensorflow/models)
We encourage you to contribute to the TensorFlow Model Garden.
Please read our [guidelines](../../wiki/How-to-contribute) for details.
**NOTE**: Only [code owners](./CODEOWNERS) are allowed to merge a pull request.
Please contact the code owners of each model to merge your pull request.
# If you open a GitHub issue, here is our policy.
* It must be a **bug**, a **feature request**, or a significant problem
with **documentation**.
* Please send a pull request instead for small documentation fixes.
* The required form must be filled out.
* The issue should be related to the repository it is created in.
General help and support should be sought on [Stack Overflow](https://stackoverflow.com/questions/tagged/tensorflow-model-garden) or other non-GitHub channels.
[![](https://img.shields.io/stackexchange/stackoverflow/t/tensorflow-model-garden)](https://stackoverflow.com/questions/tagged/tensorflow-model-garden)
TensorFlow developers respond to issues.
We want to focus on work that benefits the whole community such as fixing bugs
and adding new features.
It helps us to address bugs and feature requests in a timely manner.
---
Please understand that research models in the [research directory](https://github.com/tensorflow/models/tree/master/research)
included in this repository are experimental and research-style code.
They are not officially supported by the TensorFlow team.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment