Skip to content

GitLab

Switch branch/tag
Find file Normal viewHistoryPermalink
azure-pipelines.yml 2.43 KB
Newer Older
Shaden Smith's avatar
Add Azure Pipelines configuration.  
Shaden Smith committed
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48 

jobs:
- job: Default
  timeoutInMinutes: 360
  pool:
    name: 'GPU_testing'

  strategy:
    matrix:
      Python36:
        python.version: '3.6'
      #Python35:
      #  python.version: '3.5'
      #Python37:
      #  python.version: '3.7'
      #Python38:
      #  python.version: '3.8'


  steps:
  - task: UsePythonVersion@0
    inputs:
      versionSpec: '$(python.version)'
      addToPath: true
      architecture: 'x64'
    displayName: 'Use Python $(python.version)'

  - script: |
      python -m pip install --upgrade pip
      pip install --user -r requirements.txt
      ./install.sh
    displayName: 'Install dependencies'

  - script: |
      pre-commit run --all-files
    displayName: 'Formatting checks'

  - script: |
      pip install --user pylint
      pylint --exit-zero deepspeed/
    displayName: 'Code linter'

  - script: |
      pip install --user pytest
      pytest --verbose tests/unit/
    displayName: 'Unit tests'

  - script: |
Shaden Smith's avatar
Updating model data paths to DeepSpeedExamples  
Shaden Smith committed
49
50 
      ln -s /data/Megatron-LM/data DeepSpeedExamples/Megatron-LM/
      pip install --user -r DeepSpeedExamples/Megatron-LM/requirements.txt
Shaden Smith's avatar
Add Azure Pipelines configuration.  
Shaden Smith committed
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91 
      cd tests/model/
      pytest -s run_sanity_check.py
    displayName: 'Model tests'

  # BingBertSquad logs
  #- task: PublishPipelineArtifact@1
  #  inputs:
  #    targetPath: '$(Build.SourcesDirectory)/tests/model/BingBertSquad/test/'
  #    artifactName: BingBertSquad_logs
  #  displayName: 'BingBertSquad logs'
  #  condition: always()

  #BingBert logs
  #- task: PublishPipelineArtifact@1
  #  inputs:
  #    targetPath: '$(Build.SourcesDirectory)/tests/model/bing_bert/pretrain_test/'
  #    artifactName: BingBert_pretrain_logs
  #  displayName: 'BingBert pretrain logs'
  #  condition: always()

  #- task: PublishPipelineArtifact@1
  #  inputs:
  #    targetPath: '$(Build.SourcesDirectory)/tests/model/bing_bert/checkpoint_test_logs/'
  #    artifactName: BingBert_checkpoint_logs
  #  displayName: 'BingBert checkpoint logs'
  #  condition: always()

  # Megatron test logs
  - task: PublishPipelineArtifact@1
    inputs:
      targetPath: '$(Build.SourcesDirectory)/tests/model/Megatron_GPT2/test/'
      artifactName: Megatron_GPT2_logs
    displayName: 'Megatron GPT2 logs'
    condition: always()

  - task: PublishPipelineArtifact@1
    inputs:
      targetPath: '$(Build.SourcesDirectory)/tests/model/Megatron_GPT2/checkpoint_test_logs/'
      artifactName: Megatron_GPT2_checkpoint_logs
    displayName: 'Megatron GPT2 checkpoint logs'
    condition: always()