Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in / Register
Toggle navigation
Menu
Open sidebar
OpenDAS
nni
Commits
392c598e
Unverified
Commit
392c598e
authored
Oct 09, 2018
by
fishyds
Committed by
GitHub
Oct 09, 2018
Browse files
Change hard-coded root directory to $PWD in PAI container (#182)
parent
93546854
Changes
2
Show whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
19 additions
and
6 deletions
+19
-6
src/nni_manager/training_service/pai/hdfsClientUtility.ts
src/nni_manager/training_service/pai/hdfsClientUtility.ts
+6
-3
src/nni_manager/training_service/pai/paiTrainingService.ts
src/nni_manager/training_service/pai/paiTrainingService.ts
+13
-3
No files found.
src/nni_manager/training_service/pai/hdfsClientUtility.ts
View file @
392c598e
...
...
@@ -133,10 +133,13 @@ export namespace HDFSClientUtility {
deferred
.
resolve
(
exist
);
});
let
timeoutId
:
NodeJS
.
Timer
const
delayTimeout
:
Promise
<
boolean
>
=
new
Promise
<
boolean
>
((
resolve
:
Function
,
reject
:
Function
)
:
void
=>
{
// Set timeout and reject the promise once reach timeout (5 seconds)
setTimeout
(()
=>
deferred
.
reject
(
`Check HDFS path
${
hdfsPath
}
exists timeout`
),
5000
);
});
return
deferred
.
promise
;
return
Promise
.
race
([
deferred
.
promise
,
delayTimeout
]).
finally
(()
=>
clearTimeout
(
timeoutId
))
;
}
/**
...
...
src/nni_manager/training_service/pai/paiTrainingService.ts
View file @
392c598e
...
...
@@ -186,8 +186,8 @@ class PAITrainingService implements TrainingService {
const
nniPaiTrialCommand
:
string
=
String
.
Format
(
PAI_TRIAL_COMMAND_FORMAT
,
// PAI will copy job's codeDir into /root directory
`
/root
/
${
trialJobId
}
`
,
`
/root
/
${
trialJobId
}
/nnioutput`
,
`
$PWD
/
${
trialJobId
}
`
,
`
$PWD
/
${
trialJobId
}
/nnioutput`
,
trialJobId
,
this
.
experimentId
,
this
.
paiTrialConfig
.
command
,
...
...
@@ -343,7 +343,17 @@ class PAITrainingService implements TrainingService {
deferred
.
resolve
();
}
});
break
;
let
timeoutId
:
NodeJS
.
Timer
;
const
timeoutDelay
:
Promise
<
void
>
=
new
Promise
<
void
>
((
resolve
:
Function
,
reject
:
Function
):
void
=>
{
// Set timeout and reject the promise once reach timeout (5 seconds)
timeoutId
=
setTimeout
(
()
=>
reject
(
new
Error
(
'
Get PAI token timeout. Please check your PAI cluster.
'
)),
5000
);
});
return
Promise
.
race
([
timeoutDelay
,
deferred
.
promise
]).
finally
(()
=>
clearTimeout
(
timeoutId
));
case
TrialConfigMetadataKey
.
TRIAL_CONFIG
:
if
(
!
this
.
paiClusterConfig
){
this
.
log
.
error
(
'
pai cluster config is not initialized
'
);
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment