Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
OpenDAS
nni
Commits
392c598e
Unverified
Commit
392c598e
authored
Oct 09, 2018
by
fishyds
Committed by
GitHub
Oct 09, 2018
Browse files
Change hard-coded root directory to $PWD in PAI container (#182)
parent
93546854
Changes
2
Show whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
19 additions
and
6 deletions
+19
-6
src/nni_manager/training_service/pai/hdfsClientUtility.ts
src/nni_manager/training_service/pai/hdfsClientUtility.ts
+6
-3
src/nni_manager/training_service/pai/paiTrainingService.ts
src/nni_manager/training_service/pai/paiTrainingService.ts
+13
-3
No files found.
src/nni_manager/training_service/pai/hdfsClientUtility.ts
View file @
392c598e
...
...
@@ -133,10 +133,13 @@ export namespace HDFSClientUtility {
deferred
.
resolve
(
exist
);
});
let
timeoutId
:
NodeJS
.
Timer
const
delayTimeout
:
Promise
<
boolean
>
=
new
Promise
<
boolean
>
((
resolve
:
Function
,
reject
:
Function
)
:
void
=>
{
// Set timeout and reject the promise once reach timeout (5 seconds)
setTimeout
(()
=>
deferred
.
reject
(
`Check HDFS path
${
hdfsPath
}
exists timeout`
),
5000
);
});
return
deferred
.
promise
;
return
Promise
.
race
([
deferred
.
promise
,
delayTimeout
]).
finally
(()
=>
clearTimeout
(
timeoutId
))
;
}
/**
...
...
src/nni_manager/training_service/pai/paiTrainingService.ts
View file @
392c598e
...
...
@@ -186,8 +186,8 @@ class PAITrainingService implements TrainingService {
const
nniPaiTrialCommand
:
string
=
String
.
Format
(
PAI_TRIAL_COMMAND_FORMAT
,
// PAI will copy job's codeDir into /root directory
`
/root
/
${
trialJobId
}
`
,
`
/root
/
${
trialJobId
}
/nnioutput`
,
`
$PWD
/
${
trialJobId
}
`
,
`
$PWD
/
${
trialJobId
}
/nnioutput`
,
trialJobId
,
this
.
experimentId
,
this
.
paiTrialConfig
.
command
,
...
...
@@ -343,7 +343,17 @@ class PAITrainingService implements TrainingService {
deferred
.
resolve
();
}
});
break
;
let
timeoutId
:
NodeJS
.
Timer
;
const
timeoutDelay
:
Promise
<
void
>
=
new
Promise
<
void
>
((
resolve
:
Function
,
reject
:
Function
):
void
=>
{
// Set timeout and reject the promise once reach timeout (5 seconds)
timeoutId
=
setTimeout
(
()
=>
reject
(
new
Error
(
'
Get PAI token timeout. Please check your PAI cluster.
'
)),
5000
);
});
return
Promise
.
race
([
timeoutDelay
,
deferred
.
promise
]).
finally
(()
=>
clearTimeout
(
timeoutId
));
case
TrialConfigMetadataKey
.
TRIAL_CONFIG
:
if
(
!
this
.
paiClusterConfig
){
this
.
log
.
error
(
'
pai cluster config is not initialized
'
);
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment