The mpi API on metaserver at Caltech-Dev

Legend:

Normal status or debugging message

Notable condition which may be a non-fatal error

Error condition not fatal to job

Error condition which fatal to job

Notable condition which is not an error

Currently undefined

Condition requires email notification of the responsible administrator of this API

Condition requires phone notification of the responsible administrator of this API

Link: API Status Page for Caltech-Dev

942686755 CXX_THREAD fork_prepare@MountPointStatus.cc begin
942686755 CXX_THREAD fork_prepare@MountPointStatus.cc end
942686755 CXX_THREAD fork_prepare@MountPointStatus.cc begin
942686755 CXX_THREAD fork_prepare@MountPointStatus.cc end
942686755 CXX_THREAD fork_parent@MountPointStatus.cc begin
942686755 CXX_THREAD fork_parent@MountPointStatus.cc end
942686755 CXX_THREAD fork_parent@MountPointStatus.cc begin
942686755 CXX_THREAD fork_parent@MountPointStatus.cc end
942686755 STARTUP execOverload overloaded exec
942686755 STARTUP closeListenSock no cid registered for service 'data'
942686755 STARTUP mpi::init unused data port 10021 closed
942686755 STARTUP mpi::init port 10021 (jobstate) opened on metaserver as sock3
942686755 STARTUP bgLoop Looping process watchlogs started
942686755 STARTUP openListenSock port 10019 (operator) opened on metaserver as sock4
942686755 STARTUP openListenSock port 10020 (emergency) opened on metaserver as sock5
942686755 STARTUP setResourceLimit vmemoryuse=unlimited; datasize=unlimited; core=unlimited; maxproc=65536; descriptors=1024; memorylocked=32768; filesize=unlimited; cputime=unlimited
942686755 STARTUP leakLogger inital size of mpi API: 14348 kB
942686755 STARTUP bgLoop Looping process etchosts started
942686755 STARTUP mpi Trying API
942686755 STARTUP mpi API yes
942686755 STARTUP mpi Trying LDAS_SYSTEM
942686755 STARTUP mpi LDAS_SYSTEM yes
942686755 IDLE exec exec /usr/sbin/lsof -b -p 25063
942686755 CXX_THREAD fork_prepare@MountPointStatus.cc begin
942686755 CXX_THREAD fork_prepare@MountPointStatus.cc end
942686755 CXX_THREAD fork_prepare@MountPointStatus.cc begin
942686755 CXX_THREAD fork_prepare@MountPointStatus.cc end
942686755 CXX_THREAD fork_parent@MountPointStatus.cc begin
942686755 CXX_THREAD fork_parent@MountPointStatus.cc end
942686755 CXX_THREAD fork_parent@MountPointStatus.cc begin
942686755 CXX_THREAD fork_parent@MountPointStatus.cc end
942686755 IDLE bgLoop Looping process statpagefile started
942686755 IDLE bgLoop Looping process killedjobreaper started
942686760 IDLE setFTPandHTTPinfo (::FTPURL 'ftp://131.215.115.248') (::FTPDIR '') (::HTTPURL 'http://131.215.115.248/ldas_outgoing/jobs') (::HTTPDIR '/ldas_outgoing/jobs') (::GRIDFTPURL 'gridftp:/export/grid/ldas') (::GRIDFTPDIR '/export/grid/ldas') (::LDAS_GATEWAY 'ldas-dev 131.215.115.248') (::LDAS_SYSTEM 'ldas-dev') (::RUNCODE 'LDAS-DEV')
942686760 STARTUP mpi::killAllMpirun cleaning up for user ldas
942686760 IDLE ::exec exec /usr/bin/ssh -n -l ldas beowulf /bin/rm -rf /tmp/lam-ldas@*
942686760 CXX_THREAD fork_prepare@MountPointStatus.cc begin
942686760 CXX_THREAD fork_prepare@MountPointStatus.cc end
942686760 CXX_THREAD fork_prepare@MountPointStatus.cc begin
942686760 CXX_THREAD fork_prepare@MountPointStatus.cc end
942686760 CXX_THREAD fork_parent@MountPointStatus.cc begin
942686760 CXX_THREAD fork_parent@MountPointStatus.cc end
942686760 CXX_THREAD fork_parent@MountPointStatus.cc begin
942686760 CXX_THREAD fork_parent@MountPointStatus.cc end
942686763 IDLE ::exec exec /usr/bin/ssh -n beowulf sudo pkill -9 -u search01,search02,search03,search04,search05,search06,search07,search08,search09,search10,search11,search12,search13,search14,search15,search16 wrapperAPI
942686763 CXX_THREAD fork_prepare@MountPointStatus.cc begin
942686763 CXX_THREAD fork_prepare@MountPointStatus.cc end
942686763 CXX_THREAD fork_prepare@MountPointStatus.cc begin
942686763 CXX_THREAD fork_prepare@MountPointStatus.cc end
942686763 CXX_THREAD fork_parent@MountPointStatus.cc begin
942686763 CXX_THREAD fork_parent@MountPointStatus.cc end
942686763 CXX_THREAD fork_parent@MountPointStatus.cc begin
942686763 CXX_THREAD fork_parent@MountPointStatus.cc end
942686766 IDLE ::exec exec /usr/bin/ssh -n beowulf sudo pkill -9 -u search01,search02,search03,search04,search05,search06,search07,search08,search09,search10,search11,search12,search13,search14,search15,search16 lamd
942686766 CXX_THREAD fork_prepare@MountPointStatus.cc begin
942686766 CXX_THREAD fork_prepare@MountPointStatus.cc end
942686766 CXX_THREAD fork_prepare@MountPointStatus.cc begin
942686766 CXX_THREAD fork_prepare@MountPointStatus.cc end
942686766 CXX_THREAD fork_parent@MountPointStatus.cc begin
942686766 CXX_THREAD fork_parent@MountPointStatus.cc end
942686766 CXX_THREAD fork_parent@MountPointStatus.cc begin
942686766 CXX_THREAD fork_parent@MountPointStatus.cc end
942686769 STARTUP mpi::killAllMpirun ran kill 10 times in 9.096 seconds
942686769 STARTUP mpi::prestartLamds running lamboot for user search01
942686769 IDLE ::exec exec scp -B conf.lam search01@metaserver:
942686769 CXX_THREAD fork_prepare@MountPointStatus.cc begin
942686769 CXX_THREAD fork_prepare@MountPointStatus.cc end
942686769 CXX_THREAD fork_prepare@MountPointStatus.cc begin
942686769 CXX_THREAD fork_prepare@MountPointStatus.cc end
942686769 CXX_THREAD fork_parent@MountPointStatus.cc begin
942686769 CXX_THREAD fork_parent@MountPointStatus.cc end
942686769 CXX_THREAD fork_parent@MountPointStatus.cc begin
942686769 CXX_THREAD fork_parent@MountPointStatus.cc end
942686770 IDLE ::exec exec scp -B conf.lam search01@metaserver:
942686770 CXX_THREAD fork_prepare@MountPointStatus.cc begin
942686770 CXX_THREAD fork_prepare@MountPointStatus.cc end
942686770 CXX_THREAD fork_prepare@MountPointStatus.cc begin
942686770 CXX_THREAD fork_prepare@MountPointStatus.cc end
942686770 CXX_THREAD fork_parent@MountPointStatus.cc begin
942686770 CXX_THREAD fork_parent@MountPointStatus.cc end
942686770 CXX_THREAD fork_parent@MountPointStatus.cc begin
942686770 CXX_THREAD fork_parent@MountPointStatus.cc end
942686770 STARTUP mpi::prestartLamds Subject: LDAS-DEV mpiAPI lamboot error!; Body: lam::boot: lam::pushSchema: search01@metaserver: Permission denied (publickey,gssapi-with-mic,password). lost connection
942686770 STARTUP mpi::killAllMpirun {mpi::cleanupTmp: ldas@beowulf: ssh: connect to host beowulf port 22: No route to host} {ldas@beowulf:wrapperAPI: ssh: connect to host beowulf port 22: No route to host} {ldas@beowulf:lamd: ssh: connect to host beowulf port 22: No route to host}
942686770 IDLE setFTPandHTTPinfo (::FTPURL 'ftp://131.215.115.248') (::FTPDIR '') (::HTTPURL 'http://131.215.115.248/ldas_outgoing/jobs') (::HTTPDIR '/ldas_outgoing/jobs') (::GRIDFTPURL 'gridftp:/export/grid/ldas') (::GRIDFTPDIR '/export/grid/ldas') (::LDAS_GATEWAY 'ldas-dev 131.215.115.248') (::LDAS_SYSTEM 'ldas-dev') (::RUNCODE 'LDAS-DEV')
942686775 IDLE mpi::updateCmonNodelist updated ::beowulfNodes in cntlmonAPI to 'beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf'
942686781 IDLE mpi::updateCmonNodelist updated ::beowulfNodes in cntlmonAPI to 'beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf beowulf'
942853192 IDLE mpi::reply Manager already hung up on socket: 'sock7'.