One final point— it is possible to see tomcat running however, the web application might still not work because jdbc timed out and the only fix is to restart the instance. So, at times, I restart all tomcat instances to be proactive.

Thank you, Jeff. That's a really nice summary.  I wanted to make sure to get the part you said correct, how you said, "Some processes require manual intervention to stop before they can be started."  What I do is make sure that I'm signed in as the "ONLY execute as "userid" ", so for example, with Appworx that would be signed in as appworx. Then I do a "ps -ef | grep appw" to see everything running as appworx, then do the stop commands, then do the "ps -ef | grep appw" and kill anything still running if not now stopped, then do the start command -- that's basically what you mean by "manual intervention", just as one example w/ appworx, correct?

I'd prefer that these jobs are coordinated through the dba group however, here are a few systems which have to be started whenever the production database goes down and a dba might not be available. Some processes require manual intervention to stop before they can be started.

Check if TOMCAT is running on banapp and banapp[2-3-4-5]:
ONLY execute as "oracle": 
/u01/app/oracle/admin/tools/startup/tomcat_start_stop.shl {start|stop}
banprod

Oracle Weblogic on baninb01, baninb02 and baninb03
ONLY execute as "oracle":
/u01/app/oracle/admin/tools/startup/start_stop.shl {start|stop}

Workflow production on banapp3 and banapp4 only:
ONLY execute as "oracle":
cd /u05/app/banner/workflow/banprod
stop.sh
start.sh

Appworx production on chemungv:
ONLY execute as "appworx":
$HOME/start_stop/start_stop.shl {start|stop}

Banner Job Submission etcetera on chemungv:
ONLY execute as "banjob":
$HOME/stop_all
wait and ensure all "banjob" processes are terminated; manual intervention may be required.
$HOME/start_all

Tomcat (bu - secure apps) on ORION:
ONLY execute as "badm":
/u01/app/badm/apps/prod/apache-tomcat-prod/bin/catalina.sh stop
wait 10 seconds
/u01/app/badm/apps/prod/apache-tomcat-prod/bin/catalina.sh start



Start scripts are usually found here:

/u01/app/oracle/admin/tools/startup/

Look for "start_stop.shl" AND only run this if needed after a reboot.

Chuck and I are most familiar with the intricacies of these systems and will start other services such as Workflow, etcetera.


I'm not concerned about the application server nodes as those can be easily restarted however, data-recovery is far more problematic for a corrupt database.

  1. CHARON - banner production (banprod)
  2. GENESEE - ODS production (odsprod)
  3. BUDB - bing custom database (bprod), Applications Manager (awxprov) and HR DB (hrdb)
  4. MAXDB - Facilities Assets (maxprod)
  5. JADE - Telecom (pinnprod)
  6. R25DB - Resource 25 (r25prod)
  7. RSCDB - Events Management