--- Before & After ... Update & Reboot ---

Before-and-After-Update-Reboot.html

yum upgrade; sync; sync; sync; date
sync; sync; sync; reboot

Look at many pre-check here SEC/Update-Reboot.html --- Check TC URL's


### Top of prep file
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
ls -ld /banner/dist/*   # ls -l /etc/auto*
cat  /etc/auto.banner | egrep -v \# | awk '{print "ls -l "$1}' 
### OEL6 service --status-all | grep 'running' > /var/tmp/before-reboot
### OEL7 systemctl --no-pager | grep running > /var/tmp/before-reboot

 
---------------------
sync
sync
sync
reboot
-------------------

ls -ld /banner/dist/*   # ls -l /etc/auto*
cat  /etc/auto.banner | egrep -v \# | awk '{print "ls -l "$1}' 

### After all services have been run

### OEL6 service --status-all | grep 'running' > /var/tmp/after-reboot
### OEL7 systemctl --no-pager | grep running > /var/tmp/after-reboot
diff  /var/tmp/before-reboot /var/tmp/after-reboot 

 

------------------------------------------------------



-----------------------------------------------------
Update size? Any issues?..call for help. New Kernel?
-----------------------------------------------------
yum check-update

----------------------------------------------------------------------------------------------------------
OEL6 or 7
-----------------------------------------------------
uname -r 

If OEL7 then systemctl is used (and often /etc/init.d RC scripts)
If OEL6 only /etc/init.d RC scripts

------------------------------------------------------
Paths that must automount.
------------------------------------------------------
ls -l /etc/auto.*

cat  /etc/auto.banner | egrep -v \# | awk '{print "echo \""$1"\";ls -l "$1}' | sh | more
cat  /etc/auto.banner | egrep -v \# | awk '{print "echo \""$1"\";  ls -l "$1}' 

cat  /etc/auto.hanfs | egrep -v \# | awk '{print "echo \""$1"\";ls -l "$1}'  | sh | more
cat  /etc/auto.hanfs | egrep -v \# | awk '{print "echo \""$1"\";ls -l "$1}'

cat /etc/auto.firestone ## needs /firestone prefix  
ls -ld  /home/mriale/idcard
drwxr-xr-x 2 3615 dev 137 Aug  2  2017 /home/mriale/idcard

## SPECIAL ATTENTION for (tomcat related)
ls -ld /banner/dist/*
banner/dist -> /u01/app/banner_xe/bmui_staging
On the following servers ... 
banapp3
banapp4
banapp5
banapp6
banappt3
banappt4
banappt5
banappt6
baninb01
baninbt
chemungv
otegov


IF after a reboot some do not appear
systemctl stop autofs
systemctl start autofs
systemctl status autofs
-OR-
service autofs stop
service autofs start

### for the BANAPP servers
If ls -ld /banner/dist/*  doesn't' return contect,
Likely tomcat is not running correctly

systemctl stop tomcat
systemctl stop autofs
systemctl start autofs
ls -ld  /banner/dist 
	... if it returns content...
systemctl start tomcat 
systemctl  status tomcat

Server Specific things to check v/ AFTER REBOOTS ...
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
VENUS
128.226.6.152
continuingedtest.binghamton.edu
securetest.binghamton.edu
128.226.6.188
continuingeddev.binghamton.edu
securedev.binghamton.edu

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
http://susquehanna.cc.binghamton.edu:6128/odsdev/

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On GENESEE the database sleeps for 5 minutes before it starts...
date; ps -ef | egrep sleep, tomcat service might need to be stop|started  before the URL will work...
http://genesee.cc.binghamton.edu:6128/odsprod/twbkwbis.P_ValLogin

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
http://chenango.cc.binghamton.edu:6128/odstest/


------------------------------------------------------



###
### What Does the server do DB, App, Or something else
###
------------------------------------------------------
DB's running?
------------------------------------------------------
ps -ef | egrep pmon | egrep -v grep | awk '{print $8}'
ps -ef | egrep -i listen | egrep -v grep | awk '{print $8" "$9" "$10}'
ps -ef | egrep agent | egrep -v grep | awk '{print $8}'

IF NOT then it is likely  an app server.
ps -ef  | egrep -v \\[ |egrep   'appworx|badm|banjob|cbagent|httpd|jenkins|mysql|nginx|mariadb|oni|oracle|postgres|rabbit|tomcat|avahi|graylog' | egrep -v grep  | cut -c1-100

------------------------------------------------------
What RC scripts might need to be stopped?
------------------------------------------------------


echo "egrep -w appworx  /etc/init.d/* |cut -d: -f1 |sort -u |awk '{print \$1\" start\"}'
egrep -w badm      /etc/init.d/* |cut -d: -f1 |sort -u |awk '{print \$1\" stop\"}'
egrep -w banjob    /etc/init.d/* |cut -d: -f1 |sort -u |awk '{print \$1\" stop\"}'
egrep -w cbagentd  /etc/init.d/* |cut -d: -f1 |sort -u |awk '{print \$1\" stop\"}'
egrep -w cbagent   /etc/init.d/* |cut -d: -f1 |sort -u |awk '{print $1" stop"}'
egrep -w commvault /etc/init.d/* |cut -d: -f1 |sort -u |awk '{print $1" stop"}'
egrep -w gcstartup /etc/init.d/* |cut -d: -f1 |sort -u |awk '{print \$1\" stop\"}'
egrep -w httpd     /etc/init.d/* |cut -d: -f1 |sort -u |awk '{print \$1\" stop\"}'
egrep -w mariadb   /etc/init.d/* |cut -d: -f1 |sort -u |awk '{print \$1\" stop\"}'
egrep -w mysql     /etc/init.d/* |cut -d: -f1 |sort -u |awk '{print \$1\" stop\"}'
egrep -w nginx     /etc/init.d/* |cut -d: -f1 |sort -u |awk '{print \$1\" stop\"}'
egrep -w jenkins   /etc/init.d/* |cut -d: -f1 |sort -u |awk '{print \$1\" stop\"}'
egrep -w oni       /etc/init.d/* |cut -d: -f1 |sort -u |awk '{print \$1\" stop\"}'
egrep -w postgres  /etc/init.d/* |cut -d: -f1 |sort -u |awk '{print \$1\" stop\"}'
egrep -w oracle    /etc/init.d/* |cut -d: -f1 |sort -u |awk '{print \$1\" stop\"}'
egrep -w oussm     /etc/init.d/* |cut -d: -f1 |sort -u |awk '{print \$1\" stop\"}'
egrep -w rabbit    /etc/init.d/* |cut -d: -f1 |sort -u |awk '{print \$1\" stop\"}'
egrep -w snmptrapd /etc/init.d/* |cut -d: -f1 |sort -u |awk '{print \$1\" stop\"}'
egrep -w splunk    /etc/init.d/* |cut -d: -f1 |sort -u |awk '{print \$1\" stop\"}'
egrep -w tomcat    /etc/init.d/* |cut -d: -f1 |sort -u |awk '{print \$1\" stop\"}'" > /var/tmp/RCscan-stop.sh
sh  /var/tmp/RCscan-stop.sh

If an RC script (systemd should manage ownership) will be used to stop|start...
Confirm that there is a su oracle in the script.
egrep oracle /etc/init.d/rc-scriptname
sudo su - oracle -c /etc/init.d/rc-scriptname

chkconfig --list
chkconegrep oracle /etc/init.d/* | cut -d: -f1 | sort -u

egrep oracle /etc/init.d/* | cut -d: -f1 | sort -u
cat /etc/init.d/oracle_init |egrep 'ORA_BIN' | egrep -v if


###
### SHOULD FIND SERVICES THAT ARE RUNNING
###
systemctl | egrep -v 'waiting|target|socket|user|exited|mount|slice|plugged|path|scope' | egrep loaded | egrep active | cut -c1-120 | sort -u    
   
systemctl | egrep -i oracle     

###
### Create the commands to stop|stat|status what services are found on a partiuclar server.
### 
-- STOP
systemctl | egrep 'appworx|badm|banjob|cvunlock|cbagent|commvault|httpd|jenkins|mysql|mariadb|nginx|oni|oracle|oussm|postgres|rhns|rabbit|snmptrapd|snapd|tomcat|avahi|graylog' \
 | cut -c1-90 | egrep -v 'apparmor|target|seeded|socket|mount|session|smart|stat|user|lvm'  | sort -u | awk '{print "systemctl stop "$1}'


-- START
systemctl | egrep 'appworx|badm|banjob|cbagentcvunlock|commvault|httpd|jenkins|mysql|mariadb|nginx|oni|oracle|postgres|rhns|rabbit|snmptrapd|snmptrapd|snapd|tomcat|avahi|graylog' \
 | cut -c1-90 | egrep -v 'apparmor|target|seeded|socket|mount|session|smart|stat|user|lvm'  | sort -u | awk '{print "systemctl start "$1}'


-- STATUS
systemctl | egrep 'appworx|badm|banjob|cvunlock|cbagent|commvault|httpd|jenkins|mysql|mariadb|nginx|oni|oracle|oussm|postgres|rhns|rabbit|snmptrapd|snapd|tomcat|avahi|graylog' \
 | cut -c1-90 | egrep -v 'apparmor|target|seeded|socket|mount|session|smart|stat|user|lvm'  | sort -u | awk '{print "systemctl status "$1}'

-- All Common Services
systemctl | egrep service  | egrep -v 'user|setup|udev|mount|apparmor|seed' | cut -c-90

-- ps-ef --
ps -ef  | egrep -v \\[ |egrep  -v 'abrt|gdm|appworx|badm|banjob|cbagent|httpd|jenkins|mysql|mariadb|oni|oracle|oussm|postgres|rhns|rabbit|tomcat|avahi|graylog' | egrep -v grep

-- Comma Delimited list --
systemctl | egrep 'appworx|badm|banjob|cbagent|commvault|httpd|jenkins|mysql|mariadb|nginx|oni|oracle|oussm|postgres|rhns|rabbit|snapd|tomcat|avahi|graylog' \
    | cut -c1-90 | egrep -v 'session|smart|stat|user|lvm'  | sort -u | awk '{print $1}' | sed "s/.service/, /"  | tr -d '\n'

 
 
     
-------------------------------------------------------
What else is running, that will indicate a service?
-------------------------------------------------------
ps -ef | egrep root       | egrep -v grep | egrep -v \\[
ps -ef | egrep -v root    | egrep -v grep | egrep -v \\[
ps -ef | egrep cron       | egrep -v grep 
ps -ef | egrep badm       | egrep -v grep 
ps -ef | egrep oracle     | egrep -v grep 
ps -ef | egrep oussm      | egrep -v grep
ps -ef | egrep tomcat     | egrep -v grep 
ps -ef | egrep jenkins    | egrep -v grep 
ps -ef | egrep appworx    | egrep -v grep 

ps -ef | egrep 'appworx|badm|banjob|jenkins|oni|oracle|oussm|postgres|rabbit|tomcat' |  egrep -v grep
ps -ef | egrep 'appworx|badm|banjob|jenkins|oni|oracle|oussm|postgres|rabbit|tomcat' |  egrep -v grep |cut -c1-60

            
------------------------------------------------------
Do these scripts exist? 
------------------------------------------------------
find  /u0[1-9]/*  -name   start_stop.shl -print

find  /u0[1-9]/*  -name   idmt_cron.shl -print

find  /u0[1-9]/*  -name   dbstart       -print

------------------------------------------------------
In Oracle crontd, wait for it to run after reboot..maybe!
------------------------------------------------------
crontab -u oracle -l | egrep 'start_stop.shl|dbstart|cron'
crontab -u oracle -l 

------------------------------------------------------
Is it polling??
------------------------------------------------------
ps -ef | egrep start_stop.shl | egrep -v grep

ps -ef | egrep idmt_cron.shl  | egrep -v grep

ps -ef | egrep dbstart | egrep -v grep

When happy that all all autofs paths are ID's all services are ID'd for stop before and status and start if necessary after reboot...

yum upgrade
stop all services
sync
sync
sync
reboot
--------
Status that all services have started
responded as needed.

ls -1 /etc/rc*/* | egrep 'oracle|gcstartup|tomcat' | egrep ^S | sort -u |egrep -v unlock
ps -ef |egrep 'oracle|appworx|jenkins|tomcat|rabbit' |egrep -v grep |awk '{print $1" "$8" "$9" "$10}' |sort -u

ps -ef |egrep 'oracle|appworx|jenkins|tomcat|rabbit' |egrep -v grep |awk '{print $1" "$8" "$9" "$10}' |sort -u; printf "\n\n\n\n"; ps -ef |egrep tomcat |egrep -v grep
---
egrep oracle /etc/init.d/* |cut -d: -f1 |sort -u |awk '{print $1}'
---
cd /etc/init.d; egrep oracle * |cut -d: -f1 |sort -u |awk '{print $1}'
---
systemctl | awk '{print $1}' |egrep oracle
---
systemctl | egrep 'appworx|badm|banjob|jenkins|oni|oracle|rabbit|tomcat
systemctl | egrep 'appworx|badm|banjob|jenkins|oni|oracle|rabbit|tomcat' | cut -c1-90

ps -ef |egrep 'pmon' | egrep -v grep | egrep -v grep |awk '{print $8}'

ps -ef |egrep -v 'root|gdm|avahi|chrony|colord|rpc|rtkit|smmsp|polkitd|libstor|dbus' |egrep -v 'grep|UID' |awk '{print $1" "$8" "$9" "$10}' |sort -u
ps -ef |egrep oracle |awk '{print $1" "$8" "$9" "$10}' |sort -u |egrep -v 'sshd|grep'

ps -ef |egrep 'oracle|appworx|jenkins|tomcat|rabbit' |egrep -v grep |awk '{print $1" "$8" "$9" "$10}' |sort -u ; ps -ef |egrep tomcat |egrep -v grep |wc -l>
ps -ef |egrep -i 'oracle|appworx|jenkins|tomcat|weblogic' |egrep -v grep |wc -l
ps -ef |egrep -i 'oracle|appworx|jenkins|tomcat|weblogic' |egrep -v grep |awk '{print $1" "$8" "$9" "$10}' |sort -u > /work/pre-reboot; cat /work/pre-reboot
ps -ef |egrep -v 'root|gdm' |egrep -v 'grep|UID' |awk '{print $1" "$8" "$9" "$10}' |sort -u > /work/pre-reboot-2; cat /work/pre-reboot-2
ps -ef |egrep pmon >/work/db-names; cat /work/db-names
ps -ef |egrep 'banjob|appworx' |grep -v grep
ps -ef |egrep 'jenkins|oracle' |awk '{print $1" "$8" "$9" "$10" "$11" "$12}'
ps -ef |egrep -v 'root|gdm|avahi|chrony|colord|rpc|rtkit|smmsp' |egrep -v 'grep|UID' |awk '{print $1" "$8" "$9" "$10}' |sort -u
Check if the server is on this page... Check Server Here

If all regulare methods fail to restart oracle...

su - oracle

[UNDEF]-oracle:~$ oenv <database name>
SECRET=/u01/app/badm/admin/banprod/etc/banprod.secret
ORACLE_HOME = /u01/app/oracle/product/12.1.0/db_1
 ORACLE_SID = banprod
 BANNER_SID = banprod
       USER = oracle
[banprod]-oracle:~$ sqlplus '/as sysdba'*

SQL*Plus: Release 12.1.0.2.0 Production on Tue Sep 8 07:27:28 2020

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

Connected to an idle instance.

SQL>startup
ORA-32004: obsolete or deprecated parameter(s) specified for RDBMS instance
ORACLE instance started.

Total System Global Area 1.5032E+10 bytes
Fixed Size                  4514736 bytes
Variable Size            8891927632 bytes
Database Buffers         6106906624 bytes
Redo Buffers               29036544 bytes
Database mounted.
Database opened.