Automatic Startup and Shutdown of TK4- with Hercules on Linux

When Hercules is running as one background task of many on some server machine, it could happen that the host machine is rebooted and Hercules killed abruptly in the course of action. Thus, some means of Automatic Startup and Shutdown of Hercules is desired.
There are multiple ways to accomplish this goal. I chose the described procedure because it's still less work than integrating the said handling into the system startup procedures.

The directions provided make Hercules being run as ordinary user (not root), on on a Debian System. The TK4- install is placed directly into the home directory of the said user.

Startup

To have access to the console, Hercules is started in interactive mode within a screen-session. Actual start is handled by cron through the @reboot directive in the user's crontab.

@reboot   screen -d -m -S hercules-mvs -- ./mvs

Before activating, make sure, that the TK4--specific file ~/unattended/mode contains the word CONSOLE. If not, make the appropriate change.

Keep in mind that if you log in as root and switch users afterwards, you can't reconnect to the session with screen -r, because of TTY authorization issues. See Weblinks below for solutions.

Advanced Startup

Normally, any OS running on a real Mainframe needs a System console. Fortunately, at least TK4- was modified in a way that the system console has been redirected to the Hercules console.

For other OS, like VM "Sixpack", this is not the case. To ease startup we want to run Hercules in a Screen-Session as outlined above, but we want a second window in that session to automatically start c3270 to connect to localhost and provide a system console without further intervention.

This can be achieved with a simple ~ /.screenrc for the user running the Hercules instance:

# don't display the copyright page
startup_message off

# increase scrollback buffer size
defscrollback 10000

# create windows
screen -t hercules-vm hercules -f hercules.cnf
sleep 1
width -w 25 80
screen -t 3270-con c3270 localhost:3270
detach

It is important that the screen geometry must be set before the startup of c3270. If it's not set, it will not start. The sleep gives Hercules some time to open the Console Port for c3270 to connect to.

Afterwards, the cronjob must be modified slightly:

@reboot   screen -S hercules-vm

Network Preconfiguration

Understanding Hercules Networking can be a real challenge if you're not used to coping with TUN/TAP interfaces yet. Unfortunately, current documentation on this Facility is scarce.

While Hercules can make use of an external program hercifc, posts in the Hercules-390 group suggest there's a memory leak eating up a considerable amount of RAM. For that I opted to use a preconfigured interface.

So, I added into /etc/rc.local:

echo "Create TUN-Device for Hercules instance..."
/sbin/ip tuntap add mode tun user vmrun && \
  /sbin/ifconfig tun0 10.59.0.2 netmask 255.255.255.0

This configures a local interface for the TUN/TAP API which Hercules uses for Networking. The user parameter even enables us to get rid of the need to run Hercules as root because network access usually requires that. The peer IP has to be set in the OS running within Hercules. I assumed to have it be in the same subnet as created with the above commands. This IP subnet must be different from any subnet in your network!

In the Hercules configuration, it's just a line like the following:

0E20-0E21 CTCI tun0

This enables us to access the hosts IP stack from within Hercules. All further configuration has to be done there.

But how to access that stuff from other hosts in your network? This involves to enable IP forwarding and some tweaks on your internet gateway.

To enable IP forwarding, edit /etc/sysctl.conf and add:

net.ipv4.ip_forward = 1

Afterwards, run sysctl -p to activate the change.

Last step involves setting a static route in whatever your internet gateway is. In this example, you need to add a route with the destination address 10.59.0.0, the netmask 255.255.255.0. The gateway address is the address of your Hercules hosting Machine's Ethernet Interface.

Shutdown

The shutdown procedure is a bit more interesting, because it's necessary to initiate a system shutdown of MVS from outside MVS, in a non-interactive way. Fortunately, there's the virtual card reader device listening on Port 3505 for input and passing data to JES2.

So, my solution submits a shutdown-JCL to the card reader (and times out after one second, because the card reader ignores EOF), and waits in a loop until there's no more Hercules process to be seen. It needs netcat to be installed, to submit data to the TCP port.

#!/bin/bash
# 
# rc.local.shutdown
#
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.

echo "Signalling TK4-/Hercules to shut down, please wait at most 4 Minutes..."
/bin/nc -w1 localhost 3505 <<EOF
//EXTSHUT JOB (TK4-), 'Shutdown TK4-',
//            CLASS=A, MSGCLASS=A, MSGLEVEL=(0,0),
//            USER=HERC01,PASSWORD=CUL8TR
//SHUTDOWN EXEC SHUTDOWN
EOF

N=0

while ps ax |fgrep -v grep |fgrep -q 'hercules -f conf/tk4-.cnf'; do
        sleep 12
        ((N++))

        # Time out after 4 Minutes, let sysvrc kill hercules.
        if [ ${N} -ge 20 ]; then
                echo "Hercules shutdown timeout after 4 Minutes, giving up."
                break
        fi
done

if [ ${N} -lt 20 ]; then
        echo "Hercules shutdown finished."
fi

exit 0

Enabling the handling of rc.local.shutdown on Debian

Stock Debian installs have no shutdown-equivalent of /etc/rc.local. Therefore, changes need to be made to /etc/init.d/rc.local as outlined in the following diff:

--- /etc/init.d/rc.local	2015-04-06 17:50:43.000000000 +0200
+++ rc.local.new	2020-06-03 15:44:02.848720538 +0200
@@ -24,6 +24,16 @@
 	fi
 }
 
+do_stop() {
+	if [ -x /etc/rc.local.shutdown ]; then
+	        [ "$VERBOSE" != no ] && log_begin_msg "Running local shutdown scripts (/etc/rc.local.shutdown)"
+		/etc/rc.local.shutdown
+		ES=$?
+		[ "$VERBOSE" != no ] && log_end_msg $ES
+		return $ES
+	fi
+}
+
 case "$1" in
     start)
 	do_start
@@ -32,10 +42,14 @@
         echo "Error: argument '$1' not supported" >&2
         exit 3
         ;;
-    stop|status)
+    status)
         # No-op
         exit 0
         ;;
+    stop)
+	do_stop
+        exit 0
+        ;;
     *)
         echo "Usage: $0 start|stop" >&2
         exit 3

Weblinks

• Screen Cannot open your terminal '/dev/pts/0' - please check, Stackoverflow