Tuxedo configuration and runtime

We know how to configure various Tuxedo components quite well by now, so in this section we will discuss only the configuration relating to an Exalogic machine.

The UBBCONFIG file

In this section, we will discuss all the configuration related tasks using the UBBCONFIG file. We will be introduced to some more parameters in the UBBCONFIG file and some environment variables that need to be set for running Tuxedo on an Exalogic box. In MP mode, communication between two nodes is done through a bridge, but in Exalogic's cross-node, communication can be leveraged using remote direct memory access (RDMA). You need to specify RDMA with option values in the MACHINES section; there are four options, which are as follows:

• RDMADAEMONIP: This is the IP (IPoIB ) address, but it is not an Ethernet-based IP address, and the Msgq_daemon process is bound with it. You need to configure a one-to-one configuration for Msgq_daemon and the logical machine.

• DMADAEMONPORT: This is a listener port for the Msgq_daemon listener process.

• RDMAQSIZE: This is the queue (EMSQ) size; it's default value is 65,536 bytes if not defined in the UBBCONFIG file.

• RDMAQENTRIES: This is the entry number for a queue (EMSQ), which is the maximum number of messages in that queue.

Also, under the *RESOURCES section, you need to set the following:

• For the MODEL parameter – MP (for SHM mode, you need not use RDMA)

• For the OPTIONS parameter – LAN, EXALOGIC, and RDMA

Please make sure that the following things are checked and configured according to the guidelines mentioned:

• The shared directory for all Exalogic nodes needs to be enabled to leverage the RDMA feature. You need to make sure that the access permissions are properly set for it. The default name is /u01/common/patches/tuxtmpfile; you can also set your own directory using the EXALOGIC_SHARED_PATH environment variable. Tuxedo uses this file for message transfers when the EMSQ queue is full or the message size exceeds the queue size.

• Users from different Exalogic nodes must have read, write, or execution permission to the shared APPDIR variable, as it is shared by all nodes.

• Please set a different path for TUXCONFIG for each node.

• Please set a different path for ULOGPFX for each node.

• You need to set the /etc/securitylimits.conf parameter with the following values:

  memlock  [Msgq_daemon shared memory size] * 2 + MAXACCESSERS * 14000(KB)
  hard memlock 1853030
  soft memlock 1853030
  

So, there are not too many changes to make in your UBBCONFIG file when compared to the normal Tuxedo configuration. I put in the /etc/securitylimits parameters and the EXALOGIC_SHARED_PATH environment variable as I want to make sure that we do all the basic things right before we bring up Tuxedo on an Exalogic box.

Tuxedo Socket Direct Protocol support

The Socket Direct Protocol (SDP) feature enables Tuxedo components using BSD socket APIs that can leverage the advantages of the SDP network protocol provided by Exalogic. This feature is high-bandwidth, low-latency, and needs reduced CPU involvement. To enable SDP in Tuxedo, you must specify EXALOGIC for OPTIONS in the *RESOURCE section, which we have seen earlier, and set the relevant configuration in the UBBCONFIG file or the DMCONFIG (in the DOMAIN configuration) file. We will now show you how to use SDP for the MP mode, DOMAIN, WSL and WS Client, and JSL and WTC.

The MP mode

In the MP mode, both master and slave machines are inside the IB cluster, so only consider that SDP and IPoIB are being used inside the IB cluster; in the bootstrap phase, tmboot, tlisten, bsbridge, and bridge are using socket APIs to communicate with each other.

To configure SDP in the MP mode, you need to add sdp: as a prefix to the network address, and the network address the must be an IPoIB address. You can refer to the following example:

*NETWORK
Node1            NADDR="sdp://IB_IP: 9933"
                 NLSADDR="sdp://IB_IP: 3355"  
Node2            NADDR="sdp://IB_IP: 9933"
                 NLSADDR="sdp://IB_IP: 3355"

To start tlisten, you can use Prompt> tlisten –d /dev/tcp –l sdp://IB_IP: 3355.

To configure IPoIB in the MP mode, use the IPoIB address as the network address.

*NETWORK
Node1         NADDR="//IB_IP: 9933"
              NLSADDR="//IB_IP: 3355"
Node2         NADDR="//IB_IP: 9933"          
              NLSADDR="//IB_IP: 3355"

To start tlisten, you can use Prompt> tlisten –d /dev/tcp –l //IB_IP: 3355.

GWTDOMAIN

Functionally, if you look at the domain architecture, you may find that the GWTDOMAIN server acts as both server and client. As a server, it will listen on a configured IP address and port number in the DMCONFIG file to accept a connection request from another GWTDOMAIN. As a client, it will initiate a connection request to another GWTDOMAIN by the policy configured in the DMCONFIG file. It is more useful to use an explicit IP address when configuring GWTDOMAIN in the DMCONFIG file, though you can configure it with a hostname.

Normally, every Exalogic node has at least two types of network interface: an IB interface and an Ethernet interface. So, to configure GWTDOMAIN, you take the IB interface to bind with the IP address IB_IP, and the Ethernet interface to bind with IP address ETH_IP. We have the following four examples listed to show you how to configure GWTDOMAIN in the DMCONFIG file to use SDP or IPoIB as server and client respectively in an Exalogic environment.

Configuring GWTDOMAIN to listen on SDP

To configure the gateway domain to listen to SDP, you need to add NWADDR with a port number as shown in the following example:

*DM_LOCAL
Node1              GWGRP=DOMGRP
                   TYPE=TDOMAIN
*DM_TDOMAIN
                   Node1   NWADDR="sdp://IB_IP: 27766"

Configuring GWTDOMAIN to connect using SDP

To configure the gateway domain to connect to SDP, you need to add NWADDR with a port number as shown in the following example:

*DM_LOCAL
Node1               GWGRP=DOMGRP
                    TYPE=TDOMAIN
*DM_REMOTE
Node2               TYPE=TDOMAIN
                    DOMAINID="EXALOGIC_Node2"
*DM_TDOMAIN
                    Node2  NWADDR="sdp://IB_IP: 27766"

Configuring GWTDOMAIN to listen on IPoIB

To configure the gateway domain to listen to IPoIB, you need to add NWADDR with a port number as shown in the following example:

*DM_LOCAL
Node2            GWGRP=DOMGRP
                 TYPE=TDOMAIN
*DM_TDOMAIN
 Node2           NWADDR="//IB_IP: 27766"

Configuring GWTDOMAIN to connect using IPoIB

To configure the gateway domain to connect through IPoIB, you need to add NWADDR with a port number as shown in the following example:

*DM_LOCAL
Node2               GWGRP=DOMGRP
                    TYPE=TDOMAIN
*DM_REMOTE
Node3               TYPE=TDOMAIN
                    DOMAINID="EXALOGIC_ Node3"
*DM_TDOMAIN
Node3               NWADDR="//IB_IP: 27766"

The workstation listener (WSL)

To configure WSL on SDP or IpoIB, you need to use the following option in the DMCONFIG file:

• WSL listening on SDP – Refer to the following example on using the WSL configuration using SDP:

  *SERVERS
  DEFAULT:   CLOPT="-A"
  WSL        SRVGRP=WSGRP   SRVID=1001
             CLOPT="-A -- -n sdp://IB_IP: 11101 -m1 -M10 -x1"

• WSL listening on IPoIB – Refer to the following example on using the WSL configuration using IPoIB:

  *SERVERS
  DEFAULT:      CLOPT="-A"
  WSL           SRVGRP=WSGRP   SRVID=1001
                CLOPT="-A -- -n  //IB_IP: 11101 -m1 -M10 -x1"

The workstation (/WS) client

For the workstation (/WS) client, you need to set the WSNADDR environment variable in the following ways:

• SDP: export WSNADDR=sdp://IB_IP:1001

• IpoIB: export WSNADDR=//IB_IP:1001

The jolt service listener (JSL)

The JSL setup is similar to WSL; you need to use the following option in the DMCONFIG file:

• JSL listening on SDP – Refer to the following example on working with the JSL configuration using SDP:

  *SERVERS
  DEFAULT:   CLOPT="-A"
  JSL        SRVGRP=WSGRP SRVID=1001
             CLOPT="-A -- -n sdp: //IB_IP: 11101 -m1 -M10 -x1"

• JSL listening on IPoIB – Refer to the following example on working with the JSL configuration using IPoIB:

  *SERVERS
  DEFAULT:       CLOPT="-A"
  JSL            SRVGRP=WSGRP SRVID=1001
                 CLOPT="-A -- -n //IB_IP: 11101 -m1 -M10 -x1"

The WebLogic Tuxedo connector (WTC)

You need to do the following two steps to enable an SDP connection between the WTC and Tuxedo:

1. Specify the NWADDR value of the WTC service local/remote access pointsas follows:

   sdp://IB_IP:port

   It is the same as the GWTDOMAIN's NWADDR configuration in the DMCONFIG file.

2. You need to put an additional Java option in the WLS start-up command as follows:

   -Djava.net.preferIPv4Stack=true     java command-line
   

Databases

There is nothing special you need to do to use the database or XA interface, as this standard is widely supported on all the major database vendor products. You may want to use SDP for Oracle database invocations, but again, you don't need to perform anything special in a Tuxedo application. You do however need to configure the database to support InfiniBand. You can set the two parameters in /etc/modprobe.conf on the server node for better performance (for example, options ib_sdp sdp_zcopy_thresh=0 recv_poll=0). It's default value is 64 KB, but the recommendation is to set it with zero.

The EXALOGIC_MSGQ_CACHE_SIZE variable

The EXALOGIC_MSGQ_CACHE_SIZE environment variable can be used to improve the performance of the Tuxedo application. This value can be set between 32 and 2,048. One thing to notice is that increasing the number can improve Tuxedo's performance, but Msgq_daemon consumes more shared memory. So, setting this environment variable will help you to get better performance during the process of sending multiple messages to many queues.

Please read the following recommendation from Oracle:

If there are 40 remote Oracle Tuxedo servers providing the same service and clients call the service 100 times, setting EXALOGIC_MSGQ_CACHE_SIZE to a value equal to or greater than 40 on the client improves performance.

If there are 50 WSHs, and each WSH receives response messages from the same remote server, setting EXALOGIC_MSGQ_CACHE_SIZE to a value equal to or greater than 50 on the server environment improves performance.