Notes from my CA Unicenter Days - NSM, NPO, Software Delivery and Asset Management

NOTE: These are from my old notes. Not sure how much of this is valid or correct. I don't remember much of it. Proceed with caution !!

Internet Router --> Unable to discover

1. If the community name is customised, then go to 'pollset browser' by typing 'pollsets' in the command line. Create a new pollset for this community name.

2. Type "clean_nsm" in command line. This will stop the awservices.

3. Start "awservices" by typing "awservices start" at the command prompt. Also start "rmonapp" if it’s not already started.

4. Go to C:\NSM\SERVICES\CONFIG\AWS_NSM\DM\ or the equivalent installation directory and check if mib-2-22.dat or else check in the folder EXCLUDE for this file & copy it to C:\...\DM folder.

5. Now run reset DSM command by typing "resetdsm" at the command prompt. This will take a long time and stop awservices, which has to be restarted along with rmonapp.

6. Check in 2D map for this particular router. If it is present, delete it including child objects and rediscover it. If it’s not present, then discover it.

7. dscvrbe -r nwnsm -7 <IP ADDRESS> -5 yes -V yes -l 9 -f no -6 no -9 yes

8. After discovery, go to Open Details -> Others. Enter the DSM Server and address as the server (NPO-NSM-[nwnsm] server).

To get the object id of the router us

awget -h <IP ADDRESS> -p <PORT, usually 161> -c <Community Name> -o <OID>

awget -h <IP ADDRESS> -p <PORT, usually 161> -c <Community Name> -o 1.3.6.1.2.1.1.2.0

Various Commands

unifstat                            Unicenter Enterprise Services Status - To see active services (for troubleshooting)

unicntrl start/stop all/name of the service

                                        To start / stop Unicenter Enterprise Service

ccicntrl                            CCI --> Common CA Interface

dscvrbe -7 <IP ADDRESS> -v 9

                                        Gives SNMP information

catng2D                          2D map

caugui conlog                 Console Log

caugui msgrecord           Message Records

objview                           Object View

rmonaws                         NPO Admin Interface

caugui message              messages

awservices                     AW Services

awservices status           To see installed service

obrowser

mibbrowser                    MIB Browser

objbrows                        Object Browser

pollsets                           Opens Pollset Browser

univer                             Version of CCS and NSM

sdver                              Version of software Delivery

Version of Asset Management --> go to folder called domain, domain.ini will have the version

UNICNTRL                   Command was used when according to UNIFSTAT, there was the SNMP service as inactive.

PORTS – Don’t know if these are correct !!

Ports for SD:

TCP - 4721, 8222 & 8230 to 8235

UDP - 4104, 4725

Ports for ITRM

UDP - 4104, 4105, 4725

TCP - 4104, 4721, 8222 & 8230 to 8235

Ports used in NSM - NPO

UDP - 161, 162, 4105, 6665, 7001

TCP - 4104, 7001

Ports used for AM

TCP - 4104

UDP - 4105

Basics of NATTING - Static & Dynamic

NAT - Network Address Translation

This is not directly related to OM but since there is a lot of environments on the cloud, I thought it would be a good place to touch base on one of the basic networking concepts.

NATTING enables a LAN to use one set if IP addresses for internal traffic and a second set of addresses for external Traffic. A NAT box is located where the LAN meets the Internet. It makes all the required IP address translations.

NAT serves 3 purposes:

1.  Provides a type of firewall by hiding Internal IP Addresses

2. Enables a company to use more internal IP addresses. Since these IP addresses are used only internally, there is no possibility of conflict with IP addresses used by other companies / organizations.

3. Allows a company to combine multiple ISDN connections into a single internet connection.

There are 2 types of NAT:

Static NAT

Dynamic NAT

Static NAT: Mapping an unregistered IP address to a registered IP on a one-to-one basis. Particularly useful when a device needs to be accessible from outside network always.

Photo Courtesy : http://computer.howstuffworks.com/nat1.htm

Dynamic NAT: Maps an unregistered IP address to a registered IP address from a group of registered IP addresses.

Photo Courtesy : http://www.brainbump.net/Understanding-CiscoASA-Post-8.3-NAT-Configuration

Overloading / PAT: Port Address Translation

This kind of NAT maps multiple unregistered IP addresses to a single registered IP address by using different ports. This is also called single address NAT or port-level multiplexed NAT.

Overlapping:

When the IP addresses used on the internal network are registered IP addresses in use on another network, the router must maintain a lookup table of those addresses so that it can intercept them ans replace them with registered unique IP addresses. NAT router must also translate the "INTERNAL" addresses to registered unique addresses as well as translate the "EXTERNAL" registered addresses to addresses that are unique to the private network. This can be done through static NAT or by implementing DNS and using Dynamic NAT.

NAT Example:

a.b.c.d = Public Internet IP which is NAT'd to 192.168.1.5

Let's say you have a website www.basicesm.blogspot.com whose public DNS points to a.b.c.d, and you have an IIS website hosted on 192.168.1.5 on your internal network.  Public on the internet would open a browser type in www.basicesm.blogspot.com, which would look up the IP a.b.c.d.  It would request the webpage and hit your firewall where it would NAT to the internal IP 192.168.1.5, and return the webpage.

Remedy SPI - Config file & Rules Files

HP OM is an Event Management tool. For the complete ITIL workflow this has to be integrated with any one of the ITSM tools available in the market.

We have used Remedy Smart Plug-in(remspi) by HP to integrate HP OVO with BMC Remedy Action Request System (ARS) in our environment.

The above diagram shows a basic flow within the integration.

This post expects the reader to be well versed with HP OVO and Remedy ARS. Also refer to the SPI_RARS_AdminRef.pdf for more information.

This post gives a brief example of the cfg file and the rules file.

1. The SPI for Remedy ARS server receives the ID of an HPOM

message from the HPOM management server via a submission

program.

2. The SPI for Remedy ARS server retrieves details of the HPOM

message from the HPOM management server via an application

programmer interface (API).

3. The SPI for Remedy ARS server creates and updates action requests

using the ARS API.

4. When an action request changes, the action-request system calls the SPI for Remedy ARS client and passes on the details of the update via the ARS filter mechanism.

5. The SPI for Remedy ARS client sends the update details to the SPI for Remedy ARS server.

6. The SPI for Remedy ARS server updates the HPOM message on the HPOM management server using HPOM’s API.

7. When an HPOM message is modified, the SPI for Remedy ARS server receives a message-change event via the HPOM API.

8. HPOM sends the details of the message-change event to the SPI for

Remedy ARS server via the HPOM API.

9. The SPI for Remedy ARS server updates the action request by means of the Remedy-ARS application-programming interface (API).

The data components that are used in the data flow are:

• External Actions and Data
• Configuration file - remspi.cfg
• Rules File - remspi_rules.txt
• The SPI for Remedy ARS database

Configuration File - /etc/opt/OV/share/conf/remspi/remspi.cfg

The configuration file remspi.cfg defines the user name and passwords that the SPI for Remedy ARS uses to log in to HPOM. The passwords are encrypted.

There has to be a user in HPOM with appropriate permissions in HPOM.

Use remspipasswd  to encrypt the password of this user.

remspipasswd <new_user_password>

Example of a configuration file : 

# File: remspi.cfg

# Description: Configuration file for Remedy(tm) SPI

# Package: HP Operations SMART Plug-In for

# Remedy(tm) Action Request System(r)

# Note: Value must be on same line as keyword

REMSPI_ITO_USER remspi_server

REMSPI_ITO_PASSWD C338D5F21219E076C2000C45AA0475925A1

REMSPI_ITO_ADMIN_PASSWD C338D5F21219F076C2000C4VAA0475925A1

Rules File: /etc/opt/OV/share/conf/remspi/rules

Sample Rules file: This is an example from my test environment

SYNTAX_VERSION 2.4

SPI_RULES "SPI Rules - ProblemReport"

DESCRIPTION "Example rules for the Problem Report System"

TARGET "Probsys"

  SCHEMA      "Problem Report"

  USER        "ovouser"

  SERVER      "REMEDY_PRODSERVER"

  BACKUPUP_SERVER   "REMEDY_BACKUPSERVER"

  PASSWORD "KMIDDKLFN9R43093LNSVNDIFDFLN49RRLADMC1233NDS"

                   FIRST FIELD           2   "$ITO_SUBMITTER$"

                         FIELD           7   "New"

                   FIRST FIELD           8   "$MSG_TEXT$"

  UPDATE OVERWRITE FIRST FIELD  8000000078   "$MSG_TEXT$"

                   FIRST FIELD  8000000079   "\\n$MSG_ANNO$\\n"

  UPDATE OVERWRITE FIRST FIELD  8000000080   "$MSG_SEVERITY$"

                   FIRST FIELD  8000000081   "$MANAGEMENT_SERVERS$"

                   FIRST FIELD  8000000082   "$NODE_NAME$"

                   FIRST FIELD  8000000083   "$NODE_APPL$"

                   FIRST FIELD  8000000084   "$MSG_GRP$"

                   FIRST FIELD  8000000085   "$MSG_OBJ$"

                   FIRST FIELD  8000000086   "$MSG_ID$"

                   FIRST FIELD  8000000087   "$Instructions$"

  UPDATE OVERWRITE ALL   FIELD  8000000088   "$ACK_USER$"

                   FIRST FIELD  8000000089   "$EXTERNAL Severity$"

  ON_SUBMIT

    OWN                  #OVO message is owned after creation

    MSGTXT_PREFIX   "ARS-ID=$ARS_ID$:  "

  ITO_UPDATE

    ANNOTATE      7 "Closed" "Action request closed by $5$ on $6$"

    FORCE OWN     7 "Assigned"

    OWN           7 "Fixed"

    DISOWN        7 "Rejected"

    ACKNOWLEDGE   7 "Closed"

    UNACKNOWLEDGE 7 "Reopened"

    OP_ACTION       "operator-action"

    ESCALATE        "escalate"

    ANNOTATE        "Action request modified"

        Latest work log entry: $8000000079$

  MSGCONDITIONS

    DESCRIPTION   "Catch All"

    CONDITION

    SET TARGET "Probsys"

       FIELD 8000000083   "$NODE_APPL$"

 SUPPRESSCONDITIONS

    DESCRIPTION   "Condition Suppress"

    CONDITION

       MSGGRP "Printers"

       Severity "Warning"

    

A little bit of explanation of the rules file

8000000080 all these numbers relate to the field id in the REMEDY ARS form that gets called when remspi gets invoked. Here the form is called "PROBLEM REPORT"

MSG_SEVERITY --> 8000000080  is the Severity of the alert that comes in OM

EXTERNAL severity --> 8000000089 is the severity chosen by the operator (DC-ops uses semi-automation to create tickets)

8000000079   "\\n$MSG_ANNO$\\n"--> this goes into the work log of the ticket

MSGCONDITIONS are the conditions that have to satisfy for a ticket to be created.

SUPPRESSCONDITIONS are the conditions which have to be true for a message to be filtered out.

Sometimes you might have issues when the RemedySPI process doesn't start with ovstart. This behavior seems to be normal. The remspisrv is integrated into ovstop/ovstart through the remspisrv.lrf file. ovstop / ovstart is the official way to stop and start. ovstop stops RemedySPI, but ovstart starts the OVO processes but ovspmd doesn't know the status of the previous RemedySPI and doesn't start it again. You would have to perform “ovstart RemedySPI” to start it.

Another issue that I have faced with remspi is that remspiconfig and remspifilter produce the following errors.

Can't lock DBM database files '/var/opt/OV/share/tmp/remspi/Probys'

for target 'Probsys'. Permision Denied (SPI215-45)

&

Can't lock DBM database files '/var/opt/OV/share/tmp/remspi/_Remspi'

for target '_Remspi'. Permision Denied (SPI215-45)

Well, the technical document (Doc ID emr_na-c00917549-1) says :

These error messages occur when the Remedy SPI process is running due to the fact that the process locks these files.

Run "ovstop -c RemedySPI", then run remspiconfig

This also pertains to the remspifilter command.

Disable HTTPS agent tracing – ovtrccfg

Every weekend, my QA OM server will be stopped for a offline Oracle backup. This weekend, after the backup, OM never started. I tried to start and it wouldn't. On checking the log System.txt, I found these :

0: ERR: Mon Jul 28 06:01:16 2014: ovcd (23454/47264176451744): (ctrl-127) Could not write the ovcd pid to file: 'No space left on device'.
0: INF: Mon Jul 28 06:02:17 2014: ovc (23451/47221706264736): (ctrl-103) Could not start the Control daemon.
0: ERR: Mon Jul 28 06:02:19 2014: ovcd (25940/1103829312): (sec.cm.client-23) Error during CertificateClient initialization.
1: ERR: Mon Jul 28 06:02:19 2014: ovcd (25940/1103829312): (xpl-89) write(13)[19720B50] failed.
2: ERR: Mon Jul 28 06:02:19 2014: ovcd (25940/1103829312): (RTL-28) No space left on device
0: ERR: Mon Jul 28 06:02:19 2014: ovcd (25940/1103829312): (ctrl-63) Error initializing RPC server: Can not initialize RpcServer. error=(xpl-89) write(16)[19724D60] failed.
 (RTL-28) No space left on device.

 /var/opt/OV was 100% full.

Tracing was enabled and the trace file was almost 13GB.

I deleted the trace file, disabled tracing and tried to start opcsv.

#>>/opt/OV/support/ovtrccfg -help (used for tracing)

#>> /opt/OV/support/ovtrcadm -hosts

The following are the allowed clients for TraceServer

S. No. Client Name or IP address
-------- -------------------------

This is used to list the nodes that are configured to trace.

#>>/opt/OV/support/ovtrcadm -srvconfig 

TraceServer is running

Application Components Categories Level Enabled
----------- ---------- ---------- -------------
com.hp.openview.OvDiscoveryServer.OvDiscoveryServer
org.apache.catalina.startup.Bootstrap
java.lang.Thread
com.hp.ov.svcdisc.SvcDiscClient

gives the list of what is being traced. 

I used the following command to disable tracing.

ovtrccfg -off

This disabled the tracing and I was able to start OM using opcsv.