Monday, July 9, 2018

Troubleshooting Tips



Did the cron run

  1. In /var/log/cron{.1,.2 etc} you will find entried like the one below  -- it tellu that as wcuser the cron job was run at that time and it trigerred the script /opt/WaveCrest etc
  2. We traced this from May 6th onwards – on each day we had entries close to midnight and  21:30 hours.  
  3. So looks like the linux cron did its job of triggering the scripts
  4. So mallik is going to check why the application gave the error – and does the application have a tracing mechanism whcy says “ the script got triggered on this data and time” – so for now we need to trouble shoot the app script and see why there was an exception ,  it does not seem like a OS failed to trigger to cron job

/var/log/cron.1:May  8 21:40:01 wcapp3 crond[12394]: (root) CMD (/usr/lib64/sa/sa1 1 1)
/var/log/cron.1:May  8 21:40:01 wcapp3 crond[12395]: (wcuser) CMD (/opt/WaveCrest/scripts/ach/metabank/return/return_master.sh)
/var/log/cron.1:May  8 21:45:01 wcapp3 crond[12488]: (wcuser) CMD (/opt/WaveCrest/scripts/ach/metabank/mt_settle/mt_settle_master.sh)
/var/log/cron.1:May  8 21:45:01 wcapp3 crond[12487]: (root) CMD (date >> /root/date.txt)
/var/log/cron.1:May  8 21:50:01 wcapp3 crond[12588]: (wcuser) CMD (/opt/WaveCrest/scripts/ach/metabank/load_settle/load_settle_master.sh)
/var/log/cron.1:May  8 21:50:01 wcapp3 crond[12587]: (root) CMD (date >> /root/date.txt)
/var/log/cron.1:May  8 21:54:01 wcapp3 crond[12659]: (root) CMD (date >> /root/date.txt)
/var/log/cron.1:May  8 21:55:01 wcapp3 crond[12666]: (wcuser) CMD (/opt/WaveCrest/scripts/ach/metabank/mt_failed_email/email_master.sh)
/var/log/cron.1:May  8 22:00:01 wcapp3 crond[12712]: (root) CMD (date >> /root/date.txt)
/var/log/cron.1:May  8 22:00:01 wcapp3 crond[12713]: (wcuser) CMD (/opt/WaveCrest/scripts/ach/metabank/mt_settle_email/email_master.sh)



/var/log/cron.1:May  8 22:00:01 wcapp3 crond[12713]: (wcuser) CMD (/opt/WaveCrest/scripts/ach/metabank/mt_settle_email/email_master.sh)

Artifactory Internal Errors

  1. Some time while running the build and it is trying to deploy a new dependency – artifactory gives internal errors saying it cannot save the resource (HTTP 500)
  2. Please login to artifactory directly and see if typing the URL on the brpwser (the one which gave the error) behaves in the same way.
  3. (green star)  Please restart artifactory to solve the problem.

Debugging sql queries

  1. To enable general log :  set GLOBAL general_log=ON ;
  2. To check the path of general log : show variables like '%general_log_file' ;
    Show Variables
    mysql> show variables like '%general_log_file' ;
    +------------------+-------------------------------+
    | Variable_name    | Value                         |
    +------------------+-------------------------------+
    | general_log_file | /var/lib/mysql/newtonvm20.log |
    +------------------+-------------------------------+
    1 row in set (0.00 sec)
  3. Navigate to the path in step 2, you can check the sql queries

Local platform installation and apache forbidden errors

If you get error like below 
Forbidden
You don't have permission to access /v2/tpl/index.html on this server.
Reason IS :  you don't have write permssion on /home/wcuserChange to 755 /home/wcuser  using  chmod 755 /home/wcuser – you need super user permission



Swagger : 500  Internal Server Error

Problem:
Encountered when loading accessing REST API docs from Swagger.
Quick Fix:
The folder by the name jsonapi has a folder by the name .svn and is trying to read the contents.
Delete the folder and reload the page.

Reload
If deleting the folder doesn't fix the problem,
Refresh apisandbox and Restart tomcat and reload the swagger page.

 Redis connection issues on vargrant
Symptoms:
1. org.springframework.dao.DataAccessResourceFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:101)

2. when I tried to restart - it is saying /opt/redis-2.6.7/utils/redis_init_script start
/opt/redis-2.6.7/utils/../bin/redis.pid exists, process is already running or crashed
Cause:
vagrant up itself as vagrant box got corrupted

Solution:
Try this -- i removed pid manually and started the redis successfully
Worstcase -- my redis error is resolved only after removing the existing vagrant box and again re-adding the box

Curl command to test FIS

Here is the trace of curl command from my laptop.
Enter host password for user 'WCa2aTest':
* About to connect() to a2atest.wildcardsystems.com port 443 (#0)
*   Trying 156.55.165.26... connected
* Connected to a2atest.wildcardsystems.com (156.55.165.26) port 443 (#0)
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Request CERT (13):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using RC4-MD5
* Server certificate:
*        subject: C=US; ST=Florida; L=Jacksonville; O=Fidelity National Information Services; OU=FIS - Prepaid; CN=a2atest.wildcardsystems.com
*        start date: 2013-03-11 00:00:00 GMT
*        expire date: 2014-03-12 23:59:59 GMT
*        subjectAltName: a2atest.wildcardsystems.com matched
*        issuer: C=US; O=VeriSign, Inc.; OU=VeriSign Trust Network; OU=Terms of use at https://www.verisign.com/rpa (c)10; CN=VeriSign Class 3 Secure Server CA - G3
*        SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Server auth using Basic with user 'WCa2aTest'
> GET /a2a/documentation/toc.htm HTTP/1.1
> Authorization: Basic V0NhMmFUZXN0OldheXZLcmVzdDM5ODI=
> User-Agent: curl/7.21.4 (i386-pc-win32) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
> Host: a2atest.wildcardsystems.com
> Accept: */*
>
< HTTP/1.1 403 Forbidden
< Date: Thu, 30 Jan 2014 03:10:57 GMT
< Content-Length: 1913
< Content-Type: text/html
< Server: WWW Server/1.1
< X-Powered-By: ASP.NET

LinkageError when starting Wallet/Admin in Windows

Should you face some error like below
"java.lang.LinkageError: loader constraint violation: loader (instance of org/apache/catalina/loader/WebappClassLoader) previously initiated loading for a different type with name "javax/servlet/FilterChain"
Possible Reasons :  There is some conflict in the classes being loaded. On my IDE, I had Groovy plugin installed and there was a version mismatch between some jars. So, I tried uninstalling the plugins and tomcat was up successfully.
I was running Jdk_1.6


Skip to end of metadata
Go to start of metadata

Tomcat threads

All the threads in a threaddump marked as "http-"  are your potential tomcat managed server threads for request processing – 
1. Out of all the threads with http – you can see many classes of them
a. Threads processing requests actively
b. Threads idle in the pool - -something like a socketRead etc and waiting to process future requests
c. Also there will be some which are there for handling persistent socket connections between apache and tomcat – I think the ajp or the http type connections
d. Also there may be difference in thread handling when u use blocking socket vs non-blokcing (different connectors I think)

Tomcat threads vs Threads in the Threads dump

Actually you can do this – take a thread dump at the JVM level
You OS level thread count that you are reporting will be close to that and will consist of the following
1. Currently used tomcat threads (in the thread dump grep for thread names starting with http-“
2. All the frameworks internally used start their own threads – for example bone and c3po connection pool have their owne threads
3. Internal JVM gc threads
4. Other app threads
5. Etc
So the sum will reflect all these


Skip to end of metadata
Go to start of metadata
Issue 1: Checksum error while committing a file:

Whenever you try to check-in a file and encountered the checksum issue, there are two ways to resolve it.
  1. Remove the original file from main repository (you need BCM help to remove) and add your latest changes as a new file.
    Or
    You can go to the mentioned file: ...\.svn\text-base\MasterPropsDao.java.svn-base file and make it sync with your latest file.
  2. clean up the project
  3. commit changes




Skip to end of metadata
Go to start of metadata
We can use the tool "Tattletale" for generating the following dependencies and  reports, so that it would help us to analyse the application and make us to take decision in keeping the correct JARs as part of the War file.

Dependencies

Reports

Steps to Install and generate the reports:

  1. Download the zip file from below location :
    http://www.mastertheboss.com/jboss-application-server/216-jboss-client-jar.htm
  2. Unzip the above zip file and CD to the unzip directory.
  3. Run the below command to generate the reports
    java -Xmx512m  -jar jboss-tattletale.jar  /home/wcuser/O2-Cashier/webapps/cashier/WEB-INF/lib/ /tmp/output
  4. Open the index.html file in output folder to view the reports/results.


Skip to end of metadata
Go to start of metadata
http://prefetch.net/blog/index.php/2006/10/28/dumping-http-requests-and-reponses-to-the-apache-error-log/
To enable the module, you will first need to load the module with “LoadModule” directive. Once the module is configured to load, you can add the “DumpIOInput On” directive to to dump incoming data, and “DumpIOOutput On” to dump data sent by Apache
mod_dumpio allows for the logging of all input received by Apache and/or all output sent by Apache to be logged (dumped) to the error.log file.
The data logging is done right after SSL decoding (for input) and right before SSL encoding (for output). As can be expected, this can produce extreme volumes of data, and should only be used when debugging problems.

Changes to /etc/httpd/conf/httpd.conf

Load module DUMP IO
------------------------
LoadModule dumpio_module modules/mod_dumpio.so

Changes to /etc/httpd/conf.d/httpd_*.conf

Enable logging add to virutal host 
------------------------------------
DumpIOOutput On
DumpIOInput On
 DumpIOLogLevel debug
How to Check for loaded module
-----------------------------------------------
apachectl -t -D DUMP_MODULES | grep dump

Be Careful in Enabling this particular level of logging. This particular log level churned out logs of 265 Gigs and the server stopped responding.
Log level Warning
[wcuserro@plapp1 logs]$ ls -alh
total 264G
drwxrwxr-x 2 wcuser wcuser 4.0K Oct 26 09:13 .
drwxrwxr-x 14 wcuser wcuser 4.0K Dec 1 13:37 ..
-rwxrwxr-x 1 wcuser wcuser 864K Dec 3 12:35 httpd-static-access.log
-rwxrwxr-x 1 wcuser wcuser 264G Dec 3 12:13 httpd-static-error.log
-rwxrwxr-x 1 wcuser wcuser 804K Dec 3 12:35 httpd-static_ssl_request_log
-rwxrwxr-x 1 wcuser wcuser 3.4M Dec 3 12:35 rewrite.log
[wcuserro@plapp1 logs]$ less httpd-static-error.log
[wcuserro@plapp1 logs]$

Security Certificates

  1. Cryptography Basics Understand Key Concepts : Encryption, decryption, hashing, and digital signatures. Key terms: confidentiality, inte...