- Did the cron run
- Artifactory Internal Errors
- Debugging sql queries
- Local platform installation and apache forbidden errors
- Swagger : 500 Internal Server Error
- Curl command to test FIS
- LinkageError when starting Wallet/Admin in Windows
Did the cron run
- In /var/log/cron{.1,.2 etc} you will find entried like the one below -- it tellu that as wcuser the cron job was run at that time and it trigerred the script /opt/WaveCrest etc
- We traced this from May 6th onwards – on each day we had entries close to midnight and 21:30 hours.
- So looks like the linux cron did its job of triggering the scripts
- So mallik is going to check why the application gave the error – and does the application have a tracing mechanism whcy says “ the script got triggered on this data and time” – so for now we need to trouble shoot the app script and see why there was an exception , it does not seem like a OS failed to trigger to cron job
/var/log/cron.1:May 8 21:40:01 wcapp3 crond[12394]: (root) CMD (/usr/lib64/sa/sa1 1 1)
/var/log/cron.1:May 8 21:40:01 wcapp3 crond[12395]: (wcuser) CMD (/opt/WaveCrest/scripts/ach/metabank/return/return_master.sh)
/var/log/cron.1:May 8 21:45:01 wcapp3 crond[12488]: (wcuser) CMD (/opt/WaveCrest/scripts/ach/metabank/mt_settle/mt_settle_master.sh)
/var/log/cron.1:May 8 21:45:01 wcapp3 crond[12487]: (root) CMD (date >> /root/date.txt)
/var/log/cron.1:May 8 21:50:01 wcapp3 crond[12588]: (wcuser) CMD (/opt/WaveCrest/scripts/ach/metabank/load_settle/load_settle_master.sh)
/var/log/cron.1:May 8 21:50:01 wcapp3 crond[12587]: (root) CMD (date >> /root/date.txt)
/var/log/cron.1:May 8 21:54:01 wcapp3 crond[12659]: (root) CMD (date >> /root/date.txt)
/var/log/cron.1:May 8 21:55:01 wcapp3 crond[12666]: (wcuser) CMD (/opt/WaveCrest/scripts/ach/metabank/mt_failed_email/email_master.sh)
/var/log/cron.1:May 8 22:00:01 wcapp3 crond[12712]: (root) CMD (date >> /root/date.txt)
/var/log/cron.1:May 8 22:00:01 wcapp3 crond[12713]: (wcuser) CMD (/opt/WaveCrest/scripts/ach/metabank/mt_settle_email/email_master.sh)
Solution:
Try this -- i removed pid manually and started the redis successfully
Worstcase -- my redis error is resolved only after removing the existing vagrant box and again re-adding the box
/var/log/cron.1:May 8 22:00:01 wcapp3 crond[12713]: (wcuser) CMD (/opt/WaveCrest/scripts/ach/metabank/mt_settle_email/email_master.sh)
Artifactory Internal Errors
- Some time while running the build and it is trying to deploy a new dependency – artifactory gives internal errors saying it cannot save the resource (HTTP 500)
- Please login to artifactory directly and see if typing the URL on the brpwser (the one which gave the error) behaves in the same way.
Please restart artifactory to solve the problem.
Debugging sql queries
- To enable general log : set GLOBAL general_log=ON ;
- To check the path of general log : show variables like '%general_log_file' ;
- Navigate to the path in step 2, you can check the sql queries
Local platform installation and apache forbidden errors
If you get error like below
Forbidden
You don't have permission to access /v2/tpl/index.html on this server.
Reason IS : you don't have write permssion on /home/wcuser. Change to 755 /home/wcuser using chmod 755 /home/wcuser – you need super user permission
Swagger : 500 Internal Server Error
Problem:
500 : Internal Server Error http://localhost:8080/apisandbox/apidocs/.svn?api_key=special-key
Encountered when loading accessing REST API docs from Swagger.
Quick Fix:
The folder by the name jsonapi has a folder by the name .svn and is trying to read the contents.
Delete the folder and reload the page.
Reload
If deleting the folder doesn't fix the problem,
Refresh apisandbox and Restart tomcat and reload the swagger page.
Refresh apisandbox and Restart tomcat and reload the swagger page.
Redis connection issues on vargrant
Symptoms:
1. org.springframework.dao.DataAccessResourceFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:101)
2. when I tried to restart - it is saying /opt/redis-2.6.7/utils/redis_init_script start
/opt/redis-2.6.7/utils/../bin/redis.pid exists, process is already running or crashed
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:101)
2. when I tried to restart - it is saying /opt/redis-2.6.7/utils/redis_init_script start
/opt/redis-2.6.7/utils/../bin/redis.pid exists, process is already running or crashed
Cause:
vagrant up itself as vagrant box got corrupted
vagrant up itself as vagrant box got corrupted
Solution:
Try this -- i removed pid manually and started the redis successfully
Worstcase -- my redis error is resolved only after removing the existing vagrant box and again re-adding the box
Curl command to test FIS
Here is the trace of curl command from my laptop.
LinkageError when starting Wallet/Admin in Windows
Should you face some error like below
"java.lang.LinkageError: loader constraint violation: loader (instance of org/apache/catalina/loader/WebappClassLoader) previously initiated loading for a different type with name "javax/servlet/FilterChain"
Possible Reasons : There is some conflict in the classes being loaded. On my IDE, I had Groovy plugin installed and there was a version mismatch between some jars. So, I tried uninstalling the plugins and tomcat was up successfully.
I was running Jdk_1.6
Tomcat threads
All the threads in a threaddump marked as "http-" are your potential tomcat managed server threads for request processing –
1. Out of all the threads with http – you can see many classes of them
a. Threads processing requests actively
b. Threads idle in the pool - -something like a socketRead etc and waiting to process future requests
c. Also there will be some which are there for handling persistent socket connections between apache and tomcat – I think the ajp or the http type connections
d. Also there may be difference in thread handling when u use blocking socket vs non-blokcing (different connectors I think)
Tomcat threads vs Threads in the Threads dump
Actually you can do this – take a thread dump at the JVM level
You OS level thread count that you are reporting will be close to that and will consist of the following
1. Currently used tomcat threads (in the thread dump grep for thread names starting with http-“
2. All the frameworks internally used start their own threads – for example bone and c3po connection pool have their owne threads
3. Internal JVM gc threads
4. Other app threads
5. Etc
So the sum will reflect all these
Issue 1: Checksum error while committing a file:
Whenever you try to check-in a file and encountered the checksum issue, there are two ways to resolve it.
- Remove the original file from main repository (you need BCM help to remove) and add your latest changes as a new file.
Or
You can go to the mentioned file: ...\.svn\text-base\MasterPropsDao.java.svn-base file and make it sync with your latest file. - clean up the project
- commit changes
We can use the tool "Tattletale" for generating the following dependencies and reports, so that it would help us to analyse the application and make us to take decision in keeping the correct JARs as part of the War file.
Dependencies
- Dependants (INFO)
- Depends On (INFO)
- Graphical dependencies (INFO)
- Transitive Dependants (INFO)
- Transitive Depends On (INFO)
Reports
- Class Location (INFO)
- OSGi (INFO)
- Eliminate Jar files with different versions (WARNING)
- Invalid version (WARNING)
- Multiple Jar files (WARNING)
- Multiple Locations (WARNING)
- Black listed (ERROR)
- No version (ERROR)
Steps to Install and generate the reports:
- Download the zip file from below location :
http://www.mastertheboss.com/jboss-application-server/216-jboss-client-jar.htm - Unzip the above zip file and CD to the unzip directory.
- Run the below command to generate the reports
java -Xmx512m -jar jboss-tattletale.jar /home/wcuser/O2-Cashier/webapps/cashier/WEB-INF/lib/ /tmp/output - Open the index.html file in output folder to view the reports/results.
http://prefetch.net/blog/index.php/2006/10/28/dumping-http-requests-and-reponses-to-the-apache-error-log/
To enable the module, you will first need to load the module with “LoadModule” directive. Once the module is configured to load, you can add the “DumpIOInput On” directive to to dump incoming data, and “DumpIOOutput On” to dump data sent by Apache
To enable the module, you will first need to load the module with “LoadModule” directive. Once the module is configured to load, you can add the “DumpIOInput On” directive to to dump incoming data, and “DumpIOOutput On” to dump data sent by Apache
mod_dumpio
allows for the logging of all input received by Apache and/or all output sent by Apache to be logged (dumped) to the error.log file.
The data logging is done right after SSL decoding (for input) and right before SSL encoding (for output). As can be expected, this can produce extreme volumes of data, and should only be used when debugging problems.
Changes to /etc/httpd/conf/httpd.conf
Load module DUMP IO
------------------------
LoadModule dumpio_module modules/mod_dumpio.so
------------------------
LoadModule dumpio_module modules/mod_dumpio.so
Changes to /etc/httpd/conf.d/httpd_*.conf
Enable logging add to virutal host
------------------------------------
DumpIOOutput On
------------------------------------
DumpIOOutput On
DumpIOInput On
DumpIOLogLevel debug
How to Check for loaded module
-----------------------------------------------
-----------------------------------------------
apachectl -t -D DUMP_MODULES | grep dump
Be Careful in Enabling this particular level of logging. This particular log level churned out logs of 265 Gigs and the server stopped responding.
Log level Warning
[wcuserro@plapp1 logs]$ ls -alh
total 264G
drwxrwxr-x 2 wcuser wcuser 4.0K Oct 26 09:13 .
drwxrwxr-x 14 wcuser wcuser 4.0K Dec 1 13:37 ..
-rwxrwxr-x 1 wcuser wcuser 864K Dec 3 12:35 httpd-static-access.log
-rwxrwxr-x 1 wcuser wcuser 264G Dec 3 12:13 httpd-static-error.log
-rwxrwxr-x 1 wcuser wcuser 804K Dec 3 12:35 httpd-static_ssl_request_log
-rwxrwxr-x 1 wcuser wcuser 3.4M Dec 3 12:35 rewrite.log
[wcuserro@plapp1 logs]$ less httpd-static-error.log
[wcuserro@plapp1 logs]$
total 264G
drwxrwxr-x 2 wcuser wcuser 4.0K Oct 26 09:13 .
drwxrwxr-x 14 wcuser wcuser 4.0K Dec 1 13:37 ..
-rwxrwxr-x 1 wcuser wcuser 864K Dec 3 12:35 httpd-static-access.log
-rwxrwxr-x 1 wcuser wcuser 264G Dec 3 12:13 httpd-static-error.log
-rwxrwxr-x 1 wcuser wcuser 804K Dec 3 12:35 httpd-static_ssl_request_log
-rwxrwxr-x 1 wcuser wcuser 3.4M Dec 3 12:35 rewrite.log
[wcuserro@plapp1 logs]$ less httpd-static-error.log
[wcuserro@plapp1 logs]$