How to restart windows from command line ?

Are you running a build on windows? Is your build failing for no valid reason? Did you try every known options debugging the issue? Why don't you try your luck rebooting it?
Oops .. where is the restart restart button?

If you connected to your Windows machine through RDP (remote desktop) and if you don't find restart button, then use the below command to restart it.

   shutdown -t 0 -r -f

How to increase open files limit in Linux. Fix for java.io.FileNotFoundException (Too many open files) error

I was configuring a new build on Linux machine and it kept on failing with error

java.io.FileNotFoundException: /home/build/jenkins/workspace/OnPrem.AA-OnPrem.6.0.2.1-SP3-P4-HF50/configurations/target/classes/log4j.properties (Too many open files)

This error was occurring because our build is trying to utilize number of open File Descriptor beyond the limit.
By default in our RHEL 5.8 machine, the open Hard file limit was 1024. It can be checked with the command 
# ulimit -a|grep open      
open files (-n) 1024

or
# ulimit -Hn     
1024

#ulimit -Sn    
1024

Solution: Increase the Open file limit.  
Step 1:    vi /etc/security/limits.conf  
Step 2:  Add the below line in the end              
*                -       nofile          4096              
# End of file              

It will increase the open file limit to 4096 for all the users. 

Step 3: Verify it by running the below command in a new shell or session.
  # ulimit -a|grep open 

How to witness in a running build, a process is trying to overuse the file open limit?
Step 1: Start the build / Command. In my case I started the build from Jenkins
Step 2: Find out the process ID (PID) of the running task and note it down. In my case it is            ps -aef|grep java               
Step 3:  Then check the number of files under /proc/<PID>/fd folder            
  ls /proc/7455/fd|wc -l            
As the build progressed, I noticed the number of files keep on increasing from 20 to 1042 and immediately build failed with the above mentioned error.

What is Apache Hadoop?

Newbies can get a clean and simple introduction to Hadoop from the following Pivotal blog posts

1) Demystifying Apache Hadoop in 5 Pictures
2) Hadoop 101: Programming MapReduce with Native Libraries, Hive, Pig, and Cascading
3) 20+ Examples of Getting Results with Big Data

Highlights

• Hadoop is developed to assist in Big data Analysis
• Hadoop implements distributed computing to process large sets of data in a quick time-frame
• Hadoop divides and distributes work across large number of computers. It spreads data processing logic across 10s, 100s, or even 1000s of commodity servers.

Hadoop's main components

1) Hadoop Distributed File System (HDFS) : to help us split the data, put it on different nodes, replicate it, and manage it.

2) MapReduce: processes the data on each node in parallel and calculates the results of the job

     a) Map: Performs computation on local data set on each nodes and outputs a list of key-value pairs

     b) Reduce: The output from map step is sent to other nodes as input for the reduce step. Before reduce runs, the key-value pairs are typically sorted and shuffled. The reduce phase then sums the lists into single entries

3) Managing the Hadoop jobs: In Hadoop the entire process is called a job.

    a) Job tracker exists to divide the job into tasks and schedules tasks to run on the nodes. The job tracker keeps track of the participating nodes, monitors the processes, orchestrates data flow, and handles failures. 

    b) Task trackers run tasks and report to the job tracker. 

  With this layer of management automation, Hadoop can automatically distribute jobs on a large number of nodes in parallel and scale when more nodes are added .

Hadoop programming

There are 4 coding approaches

1) Native Hadoop library: It helps to achieve the greatest performance and have the most fine-grained control

2) Pig: similar to SQL and it is procedural, not declarative

3) Hive: Started by Facebook. It provides more SQL like interface, considered the slower of the languages to do Hadoop with.

4) Cascading: It is a set of .jars that define data processing APIs, integration APIs, as well as a process planner and scheduler

Perforce and cygwin

Are you a command-line freak ? Do you want your automated shell scripts to run on Windows ? Do you wish to work with Perforce commands on Cygwin?

You would have experienced the difficulty of running perforce commands on cygwin. The problems which occur due to cygwin's inability to convert Windows paths to it's own /cygdrive/c equivalent.
Example:
  $ p4 have pom.xml
       Path '/cygdrive/c/p4_workspaces/guruss1_p4server2_workspace/depot/On-Prem/AA/7.5.0.0/application/adaptive-authentication\pom.xml' is not under client's root 'C:\p4_workspaces\guruss1_p4server2_workspace'.

Solutions:

1) Use perforce cygwin client: 

    Perforce provide special 'p4.exe' which resolves all these issue. We just need to download it and make it available from default path.

    Where to download ?

       http://filehost.perforce.com/perforce/ 

       The available latest version which I found is     http://filehost.perforce.com/perforce/r12.1/bin.cygwinx86/ 

   The last release is made on May/2012 and hence it got the support for perforce's latest features like Streams, SSL , etc.

   But perforce announced the discontinuity of support for Cygwin p4 client from May/2013 

   Refer http://forums.perforce.com/index.php?/topic/1551-cygwin-client-support/ 

2) Add below function to your cygwin's ~/.bashrc

    function p4() {

        export PWD=`cygpath -wa .`

        /cygdrive/c/Program\ Files/Perforce/p4.exe $@

    }

    This will take care of all path related issues and we can use the same perforce client delivered along with P4V.

3) Add AltRoots: field to your workspace ( or client) spec

     Example:

       AltRoots: 

        /cygdrive/c/p4_workspaces/guruss1_p4server2_workspace   

    This doesn't solve all the problems as I have observed. It works for 'p4 sync', but it didn't for 'p4 have'

FeatureBranch v/s FeatureToggle v/s BranchByAbstraction patterns

This blog is excerpts from Martin Fowler blog and Continuous Delivery book (Jez Humble, David Farley). It explains about the concepts of
1) Feature Branch
    http://martinfowler.com/bliki/FeatureBranch.html
2) Feature Toggle
    http://martinfowler.com/bliki/FeatureToggle.html
3) Branch By Abstraction
    http://continuousdelivery.com/2011/05/make-large-scale-changes-incrementally-with-branch-by-abstraction/

    This post explains how to avoid branching in Source control.

How to restrict a Jira issue?

Why should we restrict Jira issue?
Typically Jira instance will be opened to non-developers like Customer Support, partners, QA and in some cases to Customers also. When a critical issue like security issue on our application is reported on Jira, we don't want non developers, especially partners/customers to view it. But still we may require other non-critical issues in our Jira project to be visible to all the members.

Solution:
1) We can restrict entire Jira project by creating a issue security scheme and appling it to a project.
2) We can make selective issues to be restricted.

In this blog, I'll describe to do the option 2.


1) Create an issue security scheme
Administration -> Issues -> Issue Security Schemes -> Add Issue Security Scheme -> enter a name & description -> Add

2) Add a security level to an issue security scheme
Administration -> Issues -> Issue Security Schemes -> Select your scheme -> Security Levels -> Add Security Level -> enter name and description (Ex: Development, Only developers can see this issue)
3) Add Users/Groups/Project Roles to a Security Level
Administration -> Issues -> Issue Security Schemes -> Select your scheme -> Security Levels -> Locate your security level -> Add -> Select the appropriate user, group or project role -> Add ( I selected a group, which has 5 members)
4) Assign an issue security scheme to a project
Administration -> choose your project from Projects -> Permissions -> Issues (None/Your issue security scheme) -> Select the issue security scheme that you want to associate with this project -> Associate
5) Setting Security on an Issue
Create/edit the relevant issue -> In the 'Security Level' drop-down field, select the desired security level for the issue -> When you save the issue, the issue will then only be accessible to members of that Security Level

Don't find 'Security Level' drop-down field on your issue? Here is what you need to do.
6) Modify your project permission scheme to allow users to set security levels

Administration -> Issues -> Permission Schemes
You can copy existing permission scheme and modify it or create new one. I prefer copy/modify

Select your permission scheme of your interest -> Copy -> It will create a copy -> Edit & provide name & description

Click permission -> Locate Set Issue Security -> Add -> I added a Group and Project Lead
This will allow your group members and Project Lead to set security levels on issue

7) Associate new permission scheme to your project
Administration -> choose your project from Projects -> Permissions -> Scheme -> Actions -> Use a different scheme

8) Add Security Level field to your project screen
Administration -> choose your project from Projects -> Screens -> Click on the screen scheme (MyORG Screen scheme) -> Select Screen (Defect screen)  -> Add Field -> Select Security Level -> Add

Now you should be able to find security level field on your issue. Follow Step 5, your issue shouldn't be visible to all.

Track/update Adobe Flash Player

Adobe Flash Player is a well known tool for exploitation by hackers. It is our responsibility to keep track of latest updates released by Adobe and make sure our system hosts latest Flash Player. Otherwise our systems are prone to security attacks.

Most of the browsers send alerts to upgrade Flash player, whenever latest versions are released. We just need to make few clicks and make sure it is installed.

How to check Adobe Flash version?
1)  Windows
     On windows XP/2003: Start -> Settings -> Control Panel -> Add/Remove Programs -> Locate Flash in the list -> Click on Adobe Flash Player XX Plugin -> Click here for support information
     This will display the version
     On windows 7: Start -> Control Pannel -> Programs -> Programs and Features ->  Locate Flash in the list -> You can notice version in the last column

2) Chrome Browser
    Type chrome:plugins in the address bar to open the Plug-ins page.
    On the Plug-ins page that appears, find the "Flash" listing.
    Verify the version

3) On IE or Firefox browser
    Open Flash about page http://www.adobe.com/software/flash/about/  . It displays your current version.
    Open Flash player find version page http://helpx.adobe.com/flash-player/kb/find-version-flash-player.html   . Notice your version in point 2


Download latest Adobe flash from : http://get.adobe.com/flashplayer

Why hackers penetrating Software Build infrastructure?

Many recent cyber attacks on enterprise involved penetrating into their product build infrastructure. The motive here is either to stole code signing certificates or to access source code. Build machines are the hot spots which either holds source code, certificates or it will be having access to machines which are hosting source code or certificates. In the recent attacks, the certificates are stolen from these machines and they used it to sign their own malware with well known enterprise certificates. They may lure some other customers to install their malware. Customers believe it as a genuine software, since it is signed by branded enterprise.

Hence build infrastructure, code signing environments are the critical resources of the organization which needs to be guarded with utmost care to prevent these advanced threats. The build engineers, release engineers needs to be proactive and adapt to these challenging scenarios and contribute in prevention of these advanced threats. 

A lethargic build/release engineer may become hackers bunny. Any successful attack with certificates will bring down that enterprise trust in the industry.

Refer below stories which describes how they got attacked.
Adobe: http://blogs.adobe.com/asset/2012/09/inappropriate-use-of-adobe-code-signing-certificate.html 
Bit9: https://blog.bit9.com/2013/02/08/bit9-and-our-customers-security/
RSA: http://blogs.rsa.com/anatomy-of-an-attack/

http://krebsonsecurity.com/2011/05/advanced-persistent-tweets-zero-day-in-140-characters/
https://krebsonsecurity.com/2013/02/security-firm-bit9-hacked-used-to-spread-malware/

My first experience with GlassFish Application server

GlassFish is a open source application server started by Sun Microsystems and now sponsored by Oracle. As part a POC (proof of concept) project, I started exploring GlassFish.

Installation:
I opted to install GlassFish in a Linux (RHEL 5.4) server, since I'm more comfortable on Linux.
Download GlassFish from http://glassfish.java.net/public/downloadsindex.html .
There are couple of options. I selected Open Source Distribution -> Full Platform -> GUI-based installer for Linux. It comes with a executable shell script like glassfish-3.1.2.2-unix.sh.
Problems faced:
Since my Linux server was a VMware VM, the GUI display of installer was not fitting to the full screen of vSphere console. Basically the installer opens a xterm and accessing xterm from putty remote console is not a straight forward option.
To get rid of it, I enabled vncserver on Linux machine and accessed the vnc session using UltraVNCViewer.
Now installer xterm got displayed perfectly, I chose typical installation and selected to setup service (init scripts). It created a default domain (domain1) and installation completed smoothly.


GlassFish defaults.
    Administration console: https://localhost:4848/
    How to restart?
          /etc/init.d/GlassFish_domain1 start/stop/restart

    Other details
     -----------------------
     Domain Name: domain1
     Admin port: 4848
     Http port: 8080
     Username: admin
     Server Log: <glassfish-installation-root>/glassfish/domains/domain1/logs

     -----------------------

Start/stop/list domains
asadmin command will be available in installation directory. Add it to path.
    asadmin start-domain
    asadmin stop-domain
    asadmin list-domains


Starting and Stopping the Database Server  (not used at this moment)
  To start the JavaDB server from its default location: asadmin start-database --dbhome as-install-parent/javadb
  To Stop the Java DB Server: asadmin stop-database

Starting the Administration Console
  Access: http://localhost:4848 and login.

  Problem faced: When tried to login this URL from remote machine, it gave error

     Glassfish version 3.1.2: Secure Admin must be enabled to access the DAS remotely.

  Resolution: 

    Activate the secure admin by using the following command line and restart glassfish.

        asadmin --host [host] --port [port] enable-secure-admin

  

        The host is the host name or IP address of the Glassfish Server.

        The port is the target Glassfish Server port i.e. 4848.  


Deploying and Undeploying the Sample Application Fromthe CommandLine
  asadmin deploy myapp.war

Access the application by typing the following URL in your browser:
  http://localhost:8080/myapp

To List Deployed Applications Fromthe CommandLine:
   asadmin list-applications

To Undeploy the Sample Application Fromthe CommandLine
   asadmin undeploy myapp


Deploying and Undeploying Applications by using the Administration Console
 Launch the Administration Console by typing the following URL in your browser: http://localhost:4848
 Click Applications in the left column -> Click the Deploy button -> Select Packaged File -> Browse ->     Specify a description -> click OK
 Select the check box next to the myapp application and click the Launch link to run the application.
 Access the application using https://10.31.177.78:8181/myapp
 Similarly it can be undeployed using undeploy button

Problem faced: Linux firewall has blocked the 8080 & 8181 ports. Unblocked it.


To Deploy the Sample Application Automatically
  cp myapp.war as-install/domains/domain-name/autodeploy

To Undeploy the Sample Application Automatically
  cd as-install/domains/domain-name/autodeploy
  rm myapp.war

Extracting tar.xz and tar.bz2 files

Extracting tar.xz files
xz files are one of the archive format generated using XZ Utils. XZ Utils (previously LZMA Utils) is a set of free command-line loss less data compressors, including LZMA and xz.

Check if the command xz already available in your system by running `which xz` . If not available, then download it from http://tukaani.org/xz/  and extract the source (Ex: tar xvfz xz-5.0.4.tar.gz )

To compile the source, run below commands in sequence
1) ./configure
2) make
3) make install

Make sure xz is installed succesfully by running `which xz` .

Uncompress your xz file using '-d' option.
  Ex:   xz -d coreutils-8.12.tar.xz
 
Extracting tar.bz2 files
bz2 files are bzip2 archive files. Extracting it is straight fowrward. Provide 'j' option to your tar command.
  tar xvfj glibc-2.11.3-78856c5c73f74d.tar.bz2

putty+screen: Close your putty session, without terminating the running script

Normally we connect to a remote UNIX (Linux) servers through putty. But when we disconnect a putty session, the remote shell terminates, as a result if any running script/command will end immediately.

How to continue running a script/command in putty even after closing the session?

You can use the 'screen' command, which will be available by default in most of the Linux distributions.

Screen command

  Screen command offers the ability to detach a long running process (or program, or shell-script) from a session and then attach it back at a later time.

  When the session is detached, the process that was originally started from the screen is still running and managed by the screen. You can then re-attach the session at a later time, and your terminals are still there, the way you left them.

Step by step guide

1) Login to a remote server, either through putty or secureshell.

2) Open a new screen session

     screen -S screen-name 

     Ex:   screen -S TestScreen

3) Start your command/script

     Ex: Run the below command, which will run for approx 1 min 40 secs.

     for (( c=1; c<=100;c++)) 

     do 

             echo "Hello $c"

              sleep 5

     done

4) Disconnect the current session by pressing CTRL + A + D ( or close your putty session) . Don't exit ( never run the command 'exit' or CTRL+C, CTRL+D).  

5) To check the status of previous command, open a new putty session. After login, type the below command

     screen -d -r TestScreen

     where 

        -r : Reattach to a detached screen process.

        -d: Detach the elsewhere running screen (and reattach here).

This will connect back you to the previous session and monitor the progress of your script. Again if command is not complete, don't exit.  Press CTRL A D or just close the putty session and proceed for other tasks.

     If you got only one screen session, you can also use below command to connect to it.
     screen -x TestScreen

6) Creating multiple screen tabs
     Ctrl + AC

7) Switching between screen tabs
    Ctrl + AN     ---> Switch to next tab
    Ctrl + AP      ---> Switch to previous tab
    Ctrl + AA     ---> Switch between your last 2 tabs

Note: You can use VNC sessions for the same functionality and also Citrix provides it. 

   The screen command will be useful if

      1) You don't have the commercial Citrix tool  

      2) Sometimes configuring VNC gives lots of issue. Instead of spending time on fixing this issue, you can quickly achieve this functionality using screen command. I personally still prefer VNC sessions. 

Custom fields not appearing in Jira gadgets

Are you scratching your head debugging why a custom field in Jira not appearing in Gadgets? Here is one the reason.

Scenario: We have a custom text field called 'ETA' and it is not getting listed in gadgets like 'Two Dimensional Filter Statistics’

Reason: ETA field is a text field and hence it is not appearing in gadgets like ‘Two Dimensional Filter Statistics’. When we created the same ETA field has a 'Select List field', it started appearing.