Thursday, April 19, 2012

Using BUILD_LOG_REGEX in jenkins email notification

Jenkins provide 'Email-ext' plugin, which allows to configure every aspect of email notifications.
One of my requirement is to send in mail the build package URL, which is already getting displayed in console output.

I found a token 
${BUILD_LOG_REGEX, regex, linesBefore, linesAfter, maxMatches, showTruncatedLines, substText, escapeHtml, matchedLineHtmlStyle} - Displays lines from the build log that match the regular expression. 

Here is example usage of it
${BUILD_LOG_REGEX, regex="^Build package URL:", showTruncatedLines=false}

This will extract the line starting with "Build package URL:" from the build log and send it through mail.

Wednesday, April 18, 2012

Using unix find Command to delete files older than a year

One of the boring task is to clean up data when ever disk usage exceeds the threshold limit. Among the many ways do it, here is the simple usage of Unix find command to delete files older than a year.

   find . -type f -mtime +365 -delete

Where -mtime +365 suggests to list files older than 365 days.
or
  find . -type f -atime +365 -delete

Where -atime stands for access time which is when the file was last read.


Some more examples

find . -mtime 0 # find files modified between now and 1 day ago 
# (i.e., within the past 24 hours)


find . -mtime -1 # find files modified less than 1 day ago 
# (i.e., within the past 24 hours, as before)

find . -mtime 1 # find files modified between 24 and 48 hours ago 

find . -mtime +1 # find files modified more than 48 hours ago

find . -mmin +5 -mmin -10 # find files modified between 
# 6 and 9 minutes ago


Usage of -exec option 
Here is the simple use case.
I want to find all the files with the name 'latest' in a directory and I want to run 'ls -l' command on the result, since I want to know whether 'latest' is a directory or soft link.
find . -maxdepth 2 -name latest -exec ls -l '{}' \;
Here the 
  -maxdepth option restrict the search to be conducted 2 levels of directory hierarchy.
The syntax '{}' \; is mandatory, which is tedious.

Unix find command to list files greater than 10MB
find . -size +10M -exec ls -lh '{}' \;

Unix find files older than 50 days and move them to another folder
find . -type f -mtime +50 -print| xargs -I {} mv {} /homes/siddesh/psbuild-portal-CGITemp-files

Tuesday, April 17, 2012

Using wget to download files recursively from https site

wget is a command used mostly in Linux to download files from web links (http, https or ftp). To download a file, a simple invocation of wget like
  wget http://mylibrary.com/build-kits/rel-2.7.1/prod1/prod1-2.7.1.10.tar
will download prod1-2.7.1.10.tar to the current directory.

To download set of files (ex .tar files) from a secured https site, we need to provide several options to it
  wget --no-check-certificate -r -l1 --no-parent -A.tar http://mylibrary.com/build-kits/rel-2.7.1/prod1/
where
  --no-check-certificate: don't validate the server's certificate
  -r or --recursive: specify recursive download
  -l1 or --level=NUMBER: maximum recursion depth (inf or 0 for infinite).
 --no-parent: don't ascend to the parent directory
 -A or --accept=LIST:  comma-separated list of accepted extensions.
 -nd or --no-directories: don't create directories.

Thursday, April 5, 2012

Ant script to replace tokens (VERSION) in a file (build.properties)

Version of a product ever changes for main releases, SP releases, hotfix releases, etc. We can't hard-code version value in source files. One of the effective solution to tackle this is to use tokens in the place of version values and replace it before the start of build. The most common tool used to make string replacements are scripts written in perl/shell. Though scripts works well, but why to depend on scripts which are closely tied with Unix systems, to perform token replacement operations on platform independent Java build systems. 


 Java builds are usually managed through Ant/Maven. If perl/shell scripts are not stored along with Java build system, then if anyone checkouts only java code from source control, there build may fail due to improper version characters.


Here is an illustration of performing token replacement using Ant script.

Friday, March 16, 2012

Perforce sync is slow - Checklist for debugging

Is your Perforce performance is slow while doing sync or some other operations ?


1) Check you client spec’s view (if the repository is large) and if your client has a view of the entire repository, performance will be impacted (or you may have exceeded a maxRowCount error)
2) Modify your client spec (or create a new one) with a very narrow view (for testing, maybe down a single folder)
3) Do you have a firewall setup and preventing this kind of activity?
4) Can you try sync from another machine?
5)Try and play with "p4 ping" by progressively increasing the  message count, receive and transmit lengths (one at a time). If you hit the same wall repeatedly, then this could very well be indicative of some   throttling taking place somewhere
6) p4 -z tag info
7)  p4 -vN sync //depot/path/to/file      where N = 1,2,..9
8)  Network problems  http://kb.perforce.com/article/40/isolating-network-problems

 

Friday, March 9, 2012

Difference between centralized version control (CVCS) and distributed version control systems (DVCS)


1) Repository
    CVCS: Have a single server that contains all the versioned files and metadata 
    DVCS: Each users working copy of the codebase is a bona-fide repository.

2) Checkout
   CVCS: A number of clients check out files from central repository 
   DVCS: Clients don’t just check out the latest snapshot of the files: they fully mirror the repository

3) Tracking latest revision
   CVCS: Everyone knows to a certain degree what everyone else on the project is doing. The trunk usually has the latest revision.
   DVCS: It is not plainly obvious.

Comparison of several SCM tools

Here is a link which compares many SCM tools inlcuding Distributed and Central version control tools.
It compares for few use cases, anyways take a look at http://better-scm.shlomifish.org/comparison/comparison.html

Monday, February 13, 2012

Jenkins - FATAL: Unable to produce a script file

Immediately after syncing code from perforce, a job in jenkins fails with the error message:


09:16:33  Sync complete, took 8001 ms
09:16:33  No emails were triggered.
09:16:33  FATAL: Unable to produce a script file
09:16:33  hudson.util.IOException2: Failed to create a temp file on /export/jenkins/jobs/jenkins-blr-backup/workspace

Fix: Restarted the jenkins fixed the issue
Root cause:  No idea !!!!

Monday, January 30, 2012

A build engineer's use-case of java decompiler

This is one scenario, which demonstrate the usage of java decompiler to a build engineer.

One morning, a developer walked to my cube and complained
    Developer: My changes are not picked in latest build
As a build engineer, it's a familiar complaint which I usually receive.
    Me: Cool !! provide me your change location in Perforce SCM tool and some time to debug.
That's the end of my short conversation. Now I started a massive search operation to find out, why the pretty girls changes are not picked :)

It's a Java based build system using Maven 2 and Jenkins.

Step 1: I made sure the code changes are picked up in last build by verifying the Change-list number available in perforce and the one Jenkins has picked during the build. Even I verified in the build machine, by checking the files it has synced. Everything is perfect, then I started wondering what's going wrong.

Step 2:  Since it is a maven build, I thought to check the content of jar package, our build has uploaded to nexus repository. I downloaded and extracted it and the got the mystery .class file. Now how can I prove that her changes are picked by the build using this .class file?

Step 3: I felt the need of a java decompiler. I downloaded "cavaj" java decompiler (it's free) and installed it. Opened the mystery .class file with java decompiler and I'm glad to find her changes.

Java decompiler is used to convert Java class files into Java source code. In fact, there is an arms race brewing between decompilers and so-called obfuscators, which profess to provide Java code some measure of protection from decompilers. In essence, obfuscators remove all non-essential symbolic information from your class files and, optionally, replace it with fake symbolic information designed to confuse the decompiler


Step 4: Now for curiosity I extended further to find out why the developer complaining it is not picked. The root cause of the issue is that,  developer was validating a jar file, which consumes her changes from another jar file. It's a maven dependency issue. Some how, the final jar file was not picking the right version of the dependent jar file, which is fixed later.

Wednesday, January 25, 2012

Jira Administration quick reference

1) Configuring the Layout and Design
Customising the Look and Feel
    Administration -> Global Settings -> Look and Feel -> To edit the logo, colours or time format, click the 'Edit Configuration' link at the bottom of the page
Changing the default language
  Administration -> General Configuration -> Edit Configuration > select the appropriate language in the drop-down box next to 'Default language'.
Configuring default issue navigator
  The Issue Navigator is used within JIRA to find and filter issues, and to display the search results in various formats ('views'). It is possible to
select which issue fields will be displayed as columns in the Issue Navigator.
   Administration -> Issue Fields -> Navigator Columns -> Add new column
Configuring the Default Dashboard
  Administration -> Global Settings -> System Dashboard -> Add Gadget
Configuring an Announcement Banner
  Administration -> Options & Settings -> Announcement Banner
Enabling Logout Confirmation
  Global Settings -> General Configuration -> Edit Configuration -> Logout confirmation

Tuesday, January 24, 2012

Productivity tools linkage

Reference: From slide prepared by Carl Sjoquist. 
I liked this image which depicts how various productivity tools in an organization collaborate. 

Monday, January 23, 2012

Generating ccs (Conary Change Set) file using rmake build Job-ID

To generate CCS (Conary Change Set) files, for the corresponding packages available on rBuilder server, follow these steps
  • rmake build <package name>.recipe

             This command will build the specified recipe file
             Ex: rmake build tomcat.recipe
          Note down the job number 
  • rmake changeset <jobid>  <package name>.ccs
          This command creates a changeset for the given <jobId>. The created changeset contains all the binaries built for this job.
          Ex: rmake changeset 573 tomcat.ccs 
  • conary showcs --lsl <package name>.ccs
           Use conary showcs to examine changeset file contents 
            Ex: conary showcs --lsl tomcat.ccs 

Sunday, January 15, 2012

Cygwin error: Couldn't create pipe tracker for pid

Our builds started failing to connect to a Windows machine using sshd provided by cygwin.
Then when I tried to open cygwin shell, it started throwing the error


pinfo::wait: Couldn't create pipe tracker for pid 3534
Win32 error 231
bash: fork: Resource temporarily unavailable

After googling, the reason for this error, which I came to know is
  1. It's a limitation in the number of running processes under Windows. When cygwin cannot create a new process during a fork(), you will get this behavior.
    Some people have reported success turning off (or swapping) antivirus programs, or other tricks to reduce the number of running processes on their system.
  2. Well, what that message means is that for some reason, the child process
    failed to notify the parent of its existence within a reasonable amount of
    time after it was spawned.  That could mean that it died for some reason, or
    it could mean that it was alive, but failed to notify the parent process.
    
    
    
    
    Then I felt, it's good to reboot my Windows machine once and it has solved this problem for the time being. I will keep updated if it pop up again.
    
    
    Our cygwin version is
    $ cygcheck -V
    cygcheck version 1.90.4.1
    System Checker for Cygwin
    Copyright 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006 Red Hat, Inc.
    Compiled on Jun 12 2008
    
    
    
    and Windows version is "Windows XP professional 2002 SP2"
    I guess this issue doesn't occur with latest cygwin/Windows versions.

Wednesday, January 11, 2012

nexus "Item not found on path" error

We are using nexus 1.5.0 repository to host maven packages in our organization.
Recently I started encountering the error like
   Item not found on path "/com/rsa/appliance/appliance-distribution/3.1.3-SNAPSHOT/"!
when I click to download this package from nexus GUI, even though a maven build has successfully uploaded this package to the server, which I verified by logging into the machine which is hosting nexus (a RHEL m/c)

Issue:

The Not Found Cache is getting populated with these artifacts. Need to clear the cache for the repo and try the request again. 


Solution:
 Login to Nexus as Admin and browse -> Repositories -> Right click on repository which is giving the issue (in our case it's snapshots) -> Expire Cache

This has solved our issue.

Tuesday, January 3, 2012

Encrypting using GPG - an example

GPG stands for Gnu Privacy Guard, is a tool used to sign and encrypt data. Here I'm giving a simple usage of this tool.
Usecase: We have a file on linux server, which needs to be encrypted and delivered to another machine.

Solution:
  • By default every linux server comes with "gpg" tool. You can verify by running the command "which gpg"
  • Use gpg command to encrypt your file
           cd TO_YOUR_FILE_LOCATION   (I need to encrypt a file named CONARY)
           
           gpg -c CONARY           (It will ask for the passphrase/password, give a strong one and remember it)
           It creates a file named CONARY.gpg, which is encrypted. You can take a look at the content of this file
  • Using gpg command for decryption
          Now I have transferred this encrypted file (CONARY.gpg) to another server where gpg utility is available and run below command to decrypt it
         gpg -o CONARY -d CONARY.gpg
      
          Where -o option tells to create a file CONARY, instead of printing output on console


Strengthening encryption using AES256 cipher algo


Command to Encrypt
gpg --symmetric --cipher-algo aes256 -o dpm-3.2-src.tar.gz.gpg dpm-3.2-src.tar.gz
Enter passphrase: *******
Repeat passphrase: *******

Command to decrypt
gpg -d -o dpm-3.2-src.tar.gz  dpm-3.2-src.tar.gz.gpg


gpg: AES256 encrypted data
Enter passphrase: *******







         
                 


         
           

Tuesday, December 27, 2011

how to check open ports on linux machine?

Use the commands
1) nmap -sS -O 127.0.0.1 or
2) netstat -nap   
   netstat --listen
 To display open ports and established TCP connections, enter:
   netstat -vatn
 To display only open UDP ports try the following command:
  netstat -vaun\
  if you want to see FQDN (full dns hostname), try removing the -n flag:
  netstat -vat

Friday, December 16, 2011

Jenkins most useful features

Jenkins is a CI (Continuous Integration) tool which is used to widely to manage builds and tests. In this blog, I'm listing some of the add on's which makes Jenkins richer and support it's use to manage builds using Jenkins instead of homegrown scripts. Most of these additional functionality comes through Jenkins plugin's.

1) Support to GIT source code management
    Plugin: Jenkins GIT plugin
    It checkout code from GIT repository (using http/ssh/local/git protocol), poll for changes and trigger build

2) Archive the build artifacts (Post build section)
    Provides link for download to artifacts which are built. It acts a one stop dashboard where you build and distribute artifacts to consumers. These artifacts are preserved as long as logs are preserved.
   Plugin: Comes with default Jenkins download

3) Publishing Junit test result on build dashboard (Post build section)
    If your project running junit tests, provide link to xml reports. Jenkins publish the test results , compose historical trends, web UI for viewing/tracking test results.
   Plugin: Comes with default Jenkins download

4) Publishing Javadocs on build dashboard (Post build section)
   If your project generating javadocs (for ex: mvn javadoc:javadoc), provide link to xml reports. Jenkins will publish the latest Javadoc and it can also retain older javadocs if needed.
   Plugin: Comes with default Jenkins download

5) Publishing code coverage metrics ( ex: cobertura )
   If your project calculating code coverage metrics (for ex: mvn cobertura:cobertura), provide link to xml reports generated by your coverage tools like cobertura/clover/Emma. Jenkins will publish the coverage metrics on job dashboard.
     Plugin: Jenkins cobertura plugin
  Cobertura plugin lets you to fine-tune minimum acceptable levels of code coverage.

6) Build trends
   Clicking "trend" for a project displays build time trend, timeline graph (scroll your mouse on graph to see the historical build timings).

Thursday, December 15, 2011

jenkins error: java.io.IOException: Authentication method password not supported by the server at this stage

When I tried to add a node to jenkins/hudson using ssh as launch method, the authentication keeps on failing with the below error.


[12/15/11 13:23:12] [SSH] Opening SSH connection to **.**.***.**:22.
[12/15/11 13:23:12] [SSH] Authenticating as build/******.
java.io.IOException: Password authentication failed.
at com.trilead.ssh2.auth.AuthenticationManager.authenticatePassword(AuthenticationManager.java:319)
at com.trilead.ssh2.Connection.authenticateWithPassword(Connection.java:314)
at hudson.plugins.sshslaves.SSHLauncher.openConnection(SSHLauncher.java:565)
at hudson.plugins.sshslaves.SSHLauncher.launch(SSHLauncher.java:179)
at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:199)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Authentication method password not supported by the server at this stage.
at com.trilead.ssh2.auth.AuthenticationManager.authenticatePassword(AuthenticationManager.java:289)
... 9 more
[12/15/11 13:23:12] [SSH] Connection closed.


The host has SLES OS and I was wondering what's going wrong, even though I gave correct password.

FIX: Change the sshd configuration
   In /etc/sshd_config (or /etc/ssh/sshd_config) file
  # To disable tunneled clear text passwords, change to no here!
PasswordAuthentication no
If set to no, Hudson's SSH client won't be able to connect using password authentication. If you comment this line out (which I believe is the OpenSSH default), or set it to yes, then Hudson can connect just fine.


Tuesday, December 6, 2011

How to check a machine is physical or VMware virtual machine

Run command
  • dmidecode | grep -i vmware
  • If it is a physical machine, it won't return any data
  • If it is a VMWare Virtual machine, you will get output like
        Manufacturer: VMware, Inc.
        Product Name: VMware Virtual Platform
  • What is dmidecode ?
Dmidecode reports information about your system's hardware as described in your system BIOS according to the SMBIOS/DMI standard. This information typically includes system manufacturer, model name, serial number, BIOS version, asset tag as well as a lot of other details of varying level of interest and reliability depending on the manufacturer. This will often include usage status for the CPU sockets, expansion slots (e.g. AGP, PCI, ISA) and memory module slots, and the list of I/O ports (e.g. serial, parallel, USB).
Beware that DMI data have proven to be too unreliable to be blindly trusted. Dmidecode does not scan your hardware, it only reports what the BIOS told it to.

Dmidecode is known to work on the following systems:
  • Linux i386, x86-64, ia64
  • FreeBSD i386, amd64
  • NetBSD i386, amd64
  • OpenBSD i386, amd64
  • BeOS i386
  • Cygwin i386
  • Solaris x86
  • Haiku i586


Thursday, November 10, 2011

Learning MSBuild

In my last 7 years of my career, I mostly worked on Unix based builds. Now I got curious to explore Windows builds. I learnt the 2 prominent build assistants in Windows are MSBuild and Nant. In this blog, I will share my learning experience of MSBuild.


What is MSBuild?
MSBuild is the build platform for Microsoft and Visual Studio. Programming languages that target the .NET Framework, for example, Visual C# and Visual Basic, use MSBuild project files to describe and control the application build process. When you use Visual Studio to create an MSBuild project file, the appropriate XML is added to the file automatically. 


Lets start by creating a simple C# file and compiling it through csc compiler. csc is a C# compiler, it comes along with .NET Framework (version 2.0, 3.5, or 4.0).
1) cd D:\HelloWorld
2) Create a file Helloworld.cs , with it's content as
using System;
class HelloWorld
{
    static void Main()
    {
#if DebugConfig
        Console.WriteLine("WE ARE IN THE DEBUG CONFIGURATION");
#endif
Console.WriteLine("Hello, world!");
    }
}
3) Very likely csc compiler location  won't be in your path. Set it
    set PATH=%PATH%;%WINDIR%\Microsoft.Net\Framework64\v2.0.50727
    Alternatively, if you have Visual Studio installed, you can use the Visual Studio Command Prompt, which has a path that includes the MSBuild folder.
4)  Compile it
    csc Helloworld.cs
5) Test the application by typing Helloworld.exe at the command prompt
     O/p: Hello, world!
6) Delete the application
    del helloworld.exe 
Now let's do the same thing using MSBuild project file. We are creating a minimal MSBuild project file and it's content will be saved in a file named "Helloworld.csproj".

<!-- root Project node   -->
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">


  <!--ItemGroup node contain item elements   -->
  <ItemGroup>
    <!-- An item element that refers to the application source file. -->
    <Compile Include="helloworld.cs" />
  </ItemGroup>


  <!-- A Target node to contain tasks that are required to build the application.  -->
  <Target Name="Build">
    <!-- A Task element to start the Visual C# compiler to build the application.  -->
    <Csc Sources="@(Compile)"/>
  </Target>
</Project>

Building the application using MSBuild project file
Command: msbuild Helloworld.csproj /t:Build 
This builds the Build target of the Helloworld project file by invoking the Visual C# compiler to create the Helloworld application.
Output:

Microsoft (R) Build Engine Version 2.0.50727.42
[Microsoft .NET Framework, Version 2.0.50727.42]
Copyright (C) Microsoft Corporation 2005. All rights reserved.
Build started 2011-11-11 12:05:00 PM.
Project "D:\siddesh\learning-msbuild\HelloWorld\Helloworld.csproj" (Build target(s)):
Target Build:
    C:\WINDOWS\Microsoft.NET\Framework64\v2.0.50727\Csc.exe /out:helloworld.exe helloworld.cs
Build succeeded.
    0 Warning(s)
    0 Error(s)
Time Elapsed 00:00:01.12

For detailed build, add flag
   msbuild Helloworld.csproj /t:Build /verbosity:detailed


Improving MSBuild project file - Properties

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <!-- PropertyGroup element is used to define variables/properties -->
  <PropertyGroup>
    <AssemblyName>MSBuildSample</AssemblyName>
    <OutputPath>Bin\</OutputPath>
  </PropertyGroup>
  <ItemGroup>
    <Compile Include="helloworld.cs" />
  </ItemGroup>
  <Target Name="Build">
    <MakeDir Directories="$(OutputPath)" Condition="!Exists('$(OutputPath)')" />
    <!-- The MakeDir task creates a folder that is named by the OutputPath property, provided that no folder by that name currently exists. -->
    <Csc Sources="@(Compile)" OutputAssembly="$(OutputPath)$(AssemblyName).exe" />
    <!--This instructs the Visual C# compiler to produce an assembly that is named by the AssemblyName property and to put it in the folder that is named by the OutputPath property. -->
  </Target>
</Project>

Improving MSBuild project file - Adding build targets


<Project DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
<!-- This sets the Build target as the default target. -->
  <PropertyGroup>
    <AssemblyName>MSBuildSample</AssemblyName>
    <OutputPath>Bin\</OutputPath>
  </PropertyGroup>
  <ItemGroup>
    <Compile Include="helloworld.cs" />
  </ItemGroup>
  <Target Name="Build">
    <MakeDir Directories="$(OutputPath)" Condition="!Exists('$(OutputPath)')" />
    <Csc Sources="@(Compile)" OutputAssembly="$(OutputPath)$(AssemblyName).exe" />
  </Target>
  <!-- The Clean target invokes the Delete task to delete the application. -->
  <Target Name="Clean" >
    <Delete Files="$(OutputPath)$(AssemblyName).exe" />
  </Target>
  <!-- The Rebuild target does not run until both the Clean target and the Build target have run. -->
  <Target Name="Rebuild" DependsOnTargets="Clean;Build" />
</Project>

Testing:
1) msbuild helloworld.csproj /p:AssemblyName=Greetings
    MSBuild runs the default Build target. The /p switch overrides the AssemblyNameproperty and gives it the new value, Greetings. This causes a new application, Greetings.exe, to be created in the \Bin\ folder.
2) msbuild helloworld.csproj /t:clean
  This runs the Clean task to remove the application that has the default AssemblyName property value, MSBuildSample.
3) msbuild helloworld.csproj /t:clean /p:AssemblyName=Greetings.
This runs the Clean task to remove the application that has the given AssemblyName property value, Greetings.
4) msbuild
Although a project file is not specified, MSBuild builds the helloworld.csproj file because there is only one project file in the current folder. 
Reference: http://msdn.microsoft.com/en-us/library/dd576348.aspx