Tuesday, December 2, 2014

Makefiles articles collection

Variable assignment types in Makefiles

= or := or ?= or +=

http://stackoverflow.com/questions/448910/makefile-variable-assignment 

Blunders of recursive variables

http://electric-cloud.com/blog/2009/03/makefile-performance-shell/ 

VPATH & vpath

http://www.cmcrossroads.com/article/basics-vpath-and-vpath 

Running external commands using $(shell)

http://electric-cloud.com/blog/2009/03/makefile-performance-shell/ 

Perforce trigger script to enforce JIRA ID (Perl)

A perforce trigger script written in Perl to enforce JIRA ID on Perforce changelist submission. This script also connects to JIRA server to make sure the submitted JIRA ID is a valid one.

Create a properties file p4trigger.properties with required information
cat p4trigger.properties
# Perforce pre-commit trigger to check jira-id configuration file
# %serverport%  It comes from trigger

p4user = **** 
p4port = ****:1666
p4passwd = ************************************
jiraProjects = AAAA BBBB
jiraURL = https://jira.com:8443
jiraLogin = ******
jiraPWD = /export/*****/******/****.txt


Now a perl module Utilities.pm, which consists re-usable functions.


cat Utilities.pm

Perforce to GIT migration with history

Here are the step by step guide to migrate from Perforce to GIT with history ..

  • Install git-p4 : Refer http://answers.perforce.com/articles/KB_Article/Git-P4 , in the end of this URL follow step 2 to Step 4
  • Sync the code from Perforce to a workspace (P4CLIENT).
  • Create a .p4config file with the P4PORT, P4USER & P4CLIENT values in $HOME
         cat .p4config
           P4PORT= 
           P4USER=
           P4CLIENT=
  • mkdir git-migration-root/component
  • cd  git-migration-root/component
  • cp $HOME/.p4config .
  • unset SSH_ASKPASS  # This will disable the git command attempt to open xterm. It is needed if you invoke the below command from putty.
  • git p4 clone //depot/component@all . 
        The above step will clone the code from Perforce Workspace ( with reference to .p4config P4CLIENT value). 
         In the above command "@all" option will copy the entire history. If this option is ignored, only HEAD revision is copied.
  • export GIT_SSL_NO_VERIFY=true  #Disabling SSL cert verification for push command.
  • git remote add origin ${GIT_URL}/component.git #Create a remote to point to STASH/github bare repository
  • git remote set-url origin ${GIT_URL}/component.git
  • git push origin master
       

Tuesday, November 25, 2014

FATAL: hudson.plugins.git.GitException stderr: Host key verification failed

Jenkins polling on GIT server was failing with below error

FATAL: hudson.plugins.git.GitException ...
.
.
stderr: Host key verification failed

Though error was clearly telling its a ssh host key verification issue, it took a while to figure out the root cause.
I'm running Jenkins poll from a node, not from master. But Jenkins poll was still referring to master machine, where the ssh configuration was improper.

If you install the Jenkins on Linux by using Jenkins rpm, it will setup a ssh at /var/lib/jenkins/.ssh.
And it doesn't create ssh public/private keys.
But to create a ssh key, you need to use "jenkins" user account.

By default the rpm will create below entry in /etc/passwd file
jenkins:x:494:489:Jenkins Continuous Build server:/var/lib/jenkins:/bin/false

change it to
jenkins:x:494:489:Jenkins Continuous Build server:/var/lib/jenkins:/bin/bash

Now create ssh keys

  • su jenkins
  • ssh-keygen
It will create new "id_rsa" &  "id_rsa.pub" files. 

Now copy the content of "id_rsa.pub" file to your GIT server ( we use atlassian stash).

Now try the git command which was failing


-bash-4.1$ git ls-remote -h ssh://xxxx@xxxx.com:7999/xxxx/yyyy.git rel-2.0

Are you sure you want to continue connecting (yes/no)? yes

Now it will add the host key to known_hosts file. 

That's it, your GIT polling will start working ...



Wednesday, November 12, 2014

rsync quick syntax

rsync -rltv ${source_dir}/* ${rsyncServer}:${source_dir}

Wednesday, September 24, 2014

Perforce login with ticket

The perforce server may doesn't accept the actual password as argument with -P option as shown below

bash-3.2$  p4 -u myuser -P mypassword changes -m1
                  Perforce password (P4PASSWD) invalid or unset.

But the password supplied with -P (mypassword) works, if you type it with below command
bash-3.2$ p4 login
Enter password: mypassword

User myuser logged in.

You may need to authenticate to Perforce server from scripts during automation. The alternative way to login is pass the ticket value instead of password with -P option.

How to get the ticket value?

bash-3.2$ p4 login -a -p
Enter password: ******
ABCD1234EFGH5678IJKL1234MNOP5678

You'll get a 32 character length ticket value. Use it as password along with -P

bash-3.2$  p4 -u myuser -P ABCD1234EFGH5678IJKL1234MNOP5678 changes -m1
                   Change 12345 on 2014/09/24 by idiot@idiot_box 'UI bug fix'

The above command works !!

Friday, September 19, 2014

How to extract a specific file from zip?

Lets say I have a zip file "my-archive.zip" and I want to extract only "c.zip" file from it.

First check the zip content with below content

unzip -l "my-archive.zip"

Archive:  my-archive.zip 

  Length      Date    Time    Name
---------  ---------- -----   ----
     2213  09-17-2014 15:57  a.zip
     1591  09-17-2014 15:57   b.zip
     4005  09-17-2014 15:57   c.zip
     6966  09-18-2014 14:07   d.zip
---------                     -------
   13182                     4 files
     

And now to extract only "c.zip"

unzip -j "my-archive.zip" "c.zip" -d tmp

Where it extracted "c.zip" to "tmp" directory

Tuesday, April 22, 2014

How to delete a set of old builds in Jenkins? Using CLI

You can use Jenkins command-line-interface(CLI) to delete some old build numbers. To run jenkins CLI, you need java and jenkins-cli.jar.

Where is jenkins-cli.jar?
In the machine where jenkins is hosted, you can find jenkins.war. Extract it using the command 'jar -xvf ../jenkins.war'. You can find it under WEB-INF/jenkins-cli.jar.

What is the command to delete jenkins jobs?
java -jar jenkins-cli.jar -s http://ci.jenkins-ci.org/ delete-builds JOB RANGE

where
JOB   : Name of the job to build
RANGE : Range of the build records to delete. 'N-M', 'N,M', or 'N'

Example:

java -jar WEB-INF/jenkins-cli.jar -s https://emc-jenkins:8443/jenkins delete-builds DLP-EM-9.6-SP2-HF1-Daily '22176-22190' --username guruss1
Failed to authenticate with your SSH keys.
Password:

Deleted 15 builds

It deleted build numbers from 22176 to 22190.

Wednesday, March 26, 2014

SVN Quick Reference

Creating a repository

  • mkdir svn-repos
  • svnadmin create /cygdrive/c/svn-repos
  • ls -a /cygdrive/c/svn-repos
           conf  db  format  hooks  locks  README.txt

Creating a project

  • cd temp
  • svn import -m "importing Snipe project" . file:///cygdrive/c/svn-repos/snipe/trunk
  • import tells svn to import some files to repository
  • -m to associate message to import
  • . to import contents of current directory
  • The final parameter is repo URL describing where we want to import the files
  • file:// tells svn to look on local file system for the repo

Working with a project

Checkout

  • mkdir svn_workspace; cd svn_workspace
  • svn co file:///cygdrive/c/svn-repos/snipe/trunk snipe
  • How to checkout to older version ?
  • svn co -r 5 file:///cygdrive/c/svn-repos/snipe/trunk snipe-old5

Making changes

  • vi day.txt 
  • svn status day.txt
          M       day.txt
  • M indicates that the above file is modified
  • svn diff day.txt
Index: day.txt
===================================================================
--- day.txt     (revision 1)
+++ day.txt     (working copy)
@@ -3,3 +3,5 @@
 wednesday
 thursday
 friday
+saturday
+sunday

Adding files

  • svn add mydir

Checkin

  • svn commit -m "facliltating to work on weekends"
  • svn log day.txt
------------------------------------------------------------------------
r2 | guruss1 | 2014-03-26 12:48:20 +0530 (Wed, 26 Mar 2014) | 1 line

facliltating to work on weekends
------------------------------------------------------------------------
r1 | guruss1 | 2014-03-26 12:23:27 +0530 (Wed, 26 Mar 2014) | 1 line

importing Snipe project
------------------------------------------------------------------------
  • svn log --verbose day.txt or svn log -v day.txt

Setting up external editor to provide commit message

  • export SVN_EDITOR=notepad

Mixed revision working copies

  • svn log without any other arguments shows log for the current directory and any sub-directories
  • if you run svn log immediately after commit, svn doesn't tell you about your change. We can see the change if run svn log day.txt
  • To see the log message for directory, run svn update first
  • svn update
          Updating '.':
          At revision 3.

How to check from where a working copy came from? svn info

  • svn info snipe-old5
Path: snipe-old5
Working Copy Root Path: /cygdrive/c/svn_workspace/snipe-old5
URL: file:///cygdrive/c/svn-repos/snipe/trunk
Relative URL: ^/snipe/trunk
Repository Root: file:///cygdrive/c/svn-repos
Repository UUID: d231f21e-4492-4b58-bb57-dd135cc89b28
Revision: 5
Node Kind: directory
Schedule: normal
Last Changed Author: guruss1
Last Changed Rev: 5
Last Changed Date: 2014-03-26 14:24:06 +0530 (Wed, 26 Mar 2014)

Updating a workspace

  • cd <workspace-root>; svn update
  • How to update only a specific files or directories?
  • svn update number.txt
  • Update command output indicators
  • A - added a file
  • U - updated a out-of-date file
  • D - removed a file
  • G - A file was out-of-date in the workspace, which you had modified locally. SVN merged changes from repo with your local modifications
  • C - A file was out-of-date in the workspace, hich you had modified locally. SVN tried to merge changes from repo with your local modifications, but got a conflict.
  •  U - Space followed by U, indicates changes to file properties, rather than file itself.

Properties

  • Properties are metadata associated with file and directories. It is named using simple strings and can contain any content (including binary data)
  • How to set property on a file?
  • svn propset reviewed-by "Likith AN" number.txt
  • svn status
         M      number.txt
         ?       trunk
  • svn ci -m "reviewer added"
  • How to edit a property?
  • svn propedit reviewed-by number.txt
  • How to list all the properties for a file?
  • svn proplist number.txt
  • How to print current value of a property?
  • svn propget reviewed-by number.txt
  • How to delete a property?
  • svn propdel description number.txt

Handling merge conflicts

Merging changes

  • svn co file:///cygdrive/c/svn_repos/snipe/trunk snipe_advanced
  • cd snipe; vi number.txt; svn commit -m "supporting more numbers"
  • cd ../snipe_advanced/
  • svn status -u
           *        3   number.txt
           Status against revision:      4
  • The above command tells if any updates are available for snipe_advanced directory
  • The '*' indicates the availability of update for number.txt, which is currently at revision 3.
  • svn diff -rHEAD number.txt
  • Check the difference between our version and the head revision in repo
  • -rHEAD tells to compare the current version to head revision in repo. If we doesn't provide this option, then svn compares local copy against repo version that was checked out.
  • svn update
         Updating '.':
         U    number.txt
         Updated to revision 4.
  • The above command merge changes from repo. svn prints U next to number.txt to let us know that it has updated it and tells us that our working copy has been updated to revision 4

Conflict resolution

   Scenario 1: When changes doesn't overlap
  • cd ../snipe ;  vi number.txt
  • cd ../snipe_advanced/ ; vi number.txt; svn commit -m "making 7 important"
  • cd ../snipe
  • svn commit -m "emphasizing zero"
         Sending        number.txt
         svn: E155011: Commit failed (details follow):
         svn: E155011: File '/cygdrive/c/svn_workspace/snipe/number.txt' is out of date
         svn: E160028: File '/snipe/trunk/number.txt' is out of date
  • svn update
         Updating '.':
        G    number.txt
        Updated to revision 5.
  • svn prints a G to tell us it has merged our changes with the repository version
  • svn commit -m "emphasizing zero"
 Scenario 2: When changes conflicts
  • cd ../snipe_advanced/;  vi number.txt; svn commit -m "kannada one"
  • cd ../snipe; vi number.txt; 
  • svn commit -m "hindi one"
Sending        number.txt
svn: E155011: Commit failed (details follow):
svn: E155011: File '/cygdrive/c/svn_workspace/snipe/number.txt' is out of date
svn: E160028: File '/snipe/trunk/number.txt' is out of date
  • svn update
Updating '.':
C    number.txt
Updated to revision 7.
Conflict discovered in file 'number.txt'.
Select: (p) postpone, (df) show diff, (e) edit file, (m) merge,
        (mc) my side of conflict, (tc) their side of conflict,
        (s) show all options: df
--- number.txt.r7       - THEIRS
+++ number.txt  - MERGED
@@ -1,5 +1,9 @@
 ZERO
+<<<<<<< .mine
+ek
+=======
 ondu
+>>>>>>> .r7
 two
 three
 four
Select: (p) postpone, (df) show diff, (e) edit file, (m) merge,
        (r) mark resolved, (mc) my side of conflict,
        (tc) their side of conflict, (s) show all options: p
  • Find out who made change in version 7:     svn log -r7 number.txt
  • Resolve it manually
  • svn resolved number.txt
          Resolved conflicted state of 'number.txt'
  • svn commit -m "one is hindi, two is kannada"

Wednesday, March 12, 2014

How to start a jenkins slave on linux machine without ssh?

Step 1: Login to Jenkins -> Manage jenkins -> Manage nodes -> Linux node name
Copy the command under
  Run from slave command line:
  Like java -jar slave.jar -jnlpUrl http://your-jenkins.org/jenkins/computer/linux-node-name/slave-agent.jnlp -secret ce4a8ada8d00c3b27f17bac3cb673c40a6d5d500d5bdf74084fcc5f1ea95b8b6

Step 2: Login to your linux machine

Step 3:  Run the command which you obtained in Step 1, with 'nohup' . Make sure Java is installed and is in path and also you need download slave.jar from Jenkins server.

nohup java -jar slave.jar -jnlpUrl http://your-jenkins.org/jenkins/computer/linux-node-name/slave-agent.jnlp -secret ce4a8ada8d00c3b27f17bac3cb673c40a6d5d500d5bdf74084fcc5f1ea95b8b6


Wednesday, February 12, 2014

Enabling https on Sonatype nexus

Here are the list of commands used to enable https on nexus web application. The below described example nexus application is running on Linux OS with the nexus provided jetty web server.
We are using 'keytool' which comes along with JDK for generating keys and certs.

My NEXUS_HOME = /usr/local/nexus

Step 1: Generate obfuscated password hash. This password hash will be used in the next step. You can create different password hashes for storepass and keypass, but in this example I'm using the same password.
      java -cp $NEXUS_HOME/lib/jetty-util-8.1.8.v20121106.jar org.eclipse.jetty.util.security.Password abc123
                abc123
               OBF:1igd1igf1igh1idp1idr1idt
               MD5:e99a18c428cb38d5f260853678922e03

Step 2: Generate keypair and certificate directly into keystore
          keytool -genkeypair -keystore /root/ssl-certificates/nexus-keystore.p12 -storetype PKCS12 -alias nexus -storepass OBF:1igd1igf1igh1idp1idr1idt -keypass OBF:1igd1igf1igh1idp1idr1idt -keyalg RSA -keysize 2048 -sigalg SHA256withRSA -validity 365 -v

Fill in the questions asked like your name, organization, city, state, country, etc. The above generated certificate is valid only for 365 days.

Step 3: Configuring Jetty
Make below mentioned changes to jetty.xml file in $NEXUS_HOME/conf/jetty.xml

<Configure id="Server" class="org.eclipse.jetty.server.Server">
    <Set name="threadPool">
        <New class="org.sonatype.sisu.jetty.thread.InstrumentedQueuedThreadPool"/>
    </Set>
    <Call name="addConnector">
        <Arg>
            <New class="org.eclipse.jetty.server.nio.SelectChannelConnector">
              <Set name="host">${application-host}</Set>
              <Set name="port">${application-port}</Set>
              <Set name="confidentialPort">8443</Set>
            </New>
        </Arg>
    </Call>

   <Call name="addConnector">
     <Arg>
       <New class="org.eclipse.jetty.server.ssl.SslSocketConnector">
         <Arg>
           <New class="org.eclipse.jetty.http.ssl.SslContextFactory">
             <Set name="keyStore">/usr/local/nexus/certs/nexus-keystore.p12</Set>
             <Set name="keyStorePassword">OBF:1igd1igf1igh1idp1idr1idt</Set>
             <Set name="keyManagerPassword">OBF:1igd1igf1igh1idp1idr1idt</Set>
             <Set name="trustStore">/usr/local/nexus/certs/nexus-keystore.p12</Set>
             <Set name="trustStorePassword">OBF:1igd1igf1igh1idp1idr1idt</Set>
             <Set name="certAlias">nexus</Set>
             <Set name="keyStoreType">PKCS12</Set>
             <Set name="trustStoreType">PKCS12</Set>
           </New>
         </Arg>
         <Set name="port">8443</Set>
         <Set name="maxIdleTime">30000</Set>
        <Set name="host">12.34.567.89</Set>
       </New>
     </Arg>
   </Call>
.
.
</configure>

Step 4: Redirecting http requests to https in web.xml file
Open the $NEXUS_HOME/nexus/WEB-INF/web.xml and make below changes to indicate it needs CONFIDENTIAL or INTEGRAL connections from users.

<web-app>
  ...
  <security-constraint>
    <web-resource-collection>
      <web-resource-name>Everything in the webapp</web-resource-name>
      <url-pattern>/*</url-pattern>
    </web-resource-collection>
    <user-data-constraint>
      <transport-guarantee>CONFIDENTIAL</transport-guarantee>
    </user-data-constraint>
  </security-constraint>
</web-app>

The actual changes for redirecting http requests to https is done in Step 3 (<Set name="confidentialPort">8443</Set>) .


Step 5: Restart the nexus instance
Clean up the log file : $NEXUS_HOME/logs/wrapper.log

service nexus restart

Wednesday, January 22, 2014

Working with perforce streams - command line

Streams intro

Creating a stream depot

 - Command:    p4 depot <stream-depot-name>
         Ex: # p4 depot DPM
 - Change the type to stream from local
         Type:   stream

Create a new main-line stream

 - Ex: #  p4 stream -t mainline //DPM/MAIN

 - Retain the defaults

Switch your existing workspace to the new stream

 # p4 workspace -f -s -S //DPM/MAIN

Branch files from another depot location

p4 copy -v //depot/DPM/RTJC/... //DPM/MAIN/...
p4 submit -d "Creating DPM mainline"

Now the //DPM/MAIN stream is ready to use. To work on it, switch your workspace with the command 
       # p4 workspace -f -s -S //DPM/MAIN
      # p4 sync
      # p4 edit
      # p4 submit

Branching a Development stream from mainline

# p4 stream -t development -P //DPM/MAIN //DPM/DEV

Switch your workspace to work on this new stream
# p4 workspace -s -S //DPM/DEV

Populate the DEV stream by branching files from MAIN
# p4 merge -S //DPM/DEV -r
# p4 submit -d "Branching from mainline"
# p4 resolve -am
# p4 submit -c 7

Merging files from mainline

Now there are changes in mainline, which needs to be merge down to development stream.
   p4 workspace -s -S //DPM/MAIN
   p4 sync //DPM/MAIN/startbuild
   p4 edit //DPM/MAIN/startbuild
   p4 submit -d "bugfix"

 Now switch back your workspace to DEV stream
    p4 workspace -s -S //DPM/DEV

    # p4 merge -S //DPM/DEV -r
        //DPM/DEV/startbuild#1 - integrate from //DPM/MAIN/startbuild#2
    # p4 resolve
    # p4 submit -d "Merged latest changes"

Promoting changes in Dev stream to main

Now we made changes in Dev stream, which needs copy up to main.
   p4 edit //DPM/DEV/startbuild
   p4 add info.txt
   p4 submit -d "Dev changes"

Copy-up only after merging down. So lets merge down again
  # p4 merge -S //DPM/DEV -r
    All revision(s) already integrated.

Now switch back your workspace to main
  # p4 workspace -s -S //DPM/MAIN
  # p4 update     --> To synchronize the workspace with its view of the depot

Copy the content from DEV stream to Main stream
 # p4 copy -S //DPM/DEV
     //DPM/MAIN/info.txt#1 - branch/sync from //DPM/DEV/info.txt#1
     //DPM/MAIN/startbuild#2 - sync/integrate from //DPM/DEV/startbuild#3
 #p4 submit -d "New changes from Dev stream"

Stream types

1) mainline
    Sample spec:
    ----------------------------------------------------------------------
    Stream: //DPM/MAIN
    Owner:  guruss1
    Name:   MAIN
    Parent: none
    Type:   mainline
    Options:        allsubmit unlocked notoparent nofromparent
    Paths:
        share ...
    ----------------------------------------------------------------------
2) development
  
  Sample spec:
    ----------------------------------------------------------------------
   Stream: //DPM/DEV
   Owner:  guruss1
   Name:   DEV
   Parent: //DPM/MAIN
   Type:   development
   Options:        allsubmit unlocked toparent fromparent
   Paths:
        share ...
     ----------------------------------------------------------------------
3) release
 Sample Spec:
 ----------------------------------------------------------------------
  Stream: //DPM/REL-3.5
  Owner:  guruss1
  Name:   REL-3.5
  Parent: //DPM/MAIN
  Type:   release
  Options:        allsubmit unlocked toparent nofromparent
  Paths:
        share ...
 ----------------------------------------------------------------------

Creating a Release stream from mainline

#p4 stream -t release -P //DPM/MAIN //DPM/REL-3.5

Copy up or merge to release streams - Stopping the flow

Copy up or merge to release streams from parent branches are restricted by default.

# p4 merge -S //DPM/REL-3.5 -r
  Stream //DPM/REL-3.5 not configured to 'copy' or 'merge' changes in this direction.
#$ p4 copy -S //DPM/REL-3.5 -r
   Stream //DPM/REL-3.5 not configured to 'copy' or 'merge' changes in this direction.

Flushing release stream with main content

-Switch workspace to release stream
   # p4 workspace -s -S //DPM/REL-3.5 
   # p4 sync
- Forcefully copy up the content from main to release
  # p4 copy -F -S //DPM/REL-3.5 -r
- p4 submit -d "Creating REL-3.5 from main"

Stream spec options

Default options for the stream types
  main:               allsubmit unlocked notoparent nofromparent
  development:    allsubmit unlocked toparent fromparent
  release:             allsubmit unlocked toparent nofromparent

Understanding these options
 - The “fromparent” and “toparent” options toggle whether parent-to-stream and stream-toparent flow are expected at all. The presence of 'nofromparent' in release stream avoid it from accepting changes from parent.
 - “allsubmit” -> “ownersubmit”.  Only owner submit changes to that stream. Content may be pulled from an “ownersubmit” stream into other streams.
 - “locked” - it prevents other users from modifying the stream spec

Understanding stream view Paths

1) Paths:
        share ...
    It means - Share changes with other streams. These paths appear in both client views and branch views, and are mapped to the stream. These files can be synced, submitted, and integrated.
2) Isolate
    Isolate changes from other streams. These paths appear only in client views and are mapped to the stream. That means you can sync and submit the files in these paths, but you can’t integrate them. This is useful for built binaries, generated docs, and other content that you want checked in but not branched or merged.
3) Import 
   Import a path from elsewhere. These paths appear only in client views and are mapped per the parent stream’s view (the default), or to another depot location.You can sync these files, but because they’re not mapped to the stream itself, you can’t submit them. Nor can you integrate them. All they do is mirror other paths.
4) Exclude
  Exclude a path from both client views and branch views. This keeps unneeded content out of workspaces and streams entirely.