Wednesday, June 9, 2010

Integrating Subversion with Mantis

It is useful idea to install a web-based bug tracking system. Each member of development team, QA team or even each customer can report about bug, enhancement requirent or just feedback. The other side of the coin is maintance problems. For example customer named Donkey Shon reports about the bug. At this moment, you are few days before next release, so you resolve this bug after two weeks. Another two weeks later, Shon Donkey reports that the issue is still unresolved. At this point of time you already don't remember when you fix a bug, so you don't know what revision is it and if it was in new release or in the old one. Mr. Donkey is angry and he doesn't want your next release. The problem would not been happen if you've remember to label your SVN commit with bug id and to update status of the bug and its notes, but your team was too busy with new release of product.
Everyone who experienced this use-case, wants to synchronize bag tracking system with version control system and you can find some products that aims to do it. However, if you use Subversion and Mantis you can integrate it with 100 lines of script without learning manual and installation instructions of new product, that always met only part of your requirements . There is excellent explanation how to do it, at "alt-tag.com" ( see link at the end). Although the blog mentioned, is short and clear, it has two leaks. First, the scripts code wrote for Linux ( I love Linux, but there are still some Windows servers). Second, it assumes that developer, who commits the changes, always wrote comments in same pattern. The opinion, that developers are robots, is partly true, but they do mistakes, sometimes.
In this article, I'll give you the example of scripts that check commit label before the revision committed, blocks it if it has a error, change bug status after commitment and adds commit label to bug notes at Mantis. There are 6 steps: Mantis definitions, SVN comment template definitions, post-commit svn hook, pre-commit svn hook, post-commit mantis script and pre-commit mantis script.

1. Mantis definitions( config_inc.php ):
<?php
$g_hostname = 'localhost';
$g_db_type = 'mysql';
$g_database_name = 'bugtracker';
$g_db_username = 'root';
$g_db_password = 'rcadminsin';

#User that looks in svn for comments with bug_id
$g_source_control_account = 'svn';
#comments that identifies bug in subversion
$g_source_control_regexp = '/\b(?:bug|issue)\s*[#]{0,1}(\d+)\b/i';
#definition of svn commitements that change bug status in Mantis
$g_source_control_set_status_to = RESOLVED;
$g_source_control_set_resolution_to = FIXED;
$g_source_control_fixed_regexp = '/\bfix(?:ed|es)\s+(?:bug|issue)?\s*[#]{0,1}(\d+)\b/i';
?>

2. SVN comment template: Set property "tsvn:logtemplate" recursively for all files in your project. In this case the property value can be "fixed bug #&quot, so the developer need only add Mantis bug_id.

3. Post-commit SVN hook: Every SVN repository has "hooks" folder( e.g. C:\Subversion\repository\hooks ). An executable or batch file placed in this directory will be called if its name correlates with SVN event. At this stage it is "post-commit.bat". It calls for PHP script in purpose to update bug status at Mantis and to add revision comment as bug note.
@ECHO OFF

SET REPOS=%1
SET REV=%2

SET PHP=C:\AppServ\PHP5\php.exe
SET CHECKIN=C:\AppServ\www\mantisbt-1.2.1\scripts\checkin.php
SET SVNLOOK=C:\Subversion\bin\svnlook.exe

SET LOGFILE=log_post_%REV%.txt
SET AUTHORFILE=author_post_%REV%.txt
SET OUTPUTFILE=output_post_%REV%.txt

%SVNLOOK% log -r %REV% %REPOS% > %LOGFILE%
%SVNLOOK% author -r %REV% %REPOS% > %AUTHORFILE%

ECHO SVN %REPOS% Revision %REV% > %OUTPUTFILE%
TYPE %AUTHORFILE% >> %OUTPUTFILE%
TYPE %LOGFILE% >> %OUTPUTFILE%

TYPE %OUTPUTFILE% | %PHP% %CHECKIN%

CALL DEL %LOGFILE%
CALL DEL %AUTHORFILE%
CALL DEL %OUTPUTFILE%

4. Pre-commit SVN hook: This hook calls other PHP script ( "pre-commit.bat" ):
@ECHO OFF
SET REPOS=%1
SET TX=%2

SET PHP=C:\AppServ\PHP5\php.exe

SET VALIDATOR=C:\AppServ\www\mantisbt-1.2.1\scripts\validate_bug_fix.php
SET SVNLOOK=C:\Subversion\bin\svnlook.exe
SET LOGFILE=log_pre_%TX%.txt
SET OUTPUTFILE=output_pre_%TX%.txt
%SVNLOOK% log -t %TX% %REPOS% > %LOGFILE%
ECHO SVN %REPOS% Transaction %TX% > %OUTPUTFILE%
TYPE %LOGFILE% >> %OUTPUTFILE%
TYPE %OUTPUTFILE% | %PHP% %VALIDATOR% 1>&2
IF %ERRORLEVEL% gtr 0 ( GOTO ERR ) ELSE ( GOTO RM_LOG )
:RM_LOG
CALL DEL %LOGFILE%
CALL DEL %OUTPUTFILE%

exit 0

:ERR
ECHO commit denied >&2
exit 1
The script above prevents from developers to commit revision, that aims to fix already fixed issue. If pre-commit hook has exit code other then zero, SVN will not commit the changes and will display content of STDERR. ">&2" purpose is to copy STDOUT content to STDERR, so developer can see what error prevents his commitment. Most of work are done by PHP script called (see 5 and 6).
5. Post-commit PHP script: This script is already included in Mantis distrubution. It checks if text in STDIN stream is matches the template. If true, its find bug_id changes its status in mantis and add comment as note of the issue. If you need additional functionality you can change it or override default "custom_function_checkin" fuction.

6. Pre-commit PHP script: This script is not part of Mantis, but you can easy create it, because most of it, similar to checkin.php script, included in Mantis( see 5 ). You can use Mantise's "bug_api.php" methods to add other validations.
<?php
# Detect references to issues + concat all lines to have the comment log.
# Called before SVN commits changes to ensure that bug committed is not already resolved

global $g_bypass_headers;
$g_bypass_headers = 1;
require_once( dirname( dirname( __FILE__ ) ) . DIRECTORY_SEPARATOR . 'core.php' );

$t_commit_regexp = config_get( 'source_control_regexp' );
$t_commit_fixed_regexp = config_get( 'source_control_fixed_regexp' );

# Check that the username is set and exists
$t_username = config_get( 'source_control_account' );
if( is_blank( $t_username ) || ( user_get_id_by_name( $t_username ) === false ) ) {
echo "Invalid source control account ('$t_username').\n";
exit( 1 );
}

if( !defined( "STDIN" ) ) {
define( "STDIN", fopen( 'php://stdin', 'r' ) );
}

$t_comment = '';
$t_issues = array();
$t_fixed_issues = array();
while(( $t_line = fgets( STDIN, 1024 ) ) ) {
$t_comment .= $t_line;
if( preg_match_all( $t_commit_regexp, $t_line, $t_matches ) ) {
$t_count = count( $t_matches[0] );
for( $i = 0;$i < $t_count;++$i ) {
$t_issues[] = $t_matches[1][$i];
}
}

if( preg_match_all( $t_commit_fixed_regexp, $t_line, $t_matches ) ) {
$t_count = count( $t_matches[0] );
for( $i = 0;$i < $t_count;++$i ) {
$t_fixed_issues[] = $t_matches[1][$i];
}
}
}

# If no issues found, then no work to do.
if(( count( $t_issues ) == 0 ) && ( count( $t_fixed_issues ) == 0 ) ) {
exit( 0 );
}

# If one of the issues are already resolved it is a error to commit it.
foreach( $t_fixed_issues as $t_issue_id ) {
if( bug_is_resolved( $t_issue_id ) ) {
echo "The bug with id ".$t_issue_id." is already resolved.\n";
exit( 1 );
}
}

#echo "Comment ".$t_comment." OK.\n";
exit( 0 );
?>

That's all. You can add additional functionality to php scripts described, if you need. Link to the article, that used as base:

Monday, June 7, 2010

Tomcat Https and Eclipse

One of requirements in our project is requirement for HTTPS login form. However, bosses don't want to use HTTPS in every page, so I need to do three things: create SSL key, define HTTPS in tomcat and create such definitions in Spring Security XML, that redirects anonymious requests to login form and after that to home page.
Key definition is a step that you need only at development stage. Key is part of certificate that your server needs to send in response to any HTTPS request. The certificate defined by any of free tools you can download, is not trusted by browsers and you will see certificate warning, but you can ignore it, until production stage, where you should buy a real certificate from one of organizations that sell it.
Since we use Apache Tomcat 6, I've decided to use Java keytool to create the key. This utility is supplied with JRE and you don't need to download openSSL engine. After typing
%JAVA_HOME%\bin\keytool -genkey -alias tomcat -keyalg RSA
in command line I've answered several questions and approved the data typed. Last question, about Tomcat Password, is important for next step. In the end of the process, a file named '.keystore' will be created in your user folder.
Default tomcat implementation that you have in eclipse don't support's version 2.2 of EL-expressions, that we use, and I don't want to change definitions when a project will be moved from development environment to production. So I didn't use default eclipse implementation of tomcat, but down loaded fresh Tomcat 6.0.26 from Apache website. It can be used both from eclipse and as standalone webserver, but eclipse overrides Tomcat configuration files with its own each time it noticed the change. Adding Tomcat to eclipse is simple: open Servers view, right click on it and choose 'New -> Server -> Tomcat v6.0 server'. You'll see Server runtime environment row and 'Add' link in the end of it. Click 'Add' and browse to your tomcat installation folder before you click 'Next' in 'New Server' dialog.
Now, you have eclipse project named 'Server' in any eclipse view, that shows list of projects in workspace. Open 'server.xml' file in this project and comment the element below:
<Listener SSLEngine="on" className="org.apache.catalina.core.AprLifecycleListener"/>
The reason of this operation is, that tomcat supports three types of certificates JKS, PKCS11 and PKCS12 and you don't know what certificate will be bought in future. As you can guess from its name, JKS is supported by JAVA and for other types you need openSSL or other external SSL engine. Since I don't use openSSL, I've commented the listener defintion, but didn't remove it, because I don't know what certificate will be used at production stage.
At this moment, I need to define how tomcat will deal with HTTPS connections and what certificate it will use. There is a commented connector element in 'server.xml' that defines SSL for port 8443. You can remove comments and use it as basis, but you should add attributes that define location of key file and its password. In my case It was:
<Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"
maxThreads="150" scheme="https" secure="true"
keystoreFile="${user.home}\.keystore" keystorePass="igor"
clientAuth="false" sslProtocol="TLS" />
The value of keystorePass attribute should be the same value that was entered as 'tomcat password', during key definition.
This is the end of Tomcat definitions. Right-click your server in 'Servers view' and choose 'Clean' command. This, force Eclipse to refresh Tomcat definitions and web applications, including coping updated 'server.xml' into Tomcat folder. Sometimes Eclipse has a bad day and doesn't want to notice that 'server.xml' was changed. In this case you can remove Eclipse's internal copy of it manually. You should remove workspace/.metadata/.plugins/org.eclipse.wst.server.core/tmp0/conf/server.xml
for it. If it also doesn't helps, kill eclipse and go home.
The remaining part is Spring Security configuration. I'll give our example of http element of configuration file:
<http auto-config="false" access-denied-page="/accessDenied.jsf">
<intercept-url pattern="/enter.jsf" requires-channel="https"/>
<intercept-url pattern="/j_spring_security_check"
requires-channel="https"/>
<intercept-url pattern="/**" requires-channel="http"/>
<form-login login-processing-url="/j_spring_security_check"
login-page="/enter.jsf" default-target-url="/home.jsf"
authentication-failure-url="/enter.jsf" />
<session-management session-fixation-protection="none" />
</http>
The configuration above defines 'enter.jsf' as login form and interceptor-url defines that it should use HTTPS. Last 'intercept-url' element defines that all other URLs use HTTP. The 'intercept-url' element in middle, is less obvious. 'j_spring_security_check' is request URL for all Spring Security credentials validations, so it also should use HTTPS.
This configuration only defines the protocol used for each page, but doesn't defines role's access. Roles ans grants are managed by special filter, that also redirects all not authorized request to login form.
More detailed information about HTTPS definitions of Tomcat and Spring Security can be read at these locations:
Tomcat Site:
Spring Security reference, section 2.3.2: