Tuesday, April 1, 2008

Checkout rules speed up TeamCity builds a lot

I discovered something interesting today: TeamCity allows you customize what folders are checked out from your source control (we're using subversion). When we were building before, the actual build took about 5 seconds or so but the total length of time was on average a minute and a half. People complained about this and didn't lieke doing personal builds because it took too long (personal builds allow you to send a build to TeamCity and have it try to build with your updates before committing your code. This helps stop people from committing breaking changes). I found that the reason it was taking so long is that we have a lot of stuff under version control, such as our database, the QA test projects, some tools projects, as well as a bunch of other random crap that, while important to the whole project, are not directly used by the particular build configuration we care about for the website. So, after reading the manual, I discovered that I can restrict TeamCity to only check out certain directories. Here's what I did:

My svn root for my project (this is not my real svn tree, duh) is at https://127.0.0.1:8080/projects/trunk and the various parts of the application live in sub folders. I therefore have the following:

https://127.0.0.1:8080/projects/trunk/Application - our web application
https://127.0.0.1:8080/projects/trunk/Database - the database scripts
https://127.0.0.1:8080/projects/trunk/Externals - things like Rhino, MbUnit, etc.
https://127.0.0.1:8080/projects/trunk/Tools - useful tools that we write for application admin stuff
https://127.0.0.1:8080/projects/trunk/QA - our QA department's test stuff (BVT, regression, etc)

So in order to build our web application, everything we need is in Application and Externals. However, I can't just point TeamCity to https://127.0.0.1:8080/projects/trunk/Application because then I don't get Externals. I also can't just have an externals directory on my build agents because then I have to maintain them in multiple places (DRY principle- it's not just for code; it applies to project structure and build scripts too). So, I have to check out https://127.0.0.1:8080/projects/trunk which gives me about 100mb of extra crap that I don't need to build the web application. How do I deal with this?

Under TeamCity's project configuration on the Version Control Settings section, there is a thing that says "edit checkout rules" that, when clicked, will allow you to apply rules to what gets checked out. I clicked it and here is how it works:

You put each rule on its own line. A rule starts with either a + or a -, with + meaning to check out the thing and - meaning not to check it out. You would make rules relative to the root directory, so since my version control settings for the VCS root is set to https://127.0.0.1:8080/projects/trunk, I have to make all my rules relative to that path. Therefore, to exclude Database and save about 45 seconds of checkout time, I add a rule that restricts checkout of that directory. Since the directory is https://127.0.0.1:8080/projects/trunk/Database, I do this:

-:Database

and it now ignores the database (I believe that these are case sensitive). I applied a few more rules and now all that is checked out is https://127.0.0.1:8080/projects/trunk/Application and https://127.0.0.1:8080/projects/trunk/Externals for my main application build. This reduced my build time to about nine seconds on average. Much better. There are other things that you can do here with checkout rules also but I'm not actively using any of them yet.

I have been using TeamCity for a while now, I strongly suggest you check it out. There is a free version with limited numbers of Users (20), Build Agents (3), and Build Configurations (20). You can buy additional build agents individually or you can buy the full version which gives you more features. Check it out here: http://www.jetbrains.com/teamcity/

2 comments:

Larry Bergstrom said...

I am a recruiter surfing blogs to help seek out candidates for this position, Senior Build Manager. This is an excellent opportunity in the Dallas – Fort Worth area, for someone who is a thought-leader in the strategy and creation of SCM and Build processes in an Agile, using preferably SubVersion or TeamCity, but not required. I would like to inquire with potential candidates who may be qualified for this Software Configuration Management (SCM) Manager position to ask to inform everyone of this position and/or request a resume in MS Word format and/or contact me by email and/or telephone.
Larry Bergstrom
email: Larry@CompStaff.NET
mobile: 817-723-4298
office toll-free: 888-416-8999
Computer Staff » Delivering Extraordinary Talent » www.CompStaff.NET
Senior Agile Build Manager
Under the direct supervision of the Director of Application Development, the Senior Agile Build Manager will provide guidance and leadership in the agile build strategy that includes Software Configuration Management (SCM), source code organization, continuous integration and build management, versioning/labeling, promotion and packaging of software intended for QA and production deployment.

Principal Duties and Responsibilities

• Design, coordinate and execute the strategy that facilitates ongoing parallel development, feature branching, versioning/labeling, merging and common library management for agile development team operations.

• Develop complete mastery of selected tools that provide for efficient, automated build management and continuous integration of newly developed software.

• Lead the creation of an efficient, continuous integration minded experience for build management incorporating an appropriate mix of revision, baseline, branching, labeling, code quality tools, release packaging and promotion to QA.

• Provide expert guidance in the creation of deployment packages for delivery to the Launch Manager that will in turn deploy to QA and the production environments.

• Will maintain consistent and stable integration, regression, and staging environments for developers and website designers.

• Will develop procedures for the collection and reporting of certain release engineering artifacts and metrics.

• Publish build management best practices and related information in our company wiki.

• Work with the Launch Manager, QA testers and Developers to troubleshoot and remediate issues found in the quality assurance, user assurance testing, training or production environments.

• Continuously improve our client's build management, continuous integration, and launch procedures and processes used by the product teams.


Experience

• Minimum 5 years software configuration management experience with the Microsoft.Net framework and ASP.Net.

• Associates degree in Computer Science or related field and/or equivalent experience

• Experience with Subversion and TeamCity a plus.



Skills and Abilities

• Proficiency with concurrent versioning systems (especially Subversion).

• Excellent understanding of the GAC, .Net Assemblies, Web Services, Windows Services, WCF, DLLs, EXEs, IIS, C#, VB. NET , XML and .NET platform.

• Working understanding of Windows Server 2003 operating system and its security.

• Strong understanding of IIS 6.0 and 7.0.

• Proficiency in Microsoft Visual Studio 2005 and 2008.

• Proficiency with SQL Server 2000/2005

• Ability to manage builds to meet the needs of an agile culture, specifically with respect to efficient continuous integration (fast clean/build/test/deploy)

• Ability to develop software packaging scripts and deploying .NET applications using Wise Package Studio or WinINSTALL or similar tools.

• Ability to create software configuration management standards, processes and other documents as required.

• Comfortable in a fast-paced and cutting edge agile development environment.


Core Traits

• Agile-Centric Thinker – Understands and practices the core tenets of Agile software development. Guides others in the adoption of agile build engineering practices.

• Problem Solver - Solid analytical and problem solving skills. Ability to analyze current situations with new requirements and synthesize into creative solutions.

• Hard Charger – Takes the initiative to get things done, especially in team settings with dependencies on other people.

• Adaptive – Works effectively in the faces of stress, ambiguity, difficult situations and shifting priorities.

• Innovates – Challenges the status quo thinking to generate new ideas; takes open minded approach to situations.

• Communicator - Refined written and verbal communication skills. Ability to foster open communications, listen effectively, and build strong partnership networks that result in consistent forward momentum.

• Technological Leader – Established experience in their technical field. Continually augments experience and skills with the latest research results and techniques.



Working Conditions/Physical Demands

Frequent keyboarding required, using hands to finger, handle, or feel. Specific vision abilities required by this job include close vision, and ability to adjust focus. May spend up to 8 hours per day sitting and may have occasional movement throughout the facility. Availability to support off-business hour rollouts.
Employment Type: Full-Time Employment, salary, direct hire.


Benefits: BONUS, life, health, dental, vision, 401(k) matching, short-term disability, accidental death and dismemberment, paid vacations, paid holidays, paid sick leave.


Salary: $95,000 - $110,000 per year + bonus

Location: Fort Worth, Texas
no h1b sponsorships. US citizens and US greencard holders are highly preferred. no third parties. no consulting firms. principals only.

Please email resume in MS Word format to:
Larry Bergstrom
email: Larry@CompStaff.NET
mobile: 817-723-4298
office toll-free: 888-416-8999
Computer Staff » Delivering Extraordinary Talent » www.CompStaff.NET

Gavin said...

Dude, your post will work for the build server but it wont work for agents. Team city documentation quote: "Exclude checkout rules will only speed up server-side checkouts. Agent-side checkouts emulate the exclude checkout rules by checking out all the root directories mentioned as include rules and deleting the excluded directories. So, exclude checkout rules should generally be avoided for the agent-side checkout.".

I wish they would sort this out, because we have agents running. Aside from that TeamCity rocks!