Soapbox Rants and Raves » 2009

Feb 24 2009

Dependency Management using Ivy

Category: Software Development — Phil @ 11:01 pm

I have had the good fortune to work on many software development teams over my career, and I believe this has contributed to my interest in creating effective development environments. Each time I started a new project, it seemed like I would spend a couple of days copying the best ideas from the previous project and reworking the problem areas, hopefully creating a much cleaner and simpler environment. This first step was concerned with the basic build environment, putting the ant scripts, dependent libraries, and Cruise Control into place. Unfortunately, this exercise usually consisted of a lot of cut and paste activities…

I can’t tell you how many times I have setup projects, implementing the Ant tasks for Cruise Control, PMD, Checkstyle, Clover, JavaDoc, etc… The fact that my Ant scripts were never really reusable always bothered me, but each project was independent and reuse at the corporate level never seems to pan out, so all I could do was cut and paste. Jar file management was also painful, as each project generally manged these resources in a unique fashion. I always tired to standardize my projects with a lib directory, which contained a sub-directory for each product. This sub-directory would usually include the version number in the name, so we had some idea of what version was being used. Most projects were a little more free form, they just dumped all of the jars into a lib directory with no concern for managing the component version information. Larger projects tried to create common SV

N module, just for their jars, but this was not the norm.

Enough history. I had done a lot of reading about Maven, but never took the plunge. The Maven philosophy was very different from what our teams were used to, but I was really intrigued by the dependency management aspect. This lead me to Ivy. Ivy was created to address a single problem, dependency management. Ivy was much easier to integrate into our existing environment, plugging into both Ant and Eclipse.

Ivy allows me to document, one time, all of the relationships between the open-source products in our environment. Once documented, these dependencies can be reused by all projects. Each team is no longer required to manage all of the open source jars and their dependencies, they can delegate that responsibility to Ivy. There are numerous other subtle benefits too, our code repositories are much smaller (jar files not checked in multiple times), the exact versions of each open-source product used was documented in the project it was used (project level ivy.xml files), better control over unapproved software (software must be approved before it is promoted), and it can even manage, tie together, the source code, Java documentation, and supporting files (license, css, tld, etc.).

One of the more interesting challenges was defining a strategy for integrating Ivy into the build process of large organization. Because we had multiple groups responsible for different stages of the process, we decided on a multi-repository strategy. Developers would have full access to the Sandbox repository, to install and configure new Ivy modules. Once the Sandbox modules had been tested and validated, they could be individually promoted to the External repository. The External repository is for any component, open-source or corporate, not built by our organization. The Internal repository is used to publish reusable components created by our organization. Ultimately, modules from the External and Internal repositories can be promoted to the Live repository, to be utilized by the production build process. As modules are promoted or retired from the Live repository, a baseline tag (and corresponding ivysettings.xml file) is created to capture the state of the repository. It is this tag (SVN URL) that is provided to the development teams. When each application’s schedule permits, the development teams can migrate to the most recent baseline of the Live repository.

I am completely sold on Ivy and use it for all of my home experiments; I feel it is much easier to mange an Ivy repository, than copy jar files from project to project. All I need to do is add a simple ivy.xml file and my project is ready to go. Setting up Ivy in corporate environment is a little harder, it is more about the process than what the tool actually does.

https://www.beilers.com/wp-content/plugins/sociofluid/images/digg_48.png

https://www.beilers.com/wp-content/plugins/sociofluid/images/reddit_48.png

https://www.beilers.com/wp-content/plugins/sociofluid/images/dzone_48.png

https://www.beilers.com/wp-content/plugins/sociofluid/images/stumbleupon_48.png

https://www.beilers.com/wp-content/plugins/sociofluid/images/delicious_48.png

https://www.beilers.com/wp-content/plugins/sociofluid/images/blinklist_48.png

https://www.beilers.com/wp-content/plugins/sociofluid/images/blogmarks_48.png

https://www.beilers.com/wp-content/plugins/sociofluid/images/google_48.png

https://www.beilers.com/wp-content/plugins/sociofluid/images/facebook_48.png

Comments (1)

Feb 17 2009

Interesting Electric Company Ideas…

Category: Go Green — Phil @ 6:26 pm

Have you ever wondered how much electricity you use, compared with your neighbors? Well, I do! I’m on a monthly averaging plan, but every month, the bill seems slightly higher than the last one. I basically know what is consuming most of our electricity; we have about a thousand things plugged in that are in stand-by mode! Our refrigerator and washer/dryer are about 12 years old; presumably they are consuming more juice than they should be. I would really like to know what I could do to minimize my usage, without having to resort to power strips or the manually unplugging all of the power vamps!

I would just like to know how inefficient my house really is! I saw this really cool story on the news, about the Sacramento Municipal Utility Districts (SMUD – great name!) attempt to make its customers more aware of the energy they were consuming. They started comparing each customer to their nearest 100 neighbors. Then they gave each one a grade, happy faces or frowny faces. I think the story made national news, because people actually got mad when they were given the “frowny” face. Personally, I think it is rather funny. Interestingly, SMUD removed the faces, but left the comparison bar graph. Apparently, this comparison data is having an impact on how people are managing their usage. Households that were using 3 times more electricity than their neighbors, received a real wakeup call and started to change their habits. A good thing all around, both for the environment and the pocket book.

I going send this note to my power company and see what they have to offer. Maybe they can run a custom report for me, I am just so curious about my consumption. Maybe I really don’t want to know! I promise to report back the results, should I actually hear from them.

Here are two more interesting links I discovered… The quote, taken from the Google.org site, applies to my day job in software development too…. “If you cannot measure it, you cannot improve it.” %G—%@ Lord Kelvin

Live monitoring of your electrical consumption, a SMUD service.
Live monitoring, the Google.org Energy Information Initiative

Comments (1)

Feb 15 2009

Cool WordPress Plugin…

Category: Blogging — Phil @ 12:00 am

I found a link today for some new and updated WordPress plug-ins and figured that I should try some out. I found one that would support my New Year’s Resolution; I am trying to read more books this year. I recently finished the Blink book, and am now moving on to the Think! book. The “I am reading” plug-in is kind of cool. It hooks up to the Amazon Web Services and pulls back information based on the ISBN number. Additionally, you can enter what page you are currently reading and the plug-in creates a progress bar under the image of the book cover. Probably more pain than it is worth, but I’m looking at it as a motivator…. I need to keep reading or the progress bar will never move!

On a side note, if you have not read either of the these books, I highly recommend them. Blink was very easy to read and thought provoking. It really makes you think about how people are conditioned to think and make decisions; sometimes, not the best ones!

Comments (1)

Feb 14 2009

The last word on Continuous Integration…

Category: Continuous Integration — Phil @ 9:22 am

My last rave about Continuous Integration (CI)… I happened to find a really interesting document this week, and I just had to share it… The main point that I wanted to stress is the value of CI. The study was from a very small sample, but the results where in line with my personal views. The document is a very good comparison of twelve different tools and provides a good background on why one would use CI. It is easy to read and contains lots of interesting graphs.

The document is a study of Continuous Integration tools by Fontys University of Applied Sciences in Venlo, Netherlands. This study showed that Cruise Control is still the leader in this space; probably due t o the fact that the Cruise Control has been around for almost 10 years. Hudson, which has only been around for 2 years, appears to be closing in fast on the Cruise Control market share. I found another website that shows Hudson with a significant portion of the CI market. To my surprise, there appears to only be a couple of real contenders, all of the other tools are not well represented.

I also wanted to throw in this last graph to reinforce the dominance of Subversion as a version control system. It has been interesting to watch how fast the open-source community moved away from CVS over the last couple of years. I think it will be very interesting to follow the growth of Git. It has received a lot of press over the past year and seems to create a religious debate when compared with other tools. Just what we need! I was shocked to see the significant projects using Git: Ruby on Rails, Android, Fedora, X.org, just to name a few.

Comments (0)

Feb 10 2009

Hudson Overview

Category: Continuous Integration,Testing — Phil @ 10:48 pm

I was playing with Hudson today in what I would call a non-standard, maybe non-friendly Linux box. The first time I setup Hudson, Java and Ant were installed in the standard locations; once I configured my projects, Hudson just went to work. On the non-standard box, nothing was installed in the expected locations, fortunately this was no big deal. The Hudson configuration options are simple and numerous. The system level configuration page has options to configure the JDK, Ant, Maven, Shell, and email settings; which are all specific to the machine that Hudson is installed on. At first, I thought this might be limiting, as very few projects use the same version of these tools. Luckily, the configuration page has the ability to specify the JAVA_HOME and ANT_HOME for multiple versions of each tool. This allows each project to specify which version of the tool they want to use in their build process, pretty simple.

One of the more interesting system level configurations addresses Hudson’s security. There are five different security options, from no security to per-project security. For authentication, I configured Hudson to use an LDAP server. This took a little trial and error, as this feature does not seem to be well documented. Fortunately, it only took a couple of attempts and the login screen started working. I chose to implement project-based security; this allowed me to delegate the responsibility of each user’s role to the project’s owner. This seemed like a good approach for limiting the amount of work required by the Hudson administrator. There could be simpler configurations, especially if you wanted to manage fewer people, but this approach gave me the finest grain control over each person’s ability… which might actally be overkill is practice. I would really need to see this working on a real project before I would be able to suggest the best security option, but it would seem to work in multiple environments.

The final step is to create and configure an individual project, which is also very straight forward. First, you pick the version control system and enter the repository and access information. Next, select the Ant-based project option and type in the Ant target name that is responsible for the continuous integration. There are multiple options you can configure here as well, but that is all I needed to start the process. Because Java and Ant were not installed in their normal locations, I selected the appropriate versions, as configured in the system level. That was it. I hit the build button and off it went. Hudson provides all kinds of visual feedback on the dashboard, letting you know exactly what is building, and even predicts how long until the current build finishes. One of the more interesting features is the ability to watch the build from within the dashboard (web page). Hudson uses AJAX to capture the log file of the running Ant process and shows a live update to the user. Pretty cool, it is no different than running Ant in a local shell!

This tool seems significantly simpler to get working than Cruise Control. I really like that fact that every project can be configured to match its own requirements. Some projects might want to use Checkstyle and PMD, another project might want to use FindBugs; not a problem. Once the plug-ins are installed by the administrator, Hudson makes them available to each project and automatically published the collected results. Because of this project centric view, it is even easier to integrate with multiple version control systems, each project simply chooses the appropriate back-end.

I did want to mention that Hudson seems to be a tool that extends beyond the typical developer requirements. As I mentioned in a previous post, QA groups can also use Hudson for activities other than continuous integration. One interesting feature is Hudson’s ability to monitor external jobs. This allows you to record the execution of a process run outside Hudson, even on a remote machine. It was designed so that you can use Hudson as a dashboard of your existing automation system. Hudson even have multi-server scalability built-in. This allows the environment to be configured with helper machines to distribute the processing load, taking advantage of the remote resources, while centrally managing and monitoring the process.

Hudson appears to have been pretty well thought out. Hopefully someday soon, I can use it for real!

Comments (0)

Feb 09 2009

Spring Utility Classes…

Category: Java — Phil @ 9:41 pm

Call me lazy or crazy, but I have never really taken the time to look at everything that the Spring Framework can provide. I read an interesting web page the other day, complaining about the size of different Java frameworks. The author ranked Spring as the number one in “Complexity Pain” with over 2,400 classes. If you think about it, that is really amazing . Out of curiosity, I decided to look at the JavaDoc index for the Core Spring framework. It is actually quite impressive or quite paralyzingly!

Sometime back, I discovered something in Spring that I doubt is all that well known, but probably very useful to the average Java project, the Spring Util package. I once worked with a developer who liked to put guard statements inside all of his methods. He created his own assertion class that would look for null values or null parameters, and create informative exceptions when the method was used incorrectly (the guard was true). It was a little more helpful than the basic null pointer exception and much more convenient than adding a bunch of “if” blocks. While I was not accustomed to this programming style, I ultimately became more accepting of this practice. One day when I was creating a jUnit test, I accidentally discovered the Spring Util Assert class. The Spring Assert class had many more guard options than the one we used and the additional benefit of not being in our code base! Code is a liability… as one of my friends likes to say!

I would highly recommend that anyone using Spring become familiar with the classes in the Util package. I was actually quite surprised by the variety of support; from Java 5 annotations, collection utilities, numeric conversions, to the copying of files and many more. If you really think about it, the large number of support classes should not be all that surprising. Developers always seem to need these type of classes when building non-trivial software. Obviously, the more interesting and valuable Spring classes (the reason you would actually want to use the Spring Framework!) required these reusable utility classes to provide the overall functionality. So, why don’t we take advantage of them too?

Comments (0)

Feb 07 2009

Free Norton Antivirus Upgrade…

Category: Windows — Phil @ 3:35 pm

I would not say that I’m a huge Norton fan, but I have had pretty good luck with them in the past. I’m sure there are better options, but for the price, securing 4 different machines, Norton works out well for me. The 2009 version of Norton seems smaller and less intrusive than previous versions; just my personal observation. Even more interesting, Norton released an Antivirus version for Gamers this year. I guess Norton finally realized the “bloat” their software was adding, was really getting out of hand.

Anyway, I happened to boot up my Windows laptop last night, and a Norton window popped open and said that I could upgrade my 2008 Antivirus to the 2009 version, for free. I was a little skeptical, but a quick Google search confirmed the deal. Apparently, with the release of the Norton 2009 suite of software, Norton as made available free upgrades to 2009 for anyone currently using either 2006, 2007 or 2008 versions.

Not a super big deal, but I always like installing new software, especially if it is free!

Comments (1)

Feb 07 2009

Continuous Integration

Category: Continuous Integration — Phil @ 11:07 am

I embraced continuous integration many years ago for purely selfish reasons, but it did not take long to see the real value of the process. I happened to work on a team where developers continually forgot to check in all of their work. Being an early morning person, I would be the first one to sync out the previous days changes; at least once a week, my work area would be unbuildable because of missing files. I would end up wasting time mocking up an object or figuring out how to work around the problem.

Enter Cruise Control. I originally setup the tool for notification purposes, cruise would monitor the CVS repository for changes and kick off the build script after each check in. If the build failed, anyone who made changes since the last successful build will be notified of the failure via email. To ensure that the developers would fix the problem, we enabled the spam feature. Cruise would continue to attempt building the project until it was successful. This could result in numerous email messages per hour, and was usually a pretty good motivator to fix the problem. We actually had to institute a higher level of encouragement on some projects, the “Bagel Rule”. If you were responsible for breaking the build for over 24 hours, you had to bring in bagels for the team. This rule worked wonders!

Once the developers got used to the process, we started to crank up the expectations. We could use Cruise to improve the quality of our code, enabling tools like Checkstyle and PMD. If the code did not conform to our standards or was overly complex, the build was marked as a failure. These tools essentially automated a basic code review process. They did not guarantee a good design, but did a pretty good job at ensuring that the code was well structured and maintainable. The most valuable Cruise integration was the addition of jUnit and Clover. By executing the jUnit suite after each check in, helped us discover any unintended sided effects induced by the changes. By combining the execution of the jUnit suite with code coverage data (we happened to use Clover), gave us a quantitative value of the confidence that we could place on our code. If we had low code coverage, it was impossible to have high confidence in our code’s correctness, or even in our ability to refactor the code and ensure that that it still worked. Conversely, if we had a decent quality jUnit suite and a high code coverage percentage, then we felt pretty good about the correctness o the code base and had complete confidence in our ability to refactor.

Cruise Control has been around for quite a few years and is now a very mature tool. This is one of those areas that I have not recently researched, but was surprised by how many options are now available to development teams. Cruise Control is fairly easy to setup and does exactly what it needs to do, from an continuous integration perspective. However, it does not provide that same amount of information and value that other tools appear to provide. From a commercial product perspective, I would really like to evaluate the Bamboo tool from Atlassian. I have been throughly impressed with the Atlassian product suite. I think they really understand what developers need and how they work. My favorite open source product is Hudson. It seems to have some pretty big supporters, such as JBoss QA (Check out the link, pretty cool). What makes Hudson interesting is the plug-in infrastructure and the trending aspects. It is very easy to integrate and publish a variety of metrics in the Hudson dashboard, something that Cruise Control does not do very well at all. The dashboard gives you a good feeling of the projects health and stability. Hudson keeps a history of metrics, and displays how the project has evolved over time. Pretty cool, if you are interesting in continuously improving your code base; it gives the developers a dynamic picture of their effort. Configurability is probably one of Hudson’s greatest strengths, more on that later.

What are your experiences? Just like unit testing, I would not want to develop without my continuous integration safety net!

Comments (0)

Feb 03 2009

Unit Testing and building a Ford Focus…

Category: Testing — Phil @ 10:35 pm

I was chatting with a friend today, and he told me a story that I could not believe. More interesting, was the fact that his story sounded exactly like my last post on Software Quality. His team was not on the same page, they each had different development philosophies and others used the existing environment to squash his proposal. So, what was his crazy idea? He proposed that the project use jUnit during the development process, and to add a line item to the project plan to include unit testing. He thought if unit testing was part of the project plan, the team would have a little more incentive to participate.

To his surprise and my dismay, the project manager said, “We are not building a Mercedes, we are building a Ford Focus, and doing unit testing is essentially unnecessary”. Oh my gosh, I about blew a gasket! I have been doing test driven development for so long, I would not even know how to write code without a unit test! What am am missing? Why wouldn’t a team want to create unit tests? How do they validate what they are building? Do they just use Loggers or print lines statements to generate output and then visually inspect the results?

This is one question that I have never been able to answer; why would a developer not want to create a structured unit test? Why do they prefer to add a “main” method to execute the code, rather than simply creating a jUnit test? I will not go into the benefits of using a continuous integration tool at this time, but just having a jUnit test that other developers could execute must be of some value. Right? Why would I want to continually inspect the output results of the class execution, when I could use assertions to actually validate the results of each test? The power, flexibility, speed, and maintenance of jUnit is not even comparable to value provided by a simple main method.

I personally believe that creating a unit test adds absolutely nothing to the overall cost of development. After all, the developer has to create something to validate the code, right? I have never seen a developer just sits down, write the code, and send it off to production. If code has to be executed to be validated, why not use jUnit? Throw in a tool like Eclipse, NetBeans, or IntelliJ, and it makes unit testing almost trivial. All the developer needs to do is click on a button and tool executes the unit test; these tools even have fancy windows to quickly show the result of each test, indicating success or failure.

There could be some potential downstream cost to supporting a unit test suite, but this cost has to be the same or less than than maintaining all of those of main methods. I also believe that the cost of supporting the unit test suite is offset by the total value provided by the suite. This value will be realized as the system evolves, allowing the team to refactor code, providing a validation framework for their changes.

Help me out here… Am I wasting my time building unit tests or have the other developers just not seen the light? Actually, I don’t think I am wasting my time, nor will I quit writing unit tests any time soon!

Comments (0)

Feb 01 2009

Software Quality

Category: Software Development,Testing — Phil @ 12:00 pm

What makes a high quality piece of software? Kind of a tough question to answer; everyone has their own idea of quality. I like to look at quality from a team perspective; I believe that quality software has the following signature traits:

Shared vision, documented design
Code that is easy to change and supports refactoring
Ability to prove that the code executes correctly
Code consistency

None of these traits are original and they all have their roots in traditional software development methodologies. I believe that their relevancy was renewed by the Extreme Programming and Agile development movements and actually made possible by tools which have emerged over the last few years. There are definitely other traits that convey software quality, such as efficiency, usability, and applicability. However, I believe that these are more dependent on individual efforts, rather than a product of team cohesiveness.

I know there are many alternative views and even skeptics who claim we don’t need coding standards or unit test suites. They believe their systems represent high quality software; but how do they really know? What proof do they really have? To their defense, the applications have been running, performing real business function, requiring very little maintenance, and apparently very few bugs. But, do those criteria actually imply quality? The stability of a system is definitely an important quality trait, but I would consider it a secondary trait; one that is actually a side effect of a signature trait, proof that the code actually executes properly.

While writing, I also realized there was another dimension that supersedes all other traits, the team’s cohesiveness. I believe it is very much related to having a ‘vision’, for a team to effectively produce quality software, there needs to be complete synergy. This sounds a lot like ‘assimilation’, maybe it is, but I really don’t like that word. I don’t think I have ever seen a team without a vision, but rather too many incompatible visions; they never have the opportunity to form the unified vision. This encompasses how things are designed, how they are constructed, and even how they are tested. If the team does not have synergy, achieving quality will cost a lot more. This cost will not be a corporate cost, but more of a individual cost, the additional effort required by each member of the team to ensure the project gets finished.

There is another, more personal trait that I like to apply to my projects, the desire to do things better the next time around. I’m not sure about you, but I have never built the perfect piece of software. I always think about how I could have done a better design or a better job coding. We don’t always get the chance to redo our work. But without seriously evaluating our own work, how can we ever learn from our mistakes. It is very important that the team wants to evolve, if the team is wed to their current coding patterns and practices, then achieving higher levels of quality is going to be much harder, maybe even impossible. I think quality is evolutionary; you just don’t “make” quality happen. The team has to be interested and excited about creating the best possible software, within their current environment.

Within their current environment; it is a very interesting qualifier to the above statement. I believe many teams use their environment as an excuse to keep the status quo. I think it is very important and my personal responsibility to do things better, every day, every iteration, every assignment. I have never worked on a project that did not have some amount of built-in cushion, for all of those unplanned requirements. Moreover, if it was something that the team really believed in and wanted to do, then we made it happen; if that meant some long hours, then that is what we did. Teams need to use the slow times to their advantage, adding new tool and techniques to their toolbox, improving the development process, and going after those nagging problems that never get addressed.

I have worked with and observed teams where the software development environment was like a game, developers were challenged and excited to create clean builds, remove warnings, and even create unit test. I have seen teams actually compete to generate better code coverage numbers than a rival team. <Disclaimer>High code coverage does not imply high quality code; more on that later!</Disclaimer>. The bottom line is, these teams wanted to make their code better and doing these activities was considered fun rather than a burden or something that provided no value. To further illustrate my point, I found a plug-in for the Hudson Continuous Integration tool that I would love to try, it is call the “Continuous Integration Game”. Developers actually earn (or lose) points depending on the code they check in. Break the build, lose 10 points, add a unit test and earn a point. It may seem silly, but if inexpensive, little metrics can help energize a team, then I am all for it.

I really don’t think I addressed any of my signature traits in this post, but it was important to establish a philosophical baseline for future posts on quality software development. In future posts, I hope to address how we have used tools and techniques to continuously improve our software development process and ultimately, build better software.

Comments (0)