Did We Pass On The Joel Test?
| 17th July 2011 |
One of the most popular articles on Joel on Software has been The Joel Test. If you work on a software team, you should definitely try it out. While this test is not a formula for software success but it does look like something that would help improve your team’s effectiveness.
Basically, the Joel Test has the following 12 questions you need to answer in a Yes/No.
- Do you use source control?
- Can you make a build in one step?
- Do you make daily builds?
- Do you have a bug database?
- Do you fix bugs before writing new code?
- Do you have an up-to-date schedule?
- Do you have a spec?
- Do programmers have quiet working conditions?
- Do you use the best tools money can buy?
- Do you have testers?
- Do new candidates write code during their interview?
- Do you do hallway usability testing?
Here I try to apply the test to our software team and figure out how well we fare.
1. Do you use source control?
Yes. We do use CVS (Concurrent Versions System) for source control. Since we use Eclipse as an IDE, a CVS client is already available with it.
Source control is the most important tool used during software development because:
- We always got a backup of a working copy of your software. So if one of the developers accidentally messes the code, we can always revert back to the previous version.
- Source code tools have a way to branch out your code. So you can maintain separate builds for different clients or products.
Of course there are also a few inconveniences when using source control. Source control enables two or more developers to work on the same piece of code (say a same file). So when it’s time to check in the code into the source control repository, a merging needs to be done. Source control tools are smart enough to merge changes that are independent but sometimes a manual merge is required. Also your local copy of the code will need to be updated everyday to be in sync with the repository.
There is no reason not to use a source control especially with so many good tools like CVS, SVN, and Git freely available.
2. Can you make a build in one step?
No. We can made a build in few steps though. The reason why we can’t do this is because there are a few dependencies like changing configuration files on servers. For production, it could also mean getting signed keys before you can actually make a build.
My view is that as long as we have a good process established for making builds, then it is not a problem to make builds in more than one step. Of course, it should not be something like a 100 steps but 5-10 steps would be just fine.
3. Do you make daily builds?
No. We do make builds every few days but not every day. I see no reason why this cannot be done everyday except the fact that our code base does not tend to be updated every day. We make builds only when enough tasks/defects have been resolved to warrant a build.
We do have a system that runs every few days just to check if the build is broken or any other issues were introduced in the build. But it does not create an official test build.
I think our team is quite small and it rarely happens that someone checks in code that breaks the entire build. It’s enforced strictly that developers compile the code in all possible configurations before they check in. IDEs like Eclipse make it relatively easy to ensure that you check in all the affected files and don’t skip any of them.
4. Do you have a bug database?
Yes. We use JIRA for bug tracking and find it very good. It has all the features like bug tracking for various projects, mentioning bug types, assigning priorities to bugs, assigning bugs to the concerned developers and a commenting system.
Bug tracking is essential to software development to ensure communication between developers and testers. It makes sure that all defects found by testers have been recorded for easy tracking. Systems like JIRA can also be used for task management along with bug tracking.
5. Do you fix bugs before writing new code?
Yes. Once an official test release is made and testers start detecting bugs, developers immediately start working on those bugs. So working on existing bugs is given more priority to working on new features.
However we work with Scrum as a method of the Agile framework. So work is basically divided into Sprints. Each sprint has a particular scope of work that needs to be completed within a period of say 2 weeks. This holds for the entire team of developers and testers. So it could happen that only basic testing is done during sprints and an actual complete system test is only done after completion of most of the development tasks. What this means is that it could be weeks before some of the defects are detected that could have been detected earlier simply because the priority of extensive system testing is lowered to the priority of testing the new features.
6. Do you have an up-to-date schedule?
Yes. Scrum helps us schedule the tasks that are planned for the next couple of weeks. The tasks are based on priority of the customer and tasks which are not important are cut out of the schedule.
It could happen that the customer changes his mind in time so that the priorities get changed. This causes a bit of mayhem on the earlier planned schedule. Maybe something to work on is to perform scope reduction to recalculate the priorities of the tasks, remove the unwanted tasks and again perform a re-estimation to inform the client what can and cannot be done in the remaining time.
7. Do you have a spec?
Yes. Though Scrum does not actually need requirements and design specifications. The emphasis is to start working on code as early as possible.
But since we are an onsite-offshore model, we require to have the approved requirements specifications from the customer. It could be that only some of the requirements are approved and the remaining will be done as time progresses. Same is the case with the design proposal. It could take some time for the design proposal to get approved so there is no choice but to start with the coding. As Joel mentions, this is a risk (that Scrum actually incorporates) that the requirements could change causing an expensive rework.
Till the requirements and design are finalized we start working on prototypes that we can later reuse. The advantage of this method is that we tend to know design issues or coding issues as soon as the prototyping begins. Also if there is not much change in the requirements, we can reuse parts of the prototype for the final implementation thus saving up time.
8. Do programmers have quiet working conditions?
No. Most organizations in India love to follow the cubicle way of working. Developers and testers are surrounded by each other. So noise and distractions are unavoidable.
Scrum actually promotes that the team sits in proximity to ensure that problems can be immediately communicated and resolved. Developers and testers tend to put their minds in a state of total concentration when working. So too many interruptions and distractions can be responsible for decreasing the productivity of employees. Unfortunately, the cubicle mentality is deeply embedded and there is not much that can be done to change this way of working.
9. Do you use the best tools money can buy?
Yes. Most of the development tools have been provided by our customer and they are quite good. We do tend to have issues when it comes to using the latest in computer hardware. Fortunately we have just upgraded to decent computing platforms recently.
Software organizations require that computing resources always stay upgraded to the latest technology. This is probably one of the cheapest and best ways to improve the productivity of employees especially developers. Time for activities such as compilation and debugging is significantly reduced by upgrading to the latest hardware.
10. Do you have testers?
Yes. Testers are an integral part of our software development team. They are our line of defence against defects that are missed by developers. We also have some level of automated testing for some of the tasks.
Though most bugs should ideally be found out by developers, many times it so happens that developers work on isolated features that finally get combined into the system. System tests and regressions tests are very important to find out defects that could easily pop up in production. Such testing is the forte of testers who can hit the system from all possible angles.
Of course, testers can only do the black box testing. But its possible that some parts of code are not being hit in such black box testing. Then you could have developers act like testers to find out such defects. Or you could have testers who could also think like developers and perform some white box testing.
Some of the mundane testing that needs to be done after every build could be automated by using an automated test system. This helps to detect bugs early and also frees the testers to work on finding out more ways of breaking the system.
11. Do new candidates write code during their interview?
No. This is something that we need to work on. We do ask the candidates questions to check their existing skills with a programming language and their logical ability. We do tell them to write some code but not much.
It’s quite difficult to know the level of a developer just in an interview. Making a candidate write code is a good way to know how much he/she has been in the trenches. However equally important in a developer is the ability to know how to approach solving a problem. The quality of a good developer is to search and probe ways to solve the problem before interruption another developer. Equally important is to know how passionate a developer is in working with you. It often happens that many developers have no passion for their work. Such developers end up doing work that they are assigned and nothing more. Finding developers who are constantly trying to improve themselves is a difficult task.
12. Do you do hallway usability testing?
No. Because this is one of the projects that is dependent on a domain. So an average user will have to ask 100s of questions before he becomes aware of what the system is about and what all he is supposed to do.
I think this entirely depends on the type of system that is being developed. If you are developing a web site or a online payment system, it is relatively easy to have random people perform the hallway usability testing. But if you are working on systems like a micro-controller for traffic signals or a device driver for USB support, such testing just falls out of scope.
So we scored 7 out of 12 on the Joel Test. However points 2, 3 and 12 don’t quite apply to our team. So the final score could be 7 out of 9 which is pretty decent. We can’t do much about getting quiet working conditions except maybe using noise cancelling headphones. We can surely work on the point of asking new candidates to write code during their interview.
Related Posts:
I’am not sure points 2,3 and 12 don’t apply. For point 2, you should at least be able to build automatically on your own servers. Point 3 is then easy, do dayly builds (after all it is just machine time) if there is any change in source control. You should not waste human time for that. And automating it, ensuring it is done every time something has changed is better. Typically a continuous integration system will try to build endlessly until the build is successfull or until next change in source code repository.
For point 12, I haven’t understood, do your product provide some interractions with humans or not? If yes, usability testing will be of use. Even for an hardware device. Of course if only a small group of person will use the software, usability testing will be done against that group, not the average joe.
For point 9, it seem that good software tooling is dependant of your customer. And that your harware is just on average… So its No.
In the end, you are more about 6 out of 12.
[Reply]
kevin Reply:
July 18th, 2011 at 7:01 pm
Thanks for your comment, Nicolas.
By point 2, I meant that some part of the build process requires human intervention in our case. For example, for signing keys.
Yes, point 3 could be worked on by doing automated daily builds to check nothing is broken. But the point that I was making is that when you have a small team and code doesn’t change too frequently, you can do this every few days instead of every day.
For point 12, yes, our product does provide interaction with humans. But an average person will not be able to work with the product without some sort of training.
Point 9 was kind of a mixed bag. Because the software tools are proprietary and enforced by the customer. We don’t have much choice in picking them. But those that we have are pretty good. And hardware is above average now. There was a time when we were using machines that were more than 5 years old. But those got upgraded now.
[Reply]