A few days ago in twitter, Justin Kohnen said “I really don’t like having to touch the production database. It goes against my core principles.” Almost immediately I replied with “my last W2 position…1st day. me: ‘where’s the dev/test database?’ boss: ‘yea, we don’t have one of those. use prod.’”
A couple of tweets later and I was inspired to write about that particular gig and how I was able to bring some order to an extremely chaotic development environment. When I left it was still chaotic, but I can definitely say I did my part to make future developers’ lives easier.
Note: Once again, I am NOT using real names. 
Getting Started
In early 2005, I decided to take what I hoped would be a long-term break from consulting. The company I joined isn’t much more than 5 minutes from home, the pay was really competitive and the work itself looked promising. I was actually hired to move them into the .NET world since, in 2005, they were still actively developing in Visual Basic 6. After my initial orientation and training, I was looking forward to diving in and contributing to the development effort.
The conversation I related in twitter did happen. Shortly after my training was complete, I went to my bosses office (Jon) to ask about getting started. After some hand-waving and scribbles on the whiteboard to show me how extremely complex the application was (typical of most meetings with this guy), he told me to get with one of the other devs (Mary) for instructions on grabbing the source code. Cool.
I gave Mary a call and said something to the effect of, “Jon said you could help me grab the latest source so I can start digging in.” Silly me. I was expecting something simple like “point your VSS here and get latest”, but noooo….that was not to be. 
Instead, I was told to connect to the “enterprise” server (which was actually a ratty-ass old desktop that was on its last legs, but that’s another story), copy all the sub-folders from “enterprise” to my workstation and open the project I needed. Keep in mind, this was a VB6 application that consisted of 50+ projects (mostly COM objects ala Windows DNA). Anyway, by this time, my head was about to explode when I said, “what? you guys aren’t using any source code control?”
Here’s where it gets bad…her response? “Source code control? What’s that?”
*sigh*
I asked her what happened when two developers needed to be in the same *project* at the same time. Notice I didn’t say “file”, I said “project”. She replied with, “oh, we don’t do that.”
I have *always* used some kind of source code control, so I was completely blown away by this. As crazy as it sounds, I actually had a tough time sleeping for a couple days after that.
Ouch. Ok. What’s next?
Not too long after that experience, I was given my first real assignment — figure out why a report wasn’t working correctly. I don’t remember the specifics, but at the time, I was chomping at the bit to get some real work done, so I said “no problem!”
I “got latest” — yea, I copied code from the so-called server to my dev machine and started digging into this particular issue. In order to gain better understanding of this issue specifically and the overall system in general, I checked to see where the report was getting its data. I figured once I knew what stored proc the report was using, I’d be able to dive into the database, run some queries and figure out why the report wasn’t working.
Ok. Easy enough, right?
Boss: How’s it going with that report?
Me: Fine. Uhh, I want to run some test queries. Where is the test database, or better yet, the development DB?
Boss: Yea. We don’t have those. You’ll have to use the production database.
Me:
Me:
Me:
Me: Ummmm, ok. I’ll let you know how it goes.
So, within my first two weeks, I discovered they didn’t use SCC and they had no real development environment. Not cool.
Note: At this point, I’m sure there are some readers saying, “dumbass…didn’t you ask questions during the interview?” Yea, I did, but honestly…in 2005, who the hell wouldn’t be using source code control or understand the importance of having a development / test database? Geesh, give me a break.
Making things better
I pushed on and fixed the report issue, making sure to do a diff before “checking my code in” (lol, that sounds so stupid when all I was doing was copying my files back to the server). At my status meeting that day, I asked why they didn’t have dev or test databases to work on? In fact, I went so far as to ask about the lack of SCC too. Oh, did I mention they had no issue tracking system in place either?
While I can understand his answer, “we’re all too busy”, I had a hard time accepting it. I mean, c’mon, I know everyone was busy, but how tough it is to setup SCC (even it it was just VSS)? How tough is it to grab a SQL backup and restore it somewhere else?
As it turns out, the SQL backup was an issue since the production database weighed in at around 80GB, but I knew I could come up with something. From that point on until the day I left, I made it my mission to improve things as best I could.
Source Code Control
It actually took a lot of convincing to get my boss and the other devs to see the value in source code control, but I kept pushing (gently, of course). Finally, my boss relented and told me to provide a recommendation. I knew I didn’t want to use VSS, and at that point, I wasn’t familiar enough with CVS or SVN, so I picked SourceGear Vault. I liked Vault because it uses SQL Server as its back-end instead of being file-based like VSS.
The easiest part of the whole process was importing all the existing code — that took about 20 minutes. The hardest part was convincing the team how important source code control was and explaining all the benefits. My boss was an easy sell, but a couple of the other team members took more convincing. I provided everyone with ample notice of when the file share would be closed. Over the course of a few weeks, I provided everyone with a written tutorial for using Vault as well as providing them with in-person training. It was a huge mindset change for them, but eventually everyone bought into it and started using it. Since everyone was so new to the concept of scc, I wrote a small app that emailed everyone a list of their checked out files each morning.
Issue Tracking
Along with Vault, I also convinced my boss to invest in SourceGear Dragnet for issue tracking. The only issue tracking they had in place when I got there was the IT Help Desk web site and notes my boss took during meetings. Neither was acceptable in my opinion because a) the developers almost never looked at the help desk web site and b) notes always got lost. Dragnet allowed us (the devs) to have full control over our issue list. I wrote a small app I scheduled to run every morning which emailed all the devs their open issues. This really helped!
A Dev/Test Environment
Getting everyone to use source code control was a huge victory, but it wasn’t enough. I had to get them away from their old practices of using the production database to test against. One of the first things I did was grab a SQL backup (yea, all 80gigs) and began trimming away all the fat I could. Since it was also used for testing, there was a “metric buttload” of temp tables I was able to drop. I also trimmed some of the larger tables down to a few hundred rows instead of millions. In the end, I was able to get my copy down to a reasonable 20GB. I also scripted the entire database and checked the scripts into Vault.
Once I had the database trimmed down, I had to figure out where to host it so the other devs could use it. My first thought was to use one of the existing servers, but decided to go the virtual route instead. I ended up building several virtual machines (using Microsoft Virtual PC / Virtual Server) including:
- build
- webdev
- webtest
- sqldev
- sqltest
I also built a couple VMs for client testing.
By the time I left, I had taken them from no development environment to a full-blown virtual environment that included development, testing and staging servers. I had made it possible to fit a copy of the database on a laptop, brought in a great issue tracking system and most importantly of all, gotten them to use source code control.
You know what’s crazy about the whole thing? While I was brought in to help move them to .NET, when I left they were still actively developing in VB6. I personally worked on several projects while I was there that used .NET, but my boss never let me work on moving the main application to .NET. I heard they finally did move to .NET, but it was at least 6-8 months after I left before that happened — and they ended up outsourcing/offshoring most of that work. :-\
I know I’ve heard that story before, but it’s great every time. The great moral of the story is that rather than throwing your hands in the air and saying, “this place is screwed”, you actually made a bunch of little steps and ended up buiding a nice dev environment for them. Hopefully they’ve found someone else to continue pushing them forward. I hope some young developer reads this, learns something, is motived from it, and follows in your footsteps.
Dan, let’s not go too far – we don’t need a bunch of little Michael Eatons running around!
Mike, one question I have for you – after you left, are they still using what you set up? In other words, were they converts and finally came around to see the benefits?
@Joel – nice. :-p
I know they are definitely still using Vault and Dragnet and a few months after I left, the IT guys started moving my Virtual Server VMs to VMWare. So, yes, I think I can say the overall environment I put into place is still being used.
They are still getting their morning emails letting them know about assigned tasks and checked-out files too.
I had lunch with a couple of the new (hired in after I left) devs and they both said they can’t imagine what it must have been like for me. They both thanked me for setting up Vault.
To me, that’s the true measure of success – that they didn’t just say, “Oh good,. He’s gone. Open that file share back up!” They looked at what you provided and saw the value even beyond you maintaining it. Nice work.
By the way, what made you jump back into the independant world?
@Joel – yea, I’m pretty happy that I was able to have that kind of impact on them.
Short story of why I left: a great opportunity came up.
Long story has to be told over drinks.
Been there, done that too, not much fun. Most SCM & Issue Trackers already have notficiation built in (at least they should). It’s amazing how an automated email makes people see things differently.
@Tim – it’s true that both Vault and Dragnet have methods to notify you when things happen (like ticket assignment, file check-in, etc.), but what I created were summaries that were sent in the morning as reminders.
I was just digging through some old source code and I find those apps. I might clean them up a bit and post them.
Great story.
The first company I worked for after college (in 2005) was just setting up SQL Server 2000 as their first ever database. Forget about “source control” or a “dev database”. I took the option of moseying along to another company rather than cleaning up that clusterf*ck. I commend you for having the will to stick it out and make things better.
Looking back, though, the most interesting software I’ve developed was at that company.
Actually, Mike, we don’t use much of anything that you put together. We have moved on. It seems that you need to do the same. Also, you could try to be a bit more honest in your posts.
well done…