Recently I had one of those “nightmare” days for a software developer where I was simply trying to complete one small task, and it ended up taking half a day due to a rabbit hole of problems. I had assigned myself a single issue/ticket in JIRA that I thought would be pretty easy to complete.
The ticket described an issue where the application’s “Support Request” feature is not successfully creating tickets in the support system (JIRA). The code related to this functionality is pretty light-weight, so I expect it won’t take much to fix. It’s mostly just executing a REST operation on the JIRA server using some configurable information (such as JIRA credentials).
And so it began…
8:30AM
I opened Visual Studio Code Editor. VSCE required that I login, and kept freezing when it redirected me to authenticate. I did this several times, hoping it was just a glitch. It was not.
9AM
I decided to repair VSCE. This ended up taking almost a full hour. I ended up checking email while I waited for the repair to complete. Unfortunately, I got side-tracked by the emails I was looking at, and wasn’t able to get back to messing around with VSCE (after the repair completed) for closer to 1.5 hours, when the repair only took 1 hour.
I found an article that says to “Run as admin” fixes the problem. Sure enough, it did. That’s not a good long-term solution. For the time being, however, it will have to be enough.
10AM
Now, an hour later, I’ve finally got VSCE running, I’ve launched the application in debug mode so that I can trace what’s going on. OF COURSE, I can’t reproduce the issue. When I submit a support request in the application it works just fine.
So, this leads me to believe it must be a deployment issue. It works fine on my computer but not on the production server. So, it must have something to do with the deployment configuration.
10:30AM
We use TeamCity to automate some simple tasks, such as deploying applications to certain environment, copying databases, building applications, etc.
I open TeamCity to run the build configuration that updates the application in the DEV environment. TeamCity’s license has expired, so I can’t run any of the builds. This would be a big issue for any project I was working on, since most of them have some sort of configuration in TeamCity. So, I must get this fixed.
In a larger company I might have to create an IT-Support request to ask that the IT department renew the license for TeamCity. In that scenario, it might take days to get what I needed. However, in my case, I also happen to play a prominent role in the IT department and am able to address the licensing issue (mostly) myself. Reviewing the issue with my manager, he gives the go-ahead to renew the license to TeamCity.
I go to jetbrains.com (the developers of TeamCity), find the credentials to login, find the license to team city on their website, click to renew the license, fill out the payment information, and wait for the new license to arrive in email. Once the license arrives in email, I copy/paste the license into the administrative section of the TeamCity application, and I’m now able to click the “Run” button to “Deploy to DEV”.
11:00AM
Now that the DEV environment has been updated, I test the functionality for submitting a support request. It fails. This is a good thing, as I am now able to re-produce the issue in a non-production environment.
Unfortunately, the logs are not showing what the underlying issue is. At some point, I probably skipped (or more likely forgot) to add logging into this functionality. So, now, I am trying to debug this issue which can only be reproduced on machines other than my own (where I could easily step through each of the lines of code with a debugger) without any useful logging information. I have to be able to reproduce this issue in my own development environment, or this could take forever.
11:30AM
I copy the configiruation for the DEV environment to my own personal environment. This works, as I am now able to reproduce the issue in my own environment, as well. I can step through the logic that is running when I submit support requests, and I see that the request to JIRA is failing due to invalid credentials.
While I’m here, I add some logging statements so that if this happens again in the future, I might more-easily identify what the problem is.
NOON
Again, if I were in a larger company, I would have to ask the IT-Support team to reset the password for the user in JIRA. However, I am part of IT-Support and have administrative access to JIRA. I login with my administrative credentials to reset the password for the user that the application is trying to submit a support request using.
I’m not entirely sure whether the credentials stored in the application’s configuration got changed unintentionally, or whether the password for the JIRA user account expired, or whether someone else in the IT department changed the password for the JIRA user account not realizing it was being used by this application… I don’t want to reset the password for this account and break something else.
I look through documentation for the IT department and see if I can find whether this user account is being used anywhere other than this one application. I can’t find anything that says it’s used elsewhere, so I reset the password, hoping the documentation is accurate.
1:00PM
I have reset the password and updated the password in all three environments (PROD, DEV and my local environment). I test each of the various environments to make sure the functionality works in all of them. I’ve sent out an email to other developers to notify them that they may need to update their configurations, as well.
CONCLUSION
This ticket should have been easy… It’s only a few lines of code for this functionality. But, I was stopped dead in my tracks at every corner. Had I not had issues running my IDE, renewing the license for TeamCity, finding where credentials are stored/used, this probably would have only taken an hour to fix, at the most!
Really, I hope that someone non-technical reads this and follows it even a little bit. Software development isn’t magic. It has many moving parts. When they all run smoothly, it may seem like magic. But, eventually, the parts don’t run like expected, and then it’s a nightmare.