Thanks for joining me!
Good company in a journey makes the way seem shorter. — Izaak Walton
Thanks for joining me!
Good company in a journey makes the way seem shorter. — Izaak Walton
Philosophy of unit testing and why its important.
Surprisingly when asked this question most people will answer “our QA team”. If that is how you feel about software quality it is very hard to then be AGILE and delivery software that adds business value and is of high quality. At this stage you are probably think that is harsh but step back a minute quality is own by everyone in the business if one fails the whole chain will break. As a development team we should work towards delivering high code quality so lets have a look at some best practices about testing…
This will explain the fundamentals of Pair Programming and the why and how.
Pair programming is an agile development technique in which two programmers work together at one workstation. One, the driver, writes code while the other, the observer or navigator, reviews each line of code as it is typed in. The two programmers switch roles frequently.
While reviewing, the observer also considers the “strategic” direction of the work, coming up with ideas for improvements and likely future problems to address. This frees the driver to focus all of their attention on the “tactical” aspects of completing the current task, using the observer as a safety net and guide.
This tutorial will explain the fundamentals of Docker and start you with some basic usage.
Docker is open source software to pack, ship and run any application as a lightweight container. Containers are completely hardware and platform independent so you don’t have to worry about whether what you are creating will run everywhere.
In the past virtual machines have been used to accomplish many if these same goals. However, Docker containers are smaller and have far less overhead than VMs. VMs are not portable as different VM runtime environments are very different. Docker containers are extremely portable. Finally, VMs were not built with software developers in mind; they contain no concept of versioning, and logging/monitoring is very difficult. Docker images, on the other hand, are built from layers that can be version controlled. Docker has logging functionality readily available for use.
You might be wondering what could go into a “container”. Well, anything! You can isolate pieces of your system into separate containers. You could potentially have a container for nginx, a container for MongoDB, and one for Redis. Containers are very easy to setup. Major projects like nginx, MongoDB, and Redis all offer free Docker images for you to use; you can install and run any of these containers with just one shell command. This is much easier than using a virtual machine (even with something like Vagrant).
Vanessa Love just posted this intriguing little snippet on Twitter:
In what could be the worst idea ever, i’ve opened up a google doc – add steps from your manual deployment process – https://t.co/Ag56BudiM5
— Vanessa Love (@fly401) July 31, 2017
And I got halfway through sticking some notes into the Google doc, and then thought actually this might make a fun blog post. So here’s how deployment has evolved over the 14 years since I first took over the hallowed mantle of email@example.com.
The whole site was classic ASP – no compilation, no build process, all connection credentials and other settings were managed as application variables in the global.asa file. On a good day, I’d get code running on my workstation, test it in our main target browsers, and deploy it using a visual folder comparison tool. It might have been Beyond Compare; it might have been something else. I honestly can’t remember and the whole thing is lost in the mists of time. But that was basically the process – you’d have the production codebase on one half of your screen and your localhost codebase on the other half, and you’d cherry-pick the bits that needed to be copied across.
Of course, when something went wrong in production, I’d end up reversing the process – edit code directly on live (via UNC share), normally with the phone wedged against my shoulder and a user on the other end; fix the bug, verify the user was happy, and then do a file sync in the other direction to get everything from production back onto localhost. Talk about a tight feedback loop – sometimes I’d do half-a-dozen “deployments” in one phone call. It was a simpler time, dear reader. Rollback plan was to hammer Ctrl-Z until it’s working again; disaster recovery was tape backups of the complete source tree and database every night, and the occasional copy’n’paste backup of wwwroot before doing something ambitious.
Incidentally, I still use Beyond Compare almost daily – I have it configured as my merge tool for fixing Git merge conflicts. It’s fantastic.
Once we hired a second developer (hey Dan!) the Beyond Compare approach didn’t really work so well any more, so we set up a Subversion server. You’d get stuff running on localhost, test it, maybe share an http://www.spotlight.com.dylan-pc/ link (hooray for local wildcard DNS) so other people could see it, and when they were happy, you’d do an svn commit, log into the production web server (yep, the production web server – just the one!) and do an svn update. That would pull down the latest code, update everything in-place. There was still the occasional urgent production bugfix. One of my worst habits was that I’d fix something on production and then forget to svn commit the changes, so the next time someone did a deployment (hey Dan!) they’d inadvertently reintroduce whatever bug had just been fixed and we’d get upset people phoning up asking why it was broken AGAIN.
This is where we start doing things with ASP.NET in a big way. I still dream about OnItemDataBound sometimes… and wake up screaming, covered in sweat. Fun times. The code has all long since been deleted but I fear the memories will haunt me to my grave.
Anyway. By this point we already had the Subversion server, so we had a look around for something that would check out and compile .NET code, and went with FinalBuilder. It had a GUI for authoring build pipelines and processes, some very neat features, and could deploy .NET applications to IIS servers. This was pretty sophisticated for 2006.
After one too many botched FinalBuilder deployments, we decided that a dedicated test environment and a better deployment process might be a good idea. Microsoft had just released a preview of a new deployment tool called MSDeploy, and it was awesome. We set up a ‘staging environment’ – it was a spare Dell PowerEdge server that lived under my desk, and I’d know when somebody accidentally wrote an infinite loop because I’d hear the fans spin up. We’d commit changes to Subversion, FinalBuilder would build and deploy them onto the test server, we’d give everything a bit of a kicking in IE8 and Firefox (no Google Chrome until September 2008, remember!) and then – and this was magic back in 2008 – you’d use msdeploy.exe to replicate the entire test server into production! Compared to the tedious and error-prone checking of IIS settings, application pools and so on, this was brilliant. Plus we’d use msdeploy to replicate the live server onto new developers’ workstations, which was a really fast, easy way to get them a local snapshot of a working live system. For the bits that still ran interpreted code, anyway.
By now we had separate dev, staging and production environments, and msdeploy just wasn’t cutting it any more. We needed something that can actually build different deployments for each environments – connection strings, credentials, and so on. And there’s now support in Visual Studio for doing XML configuration transforms, so you create a different config file for every environment, check those into revision control, and get different builds for each environment. I can’t remember exactly why we abandoned FinalBuilder for TeamCity, but it was definitely a good idea – TeamCity has been the backbone of our build process ever since, and it’s a fantastically powerful piece of kit.
At this point, we’d grown from me, on my own doing webmaster stuff, to a team of about six developers. Even Subversion is starting to creak a bit, especially when you’re trying to merge long-lived branches and getting dozens of merge conflicts, so we start moving stuff across to GitHub. It takes a while – I’m talking months – for the whole team to stop thinking of Git as ‘unnecessarily complicated Subversion’ and really grok the workflow, but we got there in the end.
Our deployment process at this point was to commit to the Git master branch, and wait for TeamCity to build the development version of the package. This would get built and deployed. Once it was tested, you’d use TeamCity to build and deploy the staging version – and if that went OK, you’d build and deploy production. Like very step on this journey, it was better than anything we’d had before, but had some obvious drawbacks. Like the fact we had several hundred separate TeamCity jobs and no consistent way of managing them all.
When we started migrating from TeamCity 6 to TeamCity 7, it became rapidly apparent that our “build everything several times” process… well, it sucked. It was high-maintenance, used loads of storage space and unnecessary CPU cycles, and we needed a better system.
Enter Octopus Deploy, whose killer feature for us was the ability to compile a .NET web application or project into a deployment NuGet package (an “octopack”), and then apply configuration settings during deployment. We could build a single package, and then use Octopus to deploy and configure it to dev, staging and live. This was an absolute game-changer for us. We set up TeamCity to do continuous integration, so that every commit to a master branch would trigger a package build… and before long, our biggest problem was that we had so many packages in TeamCity that the built-in NuGet server started creaking.
This started life as an experimental build of themotleyfool/NuGet.Lucene – which we actually deployed onto a server we called “Klondike” (because klondike > gold rush > get nuggets fast!) – and it worked rather nicely. Incidentally, that NuGet.Lucene library is now the engine behind themotleyfool/Klondike, a full-spec NuGet hosting application – and I believe our internal hostname was actually the inspiration for their project name. That was a lot of fun for the 18 months or so that Klondike existed but we were still running the old NuGet.Lucene codebase on a server called ‘klondike’. It’s OK, we’ve now upgraded it and everything’s lovely.
It was also in 2013 that we started exploring the idea of automatic semantic versioning – I wrote a post in October 2013 explaining how we hacked together an early version of this. Here’s another post from January 2017 explaining how it’s evolved. We’re still working on it. Versioning is hard.
So right now, our build process works something like this.
And what’s on our wishlist?
So there you go – fourteen years worth of continuous deployments. Of course, alongside all this, we’ve moved from unpacking Dell PowerEdge servers and installing Windows 2003 on them to running Chef scripts that spin up virtual machines in AWS and shut them down again when nobody’s using them – but hey, that’s another story.
Like most specialist industries, software is rife with mainstream English words that we’ve taken and misappropriated to mean something completely different. Show business is no different. The software team here at Spotlight sits smack-bang in the intersection between these two specialist fields, and so when we’re talking to our customers and product owners about the systems we build, it’s very important to understand the difference between typecasting and type casting, and exactly what sort of actor model we’re talking about. We therefore present this delightful “double glossary” of everyday terms that you’ll hear here at Spotlight Towers. Because as we all know, there’s only two hard problems in software: cache invalidation, naming things, and off-by-one errors.
Software: A mathematical model of concurrent computation that treats “actors” as the universal primitives of concurrent computation.
Showbiz: A person whose profession is acting on the stage, in films, or on television.
Software: A software agent is a computer program that acts for a user or other program in a relationship of agency
Showbiz: A person who finds jobs for actors, authors, film directors, musicians, models, professional athletes, writers, screenwriters, broadcast journalists, and other people in various entertainment or broadcast businesses.
Software: Any executable code that is passed as an argument to other code, which is expected to call back (execute) the argument at a given time.
Showbiz: A follow-up interview or audition
Software: Explicitly converting a variable from one type to another
Showbiz: Employing actors to play parts in a film, play or other production. Also the act of doing same.
Software: The opposite of a server
Showbiz: An actor, specifically in the context of the actor’s relationship with their agent or manager Internally at Spotlight we have both internal and external clients/customers
Software: A subset of the .NET framework intended to run on mobile and low-powered devices
Showbiz: An actor’s professional CV, as it appears on their agent’s’ website or in various kinds of casting software and directories
Software: Primitive data type representing a floating-point number
Showbiz: A performer who appears in place of another performer, i.e., as in a stunt.
Software: A copy of a system that updates from the original in near to real time, often a database or file storage system
Showbiz: An optical device that helps a performer check they’ve applied their makeup correctly
Software: Used in database mirroring to refer to the primary instance of the database
Showbiz: A performer with lines.
Software: The live infrastructure and code environment
Showbiz: A film, TV or stage show, such as a professional actor might list on their acting CV.
Software: Representational State Transfer – an architectural style used when building hypermedia APIs
Showbiz: What actors do between jobs.
Software: A computer program written in a scripting language
Showbiz: The written dialogue and directions for a play, film or show
Software: Standard pronunciation of SQL, referring to either the database query language. Also commonly refers to Microsoft’s SQL Server database product.
Showbiz: A published, broadcast, or recorded work that continues the story or develops the theme of an earlier one.
Software: The opposite of a client
Showbiz: Someone working as waiting staff in a restaurant. Who is quite possibly an actor moonlighting as a server to pay the bills between acting jobs.
Software: The native MacOS search application
Showbiz: Our company – www.spotlight.com, “The Home of Casting” – and the directories and services we have created since 1927. Not to be confused with the 20-odd different ‘Spotlight’ companies around the world, many of whose customers blindly email firstname.lastname@example.org whenever they have a problem. 🙂
Software: A replica of a production hosting environment used to test new features and deployments.
Showbiz: The method of presenting a play or dramatic performance; also used to refer to the stage structure itself in theatre and live performance.
Monki Gras is RedMonk’s annual UK conference about Software, Craft and Experience. I was very happy that my application for their Diversity Scholarships Programme got accepted and I gladly attended the event on the 26th and 27th of January.
Besides providing a full 2-day ticket they also had in place a Mentorship Programme for the scholars. Tracy Miranda – open source evangelist and veteran of the Eclipse community, Bryan Boreham – director of engineering at Weakework and Rachel Stephens – associate analyst at RedMonk made sure the scholars felt welcome, answered questions and facilitated networking.
While some industry events have a very corporate feel about them these days, Monki Gras was like a breath of fresh air: less pitching and more content!
The subject of 2017 was “Packaging: convenience is the killer app for great developer and user experiences” and among the speakers were Arianna Aondio from Varnish Software Group, Gordon Haff from Red Hat, Alvaro Videla who works as a distributed systems engineer and was previously a Core Developer for RabbitMq and Abby Kearns from Cloud Foundry Foundation.
My favourite talk was “Metaphors we compute by” by Alvaro Videla, who made a very interesting connection between the Linguistics and Philosophic worlds and the Software Industry. Communication is key even when you only ‘speak’ code!
The conference was held in quite an informal venue, The Bike Shed Motorcycle Club in Shoreditch and the techies were kept focused with artisan coffee, tasty food and locally brewed beer.
Looking forward to their event next year!