docker-compose
for too long and have been
meaning to upgrade to docker compose
for quite a while.
Yesterday, it was time, but I got thrown a real curve ball by the Docker
network daemon (if that’s what you call it).
I’ll spare you 4h of troubleshooting, trying to figure out why pretty much
all of my Docker-stuff was working just fine, but my surveillance system did
not.
I found the issue when I tried pinging one of my cameras in 192.168.30.0/24
.
Pinging from my workstation worked fine, but pinging from the server1
did not work.
This is weird, because both are in the 192.168.10.0/24
subnet.
When pinging from the server, I noticed that ping
used a gateway it shouldn’t:
What I finally concluded was that Docker had created a bridge network for my
Graylog deployment with the subnet mask: 192.168.16.1/20
.
This CIDR spans: 192.168.16.0 - 192.168.31.255
which is a problem, because
I have VLANs for 192.168.[10,20,30,40].0/24
.
Since the interface only existed on the server1, all my RTSP debugging on my workstation was pointless…
Luckily for me the “solution” was to do docker compose down
on the Graylog
deployment and then docker compose up
again.
This created a new non-overlapping bridge network.
After investigating further I have found that it is possible to:
I am not sure whether the second option hinders the daemon from creating new networks outside the default range or the configuration just applies to “the first network” which is created.
]]>This is why I am a big proponent of trunk-based development; the way-of-working that emphasises fast integration to a shared branch. If everyone in your organisation strive to make it easy to deliver new versions of your product to your users, you will get an answer to the above question sooner.
I have helped several organisations “work trunk-based” over the past couple of years, but I have always had to reinvent the wheel when it comes to the tooling used for automating the versioning of the product.
Conventional Semver is a small Python CLI which takes a SemVer tag and a Conventional Commit message as input, and returns “the next version”. Both SemVer and CC messages are well described by their respective standards – and the only thing my tool does is glue together existing parsing libraries – but finally I don’t have to rewrite the same scripts for every new assignment!
The tool is FOSS and available on PyPi. You’re more than welcome to use it and contribute!
The README.md on GitHub contains all the info you need, I hope, but here’s the gist.
$> pip install conventional_semver
# If your latest tag is 1.7.2 and
# your latest commit message "feat: add new shiny thing"
$> conventional_semver \
--semver "$(git describe --abbrev=0 --tags)" \
"$(git log -1 --pretty=%B | tr -d '')"
1.8.0
# Or, doing everything manually:
$> conventional_semver --semver 1.7.2 "feat: add new shiny thing"
1.8.0
I have created a dotnet sample project where you can see the CLI in action.
]]>Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
Four concise statements which identify the importance of getting things done and understanding that iterative work and continuous communication is key to building good software. My first opportunity to try these concepts out for myself – in a software development context – arose when I worked a little along side my university studies, with companies where autonomy and flexibility was imperative. Then I graduated and I got my first full time job, at a large engineering company, and my agile bubble was popped.
All of a sudden, agile meant:
and:
especially in combination with:
When us youngsters tried to discuss whether our way of working was an efficient use of time and money, we were told that agile only worked in theory and SAFe was a real world implementation. We looked at a version of the image below and sighed. The image is a perfect example of how to incorporate the right hand side of all statements above.
Fresh out of school, I figured that maybe they were right…
Fast forward to today and I have had the opportunity to work with many engineering companies, ranging from 10 to 100 000 people strong. What I have noticed is that it’s more fun – and companies tend to be better at solving their users’ problems – when they work in ways which relate to the four statements at the beginning of this post.
And I want to emphasise that SAFe is not the only culprit here. I have worked with companies who sprinkle mandatory meetings and processes all over their work weeks, all in the name of “being agile”; often motivated by a need for communication and ensuring everyone knows what’s expected of them. Great intentions, but poorly executed.
I have also noticed that the companies who embrace the statements above tend to not call themselves agile, they just do the work that needs to be done to solve their users’ problems. They live: people over processes.
Has agile done a semantic inversion and come to mean the opposite of what it was originally meant to?
When I talk to companies about DevOps, I often find that they are surprised by my focus on the agile fundamentals and developer enablement. They are mostly interested in which CSPs I am familiar with and if I have experience with their suite of tools. Often, they want me to join their “DevOps Team” and when I ask what this team does I often get a response similar to:
Our DevOps Team is responsible for managing the pipelines which build and test our products.
To which I always counter with two follow up questions:
More often than not the first question is answered with: “They don’t have the competence.” or “We want them to focus on coding.” and the second one with: “That is done by our operations department.”
Instead of tearing down the wall over which developers throw half working code for operation to try and deploy, a new wall has been erected. A wall which separates developers from their builds and tests by obscuring them in YAML and organisation-wide CI gates, imposed by someone; but likely not by the dev teams.
Too many times, “The DevOps Team” has become a part of an organisation which imposes restrictions instead of focusing on developer enablement. Instead of helping the development teams own their deliverable’s life cycle – by taking a sideline role, fully focused on support – the DevOps Team is wedged in between the developers and operation department, effectively moving the developers even further from the end-users.
And just like agile has been devoured by processes, the very thing it strived to eradicate, DevOps has become a checklist; ensuring we use certain tools.
when – at its core – DevOps is just a never ending loop. Its only purpose to provide a way for developers to take complete ownership of their deliverables’ life cycle.
I am not opposed to meetings and tools and am by no means a proponent of anarchy! There are many tools that can help an organisation be more agile, and striking a good balance between asynchronous, digital and in-person meetings is key. In my book, a daily meeting where people share information about changes to their previously communicated plan is great1 and I believe the retrospective is one of the most important meetings a team can have.
When it comes to tools, being able to deliver “working software” while also “responding to change” at any scale becomes impossible without continuous integration and proper version control; both require tools. Which tools to use depends on the organisation and what problem we intend to solve and it’s excellent that there are many good tools out there. Thy dynamic workloads enabled by cloud, the reproducibility offered by infrastructure as code, and the traceability that comes with an issue handling system like Jira; they’re all needed (to some extent), we just need to understand why.
Lastly, in many cases, a platform team is an excellent way to enable developers to take complete ownership of their deliverable’s life cycle. By providing templates and a unified way to develop, build, test, deploy and monitor applications, a platform team can help developers to focus on what they do best: “Listen to the users’ problems and solve them.” But don’t make this team another hurdle over which the developers deliverables must be thrown on its way to the end user.
The key takeaway here is that:
Everything we do in our organisation should serve the purpose of solving our users’ problems, preferably at an increasing speed. If a tool helps us, use it; if a meeting is needed, have it; but when push comes to shove, the developers are the ones who solve the users’ problems and therefore the rest of the organisation must always focus on helping them do their jobs.
I have previously written a post about planning which you can find HERE. It describes the concept of planning proportionally.
My best tips for having a daily/stand-up is to make attendance mandatory, but everyone should not be expected to share something. Don’t go around the table and expect everyone to say: “What they did yesterday, what they plan to do today, and whether they need help.”, instead, have people share whether they need help and whether things are not progressing according to plan. This way, the meeting becomes relevant and short. ↩
In a week, the 2x 30 minute long commute eats up five hours of your time.
In a month, it’s over twenty hours.
In a year1, the 30 minute commute sums up to almost 10 days.2
I don’t find any joy in commuting. On the contrary, I think commuting is a waste of time and a perfect example of unintentional time, which I wrote more about in a previous post.
Most of us trade our time for money (as opposed to piecework pay) and we only get paid when we actively do work for our employer. I.e. we don’t get paid when we’re on lunch, even if we spend lunch time in the on-premise cafeteria and it is unfeasible to leave the premise, have lunch, and be back again to resume work on time. Maybe you don’t think about the relationship between time and money in this way, but it’s there. The relationship becomes even more apparent when you work a job that bills a customer an ongoing hourly rate – e.g. most consultants and carpenters – as opposed to being a teacher or working in a department store with a monthly salary.
For some reason though, it is considered completely normal that: “the compulsory time required to transport oneself to/from one’s place of work in order to be able to carry out one’s work”, should be paid for fully by the employee.
I think it is absolutely crazy that we (as a society) find this to be a reasonable transaction.
What makes me even more baffled is that most companies charge their customers a “call-out-fee” when they leave their “homes” (i.e. their offices). How come that: When I pay a company to come home to me and do something, I need to pay for the time they spend ‘between jobs’, but when I work for a company they only pay me for the exact time I spend producing something for them?
I believe things are the way they are because there is a significant imbalance between the employer and the employee. The worker is in a position of dependency, with respect to their employer, in which the employee stand to lose more than the employer, should their relationship end. The impact on a company loosing one of their 1 000 employees is quite small, but if a person lose their job they will most likely find themselves in a very uncomfortable position. Because of this imbalance, the worker is unwilling to challenge the status quo.
I think remote work makes the equation a lot more balanced.
When an employee is given the option to work from home, (s)he can decide whether to spend the time and money required to go to the place their employer has chose for them. If they like their commute, have an errand that can be carried out adjacently to their work day, or for any other reason would like to leave their home (or workplace of choice), they can do so.
I believe that:
An employee should be reimbursed for all the time they have to dedicate to their employer in order to fulfil their work duty. This naturally includes commuting, but also things like “taking a course” and “attending a conference”. I hear about so many cases where “the course/conference is free”, as if that was compensation enough for me having to give my employer hours or even days of my time without getting paid.
If I have the option to work from home, but choose to commute, I am completely fine with “paying” for the commute myself.
We – society – must dare to challenge the status quo when it comes to remote work. I do not believe that we need laws to mandate that: “All jobs that can be done remotely shall be offered as remote positions.”, because the companies who hate remote will find ways around that. Instead, the employees must start to demand remote work.
If you – as I – believe remote work is the best kind of work, make it clear to current and future employers that you’re not interested in taking an on-premise position. Apply for jobs and make it clear that you’re only interested in the position, if you can do it from wherever you want.
Lastly, don’t be dogmatic. There are cases when gathering a group of people in one place is beneficial. Naturally I believe the employer should reimburse all attendees for the time they lose due to the gathering, but I still believe there’s value for ‘remote workers’ to meet, occasionally.
I intend to address the following topics in upcoming posts:
And maybe one or two more.
The DevOps Research and Assessment (DORA) has a great article about generative organisational culture.
In their article they conclude that:
[…] organizational culture that is high-trust and emphasizes information flow is predictive of software delivery performance and organizational performance in technology.
Not only did they see that “high-trust and information flow” correlated with high performance, the relationship was predictive: where there is trust, performance increases.
DORA also references Google’s “The five keys to a successful Google team”.1 Out of the five dynamics listed in the article, the top three are very much related to the quote above:
- Psychological safety: Can we take risks on this team without feeling insecure or embarrassed?
- Dependability: Can we count on each other to do high quality work on time?
- Structure & clarity: Are goals, roles, and execution plans on our team clear?
Why this clear relationship between trust and performance?
Running a successful company comes with an inherent risk. As a business owner, you try to understand what your customers want and therefore are willing to pay for. You risk developing the wrong thing hoping that you will eventually get things right and be able to make a profit. I wrote about this in the post about planning: the sooner we can give the customer what they want, the sooner we can move their money into our account. The feedback loop where the customer reacts to our attempts to please them is the motivation for reducing the amount of planning done in a software organisation, in favour of increasing the release cadence of the product.
When you (as a business owner) have employees, you must imbue this willingness to take risks in your organisation. This is true for every person and position throughout the organisation but, as we concluded in the planning-post, it is ultimately the developers that are responsible for realising the customers’ feedback and turn vague thoughts and ideas into code that do stuff. If developers are risk-averse, they will not dare be creative nor innovative, and find new ways to tackle problems with the product they are working on. They will be keen to always “do the safe thing”, but the safe thing will likely lead to your product being overrun by your competitors.
But don’t mistake the encouragement of taking risks with an organisation where no one is held accountable. As is stated by the second Google key above: people in an organisation must be able to count on each other to deliver high quality work on time. Accountability and psychological safety are not contradictions, they are both very much necessary in a healthy organisation.
The best way to ensure your employees or teammates are unwilling to take risks and be innovative is to distrust them. You can achieve this simply by telling them outright that you don’t believe in them, but very few leaders do this.
Another excellent way of communicating distrust is by burdening teams with responsibility but not give them the mandate to deliver on them. Even better if the responsibilities are slightly unclear and the team members are not given the opportunity to seek out answers to their questions, “because that’s what the PO does”. If you tell a team that “they must ensure their product is of super high quality and well tested”, but don’t give them the time nor the resources they say they need, you are indirectly telling them that you don’t believe they know what they are doing. The consequence will most likely be that the team take the safest possible approach to the feature, which is likely to yield the least value to the customer, which in turn gives your competition an opportunity to get ahead.
A third – very common – approach to hamper innovation is to maximise the inflexibility in your product and the ways the teams are allowed to work.
Enforce work times and locations.
Demand that everyone use the tools and techniques (preferably by strict templates).
Don’t let the developers participate in discussions concerning:
just have meetings among “non-developers” and then tell the devs what’s expected of them.
This shows the team you know what’s best and they are really just employed to do your bidding, not bring anything of their own to the table.
Everything I have experienced and read about “how to run a successful software development company” points towards the quote that is often attributed to Steve Jobs:
It doesn’t make sense to hire smart people and then tell them what to do; we hire smart people so they can tell us what to do.
– Steve Jobs
This takes us all the way back to the top of this post and the article from DORA. In self-organising teams, working in a generative organisational culture, the developers trust one another to get the job done and are fast to ask for help if they realise that they need it. Everyone involved knows that risks are necessary in order to innovate and win the marketplace, so when something goes wrong, the teams have the support, tools and authority required to handle the situation. The organisation celebrates their willingness to attempt the things they believe will be most beneficial to the company as a whole.
The teams are responsible for the complete life cycle of their product and understand its financial position in the company’s portfolio, meaning they can be engaged in business related discussions, not only technical ones. Management understands that their role is to enable the developers to do their job and “pave the way”. They focus on hiring people who want to make a difference and are willing to take responsibility for their actions, as opposed to only looking at the tech-stack on the candidates’ CVs.
Going back to my original post about developer enablement:
We want to structure our organisation in such a way that each team can take full responsibility for their deliverables. The team should own the entire life cycle: Planning, implementing, building, testing, deploying, monitoring and supporting.
You cannot achieve the above in an organisation where teams are micromanaged and constantly hampered by their leaders. You cannot achieve this without trust.
]]>We want to structure our organisation in such a way that each team can take full responsibility for their deliverables. The team should own the entire life cycle: Planning, implementing, building, testing, deploying, monitoring and supporting.
Recently, I have been thinking a lot about the first part of the life cycle above: planning. In particular, I have contemplated why different companies have very different needs when it comes to planning.
In general, we plan before we do something because there is a cost associated with doing the wrong thing.
It’s not the end of the world if you forget to buy milk1, so most people don’t spend that much time planning their grocery lists, but if your 20-storey-building-project fails you might bankrupt your company or someone might get hurt. Hence, it is understandable that some extra thought is required before you start the project.
The amount of planning we do should be proportional to the cost of failure.
So what about software development? Is it costly to do the wrong thing?
I say: No!
When you do software development right, the cost of failure is low. But I would also say that if you do software development wrong, the cost of failure is very high.
The cost associated with software development is pretty much only related to labour cost.2 Maybe you need a license for your IDE, an office chair, a computer and an ergonomic keyboard; but in relation to a developer’s salary all of these expenses are miniscule. Granted that your company’s main source of revenue comes from selling the software you develop, this implies that the best use of a developer’s time is to ship code to paying customers. All other activities that the developer partake in should aid the developer to achieve this. Furthermore, the highest priority for any non-developer in the organisation should either be related to selling the product or help the developer do their job.
As long as we deliver valuable software, money spent on labour will be money well spent, but the opposite is also true. Money spent on developing something that ultimately has no value to our customers is really costly and a poor use of resources.
This is the million dollar question – literally. Somehow we need to figure out what our customers want. Luckily, there is a very simple solution to our problem: We can ask them! Unluckily, it is often difficult for a customer to describe their wants and needs in a way that is directly translatable to source code. This means there will likely be a bit of trial and error, before we get things right.3
Best case scenario, the developers simply talk to the customer, ask them what they want, implement a first iteration of their interpretation of the request and then show the customer what they’ve done. Together they iterate their way to a solution and the famous feedback loop is born. The quicker they iterate the sooner the feature will be implemented and the sooner the customer’s money ends up in our account.
Enter: Planning sessions attempting to predict the customers’ wants for the upcoming 3 months.
If the short feedback loop, where developers tightly integrate and communicate with the customers, is the desired way of working, why do so many companies force loads of processes and planning sessions down their developers’ throats? I believe many companies over estimate the cost associated with shipping the wrong thing. They believe that getting the feature perfect – the first time – should be their highest priority and that the customer will be mad (or even lost) if we fail to do so.
Unfortunately, many companies are right.
If we ship the wrong thing, receive negative feedback from the customer, and it takes a long time to ship an updated version of our software, the customer will be mad. Faced with this, many companies dig their feet down and it becomes more and more important to ensure every delivery is perfect, which naturally takes more time, which costs more money. With fewer releases and longer in between, the developers have fewer opportunities to touch base with the customer and understand their needs, so we better make sure every release is well planned out and packed to the brim with new (hopefully) valuable functionality.
But I believe that the opposite is also true. If we ship the wrong thing, receive feedback, and have an updated version of the software in the hands of the customer quickly they will be appreciative. A happy customer is a paying customer.
Instead of spending time planning and discussing what we think the customer wants to pay us for, let’s iteratively implement our way towards their needs together.
Let’s summarise the developers’ role in the company:
goto 1
The job for the rest of the organisation is – as we discussed above – to ensure that the above steps go as smoothly as possible.
In order to maximise the amount of time our developers spend building revenue generating software, we need to minimise the amount of time they spend planning and struggling with processes. We must do everything we can to help our devs deliver the software to our customer, as this is the only way to learn whether we are doing the right thing.
Developer enablement must focus on removing hurdles and making it easier for developers to do their job. If you are not enabling your developers you are spending money on something that is not contributing to increasing your revenue.
OK… if you ask my wife, I might have forgotten to buy milk a few times too many and she is getting really annoyed with me; but that’s beside the point. ↩
If you need to make purchases in order to develop your software, this statement can start to fall apart. Maybe you need a HIL-rig mimicking a gearbox in a car, or a huge server for training ML models, costing several developer salaries… I hope you agree that even though cases like these exist, they are not recurring expenses for every feature. ↩
If we have a high cost associated with prototyping, having constantly changing requirements (or at least changing interpretations) will make it expensive to work in tight collaboration with the customer. We really need to make sure we’re not spending too much on something our customer will discard. Good thing source code is free and the only cost for producing it is the time spent hacking away at our keyboard. ↩
How could the developer of the microservice
frontend
configure their application such that it can talk tobackend
, before either microservice has been deployed?
The detailed functionality of the microservices is not relevant for the example, but it is assumed that frontend
wants
to make HTTP requests towards backend
.
By deploying their application as seen below, frontend
can be configured to communicate with
http://backend-service
1, without knowing the Cluster IP
of the backend
microservice.
Your DNS server of choice, in Kubernetes, will handle the translation.
The frontend-service
exposes a NodePort
.
This is useful if you run minikube, k3s or something
similar, and you want to test your deployment locally.
In a production scenario, you should front your service
with an
Ingress instead of connecting straight to the
service
.
# File: backend.yaml
# Documentation for deployments: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
# kubectl explain --api-version=apps/v1 Deployment
apiVersion: apps/v1
kind: Deployment
# The metadata applies to the deployment itself.
# We give it a name and put the deployment in a namespace.
# http://kubernetes.io/docs/user-guide/labels
metadata:
name: backend-deployment
namespace: default
# The specification of the desired behaviour of the Deployment.
spec:
# Number of desired pods
replicas: 3
# Existing ReplicaSets whose pods are selected by this will be the ones affected by this deployment.
# The label selector is the core grouping primitive in Kubernetes.
# It must match the pod template's labels below.
selector:
matchLabels:
app: backend
# Template describes the pods that will be created.
template:
# Standard object's metadata.
# The label app: frontend, is what groups the pod to the Deployment.
metadata:
labels:
app: backend
# Specification of the desired behaviour of the Pod.
spec:
# List of containers belonging to the pod along with settings for them.
# We use IfNotPresent to avoid pulling images from a registry if they already exist locally.
containers:
- name: backend
image: backend:latest
imagePullPolicy: IfNotPresent
ports:
- name: "backend-http"
containerPort: 80
---
# Documentation: https://kubernetes.io/docs/concepts/services-networking/service/
# kubectl explain --api-version=v1 Service
apiVersion: v1
kind: Service
# Metadata concerning the Service
metadata:
name: backend-service
namespace: default
# The specification for the Service
spec:
# Type determines how the Service is exposed.
# Defaults to ClusterIP.
type: ClusterIP
# Route service traffic to pods with label keys and values matching this selector.
# In the deployment-yaml, the pods are given the same label as the one here.
selector:
app: backend
# There are three ports specified here:
# port, is the port that will be exposed by this service
# targetPort, is the name (or number) of the port to access on the pods targeted by the service
ports:
- port: 80
targetPort: backend-http
I have omitted the comments that are simply duplicates of the backend
-YAML-comments.
# File: frontend.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: frontend-deployment
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: frontend
template:
metadata:
labels:
app: frontend
spec:
containers:
- name: frontend
image: frontend:latest
imagePullPolicy: IfNotPresent
ports:
- name: "frontend-http"
containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: frontend-service
namespace: default
spec:
# Valid options are: ExternalName, ClusterIP, NodePort, and LoadBalancer.
# https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types
# We use NodePort here since we want to expose the frontend outside of the cluster.
# Reaching the service from a browser or using curl is a bit tricky though.
# You connect using the <Node-IP>:<NodePort>.
# Retrieve the NodeIP by running: kubectl get nodes -o json | jq -r '.items[0].metadata.annotations."alpha.kubernetes.io/provided-node-ip"'
# Retrieve the NodePort by running: kubectl get svc frontend-service -o json | jq '.spec.ports[0].nodePort'
type: NodePort
selector:
app: frontend
ports:
- port: 8080
targetPort: frontend-http
It is assumed that communication between the pods is made using Port 80. ↩
I think a lot about time and in this post I want to introduce you to a concept I picked up when I started to study at university: Unintentional Time.
But before we talk about unintentional time we need to talk about its sibling: dead time.
If you work in IT you come across “dead time” every day.
It is the 15-minute-gap between one meeting that should’ve ended 10:00 and the 10:30 one.
It is the time between git push
and you getting a result back from your pipeline.
It is the time you wait for your code to compile.
Periods of dead time are usually short and you cannot really do anything meaningful with that time anyway. We accept that it exists and move on.
If dead time is: useless time that cannot really be used for anything, we need another word for: the time you planned to use for implementing a new feature but instead spent discussing a problem with someone who popped over to your desk only to realize that the problem is multifaceted, complex and require that you set aside a big chunk of time to look into it more thoroughly.
It’s the time when you search for datetime to string with timezone
for the 43rd time and find yourself (25 minutes
later) reading about why New Zeeland’s clock is currently UTC+131.
Or the 45 minute lunch that turned into 70 minutes, because you had such a good time.
Or when you grab a colleague and say: “Let’s go into this room and use the whiteboard to brainstorm a bit.”
This time is not useless (which is good), but it’s unintentional (which is really bad).
When I went to music school, I was taught the importance of practising with intention. It was not enough to “just play or sing”, I had to plan what I was going to do and then execute with focus and intentionality. Failing to do so would not be a total waste of time, but it was evident that the intentional practice sessions yielded far better results than their unintentional counterparts.
It’s the exact same thing with the 2 000 hours per year you spend doing your job. In order to get the most out of those hours, you need to plan what to do with them and then perform your plan with deliberation.
Now, someone might say: “Who cares?”, and I guess that’s fine.
To each their own.
But few things frustrates me like wasting time. Because wasted work time turns into wasted free time; and I value my free time very, very, highly.
Because when push comes to shove, I don’t work 2 000 hours per year because I love what I do. I work because I need money and because I want to occupy myself with something interesting. I think trading my skills for cash satisfy those criteria and I want the time traded to be as meaningful to me – and as valuable to the company paying me – as possible.
Because I value my time so highly, I try my best to always spend my time with intention. When I decide to do something, I do that thing. Then I do the next thing. And the next thing.
And just to be clear. I love deep diving into niche problems and 70 minute lunches, but I want to plan for them. I want them to be intentional.
A couple of years ago I heard CGP Grey talk about unintentional time in Cortex: Productivity 101. It’s worth a listen and the link takes you straight to their time-tracking discussion.
Furthermore, one reason I really like remote work is because of the opportunities it brings with respect to spending my time with intention. I have started a post about this too and hopefully it will be published soon.
It’s because of DST. ↩
DevOps is a concoction of ideas which aims to help developers deliver higher quality software at a quicker pace.
DevOps is hard to define and that is a good thing. It leaves room for flexibility and for each company to figure out what they need to do to enable autonomous and high performing development teams.
However, over the past couple of years the industry has found that certain ways of working, using certain tools, building applications in certain ways, has proven more successful than others. Like breaking your application into smaller parts (microservices), which makes it easier to modify and deploy only the part you intend to. Or ensuring that you have automated tests; not only a couple of unit tests, but enough test coverage that you feel comfortable making changes to your code base without manually verifying that everything is still working.
A lot can be said about tools, processes, architecture, leadership and many other things. I will likely do so in future posts (and I have linked to a couple of good resources at the bottom of this one) but for now, I want to highlight my favorite principal from the DevOps concoction and the one I have come to double down on over the past couple of years.
DevOps is speed, which is achievable when your operations department enables the software development teams to take complete ownership of their services.
We want to structure our organisation in such a way that each team can take full responsibility for their deliverables. The team should own the entire life cycle: Planning, implementing, building, testing, deploying, monitoring and supporting.
Why?
Because the alternative is that different parts of the organisation are responsible for different steps of the life cycle. This requires handovers. Handovers take time. Handovers implies communicating complex ideas, which is difficult. Handovers are very seldom the reason one becomes a software developer.
Handovers cost money.
But taking responsibility is not something the team can do on their own. We must give them the opportunity to take responsibility and provide them with the tools required to do their best work. Furthermore, it’s imperative that we remove as many hurdles as possible for the team.
We must enable developer ownership.
The most common hurdle I see in the industry, making it difficult to enable development teams to own their deliverables, is working in an organisation with silos.
Here are a couple of silo hurdles:
Every hurdle takes time, depletes the involved people of energy, and costs money.
When we work in silos, it is very common that each silo attempt to maximise their own goals instead of focusing on what yields the maximum utility for the end user (and therefore the company). In a silos organisation, the Operations department gets praised if all systems are operational and there is no down-time. In that same organisation, the QA department gets praised when there are no bugs in the code.
Do you know what causes down-time and bugs? Change. Change causes both.
But a company that does not change, evolve and improve will soon be left in the dust.
So instead of maximising for “not making any changes what so ever”, we should maximise for “making it super easy to quickly iterate and learn”. (In most cases, under most circumstances) it is OK to have a bug in your application, if a new release can be deployed within 5 minutes of you committing the change to source control.
We should build an organisation with a common goal (satisfy the customers’ current and future needs) and do everything we can to enable our developers to satisfy those needs.
How do we enable our developers to satisfy our customers’ needs?
We must ensure our developers get the right information. If the developers do not understand how their application is being used and what functionality their users like, dislike or lack, they will not be able to build a product that satisfy their users’ needs.
Our developers should be encouraged to interact with end users and learn from them. Additionally, our developers should definitely build their applications in such a way that they can derive meaningful metrics based on how their users interact with the product.
Best case scenario we give our developers access to some Cloud Service Provider where each team can provision resources as they see fit. This has the added benefit of fostering developers who understand the financial aspects of a company, as well as the technical.
If the team understand the cost of their infrastructure as well as the income their application generate, they can make informed decisions about new features, performance improvements, and cost reducing measures. Few organisations are willing to go this far, but the transparency is likely something many developers would welcome.
In a sufficiently large organisation, there is likely a need for an operations department (or a platform team) that ensures each development team can acquire what they need as soon as they need it, in a self-service manner. The Platform Team can help unify the tech stack of the organisation to a feasible spread and we can also get some economy of scale benefits by reusing ideas and concepts across several teams. Of the top of my head I come to think about in-house maintained Terraform modules with reasonable defaults.
If our organisation consists of just a handful of people, we might not be able to motivate a separate team to manage our platform. In such a case it becomes crucial to build your teams with people who take responsibility to acquire the skills and tools they need to handle their product’s life cycle.
Ping me on Twitter @ErikThorsell
if you have any ideas for topics in future posts
or you would like to discuss this one.
If you want to learn more about which tools, architectural principles and ways of working the industry has concluded to be successful over the past decade, take a look at the resources linked below. They are sorted according to how short they are.
We want to make it easy for everyone to get access to a repository where we write DevOps related documentation and we figured the best thing to do was to mirror our GitHub repository to Azure DevOps. That way we know that all developers have access to the same information, regardless of where in their transformation journey they are.
name: "Mirror repository to Azure DevOps"
on:
push:
branches:
- main
jobs:
mirror:
name: Mirror repository to Azure DevOps
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Setup SSH Config
run: |
mkdir -p ~/.ssh/
echo "${{ secrets.PRIVATE_RSA }}" > ~/.ssh/ado_rsa
cat << EOF >> ~/.ssh/config
Host ssh.dev.azure.com
IdentityFile ~/.ssh/ado_rsa
EOF
chmod 700 ~/.ssh
chmod 600 ~/.ssh/ado_rsa
- name: Push to ADO
run: |
git remote add ado git@ssh.dev.azure.com:v3/organization/project/repo
git push --force -u ado --all
Some things worth commenting on.
In order to avoid confusion, we opted to only mirror the main
branch of the GitHub repository.
on:
push:
branches:
- main
In order to be able to push a git repository, it is not enough to do a shallow clone (which is the default for the
action).
Setting fetch-depth: 0
clones the entire repository.
- name: Checkout repository
uses: actions/checkout@v3
with:
fetch-depth: 0
I could not get git over HTTPS to work.
I started of with HTTPS because I thought it would be easier.
Just do git push https://userame:password@url
, right?
Nope…
The issue is likely the format of Azure DevOps remote URL which has the format: https://organization@dev.azure.com/organization/project/_git/repo
.
That extra @
really throws git off – I think.
Instead, I set up a public SSH key – in Azure DevOps – for the service account we’re using to push the code and wrote some bash to get git to use the correct key. The private part of the SSH key is stored as a secret in GitHub.
- name: Setup SSH Config
run: |
mkdir -p ~/.ssh/
echo "${{ secrets.PRIVATE_RSA }}" > ~/.ssh/ado_rsa
cat << EOF >> ~/.ssh/config
Host ssh.dev.azure.com
IdentityFile ~/.ssh/ado_rsa
EOF
chmod 700 ~/.ssh
chmod 600 ~/.ssh/ado_rsa
Note that a file created using echo "something" > file
will have permissions 644
, which is too permissive for an
SSH Key.
Hence, we must chmod
the directory and file we just created.