Analyzing the State of Software Development in late 2023: Where We Stand and Where We're Going

In this issue, we are taking a deep dive into the state of software development as it stands in late 2023. Specifically, we will examine:

1. The evolving landscape of cloud platforms, rise of Hetzner and decline of Heroku

2. Current trends in frontend frameworks and languages

3. The state of automated testing and its importance in developer productivity & retaining developers

4. Emphasis on security and incorporating it into the developer workflow

5. The evolution of the day-to-day developer experience in 2023

6. The pathways to becoming a developer in 2023

I gathered insights from several industry guiding surveys such as Stack Overflow’s Annual Developer Survey, GitLab’s Annual DevSecOps Survey, and GitHub’s Innovation Graph, as well as from discussions with engineers and engineering managers in a variety of industries to confirm our findings – big thanks to everyone who helped. We've got a lot to cover, so let's dive in.

☁️ The Cloud

This year there have been some interesting developments with the cloud. AWS is still the leader amongst the big 3 cloud players with Google losing usage and Microsoft slightly gaining. Hetzner has reinvented itself, a move that has not gone unnoticed by developers. Heroku is declining sharply after the Salesforce acquisition and the removal of their free tier. What’s especially interesting is that an increasing number of organizations are making the move back to on-premises data centers from the cloud.

🥇🥈🥉The Big 3 Cloud Players

According to StackOverflow’s 2023 Developer survey, out of professional developers utilizing the big 3 cloud providers (Amazon, Microsoft, Google) there has been little shift in usage. AWS has seen a slight increase by 2.5%, and Google has seen a slight decrease by 2.8%. Azure stayed relatively flat with a 0.2% increase [1].

AWS remains the leading cloud provider used by professional developers, but not without stiff competition. Azure's focus on enterprise solutions and both Azure & GCP's heavy investment in AI services make them strong contenders, closing in on AWS's lead. Despite AWS's expansive service offerings and strong infrastructure, the other giants are not far behind. In fact, the competition has sparked innovation across the board, leading to improved services and tools for us developers. Among my favorite innovations are significant advancements in developer-centric security tools, like Amazon’s CodeGuru Security which launched this June, Microsoft bolstering GitHub Advanced Security, and Google’s OSV Scanner that launched in late 2022 (more on developer security tools later).

As we look ahead to 2024, we anticipate that the competition among cloud providers will only intensify. This fierce rivalry will hopefully continue to fuel innovation, resulting in more frameworks, tools, and infrastructure to help make us more productive.

🧗‍♂️The Rise of Hetzner

One of the standout trends in our analysis is the rise of the cloud platform Hetzner. While Hetzner’s adoption by professional developers was not included in Stack Overflow’s 2022 developer report, this year it is reported to be used by 4.44% of professional developers [1,2]. Even more telling is that Hetzner is the most admired cloud provider, with 70.09% of its users expressing a desire to continue using it—compared to AWS's 62.24% [1]. Personally, I jumped on the Hetzner bandwagon this year and couldn't be happier. The UI is simple to navigate, yet comprehensive, and paired with an open source PAAS like CapRover, deployments seamlessly integrate with your development workflow.

📉 Heroku on The Decline

Another significant shift in the cloud landscape is the steep decline of Heroku. Although Salesforce's acquisition of Heroku may be deemed successful by some from a business perspective, the platform has seen a sharp drop in usage and developer satisfaction.

The elimination of Heroku’s free tier last year might have been the final nail in the coffin, as only 21.97% of Heroku users plan on continuing to use the platform [1]. We anticipate this downward trend will continue unless substantial changes are made to regain user trust.

🏢 The Return to On-Prem and Traditional Architectures

We're also seeing a resurgence of companies turning away from cloud services in favor of more traditional on-premises or virtual machine setups. Several key events have catalyzed this shift. For instance, Amazon revealed that they slashed Amazon Prime Video's video monitoring cloud costs by 90% by transitioning from serverless functions to virtual machines. Additionally, David Heinemeier Hansson's (Creator of Ruby on Rails) company, 37 Signals, announced a complete migration from cloud services to on-premises data centers. They even released Kamal (formerly known as MRSK) to streamline the management of on-prem servers. This trend suggests a thoughtful reconsideration of the 'cloud-first' mantra that has dominated the industry.

🖥️ Frontend

🏳️ The Javascript Framework battle

The competition among JavaScript frameworks remains as fierce as ever, with no clear indication of any of them waiving the white flag soon, especially in their collective bid to dethrone React.

A couple of pivotal shifts to spotlight:

• Angular experienced a dip in popularity between 2021 & 2022 but seems to be regaining traction in 2023.

• Next.js is hot on the heels of Vue.js. Given its inherent performance advantages over conventional React and its foundation on React, we expect Next.js to continue gaining in popularity. It offers a familiar environment, making it a relatively lightweight transition for React developers compared to other FE Frameworks.

🧐 TypeScript's Moment: A Temporary Trend or Here to Stay?

As 2023 kicked off, notable figures in the developer world, like Kent C. Dodds, warmed up to TypeScript. Some accepted its benefits while others embraced it wholeheartedly.

Yet, the tide seemed to turn recently when David Heinemeier Hansson, the creator of Ruby on Rails, opted to remove TypeScript from his open-source project, Turbo 8.

While following the Twitter (X) battles between developers is certainly entertaining. I believe this is more noise rather than a signal of the decline of TypeScript.

However, the proposal to add type annotations directly into JavaScript is certainly not noise. If you can add type annotations natively to JavaScript, it would make TypeScript nearly obsolete. I plan to be closely following this proposal as it works its way through the approval process.

🧪 The State of Testing: Engineers Take the Helm

The trend of engineering teams owning the testing process, largely without the aid of a dedicated QA team, has not only taken hold but truly matured. In this section, we'll delve into how engineers have adapted to this paradigm shift in their roles.

🤖 Automated Testing Availability

Recent Stack Overflow surveys have asked developers about the availability of automated testing in their organizations. Results show a strong availability of testing automation, but only a marginal 3% increase in availability year-over-year. The data prompts intriguing areas of interest for future investigations:

• What does 'availability' entail across different organizations?

• In organizations with automated testing, where in the testing pyramid has the most automation?

• What are the adoption rates and usage patterns for automated testing tools in organizations with the tools available?

• For organizations without automated testing, is this a planned initiative or simply not a priority?

⚠️ Is Testing Still A Bottleneck in the Dev Pipeline?

GitLab's 2023 survey posed an intriguing question: 'Where does your team or organization encounter the most delays in the development process?' The survey found that 28% of respondents cited 'Testing,' and another 26% cited 'Test Data Management,' as the areas most likely to cause delays [3]. These statistics underscore the critical need for efficient testing strategies and perhaps warrant a deeper dive into best practices to mitigate these bottlenecks in future development cycles.

🧠 AI and Software Testing

In the past year we have witnessed the emergence of various AI-based developer tools. Many developers have started using generic tools such as ChatGPT, Bard, and GitHub Co-Pilot. In addition, there's been a surge of specialized AI tools for testing, including Microsoft IntelliTest, Codeo, and DiffBlue. These tools integrate seamlessly into your IDE and can generate tests for the specific code you highlight (imagine highlight-and-generate-tests instead of highlight-copy-paste).

While these AI tools offer a convenient way to quickly add unit tests to your code, they aren’t a comprehensive solution for improving overall test coverage with quality, maintainable tests. That's why I created my own AI software testing platform SoftwareTesting.ai, a GitHub app that analyzes your test coverage when tests are ran in GitHub Actions. It identifies gaps in your existing test suite and recommends test cases to fill them. If this interests you, SoftwareTesting.ai is currently free for both public and private GitHub repositories for the next month as we gather user feedback. Here is the link if you'd like to give it a try.

👨‍💻 Testing and Developer Retention

An intriguing finding from a recent GitLab survey illuminates the relationship between software testing and both developer retention as well as deployment rates. According to the survey, teams that have implemented test automation are 1.2 times less likely to face challenges in hiring and retaining developers [3]. Additionally, these teams are 1.5 times more likely to deploy to production multiple times per day [3]. In essence, test automation not only streamlines the deployment process but also enhances developer morale by enabling frequent, smaller releases.

🔒️ Security in the Development Cycle

While many organizations loudly proclaim their commitment to security, it often takes a backseat in actual practice with 41% of developers saying it is difficult to prioritize remediating security vulnerabilities [3]. The rise in notable security incidents serves as a stark reminder. From DDoS attacks on tech giants like Amazon, Google, and Cloudflare to unexpected targets like cosmetics manufacturer Estée Lauder, no company is immune. Even billboards in Israel have been hacked, illustrating that the threat landscape is far-reaching and indiscriminate.

Not only is the performance of security practices at these companies far from the standard users expect, developers are generally frustrated with the security processes, procedures, and tools they are using with 43% of developers saying their biggest frustration with security testing is that it happens too late in the development lifecycle and causes delays [3].

📈 How to Improve an Organization's Security Posture While Increasing Developer Satisfaction?

This was a key question I grappled with during my tenure as VP of Software Development at a notable telehealth startup. Conveniently, GitHub launched their Advanced Security package for GitHub Enterprise customers around the same time we began tackling this issue. The outcome was a resounding success, we were able to quickly fix many vulnerabilities and tightly weave security into our development process.

If you are interested in learning more about this topic, you can check out my session at GitHub Universe focusing on empowering developers to better own security.

You can also review how we utilized GitHub Advanced Security to prepare for our third party security audit. I'll also be releasing a more in-depth guide diving into the specifics of launching and maintaining a successful developer-centric security initiative soon, so keep an eye out.

🏖️ Life as a Developer in 2023

With the impact of Covid ushering many developers into remote work, the landscape has evolved to embrace more flexible models. Currently, 42.2% of developers are operating in a hybrid work environment [1]. It's clear that some form of remote work is here to stay, as evidenced by the mere 16.4% of developers working entirely in-person [1].

With the rise of remote and hybrid work models, there's been a notable shift toward more asynchronous forms of communication and team collaboration. Microsoft Teams and Slack lead the way in usage among professional developers [1]. This is unsurprising given their features, which not only combine chat and video conferencing but also offer extensive integrations with developer tools. It raises an intriguing question of how much these developer-centric integrations impact an organization's decision to adopt either Microsoft Teams or Slack specifically for their engineering teams.

Another essential component for enabling asynchronous work is project management software. Jira and Trello continue to dominate this space [1]. Atlassian, the company behind Jira, acquired Trello in 2017 and recently added Loom to their portfolio. Given Atlassian's dominant position in the industry, it will be interesting to see how they continue to innovate to enhance working environments for developers.

🧩 The Fragmentation of Knowledge in Remote Teams

Remote work has forced teams adapt to having less face-to-face time than before, and as a result, knowledge silos have formed. Unlike in the office, you can't simply walk up to someone's desk for a quick question while working remotely. This issue appears to be worsening. In 2022, 20.88% of developers spent less than 15 minutes per day searching for answers or solutions [2]. Now, in 2023 that number had shrunk to just 9.04%, with more developers now spending between 30 and 120 minutes per day searching for answers compared to last year [1].

🗺️ Pathway to Becoming a Developer in 2023

Even with the current economic conditions we are experiencing in late 2023 and the wave of big tech layoffs we have seen earlier this year, the demand for developers is still strong. It is interesting to see how aspiring developers are learning to code and get into the field of software development.

Nearly 80% of professional software developers hold a degree, while just over half of aspiring developers are pursing one as their way to get into the field [1]. This divergence could be attributed to the increasing popularity of online courses and 'coding boot camps.' It will be interesting to see the trend over time, and better understand how aspiring developers—who opt for learning routes outside of traditional education—fare in becoming professional developers.

Thanks for diving into this Bug Driven Development post with us! We're mixing it up a bit by switching to two deep-dive articles a month instead of our usual weekly quick takes. I’d love to hear what you think about the State of Software Development in 2023 and how you feel about our new format.

Cheers,

Justin

Follow Justin on Twitter here

Additional Sources

1. 2023 Stack Overflow Developer Survey

2. 2022 Stack Overflow Developer Survey

3. 2023 GitLab State of DevSecOps

4. 2021 Stack Overflow Developer Survey

5. 2020 Stack Overflow Developer Survey