Cloud Engineering – Software Engineering Daily

By Cloud Engineering – Software Engineering Daily

Listen to a podcast, please open Podcast Republic app. Available on Google Play Store.


Category: Software How-To

Open in iTunes


Open RSS feed


Open Website


Rate for this podcast

Subscribers: 202
Reviews: 0

Description

Episodes about building and scaling large software projects

Episode Date
Kubernetes vs. Serverless with Matt Ward
50:32

Kubernetes has become a highly usable platform for deploying and managing distributed systems.  The user experience for Kubernetes is great, but is still not as simple as a full-on serverless implementation–at least, that has been a long-held assumption. Why would you manage your own infrastructure, even if it is Kubernetes? Why not use autoscaling Lambda

The post Kubernetes vs. Serverless with Matt Ward appeared first on Software Engineering Daily.

May 29, 2020
Distributed Systems Research with Peter Alvaro
54:04

Every software company is a distributed system, and distributed systems fail in unexpected ways.  This ever-present tendency for systems to fail has led to the rise of failure testing, otherwise known as chaos engineering. Chaos engineering involves the deliberate failure of subsystems within an overall system to ensure that the system itself can be resilient

The post Distributed Systems Research with Peter Alvaro appeared first on Software Engineering Daily.

May 28, 2020
AWS Virtualization with Anthony Liguori
59:05

Amazon’s virtual server instances have come a long way since the early days of EC2. There are now a wide variety of available configuration options for spinning up an EC2 instance, which can be chosen from based on the workload that will be scheduled onto a virtual machine. There are also Fargate containers and AWS

The post AWS Virtualization with Anthony Liguori appeared first on Software Engineering Daily.

May 15, 2020
Cloudburst: Stateful Functions-as-a-Service with Vikram Sreekanti
59:35

Serverless computing is a way of designing applications that do not directly address or deploy application code to servers. Serverless applications are composed of stateless functions-as-a-service and stateful data storage systems such as Redis or DynamoDB.  Serverless applications allow for scaling up and down the entire architecture, because each component is naturally scalable. And this

The post Cloudburst: Stateful Functions-as-a-Service with Vikram Sreekanti appeared first on Software Engineering Daily.

Apr 23, 2020
Reserved Instances with Aran Khanna
59:34

When a developer spins up a virtual machine on AWS, that virtual machine could be purchased using one of several types of cost structures. These cost structures include on-demand instances, spot instances, and reserved instances. On-demand instances are often the most expensive, because the developer gets reliable VM infrastructure without committing to long-term pricing. Spot

The post Reserved Instances with Aran Khanna appeared first on Software Engineering Daily.

Apr 10, 2020
Google Cloud Networking with Lakshmi Sharma
49:54

A large cloud provider has high volumes of network traffic moving through data centers throughout the world. These providers manage the infrastructure for thousands of companies, across racks and racks of multitenant servers, and cables that stretch underseas, connecting network packets with their destination. Google Cloud Platform has grown steadily into a wide range of

The post Google Cloud Networking with Lakshmi Sharma appeared first on Software Engineering Daily.

Mar 23, 2020
Go Networking with Sneha Inguva
58:18

A cloud provider gives developers access to virtualized server infrastructure. When a developer rents this infrastructure via an API call, a virtual server is instantiated on physical machines. That virtual server needs to be made addressable through the allocation of an IP address to make it reachable from the open Internet. When the virtual server

The post Go Networking with Sneha Inguva appeared first on Software Engineering Daily.

Feb 18, 2020
Replicated Software Delivery with Grant Miller and Marc Campbell
1:08:51

Distributed systems are required to run most modern enterprise software. Application services need multiple instances for scalability and failover. Large databases are sharded onto multiple nodes. Logging services, streaming frameworks, and continuous integration tools all require the orchestration of more than one server. Deploying a distributed system has historically been difficult because the nodes of

The post Replicated Software Delivery with Grant Miller and Marc Campbell appeared first on Software Engineering Daily.

Jan 28, 2020
Lyft Kubernetes with Vicki Cheung
49:14

The ridesharing infrastructure of Lyft has a high volume of traffic that is mostly handled by servers on AWS. When Vicki Cheung joined Lyft in 2018, the company was managing containers with an internally built container scheduler. One of her primary goals at the company was to move Lyft to Kubernetes. In today’s episode, Vicki

The post Lyft Kubernetes with Vicki Cheung appeared first on Software Engineering Daily.

Jan 23, 2020
Packet: Baremetal Infrastructure with Zachary Smith and Nathan Goulding
53:41

Cloud infrastructure is usually consumed in the form of virtual machines or containers. These VMs or containers are running on a physical host machine that is also running other VMs and containers. This is called multitenancy. Servers across cloud providers such as AWS have a high utilization because there are multiple virtual instances running on

The post Packet: Baremetal Infrastructure with Zachary Smith and Nathan Goulding appeared first on Software Engineering Daily.

Jan 15, 2020
Edge Computing Platform with Jaromir Coufal
53:44

Edge computing is the usage of servers that are geographically close to the client device. The first common use case for edge computing was CDNs: content-delivery networks. A content delivery network placed media files such as images and videos on multiple servers throughout the world. These are big files, and they take lots of bandwidth

The post Edge Computing Platform with Jaromir Coufal appeared first on Software Engineering Daily.

Jan 14, 2020
Amazon EC2 with Dave Brown
33:25

Amazon EC2 (Elastic Compute Cloud) is a virtualized server product that provides the user with scalable compute infrastructure. EC2 was created in 2006 as one of the first three AWS services along with S3 and Simple Queueing Service. Since then, EC2 has provided the core server infrastructure for many of the companies that have been

The post Amazon EC2 with Dave Brown appeared first on Software Engineering Daily.

Jan 08, 2020
Amazon Kubernetes with Abby Fuller
43:49

Amazon’s container offerings include ECS (Elastic Container Service), EKS (Elastic Kubernetes Service), and Fargate. Through these different offerings, Amazon provides a variety of ways that a user can manage Kubernetes clusters and standalone container instances. The choice of which containerization system to choose depends on the needs of the user, and the tradeoffs they want

The post Amazon Kubernetes with Abby Fuller appeared first on Software Engineering Daily.

Jan 07, 2020
Kubernetes Progress with Kelsey Hightower
1:01:13

When the Kubernetes project was started, Amazon Web Services was the dominant cloud provider. Most of the code that runs AWS is closed source, which prevents an open ecosystem from developing around AWS. Developers who deploy their application onto AWS are opting into a closed, controlled ecosystem, which has both costs and benefits. The software

The post Kubernetes Progress with Kelsey Hightower appeared first on Software Engineering Daily.

Jan 06, 2020
Kubernetes at Cruise with Karl Isenberg
1:12:09

Cruise is a company that is building a fully automated self-driving car service. The infrastructure of a self-driving car platform presents a large number of new engineering problems. Self-driving cars collect vast quantities of data as they are driving around the city. This data needs to be transferred from the cars onto cloud servers. The

The post Kubernetes at Cruise with Karl Isenberg appeared first on Software Engineering Daily.

Dec 17, 2019
Linkerd Market Strategy with William Morgan
1:05:35

The container orchestration wars ended in 2016 with Kubernetes being the most popular open source tool for deploying and managing infrastructure. Since that time, most large enterprises have been implementing a “platform strategy” based around Kubernetes. A platform strategy is a plan for creating a consistent experience for software engineers working throughout an enterprise. At

The post Linkerd Market Strategy with William Morgan appeared first on Software Engineering Daily.

Dec 06, 2019
Istio Market Strategy with Zack Butcher
1:23:00

Kubernetes has created a widespread system for deploying and managing infrastructure. As Kubernetes has been increasingly adopted, companies are thinking about how to leverage that common layer of infrastructure. With the common infrastructure abstraction of Kubernetes, it becomes easier to adopt other abstractions that are uniform across the entire company.  This has created a market

The post Istio Market Strategy with Zack Butcher appeared first on Software Engineering Daily.

Dec 05, 2019
Heroku Infrastructure with Mark Turner
1:04:13

A cloud provider gives a developer low-cost compute infrastructure on-demand.  Cloud providers can be divided up into two categories: Layer 1 cloud providers and Layer 2 cloud providers. A Layer 1 cloud provider such as Amazon Web Services owns server hardware and sells compute infrastructure as a commodity. A Layer 2 cloud provider purchases compute

The post Heroku Infrastructure with Mark Turner appeared first on Software Engineering Daily.

Dec 04, 2019
Cloud Dependencies with Mya Pitzeruse
47:51

New software abstractions always take advantage of the abstractions that have been built before. Software libraries allow us to import code that sits on the same host as a new program. Open source software let us copy and paste existing code, or clone entire repositories. Cloud providers offer hosted tools and APIs that we can

The post Cloud Dependencies with Mya Pitzeruse appeared first on Software Engineering Daily.

Nov 23, 2019
Cloudflare Serverless with Zack Bloom
59:13

“Serverless” is an execution model where applications are scheduled and deployed to servers that are not directly managed by the application developer.  In serverless execution, an application only loads and operates when a user actually needs to get a response from that application. This saves on resources, because many applications do not need to run

The post Cloudflare Serverless with Zack Bloom appeared first on Software Engineering Daily.

Nov 18, 2019
GraalVM Quarkus: Java Acceleration with Guillaume Smet and Emmanuel Bernard
1:01:32

Java programs run in a different environment than they did ten years ago. Modern infrastructure runs on containers sitting in a Kubernetes cluster. The optimal configuration for a Java program in that context is different than it was for an environment dominated by virtual machines and bare metal. When you are co-scheduling your services with

The post GraalVM Quarkus: Java Acceleration with Guillaume Smet and Emmanuel Bernard appeared first on Software Engineering Daily.

Nov 14, 2019
Dark Lang with Ellen Chisa and Paul Biggar
1:07:48

Dark Lang is a programming language that is tightly integrated with the cloud.  Dark takes an opinionated approach that most developers are going to want to run their applications in the cloud, and this perspective influences how Dark looks at deployments, IDEs, exception handling, and other aspects of software development. Paul Biggar is the founder

The post Dark Lang with Ellen Chisa and Paul Biggar appeared first on Software Engineering Daily.

Oct 21, 2019
Gravity: Distributed Application Delivery with Ev Kontsevoy
1:07:13

Modern applications are distributed systems. These applications require an installation mechanism that can run and update the software across multiple nodes.  When a SaaS company starts to work with large enterprise customers, that company needs to figure out a way to deliver their product to the enterprise. This requires the SaaS company to deploy the

The post Gravity: Distributed Application Delivery with Ev Kontsevoy appeared first on Software Engineering Daily.

Oct 17, 2019
How To Build A Cloud Provider with Anurag Goel
1:15:43

Render is a cloud provider built on top of Amazon Web Services and Google Cloud. Render uses the compute abstractions provided by the major cloud providers to build a second layer cloud provider with the goal of providing a better user experience. Anurag Goel is the founder of Render, and he returns to the show

The post How To Build A Cloud Provider with Anurag Goel appeared first on Software Engineering Daily.

Oct 14, 2019
Cloud Foundry with Abby Kearns
47:57

Cloud Foundry is a system for managing distributed applications. Cloud Foundry was released in 2011, and has been widely adopted by enterprises that need a platform for deploying and scaling the applications that run within their company. The ecosystem around Cloud Foundry includes systems for continuous delivery, pubsub messaging, and containerization. Abby Kearns is the

The post Cloud Foundry with Abby Kearns appeared first on Software Engineering Daily.

Sep 24, 2019
Okta Engineering with Hector Aguilar
54:23

A new employee at a software company needs access to a variety of tools. In order to get started working, the employee might need Slack, email, Google Docs, and Amazon Web Services, and all of these require an account with a username and password. Setting up all of these accounts can be time consuming, because

The post Okta Engineering with Hector Aguilar appeared first on Software Engineering Daily.

Sep 16, 2019
Cloud-Native Applications with Cornelia Davis
55:16

Amazon Web Services first came out in 2006. It took several years before the software industry realized that cloud computing was a transformative piece of technology. Initially, the common perspective around cloud computing was that it was a useful tool for startups, but would not be a smart option for large, established businesses. Cloud computing

The post Cloud-Native Applications with Cornelia Davis appeared first on Software Engineering Daily.

Sep 13, 2019
Service Mesh Deployment with Varun Talwar
50:05

The service mesh abstraction allows for a consistent model for managing and monitoring the different components of a microservices architecture.  In the service mesh pattern, each service is deployed with a sidecar container that contains a service proxy. These sidecars are collectively referred to as the “data plane.” Each sidecar provides the service that it

The post Service Mesh Deployment with Varun Talwar appeared first on Software Engineering Daily.

Aug 14, 2019
gVisor Container Isolation with Michael Pratt and Yoshi Tamura
56:53

Software applications running within a host operating system need to be isolated. Isolation prevents security vulnerabilities, such as one application accessing the memory of another. In modern cloud environments, a single physical host might be running multiple virtual machines on top of a hypervisor. Those virtual machines might be divided up into containers. The different

The post gVisor Container Isolation with Michael Pratt and Yoshi Tamura appeared first on Software Engineering Daily.

Jul 10, 2019
Infrastructure Wars with Sheng Liang
1:01:59

Sheng Liang was the lead developer on the original Java Virtual Machine. Today he works as the CEO of Rancher Labs, a company building a platform on top of Kubernetes. Sheng joins the show to discuss his experiences in the technology industry. The container orchestration wars had many victims. The competing standards for how an

The post Infrastructure Wars with Sheng Liang appeared first on Software Engineering Daily.

Jun 19, 2019
Kubernetes Operators with Rob Szumski
1:09:07

Kubernetes has made distributed systems easier to deploy and manage. As Kubernetes has become reliable, engineers have started to look for higher level abstractions we can define on top of Kubernetes. An operator is a method of packaging, deploying, and managing a Kubernetes application. Operators are useful for spinning up distributed systems such as Kafka,

The post Kubernetes Operators with Rob Szumski appeared first on Software Engineering Daily.

Jun 18, 2019
Render: High Level Cloud with Anurag Goel
1:20:30

Cloud computing was popularized in 2006 with the launch of Amazon Web Services. AWS allowed developers to use remote server infrastructure with a simple set of APIs. But even with AWS, it was still not simple to deploy and manage a web application. In 2007, Heroku launched a platform built on top of AWS. Heroku

The post Render: High Level Cloud with Anurag Goel appeared first on Software Engineering Daily.

Jun 17, 2019
Kubernetes Vision with Joe Beda
1:12:18

Google Cloud was started with a vision of providing Google infrastructure to the masses. In 2008, it was not obvious that Google should become a cloud provider. Amazon Web Services was finding success among startups who needed on-demand infrastructure, but the traditional enterprise market was not yet ready to buy cloud resources. Googlers liked the

The post Kubernetes Vision with Joe Beda appeared first on Software Engineering Daily.

Jun 11, 2019
Kubernetes Storage with Saad Ali
1:00:30

Containers are made to fail gracefully. When your container shuts down due to a hardware or software failure, your distributed application should be able to tolerate that failure. One simple way to be able to tolerate such a failure is to make all of your application logic “stateless.” If your application does not maintain state,

The post Kubernetes Storage with Saad Ali appeared first on Software Engineering Daily.

Jun 10, 2019
Kubernetes Market with Adam Glick
1:18:41

Amazon Web Services is the leading cloud provider by a large margin. Amazon established its lead by being first to market in 2006, with Google and Microsoft taking several years to catch up to the huge business opportunity of the cloud. Since 2008, Google Cloud has been working on cloud products for developers. It started

The post Kubernetes Market with Adam Glick appeared first on Software Engineering Daily.

Jun 07, 2019
Service Mesh Interface with Lachlan Evenson
1:05:01

Containers offer a lightweight abstraction for running a server. Cloud providers are able to manage billions of containers from different users, allowing for economies of scale so that each user can pay less. Today, there is a variety of ways that users can deploy containers on a cloud provider. These containers can run in managed

The post Service Mesh Interface with Lachlan Evenson appeared first on Software Engineering Daily.

Jun 06, 2019
Multicloud Future with Bassam Tabbara
1:10:46

Each cloud provider offers a different set of services which are not always compatible with each other. What are the challenges of building an application that interoperates with multiple different clouds? The first issue is API compatibility. Most cloud providers have a managed SQL offering, a bucket storage system, and server abstractions like virtual machines

The post Multicloud Future with Bassam Tabbara appeared first on Software Engineering Daily.

Jun 05, 2019
Kubernetes Development with Tim Hockin
50:55

Kubernetes has evolved from a nascent project within Google to a thriving ecosystem of cloud providers, open source projects, and engineers. Tim Hockin is a principal software engineer who has been with Google for 15 years. Tim joins the show to talk about the early days of the Kubernetes projects, and the engineering efforts that

The post Kubernetes Development with Tim Hockin appeared first on Software Engineering Daily.

Jun 04, 2019
Google Anthos with Aparna Sinha
57:18

Google’s cloud business was long regarded as a place where startups could build a business, but not established enterprises. For serious workloads, enterprises chose Amazon almost unanimously. This phenomenon of Amazon as the default was described by a phrase that harkened back to the days of IBM’s dominance: “nobody ever got fired for choosing AWS.”

The post Google Anthos with Aparna Sinha appeared first on Software Engineering Daily.

Jun 03, 2019
Service Mesh Wars with William Morgan
1:21:11

A service mesh is an abstraction that provides traffic routing, policy management, and telemetry for a distributed application. A service mesh consists of a data plane and a control plane. In the data plane, a proxy runs alongside each service, with every request from a service being routed through the proxy. In the control plane,

The post Service Mesh Wars with William Morgan appeared first on Software Engineering Daily.

May 31, 2019
Netflix Early Days with Greg Burrell
1:05:13

Netflix started with a DVD-by-mail product. The software infrastructure and operations practices needed for the DVD business were very different from those needed by a streaming video company. Since the early days of Netflix, CEO Reed Hastings knew that the company would evolve to becoming a streaming video platform. But he did not know when

The post Netflix Early Days with Greg Burrell appeared first on Software Engineering Daily.

May 29, 2019
Scaling Intuit with Alex Balazs
1:01:35

Alex Balazs is the Intuit Chief Architect and has been working at the company for almost twenty years. Intuit’s products include QuickBooks, TurboTax, and Mint. These applications are used to file taxes, manage business invoices, conduct personal accounting, and other critical aspects of a user’s financial life. Because the applications are managing money for users,

The post Scaling Intuit with Alex Balazs appeared first on Software Engineering Daily.

May 21, 2019
Kubernetes Virtualization with Paul Czarkowski
56:55
May 07, 2019
Cloud with Eric Brewer
1:04:19

RECENT UPDATES: FindCollabs is a company I started recently The FindCollabs Podcast is out! FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the

The post Cloud with Eric Brewer appeared first on Software Engineering Daily.

Apr 26, 2019
Intricately: Mapping the Internet with Fima Leshinsky
1:03:57

RECENT UPDATES: FindCollabs is a company I started recently The FindCollabs Podcast is out! FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the

The post Intricately: Mapping the Internet with Fima Leshinsky appeared first on Software Engineering Daily.

Apr 25, 2019
gVisor: Secure Container Sandbox with Yoshi Tamura
50:04

RECENT UPDATES: Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a

The post gVisor: Secure Container Sandbox with Yoshi Tamura appeared first on Software Engineering Daily.

Apr 24, 2019
Observability Engineering with James Burns
1:02:26

RECENT UPDATES: Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a

The post Observability Engineering with James Burns appeared first on Software Engineering Daily.

Apr 23, 2019
Serverless Runtimes with Steren Giannini
57:26

RECENT UPDATES: Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a

The post Serverless Runtimes with Steren Giannini appeared first on Software Engineering Daily.

Apr 22, 2019
AWS Storage with Kevin Miller
55:17

RECENT UPDATES: FindCollabs $5000 Hackathon Ends Saturday April 15th, 2019 New version of Software Daily, our app and ad-free subscription service Software Daily is looking for help with Android engineering, QA, machine learning, and more   A software application requires compute and storage. Both compute and storage have been abstracted into cloud tools that can

The post AWS Storage with Kevin Miller appeared first on Software Engineering Daily.

Apr 08, 2019
AWS Compute with Deepak Singh
58:13

Upcoming event: FindCollabs Hackathon at App Academy on April 6, 2019 On Amazon Web Services, there are many ways to run an application on a single node. The first compute option on AWS was the EC2 virtual server instance. But EC2 is a large abstraction compared to what many people need for their nodes–which is

The post AWS Compute with Deepak Singh appeared first on Software Engineering Daily.

Apr 05, 2019
Uber Infrastructure with Prashant Varanasi and Akshay Shah
1:02:29

Upcoming events: A Conversation with Haseeb Qureshi at Cloudflare on April 3, 2019 FindCollabs Hackathon at App Academy on April 6, 2019 Uber’s infrastructure supports millions of riders and billions of dollars in transactions. Uber has high throughput and high availability requirements, because users depend on the service for their day-to-day transportation. When Uber was

The post Uber Infrastructure with Prashant Varanasi and Akshay Shah appeared first on Software Engineering Daily.

Apr 01, 2019
Workload Scheduling with Brian Grant
50:18

Upcoming events: A Conversation with Haseeb Qureshi at Cloudflare on April 3, 2019 FindCollabs Hackathon at App Academy on April 6, 2019 Google has been building large-scale scheduling systems for more than fifteen years. Google Borg was started around 2003, giving engineers at Google a unified platform to issue long-lived service workloads as well as

The post Workload Scheduling with Brian Grant appeared first on Software Engineering Daily.

Mar 29, 2019
Peloton: Uber’s Cluster Scheduler with Min Cai and Mayank Bansal
56:54

Upcoming events: A Conversation with Haseeb Qureshi at Cloudflare on April 3, 2019 FindCollabs Hackathon at App Academy on April 6, 2019 Google’s Borg system is a cluster manager that powers the applications running across Google’s massive infrastructure. Borg provided inspiration for open source tools like Apache Mesos and Kubernetes. Over the last decade, some

The post Peloton: Uber’s Cluster Scheduler with Min Cai and Mayank Bansal appeared first on Software Engineering Daily.

Mar 28, 2019
Netlify with Mathias Biilmann Christensen
1:02:25

Cloud computing started to become popular in 2006 with the release of Amazon EC2, a system for deploying applications to virtual machines sitting on remote data center infrastructure . With cloud computing, application developers no longer needed to purchase expensive server hardware. Creating an application for the Internet became easier, cheaper, and simpler. As the

The post Netlify with Mathias Biilmann Christensen appeared first on Software Engineering Daily.

Mar 08, 2019
Kubernetes Security with Liz Rice
54:36

A Kubernetes cluster presents multiple potential attack surfaces: the cluster itself, a node running on the cluster, a pod running in the node, a container running in a pod. If you are managing your own Kubernetes cluster, you need to be aware of the security settings on your etcd, your API server, and your container

The post Kubernetes Security with Liz Rice appeared first on Software Engineering Daily.

Feb 14, 2019
Replicated: On-Prem Deployments with Grant Miller
1:03:06

Cloud computing has been popular for less than twenty years. Large software companies have existed for much longer. If your company was started before the cloud became popular, you probably have a large, data center on your companies premises. The shorthand term for this software environment is “on-prem”. Deploying software to your own on-prem servers

The post Replicated: On-Prem Deployments with Grant Miller appeared first on Software Engineering Daily.

Feb 13, 2019
Knative: Serverless Workloads with Ville Aikas
59:08

Infrastructure software is having a renaissance. Cloud providers offer a wide range of deployment tools, including virtual machines, managed Kubernetes clusters, standalone container instances, and serverless functions. Kubernetes has standardized the container orchestration layer and created a thriving community. The Kubernetes community gives the cloud providers a neutral ground to collaborate on projects that benefit

The post Knative: Serverless Workloads with Ville Aikas appeared first on Software Engineering Daily.

Feb 08, 2019
VMware Kubernetes Strategy with Brad Meiseles
51:07

Virtualization software allows companies to get better utilization from their physical servers. A single physical host can manage multiple virtual machines using a hypervisor. VMware brought virtualization software to market, creating popular tools for allowing enterprises to deploy virtual machines throughout their organization. Containers provide another improvement to server utilization. A virtual machine can be

The post VMware Kubernetes Strategy with Brad Meiseles appeared first on Software Engineering Daily.

Feb 07, 2019
Scaling HashiCorp with Armon Dadgar and Mitchell Hashimoto
58:04

HashiCorp was founded seven years ago with the goal of building infrastructure tools for automating cloud workflows such as provisioning, secret management, and service discovery. Hashicorp’s thesis was that operating cloud infrastructure was too hard: there was a need for new tools to serve application developers. Hashicorp founders Mitchell Hashimoto and Armon Dadgar began releasing

The post Scaling HashiCorp with Armon Dadgar and Mitchell Hashimoto appeared first on Software Engineering Daily.

Feb 04, 2019
Prometheus Scalability with Bryan Boreham
53:24

Prometheus is an open source monitoring system and time series database. Prometheus includes a multi-dimensional data model, a query language called PromQL, and a pull model for gathering metrics from your different services. Prometheus is widely used by large distributed systems deployments such as Kubernetes and Cloud Foundry. Prometheus gathers metrics from your services by

The post Prometheus Scalability with Bryan Boreham appeared first on Software Engineering Daily.

Jan 21, 2019
Spot Instances with Amiram Shachar
1:05:40

When a developer provisions a cloud server, that server is called an “instance”. These instances can be used for running whatever workload a developer has, whether it is a web application, a database, or a set of containers. The cloud is cheap to get started on. New applications with few users can often be hosted

The post Spot Instances with Amiram Shachar appeared first on Software Engineering Daily.

Jan 18, 2019
Kubernetes in China with Dan Kohn
59:21

Chinese Internet companies operate at a massive scale. WeChat has over a billion users and is widely used as the primary means of payment by urban Chinese consumers. Alibaba ships 12 million packages per day, which is four times the amount of Amazon. JD.com, a Chinese ecommerce company, has perhaps the largest production Kubernetes installation

The post Kubernetes in China with Dan Kohn appeared first on Software Engineering Daily.

Jan 14, 2019
AWS Analysis with Corey Quinn
1:06:09

Amazon Web Services changed how software engineers work. Before AWS, it was common for startups to purchase their own physical servers. AWS made server resources as accessible as an API request, and has gone on to create higher-level abstractions for building applications. For the first few years of AWS, the abstractions were familiar. S3 provided

The post AWS Analysis with Corey Quinn appeared first on Software Engineering Daily.

Jan 11, 2019
Zeit: Serverless Cloud with Guillermo Rauch
1:11:03

Serverless computing is a technique for deploying applications without an addressable server. A serverless application is running on servers, but the developer does not have access to the server in the traditional sense. The developer is not dealing with IP addresses and configuring instances of their different services to be able to scale. Just as

The post Zeit: Serverless Cloud with Guillermo Rauch appeared first on Software Engineering Daily.

Jan 10, 2019
Cloud Events with Doug Davis
54:15

Functions-as-a-service allow developers to run their code in a “serverless” environment. A developer can provide a function to a cloud provider and the code for that function will be scheduled onto a container and executed whenever an event triggers that function. An “event” can mean many different things. It is a signal that something has

The post Cloud Events with Doug Davis appeared first on Software Engineering Daily.

Jan 09, 2019
Multicloud with Ben Hindman
1:12:42

Most applications today are either deployed to on-premise environments or deployed to a single cloud provider. Developers who are deploying on-prem struggle to set up complicated open source tools like Kafka and Hadoop. Developers who are deploying to a cloud provider tend to stay within that specific cloud provider, because moving between different clouds and

The post Multicloud with Ben Hindman appeared first on Software Engineering Daily.

Jan 08, 2019
Stateful Kubernetes with Saad Ali
1:00:08

In a cloud infrastructure environment, failures happen regularly. The servers can fail, the network can fail, and software bugs can crash your software unexpectedly. The amount of failures that can occur in cloud infrastructure is one reason why storage is often separated from application logic. A developer can launch multiple instances of their application, with

The post Stateful Kubernetes with Saad Ali appeared first on Software Engineering Daily.

Jan 07, 2019
Crossplane: Multicloud Control Plane with Bassam Tabbara
58:37

Cloud providers created the ability for developers to easily deploy their applications to servers on data centers. In the early days of the cloud, most of the code that a developer wrote for their application could run on any cloud provider, whether it was Amazon, Google, or Microsoft. These cloud providers were giving developers the

The post Crossplane: Multicloud Control Plane with Bassam Tabbara appeared first on Software Engineering Daily.

Jan 02, 2019
Google Early Days with John Looney Holiday Repeat
1:11:02

Originally posted on 16 June 2017. John Looney spent more than 10 years at Google. He started with infrastructure, and was part of the team that migrated Google File System to Colossus, the successor to GFS. Imagine migrating every piece of data on Google from one distributed file system to another. In this episode, John

The post Google Early Days with John Looney Holiday Repeat appeared first on Software Engineering Daily.

Dec 25, 2018
Service Proxying with Matt Klein Holiday Repeat
56:18

Originally posted on 14 February 2017. Most tech companies are moving toward a highly distributed microservices architecture. In this architecture, services are decoupled from each other and communicate with a common service language, often JSON over HTTP. This provides some standardization, but these companies are finding that more standardization would come in handy. At the

The post Service Proxying with Matt Klein Holiday Repeat appeared first on Software Engineering Daily.

Dec 24, 2018
Linkerd Service Mesh with William Morgan
1:00:22

Software products are distributed across more and more servers as they grow. With the proliferation of cloud providers like AWS, these large infrastructure deployments have become much easier to create. With the maturity of Kubernetes, these distributed applications are more reliable. Developers and operators can use a service mesh to manage the interactions between services

The post Linkerd Service Mesh with William Morgan appeared first on Software Engineering Daily.

Dec 19, 2018
On-Prem Cloud with Bob Fraser
1:00:49

Not every company wants to move to the public cloud. Some companies have already built data centers, and can continue to operate their business with their own servers. Some companies have compliance issues with the public cloud, and want to operate their own servers to avoid legal risk. Operating a data center is not easy.

The post On-Prem Cloud with Bob Fraser appeared first on Software Engineering Daily.

Dec 03, 2018
Cloud Costs with Ran Rothschild
57:52

Cloud computing changed the economics of running a software company. Before the cloud, a software company had to purchase physical machines which often required thousands of dollars paid up front. The cloud allowed developers to deploy their applications for free, to operate a business for cheap, and to scale without hiring a dedicated team to

The post Cloud Costs with Ran Rothschild appeared first on Software Engineering Daily.

Nov 29, 2018
Schedulers with Adrian Cockcroft Holiday Repeat
58:08

Originally published on July 6, 2016. Scheduling is the method by which work is assigned to resources to complete that work. At the operating system level, this can mean scheduling of threads and processes. At the data center level, this can mean scheduling Hadoop jobs or other workflows that require the orchestration of a network

The post Schedulers with Adrian Cockcroft Holiday Repeat appeared first on Software Engineering Daily.

Nov 19, 2018
Liquid Software with Baruch Sadogursky
59:40

The software release process is a barrier between written code and a live production environment that affects users. A software release process can involve a variety of different practices. Code might be tested for bugs using automation and manual testing. Static analysis tools can look at the code for potential memory leaks. A software release

The post Liquid Software with Baruch Sadogursky appeared first on Software Engineering Daily.

Nov 15, 2018
Fission: Serverless on Kubernetes with Soam Vasani
1:03:05

Serverless computing abstracts away the idea of a server node. Serverless lets programmers treat compute resources as high-level, reliable APIs, rather than unreliable, low-level compute nodes that might fail. Serverless dramatically improves the efficiency of programmers. Instead of thinking of a database as a set of servers that need to be sharded and replicated, the

The post Fission: Serverless on Kubernetes with Soam Vasani appeared first on Software Engineering Daily.

Nov 13, 2018
Scaling Lyft with Matt Klein
57:47

Matt Klein has worked for three rapidly growing Internet companies. At AWS, he worked on EC2, the compute-as-a-service product that powers a large percentage of the Internet. At Twitter, he helped scale the infrastructure in the chaotic days before Twitter’s IPO. Today he works at Lyft, building systems to allow for ride sharing infrastructure to

The post Scaling Lyft with Matt Klein appeared first on Software Engineering Daily.

Nov 02, 2018
Flogo: Event-Driven Ecosystem with Leon Stigter and Matt Ellis
52:16

A smart security camera takes in a high volume of video images and processes those images using a set of machine learning models. Those models can be used to identify interesting snippets of movement throughout the day, and decide which of those snippets to keep. Some of the video snippets might contain movement of birds–but

The post Flogo: Event-Driven Ecosystem with Leon Stigter and Matt Ellis appeared first on Software Engineering Daily.

Oct 25, 2018
DevSecOps with Edward Thomson
55:29

DevSecOps emphasizes moving security out of a siloed audit process and distributing security practices throughout the software supply chain. In the past, software development usually followed a waterfall development process. Each step in building software was serialized, one after another. First, software was planned. Then it was built. Then it was tested. Finally, the software

The post DevSecOps with Edward Thomson appeared first on Software Engineering Daily.

Oct 23, 2018
AWS Containers with Deepak Singh
52:55

Deepak Singh is the director of compute services at AWS, where he works on cloud products relating to containers, Linux, and High Performance Computing. In today’s show, Deepak describes how the market for containers and serverless has evolved, and how Amazon thinks about product strategy. Back in 2014, Docker containers were becoming a popular way

The post AWS Containers with Deepak Singh appeared first on Software Engineering Daily.

Oct 19, 2018
DevOps at Microsoft with Martin Woodward
1:04:56

The Windows operating system is one of the most widely used pieces of software in history. Windows was started before there was any alternative to a monolithic codebase, because Microsoft was building software before the Internet was widely used by consumers. Networked computers gave rise to web applications, and software engineers began to rethink how

The post DevOps at Microsoft with Martin Woodward appeared first on Software Engineering Daily.

Oct 12, 2018
Android Things with Wayne Piekarski
1:02:01

Internet of Things is a concept that describes lots of devices that you interact with regularly being connected to the Internet and networked together. Technologists have been dreaming of the world of IoT for many years, where our connected refrigerator can detect that we are out of food, and automatically order more food. Or our

The post Android Things with Wayne Piekarski appeared first on Software Engineering Daily.

Sep 27, 2018
Unity and WebAssembly with Brett Bibby
1:08:16

Unity is a game engine for building 2-D and 3-D experiences, augmented reality, movies, and other applications. Unity is cross-platform, so that applications can be written once and deployed to iOS, Android, web, and other surfaces. Unity has been around for 13 years, and has grown in popularity with the rise in gaming and game

The post Unity and WebAssembly with Brett Bibby appeared first on Software Engineering Daily.

Sep 25, 2018
Front Engineering with Laurent Perrin
53:55

Front is a shared inbox application that has seen rapid adoption within companies. Front allows multiple members of a company to collaborate together on a conversation–whether that conversation is in email, Twitter, or Facebook Messenger. This is useful when a customer email needs to be shared between the sales and engineering teams, or when a

The post Front Engineering with Laurent Perrin appeared first on Software Engineering Daily.

Sep 24, 2018
Android on Chrome with Shahid Hussain and Stefan Kuhne
54:35

Google has two consumer operating systems: Android and Chrome. The Android operating system has been widely deployed on mobile devices. Chrome is an operating system for laptops and tablets, originally based around the Chrome browser. For several years, these two ecosystems were mostly separate–you could not run Android apps on a Chrome operating system. Shahid

The post Android on Chrome with Shahid Hussain and Stefan Kuhne appeared first on Software Engineering Daily.

Sep 20, 2018
Kubernetes Distributions with Brian Gracely and Michael Hausenblas
1:03:32

Kubernetes is an open source container management system. Kubernetes is sometimes described as “the Linux of distributed systems” and this description makes sense: the large numbers of users and contributors in the Kubernetes community is comparable to the volume of Linux adopters in its early days. There are many different distributions of Linux: Ubuntu, Red

The post Kubernetes Distributions with Brian Gracely and Michael Hausenblas appeared first on Software Engineering Daily.

Sep 19, 2018
Continuous Delivery Pipelines with Abel Wang
49:38

Continuous integration and delivery allows teams to move faster by allowing developers to ship code independently of each other. A multi-stage CD pipeline might consist of development, staging, testing, and production. At each of these stages, a new piece of code undergoes additional tests, so that when the code finally makes it to production, the

The post Continuous Delivery Pipelines with Abel Wang appeared first on Software Engineering Daily.

Sep 18, 2018
Orchestrating Kubernetes with Chris Gaun
59:15

A company runs a variety of distributed systems applications such as Hadoop for batch processing jobs, Spark for data science, and Kubernetes for container management. These distributed systems tools can run on-prem, in a cloud provider, or in a hybrid system that uses on-prem and cloud infrastructure. Some enterprises use VMs, some use bare metal,

The post Orchestrating Kubernetes with Chris Gaun appeared first on Software Engineering Daily.

Sep 13, 2018
Kubernetes Continuous Deployment with Sheroy Marker
54:16

Engineering organizations can operate more efficiently by working with a continuous integration and continuous deployment workflow. Continuous integration is the process of automatically building and deploying code that gets pushed to a remote repository. Continuous deployment is the process of moving that code through a pipeline of environments, from dev to test to production. At

The post Kubernetes Continuous Deployment with Sheroy Marker appeared first on Software Engineering Daily.

Sep 10, 2018
Kubernetes Impact with Clayton Coleman
1:11:26

Kubernetes is in production clusters around the world with hundreds of thousands of containers. Kubernetes provides a distributed systems management environment for small startups and giant enterprises with applications ranging from microservices to machine learning pipelines. Because the use cases are already so wide-ranging, and the project has had so much adoption, the focus of

The post Kubernetes Impact with Clayton Coleman appeared first on Software Engineering Daily.

Aug 30, 2018
Android Slices with Jason Monk
57:15

The main user interfaces today are the smartphone, the laptop, and the desktop computer. Some people today interact with voice interfaces, augmented reality, virtual reality, and automotive computer screens like the Tesla. In the future, these other interfaces will become more common. Developers will want to be able to expose their applications to these new

The post Android Slices with Jason Monk appeared first on Software Engineering Daily.

Aug 28, 2018
Helm with Michelle Noorali
57:18

Back in 2014, platform-as-a-service was becoming an increasingly popular idea. The idea of PaaS was to sit on top of infrastructure-as-a-service providers like Azure, AWS, or Google Cloud, and simplify some of the complexity of these infrastructure providers. Heroku had built a successful businesses from the idea of platform-as-a-service, and there was a widely held

The post Helm with Michelle Noorali appeared first on Software Engineering Daily.

Aug 27, 2018
Build Faster with Nader Dabit
1:07:06

Building software today is much faster than it was just a few years ago. The tools are higher level, and abstract away tasks that would have required months of development. Much of a developer’s time used to be spent optimizing databases, load balancers, and queueing systems in order to be able to handle the load

The post Build Faster with Nader Dabit appeared first on Software Engineering Daily.

Aug 24, 2018
OLIO: Food Sharing with Lloyd Watkin
46:52

Food gets thrown away from restaurants, homes, catering companies, and any other place with a kitchen. Most of this food gets thrown away when it is still edible, and could provide nutrition to someone who is hungry. Just like Airbnb makes use of excess living capacity, OLIO was started to connect excess food with people

The post OLIO: Food Sharing with Lloyd Watkin appeared first on Software Engineering Daily.

Aug 21, 2018
Infrastructure Monitoring with Mark Carter
56:17

At Google, the job of a site reliability engineer involves building tools to automate infrastructure operations. If a server crashes, there is automation in place to create a new server. If a service starts to receive a high load of traffic, there is automation in place to scale up the instances of that service. In

The post Infrastructure Monitoring with Mark Carter appeared first on Software Engineering Daily.

Aug 14, 2018
GitOps: Kubernetes Continuous Delivery with Alexis Richardson
48:40

Continuous delivery is a way of releasing software without requiring software engineers to synchronize during a release.  Over the last decade, continuous delivery workflows have evolved as the tools have changed. Jenkins was one of the first continuous delivery tools and is still in heavy use today. Netflix’s open sourced Spinnaker has also been widely

The post GitOps: Kubernetes Continuous Delivery with Alexis Richardson appeared first on Software Engineering Daily.

Aug 13, 2018
Klarna Engineering with Marcus Granström
51:44

Klarna is a payments company headquartered in Sweden. Since being established in 2005 it has grown to handling $21 billion in online sales in 2017. Roughly 40% of all e-commerce sales in Sweden go through Klarna. Klarna’s original differentiator was that it allowed users to checkout of e-commerce stores without entering in credit card information.

The post Klarna Engineering with Marcus Granström appeared first on Software Engineering Daily.

Aug 10, 2018
Stripe Engineering with Raylene Yung
48:23

Stripe is a payments API that allows merchants to transact online. Since the creation of the payments API, Stripe has expanded into adjacent services such as fraud detection, business management, and billing. These other verticals leverage the existing customer base and infrastructure that Stripe has developed from the success of their payments business. Raylene Yung

The post Stripe Engineering with Raylene Yung appeared first on Software Engineering Daily.

Aug 09, 2018
GraalVM with Thomas Wuerthinger
56:09

Java programs compile into Java bytecode. Java bytecode executes in the Java Virtual Machine, a runtime environment that compiles that bytecode further into machine code, and optimizes the runtime by identifying “hot” code paths and keeping those hot code paths executing quickly. The Java Virtual Machine is a popular platform for building languages on top

The post GraalVM with Thomas Wuerthinger appeared first on Software Engineering Daily.

Aug 03, 2018
Edge Kubernetes with Venkat Yalla
57:43

“Edge computing” is a term used to define computation that takes place in an environment outside of a data center. Edge computing is a broad term. Your smartphone is an edge device. A self-driving car is an edge device. A security camera with a computer chip is an edge device. These “edge devices” have existed

The post Edge Kubernetes with Venkat Yalla appeared first on Software Engineering Daily.

Jul 30, 2018
Kubernetes in the Enterprise with Aparna Sinha
59:07

Enterprises want to update their technology faster. One way an enterprise can accelerate the adoption of new tools is to move more aggressively towards the cloud. By giving internal developers access to the cloud, it becomes easier to provision new servers–allowing for rapid experimentation, test environments, and scalability. In previous shows we have explored how

The post Kubernetes in the Enterprise with Aparna Sinha appeared first on Software Engineering Daily.

Jul 23, 2018
Git Vulnerability with Edward Thomson
55:41

Git is a distributed file system for version control. Git is extremely reliable, fast, and secure, owing to the fact that it is one of the oldest pieces of open source software. But even battle-tested software can have vulnerabilities. In this episode, we explore a subtle git vulnerability that could have potentially led to git

The post Git Vulnerability with Edward Thomson appeared first on Software Engineering Daily.

Jul 17, 2018
Shopify Infrastructure with Niko Kurtti
53:44

Shopify runs more than 600,000 small business websites. When Shopify was figuring out how to scale, the engineering teams did not have a standard workflow for how to deploy and manage services. Some teams used AWS, some teams used Heroku, some teams used other infrastructure providers. To manage all those stores effectively, Shopify has built

The post Shopify Infrastructure with Niko Kurtti appeared first on Software Engineering Daily.

Jun 27, 2018
Function Platforms with Chad Arimura and Matt Stephenson
52:58

“Serverless” is a word used to describe functions that get deployed and run without the developer having to manage the infrastructure explicitly. Instead of creating a server, installing the dependencies, and executing your code, the developer just provides the code to the serverless API, and the serverless system takes care of the server creation, the

The post Function Platforms with Chad Arimura and Matt Stephenson appeared first on Software Engineering Daily.

Jun 26, 2018
Build a Bank: Monzo with Richard Dingwall
57:03

When you interact with your bank, it probably feels different than when you interact with a software technology company. That’s because the biggest banks in the world were started before software became such a universally important tool. Their core competency is banking–not consumer software. Today, most banks make consumer-facing software. But these banks were not

The post Build a Bank: Monzo with Richard Dingwall appeared first on Software Engineering Daily.

Jun 25, 2018
Rust Networking with Carl Lerche
53:37

Rust is a systems programming language with a distinct set of features for safety and concurrency. In previous shows about Rust, we explored how Rust can prevent crashes and eliminate data races through its approach to type safety and memory management. Rust’s focus on efficiency and safety makes it a promising language for networking code.

The post Rust Networking with Carl Lerche appeared first on Software Engineering Daily.

Jun 19, 2018
Container Storage with Jie Yu
59:24

A database stores data to an underlying section of storage. If you are an application developer, you might think of your persistent storage system as being the database itself–but at a lower level, that database is writing to block storage, file storage, or object storage. A container orchestration system manages application containers. If you want

The post Container Storage with Jie Yu appeared first on Software Engineering Daily.

Jun 06, 2018
Container Native Development with Ralph Squillace
55:01

Containers have improved deployments and resource utilization. Kubernetes created a platform to manage those containers and orchestrate them into distributed applications. In today’s episode, we explore tools that improve the workflow of the application developer who is working with Kubernetes, including Helm, Draft, and Brigade. Helm is a package manager for Kubernetes, which allows users

The post Container Native Development with Ralph Squillace appeared first on Software Engineering Daily.

May 30, 2018
Container Security with Maya Kaczorowski
47:21

Deploying software to a container presents a different security model than deploying an application to a VM. There is a smaller attack surface per container, but the container is colocated on a node with other containers. Containers are meant to have a shorter lifetime than VMs, so there are generally fewer consequences if a container

The post Container Security with Maya Kaczorowski appeared first on Software Engineering Daily.

May 22, 2018
Alexa Voice Design with Paul Cutsinger
55:03

Voice interfaces are a newer form of communicating with computers. Alexa is a voice interface platform from Amazon. Alexa powers the Amazon Echo, as well as Alexa-enabled cars, refrigerators, and dishwashers. Any developer can build a device with a voice interface using a Raspberry Pi. Paul Cutsinger works on Echo and Alexa at Amazon. He’s

The post Alexa Voice Design with Paul Cutsinger appeared first on Software Engineering Daily.

May 18, 2018
Gloo: Function Gateway with Idit Levine
56:28

Gloo is a function gateway built on top of the popular open source project Envoy. The goal of Gloo is to decouple client-facing APIs from upstream APIs. Gloo is similar to an API gateway, which is a tool that software companies can use to collect all their APIs and one place and impose security, monitoring,

The post Gloo: Function Gateway with Idit Levine appeared first on Software Engineering Daily.

May 16, 2018
Cluster Schedulers with Ben Hindman
1:08:34

Mesos is a system for managing distributed systems. The goal of Mesos is to help engineers orchestrate resources among multi-node applications like Spark. Mesos can also manage lower level schedulers like Kubernetes. A common misconception is that Mesos aims to solve the same problem as Kubernetes, but Mesos is a higher level abstraction. Ben Hindman

The post Cluster Schedulers with Ben Hindman appeared first on Software Engineering Daily.

May 11, 2018
Affirm Engineering with Libor Michalek
59:52

When I buy a mattress online, I pay for it with my credit card. Behind the scenes, a complex series of transactions occur between a payment gateway, the credit card company, and a few banks. There are problems with this process–it is slow, complex, and involves the synchronization of several different parties. Some consumers will

The post Affirm Engineering with Libor Michalek appeared first on Software Engineering Daily.

May 07, 2018
Superpedestrian Robotic Wheel / Infrastructure at HubSpot Meetup Talks
1:02:21

Superpedestrian is a robotic bicycle wheel that learns how you pedal and personalizes your bicycle ride. The engineering challenges of Superpedestrian are at the intersection of robotics, software, and real-time analytics. The first half of today’s show is about Superpedestrian. Goss Nuzzo Jones and Matt Cole are engineers at Superpedestrian. The slides for their presentation

The post Superpedestrian Robotic Wheel / Infrastructure at HubSpot Meetup Talks appeared first on Software Engineering Daily.

May 05, 2018
Building Datadog with Alexis Le-Quoc
52:38

Alexis Le-Quoc started Datadog in 2010, after living through the Internet boom and bust cycle of the late 90s and early 2000s. In 2010, cloud was just starting to become popular. There was a gap in the market for infrastructure monitoring tools, which Alexis helped fill with the first version of Datadog. Since 2010, the

The post Building Datadog with Alexis Le-Quoc appeared first on Software Engineering Daily.

May 02, 2018
Google Cluster Evolution with Brian Grant
51:21

Google’s central system for managing compute resources is called Borg. On Borg, millions of Linux containers process a wide variety of workloads. When a new application is spun up, Borg provides that application with the resources it needs. Workloads at Google usually fall into one of two distinct categories: long-running application workloads (such as Gmail)

The post Google Cluster Evolution with Brian Grant appeared first on Software Engineering Daily.

Apr 27, 2018
NATS Messaging with Derek Collison
1:06:46

A message broker is an architectural component that sends messages between different nodes in a distributed system. Message brokers are useful because the sender of a message does not always know who might want to receive that message. Message brokers can be used to implement the “publish/subscribe” pattern, and by centralizing the message workloads within

The post NATS Messaging with Derek Collison appeared first on Software Engineering Daily.

Apr 24, 2018
Stripe Observability Pipeline with Cory Watson
1:08:01

Stripe processes payments for thousands of businesses. A single payment could involve 10 different networked services. If a payment fails, engineers need to be able to diagnose what happened. The root cause could lie in any of those services. Distributed tracing is used to find the causes of failures and latency within networked services. In

The post Stripe Observability Pipeline with Cory Watson appeared first on Software Engineering Daily.

Apr 23, 2018
Monitoring Kubernetes with Ilan Rabinovitch
50:31

Monitoring a Kubernetes cluster allows operators to track the resource utilization of the containers within that cluster. In today’s episode, Ilan Rabinovitch joins the show to explore the different options for setting up monitoring, and some common design patterns around Kubernetes logging and metrics gathering. Ilan is the VP of product and community at Datadog.

The post Monitoring Kubernetes with Ilan Rabinovitch appeared first on Software Engineering Daily.

Apr 16, 2018
Go Systems with Erik St. Martin
57:53

Go is a language designed to improve systems programming. Go includes abstractions that simplify aspects of low level engineering that are historically difficult—concurrency, resource allocation, and dependency management. In that light, it makes sense that the Kubernetes container orchestration system was written in Go. Erik St. Martin is a cloud developer advocate at Microsoft, where

The post Go Systems with Erik St. Martin appeared first on Software Engineering Daily.

Apr 11, 2018
Database Chaos with Tammy Butow
59:54

Tammy Butow has worked at Digital Ocean and Dropbox, where she built out infrastructure and managed engineering teams. At both of these companies, the customer base was at a massive scale. At Dropbox, Tammy worked on the database that holds metadata used by Dropbox users to access their files. To call this metadata system simply

The post Database Chaos with Tammy Butow appeared first on Software Engineering Daily.

Apr 10, 2018
Site Reliability Management with Mike Hiraga
47:17

Software engineers have interacted with operations teams since software was being written. In the 1990s, most operations teams worked with physical infrastructure. They made sure that servers were provisioned correctly and installed with the proper software. When software engineers shipped bad code that took down a software company, the operations teams had to help recover

The post Site Reliability Management with Mike Hiraga appeared first on Software Engineering Daily.

Apr 09, 2018
Cloud and Edge with Steve Herrod
1:03:44

Steve Herrod led engineering at VMWare as the company scaled from 30 engineers to 3,000 engineers. After 11 years, he left to become a managing director for General Catalyst, a venture capital firm. Since he has both operating experience and a wide view of the technology landscape as an investor, he is well-equipped to discuss

The post Cloud and Edge with Steve Herrod appeared first on Software Engineering Daily.

Feb 23, 2018
Serverless Systems with Eduardo Laureano
1:01:30

On Software Engineering Daily, we have been covering the “serverless” movement in detail. For people who don’t use serverless functions, it seems like a niche. Serverless functions are stateless, auto-scaling, event driven blobs of code. You might say “serverless sounds kind of cool, but why don’t I just use a server? It’s a paradigm I’m

The post Serverless Systems with Eduardo Laureano appeared first on Software Engineering Daily.

Feb 22, 2018
Cloud Foundry Overview with Mike Dalessio
1:02:40

Earlier this year we did several shows about Cloud Foundry, followed by several shows about Kubernetes. Both of these projects allow you to build scalable, multi-node applications–but they serve different types of users. Cloud Foundry encompasses a larger scope of the application experience than Kubernetes. Kubernetes is lower level, and is actually being used within

The post Cloud Foundry Overview with Mike Dalessio appeared first on Software Engineering Daily.

Feb 21, 2018
Box Kubernetes Migration with Sam Ghods
56:17

Over 12 years of engineering, Box has developed a complex architecture of services. Whenever a user uploads a file to Box, that upload might cause 5 or 6 different services to react to the event. Each of these services is managed by a set of servers, and managing all of these different servers is a

The post Box Kubernetes Migration with Sam Ghods appeared first on Software Engineering Daily.

Feb 13, 2018
Scaling Box with Jeff Quiesser
48:11

When Box started in 2006, the small engineering team had a lot to learn. Box was one of the earliest cloud storage companies, with a product that allowed companies to securely upload files to remote storage. This was two years before Amazon Web Services introduced on-demand infrastructure, so the Box team managed their own servers,

The post Scaling Box with Jeff Quiesser appeared first on Software Engineering Daily.

Feb 12, 2018
Load Testing Mobile Applications with Paulo Costa and Rodrigo Coutinho
56:54

Applications need to be ready to scale in response to high-load events. With mobile applications, this can be even more important. People rely on mobile applications such as banking, ride sharing, and GPS. During Black Friday, a popular ecommerce application could be bombarded by user requests–you might not be able to complete a request to

The post Load Testing Mobile Applications with Paulo Costa and Rodrigo Coutinho appeared first on Software Engineering Daily.

Feb 08, 2018
Serverless at the Edge with Kenton Varda
59:48

Over the last decade, computation and storage has moved from on-premise hardware into the cloud data center. Instead of having large servers “on premise,” companies started to outsource their server workloads to cloud service providers. At the same time, there has been a proliferation of devices at the “edge.” The most common edge device is

The post Serverless at the Edge with Kenton Varda appeared first on Software Engineering Daily.

Feb 06, 2018
Linkedin Resilience with Bhaskaran Devaraj and Xiao Li
50:16

How do you build resilient, failure tested systems? Redundancy, backups, and testing are all important. But there is also an increasing trend towards chaos engineering–the technique of inducing controlled failures in order to prove that a system is fault tolerant in the way that you expect. In last week’s episode with Kolton Andrus, we discussed

The post Linkedin Resilience with Bhaskaran Devaraj and Xiao Li appeared first on Software Engineering Daily.

Feb 05, 2018
Chaos Engineering with Kolton Andrus
59:31

The number of ways that applications can fail are numerous. Disks fail all the time. Servers overheat. Network connections get flaky. You assume that you are prepared for such a scenario, because you have replicated your servers. You have the database backed up. Your core application is spread across multiple availability zones. But are you

The post Chaos Engineering with Kolton Andrus appeared first on Software Engineering Daily.

Feb 02, 2018
How to Change an Enterprise’s Software and Culture with Zhamak Dehghani
57:46

On this show, we spend a lot of time talking about CI/CD, data engineering, and microservices. These technologies have only been widely talked about for the last 5-10 years. That means that they are easy to adopt for startups that get founded in the last 5-10 years, but not necessarily for older enterprises. Within a

The post How to Change an Enterprise’s Software and Culture with Zhamak Dehghani appeared first on Software Engineering Daily.

Feb 01, 2018
Serverless Containers with Sean McKenna
55:55

After two weeks of episodes about Kubernetes, our in-depth coverage of container orchestration is drawing to a close. We have a few more shows on the topic before we move on to covering other aspects of software. If you have feedback on this thematic format (whether you like it or not), send me an email:

The post Serverless Containers with Sean McKenna appeared first on Software Engineering Daily.

Jan 25, 2018
Container Instances with Gabe Monroy
54:24

In 2011, platform-as-a-service was in its early days. It was around that time that Gabe Monroy started a container platform called Deis, with the goal of making an open source platform-as-a-service that anyone could deploy to whatever infrastructure they wanted. Over the last six years, Gabe had a front row seat to the rise of

The post Container Instances with Gabe Monroy appeared first on Software Engineering Daily.

Jan 22, 2018
Service Mesh Design with Oliver Gould
59:38

Oliver Gould worked at Twitter from 2010 to 2014. Twitter’s popularity was taking off, and the engineering team was learning how to scale the product. During that time, Twitter adopted Apache Mesos, and began breaking up its monolithic architecture into different services. As more and more services were deployed, engineers at Twitter decided to standardize

The post Service Mesh Design with Oliver Gould appeared first on Software Engineering Daily.

Jan 19, 2018
Kubernetes Storage with Bassam Tabbara
1:00:39

Modern applications store most of their data on hosted storage solutions. We use hosted block storage to back databases, hosted object storage for objects such as videos, and hosted file storage for file systems. Using a cloud provider for these storage systems can simplify scalability, durability, and availability–it can be less painful than taking care

The post Kubernetes Storage with Bassam Tabbara appeared first on Software Engineering Daily.

Jan 18, 2018
Kubernetes State Management with Niraj Tolia
55:26

A common problem in a distributed system: how do you take a snapshot of the global state of that system? Snapshot is difficult because you need to tell every node in the system to simultaneously record its state. There are several reasons to take a snapshot. You might want to take a picture of the

The post Kubernetes State Management with Niraj Tolia appeared first on Software Engineering Daily.

Jan 17, 2018
Kubernetes Operations with Brian Redbeard
52:09

In the last four years, CoreOS has been at the center of enterprise adoption of containers. During that time, Brian Harrington (or “Redbeard”) has seen a lot of deployments. In this episode, Brian discusses the patterns he has seen among successful Kubernetes deployments–and the pitfalls of the less successful. How should you manage configuration? How

The post Kubernetes Operations with Brian Redbeard appeared first on Software Engineering Daily.

Jan 16, 2018
FluentD with Eduardo Silva
53:03

A backend application can have hundreds of services written in different programming frameworks and languages. Across these different languages, log messages are produced in different formats. Some logging is produced in XML, some is produced in JSON, some is in other formats. These logs need to be unified into a common format, and centralized for

The post FluentD with Eduardo Silva appeared first on Software Engineering Daily.

Jan 15, 2018
The Gravity of Kubernetes
1:02:41

Kubernetes has become the standard way of deploying new distributed applications. Most new internet businesses started in the foreseeable future will leverage Kubernetes (whether they realize it or not). Many old applications are migrating to Kubernetes too. Before Kubernetes, there was no standardization around a specific distributed systems platform. Just like Linux became the standard

The post The Gravity of Kubernetes appeared first on Software Engineering Daily.

Jan 13, 2018
Kubernetes Vision with Brendan Burns
57:26

Kubernetes has become the standard system for deploying and managing clusters of containers. But the vision of the project goes beyond managing containers. The long-term goal is to democratize the ability to build distributed systems. Brendan Burns is a co-founder of the Kubernetes project. He recently announced an open source project called Metaparticle, a standard

The post Kubernetes Vision with Brendan Burns appeared first on Software Engineering Daily.

Jan 12, 2018
High Volume Distributed Tracing with Ben Sigelman
57:46

You are requesting a car from a ridesharing service such as Lyft. Your request hits the Lyft servers and begins trying to get you a car. It takes your geolocation, and passes the geolocation to a service that finds cars that are nearby, and puts all those cars into a list. The list of nearby

The post High Volume Distributed Tracing with Ben Sigelman appeared first on Software Engineering Daily.

Jan 11, 2018
Kubernetes on AWS with Arun Gupta
52:24

Since Kubernetes came out, engineers have been deploying clusters to Amazon. In the early years of Kubernetes, deploying to AWS meant that you had to manage the availability of the cluster yourself. You needed to configure etcd and your master nodes in a way that avoided having a single point of failure. Deploying Kubernetes on

The post Kubernetes on AWS with Arun Gupta appeared first on Software Engineering Daily.

Jan 10, 2018
Istio Motivations with Louis Ryan
57:05

A single user request hits Google’s servers. A user is looking for search results. In order to deliver those search results, that request will have to hit several different internal services on the way to getting a response. These different services work together to satisfy the user request. All of these services need to communicate

The post Istio Motivations with Louis Ryan appeared first on Software Engineering Daily.

Jan 09, 2018
Kubernetes Usability with Joe Beda
1:08:24

Docker was released in 2013, and popularized the use of containers. A container is an abstraction for isolating a well-defined portion of an operating system. Developers quickly latched onto containers as a way to cut down on the cost of virtual machines–as well as isolate code and simplify deployments. Developers began deploying so many containers

The post Kubernetes Usability with Joe Beda appeared first on Software Engineering Daily.

Jan 08, 2018
Cloud R&D with Onsi Fakhouri
58:03

In the first 10 years of cloud computing, a set of technologies emerge that every software enterprise needs; continuous delivery, version control, logging, monitoring, routing, data warehousing. These tools were built into the Cloud Foundry project, a platform for application deployment and management. As we enter the second decade of cloud computing, another new set

The post Cloud R&D with Onsi Fakhouri appeared first on Software Engineering Daily.

Jan 05, 2018
Cloud Foundry with Rupa Nandi
57:08

Cloud Foundry is an open-source platform as a service for deploying and managing web applications. Cloud Foundry is widely used by enterprises who are running applications that are built using Spring, a popular web framework for Java applications, but developers also use Cloud Foundry to manage apps built in Ruby, Node and any other programming

The post Cloud Foundry with Rupa Nandi appeared first on Software Engineering Daily.

Jan 03, 2018
High Volume Logging with Steve Newman
1:03:51

Google Docs is used by millions of people to collaborate on documents together. With today’s technology, you could spend a weekend coding and build a basic version of a collaborative text editor. But in 2004 it was not so easy. In 2004 Steve Newman built a product called Writely, which allowed users to collaborate on

The post High Volume Logging with Steve Newman appeared first on Software Engineering Daily.

Dec 15, 2017
Scala at Duolingo with Andre Kenji Horie
50:01

Duolingo is a language learning platform with over 200 million users. On a daily basis millions of users receive customized language lessons targeted specifically to them. These lessons are generated by a system called the session generator.   Andre Kenji Horie is senior engineer at Duolingo. He wrote about the process of rewriting the session

The post Scala at Duolingo with Andre Kenji Horie appeared first on Software Engineering Daily.

Dec 14, 2017
Cloud Marketplace with Zack Bloom
1:02:56

Ten years ago, if you wanted to build software, you probably needed to know how to write code. Today, the line between “technical” and “non-technical” people is blurring. Website designers can make a living building sites for people on WordPress or Squarespace–without knowing how to write code. Salesforce integration experts can help a sales team

The post Cloud Marketplace with Zack Bloom appeared first on Software Engineering Daily.

Dec 12, 2017
Scalable Multiplayer Games with Yan Cui
1:13:34

Remember when the best game you could play on your phone was Snake? In 1998, Snake was preloaded on Nokia phones, and it was massively popular. That same year Half-Life won game of the year on PC. Metal Gear Solid came out for Playstation. The first version of Starcraft also came out in 1998. In

The post Scalable Multiplayer Games with Yan Cui appeared first on Software Engineering Daily.

Dec 11, 2017
Decentralized Objects with Martin Kleppman
1:15:16

The Internet was designed as a decentralized system. Theoretically, if Alice wants to send an email to Bob, she can set up an email client on her computer and send that email to Bob’s email server on his computer. In reality, very few people run their own email servers. We all send our emails to

The post Decentralized Objects with Martin Kleppman appeared first on Software Engineering Daily.

Dec 08, 2017
Serverless Applications with Randall Hunt
46:59

Developers can build networked applications today without having to deploy their code to a server. These “serverless” applications are constructed from managed services and functions-as-a-service. Managed services are cloud offerings like database-as-a-service, queueing-as-a-service, or search-as-a-service. These managed services are easy to use. They take care of operational burdens like scalability and outages. But managed services

The post Serverless Applications with Randall Hunt appeared first on Software Engineering Daily.

Dec 07, 2017
Serverless Scheduling with Rodric Rabbah
1:10:13

Functions as a service are deployable functions that run without an addressable server. Functions as a service scale without any work by the developer. When you deploy a function as a service to a cloud provider, the cloud provider will take care of running that function whenever it is called. You don’t have to worry

The post Serverless Scheduling with Rodric Rabbah appeared first on Software Engineering Daily.

Dec 04, 2017
React and GraphQL at New York Times
58:15

Are we a media company or a technology company? Facebook and the New York Times are both asking themselves this question. Facebook originally intended to focus only on building technology–to be a neutral arbiter of information. This has turned out to be impossible. The Facebook newsfeed is defined by algorithms that are only as neutral

The post React and GraphQL at New York Times appeared first on Software Engineering Daily.

Nov 30, 2017
How IBM Runs Its Cloud with Jason McGee
1:00:41

Cloud computing changed the economics of running a software company. A cloud is a network of data centers that offers compute resources to developers. In the 1990s, software companies purchased servers–an upfront capital expense that required tens of thousands of dollars. In the early 2000s, cloud computing started, and turned that capital expense into an

The post How IBM Runs Its Cloud with Jason McGee appeared first on Software Engineering Daily.

Nov 29, 2017
Thumbtack Infrastructure with Nate Kupp
46:53

Thumbtack is a marketplace for real-world services. On Thumbtack, people get their house painted, their dog walked, and their furniture assembled. With 40,000 daily marketplace transactions, the company handles significant traffic. On yesterday’s episode, we explored how one aspect of Thumbtack’s marketplace recently changed, going from asynchronous matching to synchronous “instant” matching. In this episode,

The post Thumbtack Infrastructure with Nate Kupp appeared first on Software Engineering Daily.

Nov 28, 2017
Marketplace Matching with Xing Chen
56:42

The labor market is moving online. Taxi drivers are joining Uber and Lyft. Digital freelancers are selling their services through Fiverr. Experienced software contractors are leaving contract agencies to join Gigster. Online labor marketplaces create market efficiency by improving the communications between buyers and sellers. Workers make their own hours, and their performance is judged

The post Marketplace Matching with Xing Chen appeared first on Software Engineering Daily.

Nov 27, 2017
Load Balancing at Scale with Vivek Panyam
53:03

Facebook serves interactive content to billions of users. Google serves query requests on the world’s biggest search engine. Uber handles a significant percentage of the transportation within the United States. These services are handling radically different types of traffic, but many of the techniques they use to balance loads are similar. Vivek Panyam is an

The post Load Balancing at Scale with Vivek Panyam appeared first on Software Engineering Daily.

Nov 22, 2017
Incident Response with Emil Stolarsky
57:12

As a system becomes more complex, the chance of failure increases. At a large enough scale, failures are inevitable. Incident response is the practice of preparing for and effectively recovering from these failures. An engineering team can use checklists and runbooks to minimize failures. They can put a plan in place for responding to failures.

The post Incident Response with Emil Stolarsky appeared first on Software Engineering Daily.

Nov 21, 2017
Run Less Software with Rich Archbold
1:00:14

There is a quote from Jeff Bezos: “70% of the work of building a business today is undifferentiated heavy lifting. Only 30% is creative work. Things will be more exciting when those numbers are inverted.” That quote is from 2006, before Amazon Web Services had built most of their managed services. In 2006, you had

The post Run Less Software with Rich Archbold appeared first on Software Engineering Daily.

Nov 20, 2017
High Volume Event Processing with John-Daniel Trask
1:02:39

A popular software application serves billions of user requests. These requests could be for many different things. These requests need to be routed to the correct destination, load balanced across different instances of a service, and queued for processing. Processing a request might require generating a detailed response to the user, or making a write

The post High Volume Event Processing with John-Daniel Trask appeared first on Software Engineering Daily.

Nov 16, 2017
Fiverr Engineering with Gil Sheinfeld
59:47

As the gig economy grows, that growth necessitates innovations in the online infrastructure powering these new labor markets. In our previous episodes about Uber, we explored the systems that balance server load and gather geospacial data. In our coverage of Lyft, we studied Envoy, the service proxy that standardizes communications and load balancing among services.

The post Fiverr Engineering with Gil Sheinfeld appeared first on Software Engineering Daily.

Nov 15, 2017
Serverless Event-Driven Architecture with Danilo Poccia
57:44

In an event driven application, each component of application logic emits events, which other parts of the application respond to. We have examined this pattern in previous shows that focus on pub/sub messaging, event sourcing, and CQRS. In today’s show, we examine the intersection of event driven architecture and serverless architecture. Serverless applications can be

The post Serverless Event-Driven Architecture with Danilo Poccia appeared first on Software Engineering Daily.

Nov 14, 2017
Netflix Serverless-like Platform with Vasanth Asokan
59:10

The Netflix API is accessed by developers who build for over 1000 device types: TVs, smartphontes, VR headsets, laptops. If it has a screen, it can probably run Netflix. On each of these different devices, the Netflix experience is different. Different screen sizes mean there is variable space to display the content. When you open

The post Netflix Serverless-like Platform with Vasanth Asokan appeared first on Software Engineering Daily.

Nov 07, 2017
Serverless Authentication with Bobby Johnson
1:00:03

Serverless architecture is software that runs without an addressable server. Serverless is made possible by two types of technology: platform as a service providers like Auth0, and functions as a service like AWS Lambda. With both of these technologies, we can program logic that runs without being deployed to a server. Functions as a service

The post Serverless Authentication with Bobby Johnson appeared first on Software Engineering Daily.

Nov 06, 2017
Augmented Reality with Scott Montgomerie
59:10

Augmented reality applications are slowly making their way into the world of the consumer. Pokemon Go created the magical experience of seeing Pokemon superimposed upon the real world. IKEA’s mobile app lets you see how a couch would fit into your living room, which has a significant improvement on the furniture buying process. Augmented reality

The post Augmented Reality with Scott Montgomerie appeared first on Software Engineering Daily.

Nov 01, 2017
Elastic Load Balancing with Ranga Rajagopalan
57:02

Computational load is the amount of demand that is being placed on a computer system. “Load” can take the form of memory, CPU, network bandwidth, disk space, and other finite resources. When we design systems, we need to prepare for high-load events. On a social network, people are much more active in the mornings. On

The post Elastic Load Balancing with Ranga Rajagopalan appeared first on Software Engineering Daily.

Oct 31, 2017
IFTTT Architecture with Nicky Leach
59:21

It’s 9pm at night, and you are hungry. You order a pizza from Domino’s. You live on a street that’s dark, and so you have installed a smart lightbulb in front of your mailbox that lights up the address. When the pizza at Domino’s is ready, you want the lightbulb on your mailbox to light

The post IFTTT Architecture with Nicky Leach appeared first on Software Engineering Daily.

Oct 26, 2017
Quantum Computing Introduction with Zlatko Minev
1:00:30

Computer chips have physical limitations. When transistors get too small, electrons start to behave in ways that make the hardware modules less reliable. Our reliable technological progress has been enabled by Moore’s Law: the idea that the number of components we can fit on a chip doubles roughly every 12-18 months. We can’t keep shrinking

The post Quantum Computing Introduction with Zlatko Minev appeared first on Software Engineering Daily.

Oct 23, 2017
Internet Monitoring with Matt Kraning
58:12

How would you build a system for indexing and monitoring the entire Internet? Start by breaking the Internet up into IP address ranges. Give each of those address ranges to servers distributed around the world. On each of those servers, iterate through your list of IP addresses, sending packets to them. Depending on what sorts

The post Internet Monitoring with Matt Kraning appeared first on Software Engineering Daily.

Oct 17, 2017
Scala Native with Denys Shabalin
56:45

Scala is a functional and object oriented programming language built on the JVM. Scala Native takes this language, loved by many, and brings it to bare metal. Scala Native is an optimizing ahead-of-time compiler and lightweight managed runtime designed specifically for Scala. Denys Shabalin is a Research Assistant at the EPFL and the primary creator

The post Scala Native with Denys Shabalin appeared first on Software Engineering Daily.

Oct 16, 2017
Tinder Engineering Management with Bryan Li
58:33

Tinder is a rapidly growing social network for meeting people and dating. In the past few years, Tinder’s userbase has grown rapidly, and the engineering team has scaled to meet the demands of increased popularity. On Tinder, you are presented with a queue of suggested people that you might match with, and you swipe left

The post Tinder Engineering Management with Bryan Li appeared first on Software Engineering Daily.

Oct 09, 2017
Video Infrastructure with Matt McClure and Jon Dahl
58:52

Playing a video on the Internet seems simple. You press play, the video gets delivered, and boom–you are watching Game of Thrones, right? It’s a bit more complicated. Unless you have built an application that involves video, you probably have not dealt with the world of codecs, bitrates, and streaming. Depending on the bandwidth between

The post Video Infrastructure with Matt McClure and Jon Dahl appeared first on Software Engineering Daily.

Sep 27, 2017
Tinder Growth Engineering with Alex Ross
1:01:20

Tinder is a popular dating app where each user swipes through a sequence of other users in order to find a match. Swiping left means you are not interested. Swiping right means you would like to connect with the person. The simple premise of Tinder has led to massive growth, and the app is now

The post Tinder Growth Engineering with Alex Ross appeared first on Software Engineering Daily.

Sep 21, 2017
Spotify Event Delivery with Igor Maravic
1:01:00

Spotify is a streaming music company with more than 50 million users. Whenever a user listens to a song, Spotify records that event and uses it as input to learn more about the user’s preferences. Listening to a song is one type of event–there are hundreds of others. Opening the Spotify app, skipping a song,

The post Spotify Event Delivery with Igor Maravic appeared first on Software Engineering Daily.

Sep 18, 2017
Cloud-Native SQL with Alex Robinson
1:01:04

Applications built in the cloud are often serving requests from all around the world. A user in Hong Kong could have written to a database entry at the moment just before a user in San Francisco and a user in Germany simultaneously try to read from that database. If the user in San Francisco is

The post Cloud-Native SQL with Alex Robinson appeared first on Software Engineering Daily.

Aug 28, 2017
Error Diagnosis with James Smith
58:04

When a user experiences an error in an application, the engineers who are building that application need to find out why that error occurred. The root cause of that error may be on the user’s device, or within a piece of server-side logic, or hidden behind a black box API. To fix a complex error,

The post Error Diagnosis with James Smith appeared first on Software Engineering Daily.

Aug 18, 2017
Open Compute Project with Steve Helvie
59:10

Facebook was rapidly outgrowing its infrastructure in 2009. Classic data center design was not up to the task of the rapid influx of new users and data, photos and streaming video hitting Facebook’s servers. A small team of engineers spent the next two years designing a data center from the ground up to be cheaper,

The post Open Compute Project with Steve Helvie appeared first on Software Engineering Daily.

Aug 14, 2017
Serverless Continuous Delivery with Robin Weston
1:02:10

Serverless computing reduces the cost of using the cloud. Serverless also makes it easy to scale applications. The downside: building serverless apps requires some mindset shift. Serverless functions are deployed to transient units of computation that are spun up on demand. This is in contrast to the typical model of application delivery–the deployment of an

The post Serverless Continuous Delivery with Robin Weston appeared first on Software Engineering Daily.

Aug 07, 2017
Serverless Startup with Yan Cui
57:56

After raising $18 million, social networking startup Yubl made a series of costly mistakes. Yubl hired an army of expensive contractors to build out its iOS and Android apps. Drama at the executive level hurt morale for the full-time employees. Most problematic, the company was bleeding cash due to a massive over-investment in cloud services.

The post Serverless Startup with Yan Cui appeared first on Software Engineering Daily.

Aug 04, 2017
Platform Continuous Delivery with Andy Appleton
56:24

Continuous delivery is a model for deploying small, frequent changes to an application. In a continuous delivery workflow, code changes that are pushed to a repository set off a build process that spins up a new version of the application. Testing is performed against that new build before advancing it to production, merging it with

The post Platform Continuous Delivery with Andy Appleton appeared first on Software Engineering Daily.

Aug 02, 2017
Reinforcement Learning with Michal Kempa
45:19

Reinforcement learning is a type of machine learning where a program learns how to take actions in an environment based on how that program has been rewarded for actions it took in the past. When program takes an action, and it receives a reward for that action, it is likely to take that action again

The post Reinforcement Learning with Michal Kempa appeared first on Software Engineering Daily.

Jul 21, 2017
Apparel Machine Learning with Colan Connon and Thomas Bell
55:53

In its most basic definition, machine learning is a tool that makes takes a data set, finds a correlation in that data set, and uses that correlation to improve a system. Any complex system with well-defined behavior and clean data can be improved with machine learning. Several precipitating forces have caused machine learning to become

The post Apparel Machine Learning with Colan Connon and Thomas Bell appeared first on Software Engineering Daily.

Jul 20, 2017
Backups with Kenny To
1:02:19

Every software company backs up critical data sources. Backing up databases is a common procedure, whether a company is in the cloud or on-prem. Backing up virtual machine instances is less common. Rubrik is a company that is known for building backup infrastructure for enterprises. Their main product is an appliance that sits on prem

The post Backups with Kenny To appeared first on Software Engineering Daily.

Jul 18, 2017
Deployment with Avi Cavale
1:03:13

Software deployment evolves over time. In the 90s, a “deployment” might have meant issuing a new edition of your software via CD-ROM. Today, a deployment is often a multi-stage process. A new software build will undergo automated unit tests and integration tests, before being deployed to users.  The deployment might only go out to a

The post Deployment with Avi Cavale appeared first on Software Engineering Daily.

Jul 11, 2017
Kafka in the Cloud with Neha Narkhede
59:01

Apache Kafka is an open-source distributed streaming platform. Kafka was originally developed at LinkedIn, and the creators of the project eventually left LinkedIn and started Confluent, a company that is building a streaming platform based on Kafka. Kafka is very popular, but is not easy to deploy and operationalize. That is why Confluent has built

The post Kafka in the Cloud with Neha Narkhede appeared first on Software Engineering Daily.

Jul 10, 2017
Istio Service Mesh with Varun Talwar and Louis Ryan
48:36

Modern software applications are often built out of loosely coupled microservices. These services can be written in different languages, by different people, but communication between services needs to be standardized. For this reason, a service proxy is useful. A service proxy is a sidecar container that sits next to a service and facilitates communications with

The post Istio Service Mesh with Varun Talwar and Louis Ryan appeared first on Software Engineering Daily.

Jun 27, 2017
Service Mesh with William Morgan
59:05

Containers make it easier for engineers to deploy software. Orchestration systems like Kubernetes make it easier to manage and scale the different containers that contain services. The popular container infrastructure powered by Kubernetes is often called “cloud native.” On Software Engineering Daily, we have been exploring cloud native software to get a complete picture of

The post Service Mesh with William Morgan appeared first on Software Engineering Daily.

Jun 26, 2017
Software Architecture with Simon Brown
44:01

Software architecture address the challenge of communicating and navigating large, complex systems to stakeholders, both technical and non-technical.  Over the years software architecture has gone in and out of fashion.  Today we discuss why software architecture is important, what it means to have software architecture, and how to properly structure teams and incorporate architecture. Today’s

The post Software Architecture with Simon Brown appeared first on Software Engineering Daily.

Jun 20, 2017
IoT Edge with Olivier Bloch
57:23

A self-driving car needs to be able to quickly respond to changes in driving conditions. A factory needs to be able to quickly respond to changes in workplace safety. For these kinds of applications, we need processing power closer to the user of the application. If we put all of our application logic in the

The post IoT Edge with Olivier Bloch appeared first on Software Engineering Daily.

Jun 19, 2017
Google Early Days with John Looney
1:11:02

John Looney spent more than 10 years at Google. He started with infrastructure, and was part of the team that migrated Google File System to Colossus, the successor to GFS. Imagine migrating every piece of data on Google from one distributed file system to another. In this episode, John sheds light on the engineering culture

The post Google Early Days with John Looney appeared first on Software Engineering Daily.

Jun 16, 2017
Container Engines with David Aronchick and Chen Goldberg
51:10

Kubernetes makes it easier for engineering teams to manage their distributed systems architecture. But it’s still not simple to deploy and operate a Kubernetes cluster. Google Container Engine (GKE) is a managed control plane for Kubernetes. Just as developers can use Google App Engine to easily deploy monolithic apps against a platform as a service,

The post Container Engines with David Aronchick and Chen Goldberg appeared first on Software Engineering Daily.

Jun 08, 2017
DNS with Phil Stanhope
1:02:28

DNS stands for domain name system. This is the naming system that maps the entire internet. It associates information with domain names. More specifically, DNS specifies mappings between numerical IP addresses and domain names. Most engineers know these basic facts about DNS, but they may not know how much engineering a complex company like Etsy

The post DNS with Phil Stanhope appeared first on Software Engineering Daily.

Jun 06, 2017
GitLab with Pablo Carranza
56:03

On January 31st 2017, GitLab experienced a major outage of their online repository hosting service. The primary database server experienced data loss due to a combination of malicious spam attacks and engineering mistakes that occurred while trying to respond to those spam attacks. GitLab responded to the event transparently. The company put up a postmortem

The post GitLab with Pablo Carranza appeared first on Software Engineering Daily.

Jun 02, 2017
Healthcare Engineering with Isaac Councill
55:10

Healthcare is a complex business. Oscar is a company that wanted to build a new insurance provider–but realized that healthcare is so interconnected that in order to build a new insurance provider, realized it actually needed to build an entire healthcare business too, complete with patient management and facilities. Since Oscar is a modern technology

The post Healthcare Engineering with Isaac Councill appeared first on Software Engineering Daily.

May 23, 2017
Microservices Transition with Cassandra Shum
49:02

Many companies are transitioning from a monolith to microservices architecture. Tools for cloud computing, containerization, and continuous delivery are making this easier. But there are still technological and organizational challenges that a company will encounter while making this transition. Cassandra Shum is an engineer with ThoughtWorks. She has worked with major financial institutions and other

The post Microservices Transition with Cassandra Shum appeared first on Software Engineering Daily.

May 22, 2017
Firebase with Doug Stevenson
52:55

Firebase is a backend-as-a-service. The key efficiency of a backend-as-a-service is that it enables developers to go from having a 3-tier architecture (client, server, database) to a 2-tier architecture (client, backend-as-a-service). The team who started Firebase built it as a pivot. They had started a social network, and then they realized there wasn’t a good

The post Firebase with Doug Stevenson appeared first on Software Engineering Daily.

May 17, 2017
Spring Boot with Josh Long
41:50

Spring Framework is an application framework for Java and JVM languages. Spring was originally built around dependency injection, but grew to become an entire ecosystem of tools and plugins for Java developers. Spring was originally released 15 years ago, and since then a lot has changed around application development. For example, many engineers deploy applications

The post Spring Boot with Josh Long appeared first on Software Engineering Daily.

Apr 26, 2017
Microservices Practitioners with Austin Gunter and Richard Li
58:03

The word “microservices” started getting used after a series of events–companies were moving to cloud virtual machines. Those VMs got broken up into containers, and the containers can fit to the size of the service. Services that are more narrowly defined take up smaller containers, and can be packed more densely into the virtual machines–hence

The post Microservices Practitioners with Austin Gunter and Richard Li appeared first on Software Engineering Daily.

Apr 20, 2017
Elasticsearch with Philipp Krenn
58:30

Search is a common building block for applications. Whether we are searching Wikipedia or our log files, the behavior is similar: a query is entered and the most relevant documents are returned. The core data structure for search is an inverted index. Elasticsearch is a scalable, resilient search tool that shards and replicates a search

The post Elasticsearch with Philipp Krenn appeared first on Software Engineering Daily.

Apr 12, 2017
API Design Standards with Andy Beier
54:17

There are various standards at play when creating and consuming Application Program Interfaces (APIs).  These standards, though, are mostly technical and mostly lower-level than the content of the API. Andy Beier has experienced the broad range of API quality in his role with Domo in creating integrations with other businesses.  He has made standardization of

The post API Design Standards with Andy Beier appeared first on Software Engineering Daily.

Apr 05, 2017
Failure Injection with Kolton Andrus
54:17

Servers in a data center fail. Sometimes entire data centers have a power outage. Bugs in an application make it into production. Human operators make mistakes and cause data to be deleted. Failure is unavoidable. We make backups and replicate our servers so that when a failure occurs, we can quickly respond to it without

The post Failure Injection with Kolton Andrus appeared first on Software Engineering Daily.

Mar 29, 2017
Software Psychology with Bjorn Freeman Benson
54:44

Designers and software engineers need to communicate with each other. From Apple to Slack to Uber, the emphasis on visual design within a product is rising in importance. Much like development and operations siloes have been bridged with the DevOps movement, design and engineering teams are working more closely together to align the vision of

The post Software Psychology with Bjorn Freeman Benson appeared first on Software Engineering Daily.

Mar 28, 2017
Stripe Infrastructure with Evan Broder
49:41

If you are building a service that processes payments, your software architecture has a lot of requirements. Not only do you need to be highly available, consistent, and fast–you need to be PCI compliant. In this episode, we explore the infrastructure of Stripe with Evan Broder, who has been with the company for five years.

The post Stripe Infrastructure with Evan Broder appeared first on Software Engineering Daily.

Mar 16, 2017
Stripe Observability with Cory Watson
1:06:50

Observability allows engineers to understand what is going on inside their systems. In its most raw form, observability comes from log data. Modern systems have many layers of logs–virtualized cloud infrastructure, container orchestration, the container runtime itself, and the application logic running within the container. With all of these layers, it is not practical for

The post Stripe Observability with Cory Watson appeared first on Software Engineering Daily.

Mar 15, 2017
Using CQRS to Make Controllers Lean with Derek Comartin
49:53

Command Query Responsibility Segregation (CQRS) is a powerful concept that has the potential to make for reliable and maintainable systems.  It is also broadly misunderstood and means different things to different people. Derek Comartin learned about the idea after viewing some talks by Greg Young and has since successfully applied the approach with great success

The post Using CQRS to Make Controllers Lean with Derek Comartin appeared first on Software Engineering Daily.

Mar 10, 2017
Load Testing with Mark Gilbert
51:03

Load testing measures performance of a system undergoing a large volume of requests. Before an application is pushed to production, engineers will often load test their software to ensure it is resilient in the face of high traffic. As web applications have changed, the requirements around load testing have changed as well. External APIs, internal

The post Load Testing with Mark Gilbert appeared first on Software Engineering Daily.

Mar 08, 2017
Parse and Operations with Charity Majors
1:05:32

Parse was a backend as a service company built in 2011 before being acquired by Facebook in 2013. Building a backend as a service for developers requires walking a thin line between giving engineers lots of control and preventing those engineers from shooting themselves in the foot. While she was at Parse, Charity Majors learned

The post Parse and Operations with Charity Majors appeared first on Software Engineering Daily.

Mar 01, 2017
Heroku Autoscaling with Andrew Gwozdziewycz
59:35

When an application is using all of its available resources, that application needs to be scaled. Scaling an application means giving it more resources–typically servers. Autoscaling is an engineering practice where an application is automatically given more or less resources based on how healthy the application performance is at a given time. Applications on Heroku

The post Heroku Autoscaling with Andrew Gwozdziewycz appeared first on Software Engineering Daily.

Feb 28, 2017
Data Warehousing with Mark Rittman
58:45

In the mid 90s, data warehousing might have meant “using an Oracle database.” Today, it means a wide variety of things. You could be stitching together a big data pipeline using Kafka, Hadoop, and Spark. You could be using managed tools like BigQuery from Google. How did we get from the simple days of Oracle

The post Data Warehousing with Mark Rittman appeared first on Software Engineering Daily.

Feb 27, 2017
Service Proxying with Matt Klein
56:18

Most tech companies are moving toward a highly distributed microservices architecture. In this architecture, services are decoupled from each other and communicate with a common service language, often JSON over HTTP. This provides some standardization, but these companies are finding that more standardization would come in handy. At the ridesharing company Lyft, every internal service

The post Service Proxying with Matt Klein appeared first on Software Engineering Daily.

Feb 14, 2017
Infrastructure with Datanauts’ Chris Wahl and Ethan Banks
49:06

Infrastructure is a term that can mean many different things: your physical computer, the data center of your Amazon EC2 cluster, the virtualization layer, the container layer–on and on. In today’s episode, podcasters Chris Wahl and Ethan Banks discuss the past, present, and future of infrastructure with me. Ethan and Chris host Datanauts, a podcast

The post Infrastructure with Datanauts’ Chris Wahl and Ethan Banks appeared first on Software Engineering Daily.

Feb 13, 2017
Giphy Engineering with Anthony Johnson
56:52

Giphy is a search engine for gifs, the short animated graphics that we see around the Internet. Giphy is also a creative platform where people create new gifs. Every search engine requires the construction of a search index, which is a data structure that responds to search queries efficiently. Since Giphy is a search engine

The post Giphy Engineering with Anthony Johnson appeared first on Software Engineering Daily.

Feb 06, 2017
Twilio Engineering with Pat Malatack
59:24

Back in 2008, the range of tools that engineers could use to connect computer systems together were getting quite good. Cloud computing was democratizing access to servers. But the telephony ecosystem was still inaccessible to the average developer. If you needed your program to make a phone call and connect a user to a customer

The post Twilio Engineering with Pat Malatack appeared first on Software Engineering Daily.

Jan 31, 2017
Email Infrastructure with Chris McFadden
1:02:46

A company like Pinterest has millions of transactional emails to send to people. The scalability challenges of sending high volumes of email mean that it makes more sense for most companies to use an email as a service product rather than building their own. Chris McFadden is the VP of engineering and cloud operations at SparkPost

The post Email Infrastructure with Chris McFadden appeared first on Software Engineering Daily.

Jan 10, 2017
Meetup Architecture with Yvette Pasqua
58:36

Meetup is an online service that allows people to gather into groups and meet in person. Since 2002, the company has been growing and its technology stack has been changing. Today, they are in the process of migrating to the cloud, using both Amazon Web Services and Google Compute Platform. Yvette Pasqua is the CTO

The post Meetup Architecture with Yvette Pasqua appeared first on Software Engineering Daily.

Jan 06, 2017
Evolutionary Architecture with Neal Ford
57:25

When a useful new technology comes out, companies that are in a position to adopt that new technology can gain an edge over competitors. As our industry grows and moves faster, these kinds of changes are coming faster–some recent examples are Docker, ReactJS, and Kubernetes. Evolutionary architecture supports incremental, guided change as a first principle

The post Evolutionary Architecture with Neal Ford appeared first on Software Engineering Daily.

Jan 05, 2017
Self-Contained Systems with Eberhard Wolff
58:48

Self-contained systems is an architectural approach that separates the functionality of a system into many independent systems. Each self-contained system is an autonomous web application, and is owned by one team. Communication with other self-contained systems or 3rd party systems is asynchronous where possible. As Eberhard Wolff explains in this episode, self-contained systems is not

The post Self-Contained Systems with Eberhard Wolff appeared first on Software Engineering Daily.

Jan 03, 2017
Performance Monitoring with Andi Grabner
1:00:23

Application performance monitoring helps an engineer understand what is going on with an application. An application on a single machine is often monitored by inserting bytecode instructions into the application after it has been interpreted. Distributed cloud applications with functionality broken up across multiple servers often use distributed tracing. Andi Grabner from Dynatrace joins today’s

The post Performance Monitoring with Andi Grabner appeared first on Software Engineering Daily.

Dec 27, 2016
Antifraud Architecture with Josh Yudaken
56:54

Online marketplaces and social networks often have a trust and safety team. The trust and safety team helps protect the platform from scams, fraud, and malicious actors. To detect these bad actors at scale requires building a system that classifies every transaction on the platform as safe or potentially malicious. Since every social platform has

The post Antifraud Architecture with Josh Yudaken appeared first on Software Engineering Daily.

Dec 23, 2016
Reactive Microservices with Jonas Boner
1:01:10

For many years, software companies have been breaking up their applications into individual services for the purpose of isolation and maintainability. In the early 2000s, we called this pattern “service-oriented architecture”. Today we call it “microservices”. Why did we change that terminology? Did the services get smaller? Not exactly. Jonas Boner suggests that the movement

The post Reactive Microservices with Jonas Boner appeared first on Software Engineering Daily.

Dec 19, 2016
Scale API with Lucy Guo and Alexandr Wang
51:26

Some tasks are simple, but cannot be performed by a computer. Audio transcription, image recognition, survey completion–these are simple procedures that almost any human could execute, but the machine learning models have not gotten consistent enough to do them accurately. Scale is an API for human labor, created by Lucy Guo and Alexandr Wang. Similar

The post Scale API with Lucy Guo and Alexandr Wang appeared first on Software Engineering Daily.

Dec 16, 2016
Netflix Caching with Scott Mansfield
53:12

Caching is a fundamental concept of computer science. When data is accessed frequently, we put that data in a place where it can be accessed more quickly–we put the data in a cache. When data is accessed less often, we leave it in a place where the access time is slow or expensive. Netflix has

The post Netflix Caching with Scott Mansfield appeared first on Software Engineering Daily.

Dec 09, 2016
Developer Tools with Josh Varty
48:03

When you are working on a program, a lot of things are going through your head. In some sense, you become part machine when you are programming. Learnable Programming is a concept that facilitates this, by showing developers what the computer is doing in real time, before compiling. In this episode, Josh Varty, co-founder of

The post Developer Tools with Josh Varty appeared first on Software Engineering Daily.

Dec 07, 2016
Microservices with Rafi Schloming
48:29

Microservices are a widely adopted pattern for breaking an application up into pieces that can be well-understood by the individual teams within the company. Microservices also allow these individual pieces to be scaled independently and updated in isolation. Past Software Engineering Daily episodes have covered the microservice architectures of Twitter, Netflix, Google, Uber and other

The post Microservices with Rafi Schloming appeared first on Software Engineering Daily.

Nov 22, 2016
Slack Bots with Amir Shevat
58:01

Slack is a chat client that has reached wide adoption. The rise of Slack has coincided with the rise of chatbots. A chatbot is a simple, conversational interface into a computer program that may have simple functionality, like telling you some simple statistics, or more complex functionality, like helping you manage your continuous integration pipeline.

The post Slack Bots with Amir Shevat appeared first on Software Engineering Daily.

Nov 16, 2016
AWS Open Guide with Joshua Levy
1:07:01

Amazon Web Services changed the economics of building an internet application. Instead of having to invest tens of thousands of dollars up front for hardware, developers can pay for services over time as their application scales. As AWS has grown to be a gigantic platform, the documentation about how to use cloud infrastructure has become

The post AWS Open Guide with Joshua Levy appeared first on Software Engineering Daily.

Nov 14, 2016
Infrastructure Mistakes with Avi Freedman
1:05:10

The blueprint for a typical startup involves investing heavily in cloud services–either from Amazon, Google, or Microsoft. The high costs can quickly eat away at all of the money that startup has raised. In today’s episode, Avi Freedman outlines some of the infrastructure mistakes that can set back a company severely–cloud jail, hipster tools, and

The post Infrastructure Mistakes with Avi Freedman appeared first on Software Engineering Daily.

Nov 11, 2016
ChatOps with Jason Hand
57:36

Chat bots are your newest co-worker. Slack, HipChat, and other chat clients allow developers and other team members to communicate more dynamically than the limits of email. Companies have started to add bots to their chat rooms. These bots can give you technical information, restart a server, or notify you that a build has finished.

The post ChatOps with Jason Hand appeared first on Software Engineering Daily.

Nov 02, 2016
Managed Kafka with Tom Crayford
51:38

Kafka is a distributed log for producers and consumers to publish messages to each other. We’ve done many shows about Kafka as a key building block for distributed systems, but we often leave out the discussion of the complexities of setting up Kafka and monitoring it. Kafka deployments can be a complex piece of software

The post Managed Kafka with Tom Crayford appeared first on Software Engineering Daily.

Oct 25, 2016
Google Cloudbuilding with Joe Beda
1:00:47

Google Compute Engine is the public cloud built by Google. It provides infrastructure- and platform-as-a-service capabilities that rival Amazon Web Services. Today’s guest Joe Beda was there from the beginning of GCE, and he was also one of the early engineers on the Kubernetes project. Google’s internal systems have made it easy for employees to

The post Google Cloudbuilding with Joe Beda appeared first on Software Engineering Daily.

Oct 20, 2016
Docker Cloudcasting with Brian Gracely
1:04:25

Cloud computing was something much different in 2011, when Brian Gracely and Aaron Delp started The Cloudcast, a podcast I listen to on a regular basis. The Cloudcast features technical discussions about cloud infrastructure technology, and one of the most recent shows was a monologue by Brian Gracely where he explained his perspective on the

The post Docker Cloudcasting with Brian Gracely appeared first on Software Engineering Daily.

Oct 19, 2016
Kafka Event Sourcing with Neha Narkhede
1:01:27

When a user of a social network updates her profile, that profile update needs to propagate to several databases that want to know about such an update–search indexes, user databases, caches, and other services. When Neha Narkhede was at LinkedIn, she helped develop Kafka, which was deployed at LinkedIn to help solve this very problem.

The post Kafka Event Sourcing with Neha Narkhede appeared first on Software Engineering Daily.

Oct 14, 2016
DevOps Handbook with Gene Kim
52:46

The intent of the DevOps movement is to get organizations moving faster and more effectively by breaking down siloes, and improving communication. Gene Kim’s book The Phoenix Project illustrated this by telling the fictional story of a company adopting a DevOps mentality. Although that book was fiction, Gene is an experienced engineer, having worked as

The post DevOps Handbook with Gene Kim appeared first on Software Engineering Daily.

Oct 13, 2016
Netflix Scheduling with Sharma Podila
50:48

  At Netflix, developers write applications with a variety of requirements–from simple requests for a list of movies to more resource-intensive requests like a complex machine learning workflow. Netflix wants developers to be able to request the resources they need from a compute cluster and receive those resources on-demand, without thinking too much about the

The post Netflix Scheduling with Sharma Podila appeared first on Software Engineering Daily.

Oct 12, 2016
Monitoring Architecture with Theo Schlossnagle
58:57

Building a monitoring system is a complex distributed systems problem. Events are produced from different points in an application and must be aggregated in order to form metrics. These events are often ingested by a time series database, which forms the backbone of our monitoring system. Theo Schlossnagle is the CEO of Circonus, where he

The post Monitoring Architecture with Theo Schlossnagle appeared first on Software Engineering Daily.

Oct 11, 2016
Continuous Delivery with David Rice
53:46

In order to move software updates from the development team to production, companies do a variety of things. Some teams might email files to each other or use FTP or even floppy disks. Most companies today at least use version control systems like Git together with separate servers for development and production. When code is

The post Continuous Delivery with David Rice appeared first on Software Engineering Daily.

Oct 10, 2016
Kafka Streams with Jay Kreps
1:00:19

Kafka Streams is a library for building streaming applications that transform input Kafka topics into output Kafka topics. In a time when there are numerous streaming frameworks already out there, why do we need yet another? To quote today’s guest Jay Kreps “the gap we see Kafka Streams filling is less the analytics-focused domain these

The post Kafka Streams with Jay Kreps appeared first on Software Engineering Daily.

Oct 07, 2016
Platform as a Service with Sinclair Schuller
59:32

Platform as a service can mean different things to different people. The most prominent feature of a PaaS is the ability to abstract away issues that every developer within an organization has to deal with. As an example, developers today don’t need to fear scalability and load balancing issues as much as engineers of the

The post Platform as a Service with Sinclair Schuller appeared first on Software Engineering Daily.

Oct 04, 2016
Cloud Clients with Jon Skeet
1:03:06

Google builds cloud services for developers, such as PubSub, Cloud Storage, BigQuery, and Cloud DataStore. On Software Engineering Daily, we’ve done lots of shows about how these types of services are built. In this episode, we are zooming in on the interaction between the developer using a cloud service and the design and engineering of

The post Cloud Clients with Jon Skeet appeared first on Software Engineering Daily.

Sep 20, 2016
Cloud Dataflow with Eric Anderson
1:03:23

Batch and stream processing systems have been evolving for the past decade. From MapReduce to Apache Storm to Dataflow, the best practices for large volume data processing have become more sophisticated as the industry and open source communities have iterated on them.   Dataflow and Apache Beam are projects that present a unified batch and

The post Cloud Dataflow with Eric Anderson appeared first on Software Engineering Daily.

Sep 16, 2016
Slack’s Architecture with Keith Adams
1:00:34

Slack is a chat application that is rapidly growing in popularity. The focus of Slack is to create a polished, responsive tool for productivity that cuts down on the emailing, context switching, and useless meetings that take place at a typical enterprise.   Keith Adams, the chief architect at Slack, joins the show to explain

The post Slack’s Architecture with Keith Adams appeared first on Software Engineering Daily.

Sep 12, 2016
Uber’s Ringpop with Jeff Wolski
59:21

Uber has a software architecture with unique requirements. Uber does not have the firehose of user engagement data that Twitter or Facebook has, but each transaction on Uber is both high value and time-sensitive. Users are paying for transportation that they expect to be available and reasonably close by. When Uber’s system is trying to

The post Uber’s Ringpop with Jeff Wolski appeared first on Software Engineering Daily.

Aug 26, 2016
Kubernetes Migration with Sheriff Mohamed
54:40

Kubernetes is a cluster management tool open sourced by Google. On Software Engineering Daily, we’ve done numerous shows on how Kubernetes works in theory. Today’s episode is a case study in how to deploy Kubernetes to production at a company with existing infrastructure.   GolfNow is a fifteen year-old application written in C# .NET. It

The post Kubernetes Migration with Sheriff Mohamed appeared first on Software Engineering Daily.

Aug 25, 2016
Distributed Tracing with Reshmi Krishna
50:12

In a microservices architecture, a user request will often make its way through several different services before it returns a result to the end user. If a user experiences a failed request, the root cause could be in any of the services along that request path. Even more problematic is the challenge of debugging latency

The post Distributed Tracing with Reshmi Krishna appeared first on Software Engineering Daily.

Aug 24, 2016
Serverless Architecture with Mike Roberts
55:25

“Serverless” usually refers to an architectural pattern where the server side logic is run in stateless compute containers that are event-triggered and ephemeral. Mike Roberts has written a series of articles about serverless computing, in which he discusses theories and patterns around serverless architecture. In this episode, Mike and I discuss how to reimagine our

The post Serverless Architecture with Mike Roberts appeared first on Software Engineering Daily.

Aug 23, 2016
Apache Beam with Frances Perry
1:00:49

Unbounded data streams create difficult challenges for our application architectures. The data never stops coming, and we are forced to assume that we will never know if or when we have seen all of our data. Some streaming systems give us the tools to deal partially with unbounded data streams, but we have to complement

The post Apache Beam with Frances Perry appeared first on Software Engineering Daily.

Aug 19, 2016
Prometheus Monitoring with Brian Brazil
56:31

Prometheus is a tool for monitoring our distributed applications. It allows us to focus on the services we are deploying rather than the individual machines that make up instances of that service.   The monitoring service itself is a portion of a distributed system that is treated differently than the services we are monitoring. We

The post Prometheus Monitoring with Brian Brazil appeared first on Software Engineering Daily.

Aug 10, 2016
The Art of Monitoring with James Turnbull
1:04:57

Monitoring translates machine data into actionable business metrics, and is a key component of a modern software company. James Turnbull’s new book “The Art of Monitoring” describes how organizations can build their monitoring infrastructure.     James joins the show today to outline the strategies that a company can use to proactively monitor their systems.

The post The Art of Monitoring with James Turnbull appeared first on Software Engineering Daily.

Jul 28, 2016
Scalable Architecture with Lee Atchison
55:47

Lee Atchison spent seven years at Amazon working in retail, software distribution, and Amazon Web Services. He then moved to New Relic, where he has spent four years scaling the company’s internal architecture. From his decade of experience at fast growing web technology companies, Lee has written the book Architecting for Scale, from O’Reilly.  

The post Scalable Architecture with Lee Atchison appeared first on Software Engineering Daily.

Jul 09, 2016
Schedulers with Adrian Cockcroft
58:08

Scheduling is the method by which work is assigned to resources to complete that work. At the operating system level, this can mean scheduling of threads and processes. At the data center level, this can mean scheduling Hadoop jobs or other workflows that require the orchestration of a network of computers. Adrian Cockcroft worked on

The post Schedulers with Adrian Cockcroft appeared first on Software Engineering Daily.

Jul 07, 2016
Cloud Providers with Don Pezet
56:54

In 1999, it took $50,000 to buy a server. Once you bought that server, you had to know how to operate and maintain it. Today, cloud service providers have changed how we build software. Servers, load balancers, networking, storage–these hardware concerns have been turned into software. Don Pezet joins the show today to discuss the

The post Cloud Providers with Don Pezet appeared first on Software Engineering Daily.

Jul 05, 2016
Scaling Twitter with Buoyant.io’s William Morgan
59:55

Six years ago, Twitter was experiencing outages due to high traffic. Back in 2010 Twitter was built as a monolithic Ruby on Rails application. Twitter migrated to a microservices architecture to fix these problems. During this migration, the engineers at Twitter learned how to build and scale highly distributed microservice architectures. William Morgan was an

The post Scaling Twitter with Buoyant.io’s William Morgan appeared first on Software Engineering Daily.

Jun 23, 2016
Manufacturing and Microservices with Cimpress’ Jim Sokoloff and Maarten Wensveen
57:01

Mass customization is the process of making customized, personalized products that are accessible to individuals and small businesses. The process involves manufacturing, assembly lines, supply chains, and software at every step along the way. Today’s guests are Jim Sokoloff and Maarten Wensveen, who work on infrastructure and technology at Cimpress, a mass customization platform. Cimpress

The post Manufacturing and Microservices with Cimpress’ Jim Sokoloff and Maarten Wensveen appeared first on Software Engineering Daily.

Jun 22, 2016
Serverless Code with Ryan Scott Brown
59:30

The unit of computation has evolved from on premise servers to virtual machines in the cloud to containers running in those virtual machines. Serverless computation is another stage in the evolution of computational unit management. With a serverless architecture, a function call to the cloud spins up a transient container, calls the function on that

The post Serverless Code with Ryan Scott Brown appeared first on Software Engineering Daily.

Jun 21, 2016
Google’s Site Reliability Engineering with Todd Underwood
57:13

Google’s site reliability engineers are responsible for maintaining the highly available services that power the Google software that we all use on a regular basis. O’Reilly recently published the book “Site Reliability Engineering: How Google Runs Production Systems”, and the book provides a comprehensive window into how the site reliability engineering role works. Todd Underwood

The post Google’s Site Reliability Engineering with Todd Underwood appeared first on Software Engineering Daily.

Jun 15, 2016
Dropbox’s Magic Pocket with James Cowling
54:55

Dropbox has been storing files on Amazon Web Services for 8 years, and Dropbox’s core business is storing files. For the past three years, Dropbox has been working on a project to migrate its file storage from Amazon Web Services to its own custom built infrastructure. Magic Pocket is the name of Dropbox’s new infrastructure

The post Dropbox’s Magic Pocket with James Cowling appeared first on Software Engineering Daily.

May 18, 2016
Distributed Systems Tradeoffs with Camille Fournier
59:21

Distributed systems products are often marketed with terms like “real-time data” and “hassle-free scaling”, but what do those terms actually mean? Is data in a distributed system ever reliably “real time”? Do we ever have strong enough plans about our scalability strategy to say that scaling will be “hassle free”? Camille Fournier joins us today

The post Distributed Systems Tradeoffs with Camille Fournier appeared first on Software Engineering Daily.

May 16, 2016
Distributed Systems and Exception Monitoring with Brian Rue
37:26

Exception monitoring services and log management services are two sides of a gradient. Exception monitoring services capture and aggregate the problems that occur on your application. Log management services aggregate all of your logs, so that you can decide for yourself what constitutes a problem. Brian Rue from Rollbar joins the show today to talk

The post Distributed Systems and Exception Monitoring with Brian Rue appeared first on Software Engineering Daily.

Apr 29, 2016
Google’s Container Management with Brendan Burns
46:29

Kubernetes is an open source system for automating deployment, operations, and scaling of containerized applications. Google developed Kubernetes after fifteen years of running containers in production. Brendan Burns is a founder of the Kubernetes project, and he joins us to talk about the lessons learned as Google has built containerized applications to distribute across its

The post Google’s Container Management with Brendan Burns appeared first on Software Engineering Daily.

Apr 20, 2016
Search as a Service with Julien Lemoine
52:02

“You need to build more things yourself to be highly available, but one of the very good consequences of being bare metal is that the prices are very low compared to what you could get on the cloud provider.” Engineers who want to add search to their application usually deploy Elasticsearch, or write their own

The post Search as a Service with Julien Lemoine appeared first on Software Engineering Daily.

Apr 18, 2016
Managing a CDN with Carl Gustas
40:57

“We’re not always in control of other people’s networks.” CDN stands for content delivery network. A content delivery network is a system of distributed servers that delivers web pages and other web content. Without CDNs, the internet would be much slower, because CDNs function as a caching layer for most web resources. Carl Gustas is

The post Managing a CDN with Carl Gustas appeared first on Software Engineering Daily.

Apr 15, 2016
Logging and NoOps with Christian Beedgen
55:24

“You write the code, but you don’t run it? That’s just preposterous.” Software applications are constantly generating logs. These logs are necessary to understand how an application is functioning, and logs are key to debugging. As applications have gotten more complex, logging infrastructure has become complex as well. Storing and managing all of our log

The post Logging and NoOps with Christian Beedgen appeared first on Software Engineering Daily.

Apr 12, 2016
Scaling Email with J.R. Jasperson
51:20

“As the scale continues to increase, certain effects of architecture become less and less efficient.” When you spend money online, you expect a receipt to come in your email. When you register for a new web site, you need to verify your sign up in your email. These types of emails are called “transactional email”

The post Scaling Email with J.R. Jasperson appeared first on Software Engineering Daily.

Apr 08, 2016
Automating Infrastructure at HashiCorp with Mitchell Hashimoto
1:01:23

“SaaS, whether we want it or not, in enterprise technology or in our data centers, is coming.” Application delivery has become more complex as software architectures have moved into the cloud. Data center infrastructure has turned into code to be manipulated, and software engineering teams are adjusting their strategies. HashiCorp is a company that builds

The post Automating Infrastructure at HashiCorp with Mitchell Hashimoto appeared first on Software Engineering Daily.

Apr 06, 2016
Bootstrapping a SaaS for Developers with Itai Lahan
1:03:45

“It’s an amazing era for software developers – we have all this amazing infrastructure behind the scenes that we can build upon.” Ten years ago, building a highly scalable image delivery service would require millions of dollars in upfront costs, and hours of work configuring hardware server infrastructure. Today, it is possible to bootstrap this

The post Bootstrapping a SaaS for Developers with Itai Lahan appeared first on Software Engineering Daily.

Mar 31, 2016
Developer Analytics with Calvin French-Owen
54:00

“Its sort of like the old joke in computer science – what do you do when you have a problem? Well, add a layer of abstraction.” Today’s guest is Calvin French-Owen, the CTO of Segment, a tool that companies use to aggregate their analytics into once place. As Segment has scaled, the company has had

The post Developer Analytics with Calvin French-Owen appeared first on Software Engineering Daily.

Mar 25, 2016
Continuous Delivery and Test Automation with Flo Motlik
54:13

“It’s Friday night and you’re basically out of the office on your way to meet with friends. And you just merge this thing and put it into production because you have that trust – that the system will capture any kind of problem.” Continuous integration and deployment are important tools for modern software development. With

The post Continuous Delivery and Test Automation with Flo Motlik appeared first on Software Engineering Daily.

Mar 01, 2016
Distributed Systems with Leslie Lamport
50:06

This episode is a republication from my interview with Leslie Lamport on Software Engineering Radio. Leslie Lamport won a Turing Award in 2013 for his work in distributed and concurrent systems. He also designed the document preparation tool LaTex. Leslie is employed by Microsoft Research, and has recently been working with TLA+, a language that is

The post Distributed Systems with Leslie Lamport appeared first on Software Engineering Daily.

Feb 27, 2016
Engineering Cloud Services with Sam Kottler
48:52

“A lot of our customers are kind of AWS refugees who really don’t want all the stuff amazon has, and they just want a really easy to use system that is reliable.”

Continue reading…

The post Engineering Cloud Services with Sam Kottler appeared first on Software Engineering Daily.

Feb 05, 2016
Moving to Microservices at SoundCloud with Lukasz Plotnicki
49:59

“You can have a monolith, and it can be a perfectly good thing.”

Continue reading…

The post Moving to Microservices at SoundCloud with Lukasz Plotnicki appeared first on Software Engineering Daily.

Feb 04, 2016
Engineering at Quora with Shreyes Seshasai
53:16

“If an engineer is doing something repeatedly over and over again, their mind is immediately going to jump to that place where it’s like ‘Okay how can I make this faster?’ or ‘How can I save myself time?’” Quora is a Q&A website where questions are asked, answered, edited and organized by its community of

The post Engineering at Quora with Shreyes Seshasai appeared first on Software Engineering Daily.

Dec 28, 2015
Scaling Uber with Matt Ranney
45:14

“If you can make a system that can survive this random failure testing, then you will more or likely survive whatever other chaotic conditions exist.”

Continue reading…

The post Scaling Uber with Matt Ranney appeared first on Software Engineering Daily.

Dec 04, 2015
Netflix Genie with Tom Gianos
56:57

“Sometimes there’s a misconception that Genie is a job scheduling platform... Genie really represents our extraction layer, from what our computational resources are, to our end user jobs.”

Genie is an open-source tool that provides job and resource management for the Hadoop ecosystem in the cloud.

Continue reading…

The post Netflix Genie with Tom Gianos appeared first on Software Engineering Daily.

Oct 13, 2015
Taming Distributed Architecture with Caitie McCaffrey
53:51

Distributed systems programming will always be a world of tradeoffs -- there is no silver bullet in the future. But life can be made easier with tactics such as the actor pattern and the use of conflict-free replicated data types (CRDTs).

Caitie McCaffrey is a distributed systems engineer who currently works at Twitter. She previously worked on Halo 4 at Microsoft and 343 Industries. At QCon San Francisco, she will be hosting the track Taming Distributed Architecture.

Continue reading…

The post Taming Distributed Architecture with Caitie McCaffrey appeared first on Software Engineering Daily.

Sep 10, 2015
Origin of DevOps with John and Damon from DevOps Cafe
47:50

“DevOps is not a thing. It is a set of problem statements and solution possibilities that are always growing.” The hosts of DevOps Cafe joined Software Engineering Daily for a conversation about DevOps culture and misconceptions. Questions What do software engineers need to know about DevOps? What are the biggest misconceptions around DevOps? Is DevOps

The post Origin of DevOps with John and Damon from DevOps Cafe appeared first on Software Engineering Daily.

Aug 31, 2015
Continuous Delivery with Jenkins Creator Kohsuke Kawaguchi
53:44

Jenkins is an extensible open source continuous integration server. Kohsuke Kawaguchi is the primary developer of Jenkins CI and the CTO of CloudBees, a provider of enterprise Jenkins. Questions: How does continuous integration affect DevOps? What has changed in the five years since Jenkins was created? In what ways is Jenkins opinionated? What are the

The post Continuous Delivery with Jenkins Creator Kohsuke Kawaguchi appeared first on Software Engineering Daily.

Aug 27, 2015
Containers with Bryan Cantrill from Joyent
55:06

Container infrastructure has benefits of security, scalability and efficiency. Containers are a central component of the DevOps movement. Joyent provides simple, secure deployment of containers with bare metal speed on container-native infrastructure Bryan Cantrill is the CTO of Joyent, the father of DTrace and an OS kernel developer for 20 years. Questions: Why are containers

The post Containers with Bryan Cantrill from Joyent appeared first on Software Engineering Daily.

Aug 26, 2015
Hadoop Ops: Rocana CTO Eric Sammer Interview
56:39

Rocana applies big data, advanced analytics, and visualizations to dev ops in order to guide users to the root causes of problems. Eric Sammer is the co-founder and CTO of Rocana. At Cloudera, he served as an Engineering Manager responsible for tools and partner integrations. Within that role, he developed many of Cloudera’s best practices for

The post Hadoop Ops: Rocana CTO Eric Sammer Interview appeared first on Software Engineering Daily.

Aug 05, 2015