Kubecon + Cloudnative Europe 2023 occurred in Amsterdam from April 19 to April 21. This year, the event attracted more than 10,000 participants from all over the world.
The conference featured keynote speakers, technical sessions, workshops, and networking opportunities for developers, operators, and other professionals interested in cloud-native technologies. Attendees learned about the latest developments in Kubernetes and other open-source projects and best practices for deploying and managing cloud-native applications.
As a sponsor of Kubecon + Cloudnative Europe 2023, our SRE team attended many high quality and highly technical talks. They had the opportunity to learn about the latest developments in Kubernetes and other open-source projects and best practices for deploying and managing cloud-native applications. They also met many experts in this industry, such as representatives from leading technology companies, startups, and open-source communities.
Takeaways from Criteos
Benjamin Arezki’s favorite talk: InSPIREing Progress
Attending my first KubeCon proved to be an incredible experience, boasting over 10,000 attendees and a plethora of engaging talks. With numerous appealing sessions scheduled simultaneously, it was difficult to choose which to attend. Fortunately, replays are available online, ensuring attendees can catch up on any missed content. KubeCon provided a valuable opportunity for networking and discussing common challenges faced in our daily work with other professionals. Additionally, it allowed us to meet the dedicated contributors behind the open-source projects we use every day, further enhancing the overall experience.
My favorite session was InSPIREing Progress: How We’re Growing SPIFFE and SPIRE in 2023 and Beyond — Daniel Feldman, Hewlett Packard Enterprise & Andrés Vega, ControlPlane. During this presentation, the speakers provided an in-depth look at the latest developments and the future roadmap of SPIFFE and SPIRE. As cloud-native architectures continue to evolve, the need for secure authentication and identity management solutions becomes ever more critical.
The presentation began by highlighting the security motivation behind the project, which is the need to reduce trust to zero. As cloud-native architectures become increasingly complex with more actors involved, trust becomes a crucial issue. Traditional methods of storing and encrypting secrets typically require another secret for access and decryption, creating a need for a more secure solution.
SPIFFE (Secure Production Identity Framework for Everyone) addresses this issue by defining a set of interfaces, including APIs and documents, for proving, validating, and obtaining service or workload identities. SPIRE, which implements the SPIFFE interface, provides a toolchain for establishing trust between software systems. The concept behind SPIFFE has its roots in papers from Bell Labs Factotum, published in 2002. SPIFFE and SPIRE joined the Cloud Native Computing Foundation (CNCF) in 2017 and graduated in 2022.
The primary benefits of SPIFFE and SPIRE include delivering mutual authentication across untrusted environments, leveraging strong attestation and cryptographic material, managing lifecycle, enabling mutual TLS (mTLS), and providing secret-less authentication to third parties.
The presentation also covered the details of the SPIFFE specifications, including SPIFFE IDs, SPIFFE Verifiable Identity Documents (SVIDs), the SPIFFE Workload API, the SPIFFE Trust Bundle, and SPIFFE Federation. These components work together to create a robust and secure solution for managing and authenticating service identities in complex environments.
The presenters also shared notable examples of SPIFFE and SPIRE end users, such as Netflix, Uber, and Bloomberg. They also highlighted the extensive integration of SPIFFE and SPIRE with various components, including Kubernetes, Nginx, Envoy, Sigstore, and Traefik. Additionally, there is now an official Helm chart available.
The presentation also discussed new features added since the last KubeCon, with an in-depth explanation of each: TPM integration, Istio integration, Sigstore integration, Bearer tokens, and Credential composers. These enhancements contribute to the versatility and robustness of the SPIFFE and SPIRE ecosystem.
Lastly, the presenters shared a glimpse of their roadmap, revealing ongoing work in areas such as flexible federation and confidential computing. This information showcased the commitment to continuous improvement and innovation within the SPIFFE and SPIRE projects, making them increasingly essential in securing cloud-native environments.
To conclude, this presentation was truly inspiring and informative, providing valuable insights into the growth and future direction of SPIFFE/SPIRE. The knowledge shared during the talk is highly beneficial for companies of all sizes and industries, as secure authentication and identity management play an increasingly crucial role in the rapidly evolving world of cloud-native architectures. If you want to know more about this subject, you can take a look at this book.
Julien Pepy’s highlights: Setting up Etcd with Kubernetes to Host Clusters with Thousands of Nodes
I was surprised by the number of quality talks I attended (well, at least the ones I managed to get to, thanks to 10k+ people on site!), stretching well beyond the obvious trend for security and network/epbf related talks. There were a lot of enlightening security-oriented talks raising the alarm on the simplicity of attack vectors on Kubernetes, with live demos on how to hack a cluster or how to apply live-countermeasures on an identified, ongoing intrusion. That being said, one particularly compelling and captivating talk was Setting up Etcd with Kubernetes to Host Clusters with Thousands of Nodes by Isovalent & Datadog.
It started with a 101 on setting up a Kubernetes control plane for scalability, from a naive 1-node development setup to an optimised and refined architecture, with a split etcd cluster dedicated to the network hungry Events resources. I was especially impressed by the fluidity and simplicity with which the speaker managed to make accessible such a technical topic to the wide audience of KubeCon (given the numbers, I’d bet that the majority of attendants are using managed Kubernetes offerings from the big cloud providers).
The talk then shifted to the client parts which, in a Kubernetes cluster, could put pressure on apiserver and etcd cluster: other control plane components (controller manager, scheduler), side-players (coredns, autoscaler, various controllers including ingress…), worker-side (kubelet, kube-proxy, node-level applications — typically daemon sets), kubectl commands (controlled or rogue), and even apiserver which can call itself! The main focus was on apparently inoffensive read calls from Kubernetes CLI or clients due to hidden defaults (resource version), the unanticipated pressure it can generate on etcd, and recommendations on how to address it. It was supported by an entertaining real-life incident report showing how even a single app can have a devastating impact on the Kubernetes control plane. The fact that, at Criteo, we actually fully manage Kubernetes clusters by ourself made me especially receptive to this kind of topic.
Despite being actually highly technical, this talk was fairly accessible and certainly hit its target of making a wider audience aware of the unexpected side-effect of their client-side usage of Kubernetes API. There was only one absent topic which is still quite debated by the community, especially in the context of scalability: the usage of CRDs at scale and its impact on apiserver/etcd.
Geoffrey Beausire’s theme: Stateful application in Kubernetes
As part of the NoSQL team and having worked with Kubernetes for over 3 years, it was great seeing how other people operate databases under Kube. KubeCon is a big conference, the largest I have attended yet.
Being surrounded by some many Kubernetes enthusiasts was thrilling, we were lucky enough to have our own booth showing all the technologies we use everyday which led to many interesting technical discussions. The hardest part of the conference was choosing which talk to attend as there were so many happening in parallel.
Instead of choosing one talk, I chose to focus on a particular theme: stateful application in Kubernetes. The first talk is Distributing Pod Disruption Budgets Across Multiple Clusters by Illya Chekrygin. Pod Disruption Budget (PDB) is a great way to protect workload against planned disruptions. However, PDB is not perfect: it can only consider pods of one namespace and you cannot use multiple PDBs to protect one pod). In his talk, Illya proposes a distributed PDB: a way to chain and federate multiple PDBs together. By interlocking multiple PDBs, it becomes possible to expose more complex constraints to Kubernetes. Pod Disruption Budgets are of particular interest for us as we use this as the main interface to our maintenance system.
The second talk I attended was Availability and Storage Autoscaling of Stateful Workloads on Kubernetes — Leila Abdollahi Vayghan, Shopify. They explained how they scale up their storage for Elasticsearch clusters and showed the limitations for scaling down. Most cloud provider provide expandable persistent volumes via the Kubernetes API. However, by default, StatefulSet doesn’t support changing how much storage is requested. To work around this issue, they use a custom controller that is performing a shallow delete of the statefulset (deleting only the statefulset itself, not the pods it is managing) then recreate it with the current configuration. While not the main focus of the talk, it shows the power of using custom controllers to implement custom logic within clusters.
My third talk was Scaling Databases at Activision — Greg Smith & Vladimir Kovacik, Activision/Blizzard. In this talk, Greg and Vladimir shared how they are migrating their aging MySQL based infrastructure to Vitess. Their talk was great because they showed the different steps they went through for their migration: from proof of concept v1 using Helm to a production-grade system using operators and Vitess’ Orchestrator. Migrating an existing service and ensuring it’s performing as expected is often the most complicated part of deploying a new technology. I especially liked their approach of migrating both a simple use case as well as one of their biggest use cases as a first step of the migration, as it is also the approach we chose for our own migrations.
Raphael Bizos’ highlights
I was very enthusiastic about attending Kubecon this year as there were maintainers of several projects we extensively use in my team, Observability. Namely: Prometheus, OpenTelemetry and VictoriaMetrics. Indeed, I even had the chance to talk about a long lasting pull request of mine on Prometheus with the maintainers. The best part is that it may be the first step in more significant change aimed at adding a metadata storage in Prometheus and it makes sense in the context of making Prometheus and OpenTelemetry more compatible. I had a long discussion about VictoriaMetrics with Bartlomiej Plotka, one of the creators and maintainers of Thanos who was pretty interested in hearing our feedback about it. In my conversations with the VictoriaMetrics team, I was able to share what we love (and love less) about their project, and they were very receptive to my feedback. They even gave me a stylish VictoriaMetrics sweatshirt that I’ll proudly wear!
One presentation that I attended was Show Me the Metrics: How a Huge Bank Does Observability with Multi-Tenancy Prometheus and Thanos. It was about using Thanos for Prometheus retention infrastructure on a scale comparable to ours (the realm of millions of datapoints per second). This was particularly interesting as we are currently experimenting with Thanos as our aggregating layer on top of Prometheus.
Another presentation that stood out for me was Effortless Open Source Observability with Cilium, Prometheus and Grafana — LGTM! which explained how to add metrics to applications without having to instrument the code, using Cilium and Hubble. This leverages eBPF to automatically understand network events from low-level network up to L7 application layers such as HTTP. Hubble has a great interface to investigate issues and with cillium it’s able to make sense of pretty complex metrics (especially with service mesh activated). Also it’s eBPF so it comes with points for style and hype.
Last buy not least, I attended Smarter Golden Signals! which focused on using machine learning pipelines for anomaly detection in Kubernetes clusters. The presenter introduced a project called Numaproj that they used for this purpose. I was pretty excited to learn about this approach, as it’s a topic my team have been talking about for a long time as a future feature that we might decide to PoC someday.
To conclude, Kubecon + Cloudnative Europe 2023 was an amazing experience for our team to attend. It was a great opportunity for our teams to exchange with many experts and to learn a lot during the conferences and workshops they attended. It was also an opportunity to showcase Criteo to our many interactions with the participants. We gained much knowledge during this event and can’t wait to share it with our colleagues and clients.
We thank the Kubecon + Cloudnative Europe 2023 organizers, who have done a remarkable job organizing such a large-scale event. Everything was smooth and easy for both participants and sponsors. We are grateful to have had the opportunity to participate in this event.
We look forward to next year and hope to continue to be part of future editions.
Thanks again to the Kubecon + Cloudnative Europe 2023 organizers and to all those who contributed to the success of this event.
