Agenda

The world is changing, it’s pretty clear by this point.

With the advent of large language models (LLMs) and generative AI, we are witnessing the boundaries constantly being pushed on what is possible for a machine to understand and do, simply by using words. This revolution is not only reshaping technological possibilities but also redefining people’s expectations.
However, while this revolution is happening, our dear old friend SQL is officially celebrating its 50th birthday, proving itself once more as a survivor in the world of data.
So what is it going to be? Who will control the future of data? Words or SQL?

Let’s take a journey into the past, present and future of querying data.

We will talk about how SQL became so popular, how it survived for so long and what is still making it so sticky 50 years later. We will talk about the LLM revolution, the challenges of leveraging AI to transition from SQL to NLQ (Natural language querying) and some hints to the technologies that could get us closer such as RAG, Semantic Layers and Knowledge Graphs.
Lastly we will peek into the crystal ball and try to figure out what the future might bring, how it might change the way we see modern data platforms and data self service, and what all of this might mean for us data professionals.

In a rapidly evolving world, data is the future. This panel brings together four leading investors from diverse backgrounds to explore the opportunities and challenges that lie ahead in the data domain. From transformative technologies to emerging startups, our experts will share their insights on which trends will shape tomorrow. Join us for a forward-looking discussion that will uncover how data-driven innovation is unlocking new paths for growth and success.

In recent years, we have witnessed the emergence of powerful frameworks and libraries, such as Apache DataFusion, DuckDB, and Polars, that enable us to query and transform vast amounts of data at blazing speeds, even with modest compute resources. These frameworks not only enhance our capabilities but also pave the way for new kinds of data architectures and design patterns that challenge the traditional design of our ETL processes and query infrastructure. Workloads that once required a distributed Spark cluster to complete in a reasonable time, or queries that needed data warehouses to scan vast datasets, can now be processed more efficiently and with less overhead.

In this session, I will focus on three major frameworks: DuckDB, Apache DataFusion, and Polars, which are central to what many refer to as the ‘new data stack’ and its transformative capabilities. We will discuss the significant advantages these frameworks offer to data engineers, especially in cloud-native environments, and demonstrate the new data design patterns they enable through real use cases that utilize them.

In this session, we explore the critical role of cost-aware architecture in achieving optimal cost efficiency for modern software systems. As organizations increasingly rely on cloud services, microservices, and distributed computing, understanding and managing costs becomes paramount.

We will discuss the shift from traditional architecture to cost-aware design and explore how architectural decisions impact operational expenses. We will showcase a real life example and the lessons learned when evaluating the different cloud providers and their services – based on cost models and show how rearchitecting the solution with cost in mind made the difference.

Attendees will gain insights into designing cost-efficient systems that balance performance, scalability, and financial constraints. By adopting a cost-aware mindset, organizations can optimize their architecture and drive sustainable growth.

As a user explores the open web, our mission is to recommend the most suitable content for them to read next out of millions of potential items. Our goal is optimizing for clicks and conversions – And we have to do so (very) very fast.

In this talk, we’re gonna talk about how in Taboola we reduced this problem by leveraging the power of Vespa – a high scale Vector DB. Unlike other DBs – Vespa is both a search AND a vector DB. We’ll discuss Vespa’s architecture, topology and ranking features. How we implemented near real time data updates (using Debezium, Kafka and in-house mirroring), filters to reduce the document space, the optimizations we implemented along the way – and the tools and techniques we used to measure their impact, and how we embedded deep learning model estimations into simple 32-bit vectors to generate CTR and CVR estimations to use in ranking.

I got some new gray hair on the spot.

After that, I took a deep breath and called my tech lead to discuss the issue. We ended up recovering in less than half a day to just coin our motto: “Write your code as if you’re going to delete production”.

Let me take you through this nightmare that taught me 3 important lessons. 3 lessons that separate tech leads from the others.

Classification model inference is relatively a straight forward issue unless the real world input data contains examples from classes the model has not seen before. In this case, the feature vector of an ‘unseen’ input might coincide with one of typical vectors of a class the model was trained on. In this case, an unseen class might have a prediction with hight certainty level. We shall discuss this problem and it’s possible and popular solutions.

Forget the titles, no one is born a Data Engineer. With the right guidance, attitude and passion, the transition to data engineering is not only possible, but also brings many advantages. In this talk, I will take you through my own journey of leaving the world of APIs and services as a backend engineer to enter the world of variety and scale as a data engineer. I will cover why I was passionate about this transition, the challenges I faced, the change in perception, and the benefits of being a data engineer with a background in backend engineering. By the end of this talk, you will be convinced that one of the best ways to scale your team is by incorporating new talent from various backgrounds, giving your team a diverse set of knowledge and skills.

As you all know, data professions have become some of the most required professions in the world.

We at the Ministry of Education felt an obligation to prepare the students for this important field, therefore we established the National Data Analysis Major for High Schools in Israel.

In this talk, we’ll tell you about the dilemmas we’ve had about what should be taught in the data analysis program and what skills would prepare the student best to be data analysis-oriented.

We will tell you about the global interest in our Major and the research being done worldwide on our groundbreaking program.
Above all, we will tell you about you- data people, and your enormous contribution to upgrading and improving the Major.

You have a very important job, you need to keep production up & running 24/7! You setup alerts so you will know when something is wrong with it, but now there are alerts! Some of them are very important! So important that you need to leave everything that you do and start to handle them but some of them are less important and you can look at them in the morning or even ignore them.

Maybe we have too many alerts?
Or maybe we are missing alerts?
What is the purpose of an alert?

Did you try to think about the KPI of a good alert? How do we measure if an alert is good or bad?
In this talk we are going to define what is a good alert, how to create a KPI on alerts so you can make better alerts that are more accurate and help keep production up & running.

In an era defined by data abundance, the ability to harness its potential has become a competitive imperative. From startups to multinational corporations, the pursuit of a data-driven culture has never been more urgent. But amidst the buzzwords and promises, what truly distinguishes success and failure?

This talk offers an anthology of real-world stories that illuminate the journey toward building and sustaining a data-driven culture. Through some narratives drawn from diverse industries, we uncover the challenges, fails, triumphs, and transformative strategies employed by organizations committed to unleashing the power of their data.

Attendees will gain invaluable insights into practical approaches that have proven effective in fostering a data-driven culture, as well as lessons learned from initiatives that fell short of expectations, ensuring that attendees leave equipped with real-world knowledge that can be applied immediately within their organizations.

In today’s data-driven world, managing vast amounts of data from various online and offline sources to various destinations is both a challenge and an opportunity. At Riskified, we have established a robust data contracts platform to ensure trust and quality across our data ecosystem.

In this session, we will explore how we have built comprehensive data contracts that include strict testing, detailed documentation, and an extensive data catalog based on open-source products like DBT, Datahub, Great Expectations, and Elementary.

We will share our methodologies for maintaining data integrity, the tools we use for quality assurance, and the best practices for documenting and cataloging data pipelines and tables.

Join us to discover how our approach to data contracts can enhance data reliability and support informed decision-making within your organization.

As engineers, we are used to keeping the data we produce, usually within the guidelines of our business needs (like when we create retention as the data is too old to be used) or limitations (we cannot afford more than X TB of data).

However, sometimes we might create “too much” data, which we just cannot support, and an instant drop of the data is in order.
In Outbrain, as part of the data democratization strategy, a developer is free to send (almost) any kind of log/message that he wants/needs from his microservice/application. Therefore, we had to impose mechanisms to potentially drop data base on various dimensions of the sent message (and the sending rate/volume).

In this session I will present the idea behind the system and how it works.

Apache Iceberg provides a powerful framework for managing large-scale data with advanced features that are crucial for today’s data-intensive applications. This talk will focus on the various metadata tables that Iceberg offers and how they can be leveraged to enhance data management practices across diverse use cases, such as compaction, incremental processing, and even monitoring.

Real Time and Data Lakes are becoming a single beast, these worlds are converging. Kafka became the de facto streaming platform, integrating with modern data lakes. Flink became super popular and adopted by big vendors such as Amazon, Microsoft and Alibaba. Confluent, not only adapted flink but also introduced TableFlow solution that allows to seamlessly create data lake tables based on kafka topics.

This convergence signifies the necessity for real-time and data lake systems to share a common language and standards

In this talk, I will share Yotpo’s journey in data generation and ingestion into the lake. Beginning with the transition from operational databases to daily snapshotting for lake availability, we progressed to implementing DB-to-lake streaming using CDC and Debezium. Currently, we are advancing towards well-defined Async APIs and a unified architecture ensuring better performance and cost efficiency while treating data as a first-class citizen from the very beginning of a service design. I will delve into the key components enabling this architecture, including Kafka, Outbox pattern, Flink, Cloudevents, and AsyncApi, explaining why and how we decided to put everything together in order to execute this exciting transition.

If you think the real-time layer should cooperate better with data lake, this talk is for you.

In this talk, we present our successful optimization of Spark-based big data pipelines at PayPal, achieving 70% of cost reductions through strategic GPU utilization. We discuss the key challenges we encountered in our big data and machine learning domains, and how we used Spark RAPIDS to solve them. In addition, we’ll share a glimpse of the new GPU-accelerated data processing domain.

User expectations are sky-high, while at the same time users have increasing difficulty articulating their complex needs in a simple search bar on a website. This talk dives into leveraging generative AI and vector search to transform vague user queries into results the user actually wanted, even if they did not know initially what they clearly wanted. We will explore why traditional search methods fall short in grasping user intent and address the common problems users face with ambiguous search queries.

Learn how GenAI generates embeddings to capture the context and semantics of queries. This talk though is not only about the theory. In the talk you will be shown practical examples of how to generate embeddings, and how to set up vector indexes.

See how these advanced search capabilities can transform user interaction and, as a result, business outcomes, making user uncertainty into certainty with GenAI and vector search.