Data Teams Summit is all about peer-to-peer empowerment, led by data rockstars at future-forward organizations, talking about how they are establishing predictability, increasing reliability, and creating economic efficiencies with their data pipelines.
I had the pleasure to be part of this event, participating in a panel about transitioning from IC to Manager. It was great to see so many familiar faces and make new friends at this conference. Below, I share with you a summary of my favorite sessions.
Panel: Winning Strategies to Unleash your Data Team
Kunal Agarwal, Co-founder & CEO @ Unravel Data
Sanjeev Mohan, Principal @ SanjMo
Benjamin Rogojan, Owner & Data Consultant @ Seattle Data Guy
Great data outcomes depend on successful data teams. Every single day, data teams deal with hundreds of different problems arising from the volume, velocity, variety—and complexity—of the modern data stack.
Learn best practices and winning strategies for what works (and what doesn’t) to help data teams tackle the top day-to-day challenges and unleash innovation.
How are teams ensuring that their pipelines are reliable?
Ben mentioned that one thing you can implement are SQL checks as a reliability component to make sure everything is accurate. There is so much tooling that has recently come into play to use and build trust, but sometimes these tools just allow you to make bad decisions faster. So be sure to figure out what is the right solution for your use case.
Sanjeev brought in a neat perspective by sharing a common situation where the stakeholder looks at a dashboard, finds that ‘something is suspiciously off’, and then alerts the data team that it needs to be fixed. This is nearly impossible and would require hours of research if you don’t have visibility into your pipelines. In a perfect world, you could proactively determine where things could go wrong and fix the issue before it happens (or before someone catches it).
With more organizations moving to the cloud, how are costs of the cloud being managed?
Ben reminds us that at first, the concept of using data science and being data driven was being done at any cost, money didn't really matter. Now, money matters and many of us see that with companies trying to reduce costs. If we push for ownership of pipelines, that can help drive costs since the owner would be aware of those one-off queries that cost $100 per day but are not serving much of a purpose.
Sanjeev believes that with the rise of FinOps, we will start to understand where the organizations are incurring costs which could help manage these costs for cloud. Although, you might need to do a little bit of digging / searching because apparently Accenture’s AWS bill can be 10s of millions of lines long!
How are companies filling roles today and how can they do better?
Ben created a survey recently and found that the biggest problem that teams are facing is finding new talent. He realizes, from experience, that companies need to be up front with what they are looking for, in both skills and types of projects. Oftentimes, someone accepts a new role only to find out 6 months in that the projects they are working on do not fully leverage their skills. Sure there are new tools that you likely have to learn, but companies have to be specific with their job descriptions.
Online resources were not available like they there are today for Sanjeev. If you came out of college and needed to learn a skill for work, there was a lot of self-training. He shared that this is the golden era of data and you own your career. If you want to learn something, there are several free online resources, Slack communities, and social media posts available for learning.
Kunal also brought up a solid point that with the modern data stack today and how often it changes, you pretty much have no choice but to upskill because what you know today is going to be phased out in about 2 years.
What are your predictions for 2023 for DataOps and Data Teams?
Ben sees an increased focus on the basics, like data modeling and data quality. Companies are realizing the benefits of having reliable data sets and the affect they have on the value to the business.
Sanjeev sees more organizations using more than one cloud because there is no single pane of glass that can easily show everything. He also thinks that the metadata space is going to get more visibility because it is currently a mess and needs some help. Lastly, the hope of companies shifting from talking about tooling, teams, etc. to focusing on outcomes.
Becoming a Data Engineering Team Lead
Matthew Weingarten, Senior Data Engineer @ Disney Streaming
As you progress up the career ladder for data engineering, responsibilities shift as you start to become more hands-off and look at the overall picture rather than a project in particular.
How do you ensure your team's success? It starts with focusing on the team members themselves.
In this talk, Matt Weingarten, a lead Data Engineer at Disney Streaming, will walk through some of his suggestions and best practices for how to be a leader in the data engineering world.
Matthew started out strong by sharing one thing I am passionate about, highlighting that data has a technical and a human element.
As a team lead, you will be the most successful if you have some technical abilities that you can leverage to get the work done. Most importantly is the ability to delegate, which requires the ability to understand the requirements of the task and the abilities/skills of your team. This will equip you to pick the right person for the job.
Another great takeaway for a team lead is the importance of the 1:1 meeting. This needs to be more than just a meeting you have so you can check a box. You want to make sure to include honest feedback to help your team understand areas of improvement which they can then use to help themselves grow in their career.
High level tips:
Focus on the bigger picture
Always think about improvement
Bring people out of their shell
Culture means a lot more than you think
Panel: The Habits of Successful Data Engineers
Loris Marini, Data Scientist, Founder & Host @ Mars
Vishal Ramrakhyani, VP Engineering @ Zoomcar
Swati Vishwanathan, Data Engineer @ Swinerton
Tobias Zwingmann, Data Science Mentor @ Springboard
What does it mean to build relationships?
Tobias starts off the session with a quotable tip, 'you need to build your network before you need it'. This was such a great tip that Loris agreed to have t-shirts made with this phrase, coming soon! Tobias elaborated by mentioning that relationships should not be treated as transactional interactions. You should not just reach out to someone because you need something.
Vishal highlighted how you can build relationships with different types of groups. One example being with customers, which could be considered internal stakeholders. In this case, you can build a relationship by understanding their problems and trying to help them solve those problems.
Swati thinks that relationship building is a topic that is sometimes forced on professionals in their career and can be hard for some. What she did notice is that face-to-face presence means a lot to others (this can include turning on your camera during virtual meetings). Additionally, the face time helps make an impression for you to be a go-to down the road.
Why do you think that Engineers do not take time to connect with people that they do not know?
Tobias shared some deep insights here by mentioning that Engineers are trained on producing output. Since meetings and relationships don’t have an output and it is hard to measure, it feels like a waste of time. He shared that he needs 5X more time to get to know a person than other people would, which makes it especially hard when you first enter a new company or a new team.
How do you build a rapport with others?
Tobias, again with the deep insights, thinks that there is no golden rule to building a rapport. Some people like small talk while others do not. Some people talk technical while others talk business. The key to building rapport is to figure out, within the first few seconds, how to communicate. This is not an easy skill to build, but can be developed with practice.
Vishal shared that when he transitioned from Engineer to VP, it was a challenge because he had such a technical engineering mindset but needed to pivot to connect with his peers. To do this, he started a Fun Friday quiz, not related to Engineering but related to business questions. This got a lot of visibility and his peers enjoyed the interaction.
Swati reminded us to be ourselves and that not every day is a Shark Tank presentation. You can talk about things like books, music, anything that brings you joy. She closes out the panel with a wonderful reminder: leadership is human too!
Panel: Bridging the Gap: Going from Individual Contributor to Manager
Matthew Blasa, Data Scientist & YouTuber @ DataLife360
Rob Albritton, VP, AI Practice Lead @ Octo
Monica Kay Royal, Founder & Chief Data Enthusiast @ nerdnourishment
Puppy Tsai, Associate Product Manager @ Coach Art
What was the hardest transition from Individual Contributor to Manager?
Monica shared that it was not being able to play in the weeds anymore and it was hard to help others without feeling like you were stepping on their toes. Rob found it the most difficult going from being responsible for himself only to being responsible for many individuals. This responsibility came with a lot of paperwork, including timesheets, PTO requests, and other administrative tasks. Additionally, he felt a lot of responsibility for other things such as the hiring process. Puppy mentioned that the transition in mindset was something to get used to, which was a little strange when having to still be required to complete general trainings.
How did the field of Data Science change during the transition?
Rob gave a callback to Monica’s point about not being able to get into the weeds as much anymore. He realized as he progressed throughout his career that he has become more of a generalist by trade. Probably due to being in charge of bigger areas of the business and the need to be only an inch deep everywhere. However, he observed that employees do not respect leaders that do not take time to understand each of the different areas of the business. Monica agrees with the generalist point, she said that she feels like Iron Man in front of his AI machine. You see the big picture, know how everything relates, and know where things need to go. This comes in handy because you know who to talk to when you have a specific question and who needs to be in certain meetings. Sometimes, being a generalist meant that you were the translator between the business and the technical side of the house.
How has being a Product Manager helped you lead a team?
Puppy mentioned that you need to understand the user and having a product mindset allows you to work better. It also saves you a lot of time and energy in communication which makes you more efficient. Monica thinks that the Product Manager title is just another buzzword and has noticed throughout her career that everyone on a data team has the same goal: work together, as a team, and get things done, together. The only difference is that since you have more experience, you become the go-to person when someone has questions and needs direction.
What kind of resources did you use to make this IC to Manager transition?
Rob is a huge advocate of mentors and leveraging insights of those that have been where you want to be. A lot of people are willing to be a mentor if you ask. Some Universities set you up with career coaches and resources to help as well. Monica found her mentor through working with her manager. Being honest and transparent with your manager is the best advice for situations that are both working well and not working so well.
How do you motivate and inspire direct reports?
Rob emphasizes the need to build a strong culture. Creating a common vision and goal for the team makes everyone closer. This makes it easier to work together and trust each other because no one wants to let down the team. He said it’s all about building that family environment. Puppy mentions to try to be a role model and show that you care and trust others. Make sure that others can come to you to ask questions. Monica says that when she meets with a group, she is honest and up front. She also makes sure that if anyone is uncomfortable in the group meeting setting, they can ask to meet separately. She said that she strives to cater to other people’s styles and include everyone. Rob added that it is always good to create a safe environment where people can speak their mind.
How do new managers get used to delegating?
Matthew mentioned before that there are two kinds of delegation for a project, finding the best person to complete the task and finding the best person to present the completed task to the stakeholders.
Monica jumped in and disclosed that she does not know how to delegate and asked Rob and Puppy for help. Rob mentioned that he is still learning how to delegate himself, and added that one of the hardest things to learn is how to identify which tasks to delegate. We oftentimes like to say yes but decide to get things done ourselves because it is ‘easier’; however, that is not scalable. Puppy was the savior of this question and shared the use of the Eisenhower Box. The items in the lower level quadrants are those to delegate as they are not as important.
Panel: What’s in store for the Future of Data Engineering
Shane Murray, Field CTO @ Monte Carlo
Eric Callahan, Sr. Data Consultant @ Pickaxe
Brandon Beidel, Director of Product Management @ Red Ventures
With budgets tightening and data use cases skyrocketing, how can data engineering teams set themselves up for success in 2023?
In this panel conversation, Shane Murray, Field CTO at Monte Carlo, Brandon Biedel, Director of Product Management, Red Ventures, and Eric Callahan, Sr. Data Consultant at Pickaxe Foundry and former Head of Data at Understood, will discuss their predictions for the year’s top challenges - and opportunities - facing data engineering teams in the New Year.
Their conversation will include the future of BI tooling, operationalizing distributed environments like data mesh, implementing data quality initiatives like data contracts, making the case for data with your finance team, and other pressing considerations.
What are your 2023 trends and predictions for Data Teams?
Eric foresees an increase in the investment in tools. With the many headcount cuts at companies lately, we are now tasked to do more with less. Brandon also sees this theme of doing more with less and adds a financial layer. He thinks that companies acknowledge the need for tools but also the need to make investments with financial awareness.
Are you seeing technologies or trends where we might be able to reduce costs or help with reduction in headcount?
Brandon advices companies to answer basic questions about where their data is at today, before making any investments. Some of these basic questions include:
Is the data that we are creating useful?
Are we using our data?
Do the right people have access to the data?
Are our systems reliable?
Eric adds in the concept of orchestration. He advises companies to look for inefficiencies and look for where things are not talking to each other. Additionally, you want to automate and reduce manual processes.
With the new approaches such as data mesh and data lakes, what other breakout trends do you see coming in 2023?
Eric comes from an analytic background so he got excited about SQL and Python breaking down the walls thus allowing the ability to create metrics using Python. Brandon thinks the lines between data tools will overlap more and more, possibly resulting in the consolidation of cloud providers to draw in more toolsets for their offerings.
How do you scale a data platform strategy?
Brandon warns not to get pulled into a particular tool and to focus on the objectives and outcomes. The end user doesn’t really care what tools you use, they just want to see the outcome and use the product. Eric agrees and adds that it helps to understand all aspects of how the business works to get the data right. The data should be a representation of the business.
What are your thoughts around self-serve data?
Brandon thinks it is helpful to break down the lifecycle of data itself first to make it easier for users to bring in the data sources on their own. Eric agrees with that approach but emphasizes the importance of the data maturity of the organization. If they are not ready to offer self-serve data then it would be a waste of time and effort.
What is the decision process for buying tools?
Eric gives these three points:
Depends on the company and the use case
Depends on what the specific tool does and if it fits the use case
Depends on where you are at as a company and if existing tools can integrate with new tools
Brandon says that you need to do more than just going to a vendor page and purchase a tool. Start with the use case, figure out if the tool gives you what you need, then fill in the gaps. It is also important to note that there will be tradeoffs everywhere.
How do you think about team structures (centralization vs. decentralized vs. hub and spoke vs. other)?
Eric circles back to the importance of the organization's data maturity. If you are new to data, it does not serve you well to have people sit on different teams since you are trying to build out the governance guard rails, standard processes, and such. Once you are on solid footing, you can start sending people out to sit in the product teams. Brandon believes that there is not a one size fits all solution and it depends on where you are on the journey. He ends by sharing that it is important that you are able to have a mentorship program to develop the necessary skills within the team.
Building the Best Data Team is Not Rocket Science
Nirmal Budhathoki, Senior Data & Applied Scientist @ Microsoft
Nirmal kicks us off with defining what the ‘best team’ means by stating that it is not a perfect team but a complete team. The data team needs to align with the organization's vision and it is important to note the challenges.
I liked this idea of ‘The Magic Number of 3’, outlined below, and how it relates to building a team.
Company Sizes: Small, Medium, Large
Scopes: Analytics, Strategy, Governance
Structures: Centralized, Decentralized, Federated
Teams: Data Science, Data Engineering, Data Analytics / BI
How the Company Size Affects a Data Team
Small (start-ups)
Budgeting could be limited
Breadth in skills in data roles
Medium
Some structure for teams
Balance in breadth and depth of skills in data roles
Large
Well structured
Depth of skills in data roles
How Scope Affects a Data Team
Analytics
Raw data > Business insights
Good place to start
Strategy
Organizational Roadmap / Planning
Focus on current and future data usage
Governance
Data regulations, policies, processes, and standards (mandatory / company-driven)
Data trust, accuracy, availability
How Structure Affects a Data Team (the pictures were great here!!)
Different Data Teams
Data Analysts
Collecting raw data, and generating business insights
Presenting results or metrics - reporting tools
Skills Needed: SQL, some Python, Data Visualization tools (Tableau, PowerBI)
Data Scientists
Data analysis (exploration) + predictive analytics
Building machine learning models
Skills needed: Python (or R), Statistics / Math, (some SQL), Domain knowledge
Data Engineers
Building infrastructure / pipelines needed for analytics (Data Plumbers)
Orchestrating the ETL (Extract, Transform, Load) jobs
Skills needed: DevOps, Building Pipelines, Python
Key Takeaway
Diversity is powerful… it helps your Data Team
Create a good culture
Bring different mindsets together
Be inclusive
Learn and grow
Panel: What Happens When your Infrastructure Doesn’t Scale Anymore
Mark Freeman II, Senior Data Scientist @ On the Mark Data
Richad Nieves-Becker, Sr. Assoc. VP, Data Science @ Revantage
Ben Doremus, Chief Technology Officer @ Magenta
Sarah Floris, Senior Data & ML Engineer @ ZWIFT
What are the early signs that your data infrastructure isn’t scaling anymore?
Sarah starts us off by sharing that she sees this problem on a regular basis, and has experienced scaling issues at startups and medium companies. The first sign is when you start getting alerts, a lot of them! The engineers are then tasked to immediately fix everything and are typically working with a reactive mindset. To help this reactive mindset shift, she likes to set things up so that things are monitored correctly and there is a backup of logs available. Those are nice for the ‘what happened’ questions from leadership.
Ben amusingly shared that you can simply look at the velocity of the backlog and know that things are not right. He shares that if you have manual tasks that start overloading, then you are in need of re-structuring. He painted a picture of a company with all automated tasks, except for one > the initial load of the data (by Martha). Now just imagine what happens when Martha goes on vacation…
Richad mentioned that you should keep track of the conversations that you have on your direct reports. High level themes of what is going well / not so well. This could highlight potential problem areas in the infrastructure and potential improvement opportunities. One other sign, related to Ben’s first point, is if the product roadmap becomes a wish list. This is because you have had to push everything out to fit the emergency / ad hoc / reactive requests.
Can you share your experiences with the build vs. buy debate?
Ben said that if you have a wide open budget, you can do anything. If not, you need to figure out what will be the most valuable. Cloud is more popular because people don’t want to have to manage that side of the house, so they can focus on other things which sometimes is worth it. As an example, everyone has Salesforce but not a lot of people want to build out the integrations needed for their business use cases.
Who is doing the research on this build vs. buy decisions?
Sarah says that it depends on your approach. She agrees with Ben in his definition of the debate and adds that it really depends on how quickly things can get up and running. This includes development, implementation, and onboarding. Richad shares his perspective on how this works at larger companies and says it basically takes longer for decisions to be made. Also, if you are looking to buy, ideally you want to buy at a variable rate so that you can start small in costs and the tool can scale with your business. This is a plus for the vendors too since they love these long-term relationships. Ben wrapped us up with the concept of time = money and if you want to get things done > do what it takes to get them done.
Sometimes you have to endure up front costs for long-term ROI. How do you get leadership onboard?
Richad shares his corporate perspective with two options. 1) Give an executive your MVP version of the app (it will be very slow) and let the infrastructure speak for itself ..or.. 2) become a great storyteller. With storytelling, the goal is to paint a picture of what the experience will be like in a concrete manner and then tie it to business outcomes, where you want to be in 3 years, and how that relates to your roadmap. Remember to use language that they already know and a good tip is to work backwards from value. Ben reminds us that the problem is not always the money, it’s about getting all the teams to understand that it is a priority to get the work done.
Closing Thoughts?
Richad is happy to see that infrastructure is starting to become a bigger topic. Hopefully one day we can tackle data cleaning, although that can be very hard to automate.
Sarah emphasizes that this work is an iterative process, and always will be. Giving power and strength to get things started is the biggest hurdle, but trust the system and the work will get done.
Ben officially closes us out with this: be aware, talk to people, try to deal with problems before they get too big.
Resources
If you would like to watch any and all of the sessions from this year's event, or past year's events, visit the Data Teams Summit website
Thank you for reading, Thank you for supporting
Comments