Data scientists won’t need to switch tools as often, Google said.
Google Cloud is adding new features to Colab in BigQuery that, it said, will boost the productivity of data scientists.
The role of the data scientist is undergoing a rapid transformation, according to Yasmeen Ahmad, managing director of Data Cloud at Google. In a blog post, Ahmad contends that, while data scientists have traditionally focused on analyzing historical data to build predictive models that guide business decisions, this approach is no longer sufficient.
“The market now demands that data scientists build the future by designing and deploying autonomous agents that can reason, act, and learn on behalf of the enterprise,” she wrote.
Although data scientists’ roles have evolved, the tools they work with haven’t, and one of the biggest challenges they face is context-switching: writing SQL in one client, exporting data, loading it into a Python notebook, configuring a separate Spark cluster for heavy lifting, and then switching to a BI tool just to visualize results, she said.
This switching between different tools, according to Ahmad, kills productivity and in order to bypass these issues, Google has made enhancements to Colab Enterprise, its managed data science notebook environment, inside BigQuery and VertexAI.
These enhancements, which are in preview, include Native SQL Cells, Rich Interactive Visualization Cells, and added support via the Data Science Agent.
The Native SQL Cells will help data scientists iterate on SQL queries and Python code in the same place and also allow them to immediately pipe SQL query results into a BigQuery DataFrame to build models in Python, the Rich Interactive Visualization Cells helps by automatically generating editable charts from data, giving quicker access to analysis, Ahmad wrote.
Additionally, the Data Science Agent in Colab has been updated and it can now generate detailed plans that will guide data scientists on when to use BigQuery ML, BigQuery DataFrames, or large-scale Spark transformations, she added.
Analysts see these enhancements benefitting data scientists.
“By unifying SQL, Python, Spark, and visualization inside BigQuery, via notebooks, Google is eliminating brittle handoffs and giving engineers back hours lost to tool-juggling,” Michael Ni, principal analyst at Constellation Research, said.
“In addition, with the enhancements to the Data Science Agent that guide choices like Spark versus DataFrames, teams avoid trial-and-error and the frustration that comes with it,” Ni added.
“Google is the first to collapse all functions and tools into a single developer flow. Not a single vendor has consolidated as many typical data engineering tools that drive the swivel-chair problem like BigQuery just did. This is a productivity play that translates directly into faster time-to-insight” Ni said.
