How to Register & Prepare If you're interested in advancing your skills with Databricks through a Specialist Session, here's a clear guide on how to register and what free courses you can take to prepare effectively. How to Begin Your Learning Path S...
- 1016 Views
- 0 replies
- 3 kudos
2 weeks ago
BrickCon 2025 | Dec 3–5, 2025 A community-driven conference built by and for Databricks users around the world. Get ready for an unforgettable experience at BrickCon 2025 — the first-ever community-led Databricks conference bringing together da...
- 555 Views
- 3 replies
- 4 kudos
2 weeks ago
We are excited to spotlight Nayanjyoti Sonowal, one of our most active and insightful contributors in the Databricks Community, whose journey reflects the spirit of learning, collaboration, and innovation we value most. Dive into his story, told in h...
- 602 Views
- 5 replies
- 11 kudos
Tuesday
Healthy gaming communities start with positive interactions. With this Databricks Solution Accelerator, you can: Detect toxic comments in real time using NLP techniquesFlag questionable chat messages for moderationIntegrate seamlessly into existing c...
- 1324 Views
- 4 replies
- 4 kudos
2 weeks ago
We kept the October momentum going strong — with vibrant technical discussions, creative experiments, and deep dives into AI, streaming, and governance. Here’s a look at what stood out this week in the Databricks Community Fresh Content & Articles...
- 399 Views
- 1 replies
- 4 kudos
Wednesday
As we close out September and step into October, it’s incredible to see how our top contributors continue to lead with consistency, clarity, and collaboration. Week after week, they’ve stayed at the forefront of discussions, setting the tone for know...
- 1140 Views
- 4 replies
- 9 kudos
2 weeks ago
Learn, Level Up, and Get Rewarded! Save the dates: October 10 – October 31, 2025! We're celebrating learning, growth, and career advancement with an exciting opportunity to level up in data engineering, data analysis, machine learning, and genera...
- 150052 Views
- 313 replies
- 111 kudos
08-07-2025
Data + AI World Tour continues the momentum from Data + AI Summit, the world’s premier event for the data, analytics and AI community. Now we’re coming to a city near you — with 19 stops across five continents. Find out what’s next in data and AI, an...
- 3178 Views
- 8 replies
- 2 kudos
a month ago
Community Activity
- 87 Views
- 1 replies
- 0 kudos
Results from the spark application to driver
I tried to read many articles but still not clear on this:The executors complete the execution of tasks and have the results with them.1. The results(output data) from all executors is transported to driver in all cases or executors persist it if tha...
- 87 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @raghvendrarm1 - Have a look at the section "Apache Spark’s Distributed Execution" in chapter 1 of Learning Spark, 2nd Edition (https://www.oreilly.com/library/view/learning-spark-2nd/9781492050032/ch01.html). Have a look at teh picture -
- 0 kudos
- 63 Views
- 1 replies
- 0 kudos
Cannot import pyspark.pipelines module
Question: What could be a cause of the following error of my code in a Databricks notebook, and how can we fix the error? I'm using latest Free Edition of Databricks that has runtime version 17.2 and PySpark version 4.0.0.Error:ImportError: cannot im...
- 63 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @Saf4Databricks - Are you trying to use it from a standalone Databricks notebook? You should only use it from with Lakeflow Declarative Pipeline (LDP). The link you shared is about LDP. Here is an example where I used it.
- 0 kudos
- 1226 Views
- 10 replies
- 8 kudos
Resolved! Connect to azure data lake storage using databricks free edition
Hello guys, i'm using databricks free edition (serverless) and i am trying to connect to a azure data lake storage.The problem I'm having is that in the free edition we can't configure the cluster so I tried to make the connection via notebook using ...
- 1226 Views
- 10 replies
- 8 kudos
Latest Reply
- 8 kudos
@TalessRocha thanks for getting back to us! Glad to hear you got it working, that's awesome. Best of luck with your projects.All the best,BS
- 8 kudos
- 351 Views
- 8 replies
- 1 kudos
getting 500 on embedding model invocation call
I'm getting the following error message "{"error_code": "INTERNAL_ERROR", "message": "The server received an invalid response from an upstream server."}" when making a call to bge-large-en embedding model.
- 351 Views
- 8 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @tefrati - Can you please go to the model serving endpoint and click on the use drop down (as shown in the picture below) and try the simple python or sql and see if that works for you. from openai import OpenAI import os # How to get your Databr...
- 1 kudos
- 18 Views
- 0 replies
- 0 kudos
How to make a song for peace is a love you so jao ab
How to make sure you will be sala Uddin Ahmed to deewane season of the same hai bhai send your number and the first to make it to you too baby
- 18 Views
- 0 replies
- 0 kudos
- 3087 Views
- 2 replies
- 0 kudos
New default notebook format (IPYNB) causes unintended changes on release
Dear Databricks,We have noticed the following issue since the new default notebook format has been set to IPYNB. When we release our code from (for example) DEV to TST using a release pipeline built in Azure DevOps, we see unintended changes popping ...
- 3087 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @Rvwijk, please take a look at this. This should solve your issue. I suspect the mismatch is happening due to the previous ones, including output for the notebook cells. You may need to perform a rebase of your repository and allow the output to b...
- 0 kudos
- 21 Views
- 0 replies
- 1 kudos
Migrate External Tables to Managed
With managed tables, you can reduce your storage and compute costs thanks to predictive optimization or file list caching. Now it is really time to migrate external tables to managed ones, thanks to ALTER SET MANAGED functionality. Read more: - h...
- 21 Views
- 0 replies
- 1 kudos
- 26 Views
- 1 replies
- 0 kudos
Databricks Certification: Booking Slot While Taking Partner Academy Course?
Hi team,I'm currently diving into the Partnered Academy learning plan (e.g., Data Engineer Associate, ML Practitioner, etc.) and I'm looking to schedule my certification exam.I'm wondering about the official policy: Is it mandatory to complete the en...
- 26 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
From my experience everything is independent from each other
- 0 kudos
- 35 Views
- 0 replies
- 0 kudos
Cannot export embedded dashboard widget as CSV or other formats except PNG
I’ve integrated a Databricks dashboard into my web application for all my users, following the guidelines in this article:Embedding Databricks Dashboards.This integration worked perfectly initially. However, I’m now encountering an issue with exporti...
- 35 Views
- 0 replies
- 0 kudos
- 34824 Views
- 47 replies
- 30 kudos
community edition : "User is not a member of this workspace"
Something very strange has happened. When trying to login to my databricks community edition account. I'm getting the email with my verification code. but after entring that code, I'm getting the error message : "User is not a member of this workspac...
- 34824 Views
- 47 replies
- 30 kudos
Latest Reply
- 30 kudos
I'm also facing the same error on registering to community edition. After providing correct verification code, I'm getting error "User is not a member of this workspace". The url -> https://community.cloud.databricks.com/verify-code?tuuid=2cd7f036-38...
- 30 kudos
- 1842 Views
- 3 replies
- 1 kudos
Best practices for optimizing Spark jobs
What are some best practices for optimizing Spark jobs in Databricks, especially when dealing large datasets? Any tips or resources would be greatly appreciated! I’m trying to analyze data on restaurant menu prices so that insights would be especiall...
- 1842 Views
- 3 replies
- 1 kudos
Latest Reply
- 1 kudos
Their menu is full of creative options — it’s hard to pick just one favorite because everything tastes amazing!”
- 1 kudos
- 2647 Views
- 2 replies
- 0 kudos
Resolved! Whether Databricks has any plans to Integrate Databricks Genie with Azure OpenAI?
Hi,I am checking to see if Databricks has any plans to Integrate Databricks Genie with Azure OpenAI? as I know Databricks Genie is designed to work with Databricks' data ecosystem itself. But is there any plans to enhance this capability where we can...
- 2647 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Thank you Louis, yes I believe when I posted in May it wasn’t available. Now I can see the option. Thanks for the reply.
- 0 kudos
- 181 Views
- 1 replies
- 0 kudos
AIM with Entra ID Groups – Users and Service Principals not visible in Workspace
Hello Community, I am testing Automatic Identity Management (AIM) in Databricks with Unity Catalog enabled. Steps I did: • AIM is activated • In Microsoft Entra ID I created a group g1 and added user u1 and service principal sp1 ...
- 181 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
In Azure Databricks, when AIM is enabled, Entra users, service principals, and groups are available in Azure Databricks as soon as they’re granted permissions. Group memberships, including nested groups, flow directly from Entra ID, so permissions al...
- 0 kudos
- 11426 Views
- 16 replies
- 3 kudos
Is it possible to view Databricks cluster metrics using REST API
I am looking for some help on getting databricks cluster metrics such as memory utilization, CPU utilization, memory swap utilization, free file system using REST API.I am trying it in postman using databricks token and with my Service Principal bear...
- 11426 Views
- 16 replies
- 3 kudos
Latest Reply
- 3 kudos
Is there any solution found to get cpu, memory metrics for Hive meta store backed workloads ? We are not using UC. So can't use system tables
- 3 kudos
- 71 Views
- 2 replies
- 0 kudos
DLT Pipeline Stopped working
Hello, Suddenly our DLT pipelines we're getting failures saying thatLookupError: Traceback (most recent call last): result_df = result_df.withColumn("input_file_path", col("_metadata.file_path")).withColumn( ...
- 71 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
May be there is internally some updates from databricks Can Check and Switch Your Pipeline Channel, In the DLT pipeline settings (under Advanced > Channel), confirm if it's set to "Preview". Switch to "Current" for a more stable engine version, then...
- 0 kudos
Announcements
| User | Count |
|---|---|
| 2109 | |
| 1151 | |
| 946 | |
| 922 | |
| 754 |
Latest from our Blog
Azure Databricks — Serverless Private Connectivity to Customer’s Resources (Part-1)
Authors: Kiran Anand, Suraj Karuvel Introduction The serverless model provides customers with a fast and easy option to run their workloads in Azure Databricks. In this model, the compute runs in Dat...
How to Deploy Agent-to-Agent (A2A) Protocol on Databricks Apps?-> End-to-End Guide
Modern AI systems rarely live alone. They collaborate. Google’s Agent-to-Agent (A2A) protocol gives agents a common language (JSON-RPC over HTTP/S, with streaming via SSE) plus a discovery mechanism (...
IoT Ingestion Using Append Flows
IoT (Internet-of-Things) devices are disrupting the manufacturing industry. While estimates vary, most sources agree that there are tens of billions of IoT devices currently operating in manufacturing...
Turning GPS Pings into Insights: Mobile Fleet Analytics with Databricks Spatial SQL
Turning GPS Pings into Insights: Mobile Fleet Analytics with Databricks Spatial SQL Energy companies manage large and complex fleets of mobile equipment - from service trucks in a utility company, to ...