Databricks Platform Discussions
Dive into comprehensive discussions covering various aspects of the Databricks platform. Join the co...
14948 Posts
Activity in Discussions
- 87 Views
- 1 replies
- 0 kudos
Results from the spark application to driver
I tried to read many articles but still not clear on this:The executors complete the execution of tasks and have the results with them.1. The results(output data) from all executors is transported to driver in all cases or executors persist it if tha...
- 87 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @raghvendrarm1 - Have a look at the section "Apache Spark’s Distributed Execution" in chapter 1 of Learning Spark, 2nd Edition (https://www.oreilly.com/library/view/learning-spark-2nd/9781492050032/ch01.html). Have a look at teh picture -
- 0 kudos
- 63 Views
- 1 replies
- 0 kudos
Cannot import pyspark.pipelines module
Question: What could be a cause of the following error of my code in a Databricks notebook, and how can we fix the error? I'm using latest Free Edition of Databricks that has runtime version 17.2 and PySpark version 4.0.0.Error:ImportError: cannot im...
- 63 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @Saf4Databricks - Are you trying to use it from a standalone Databricks notebook? You should only use it from with Lakeflow Declarative Pipeline (LDP). The link you shared is about LDP. Here is an example where I used it.
- 0 kudos
- 1226 Views
- 10 replies
- 8 kudos
Resolved! Connect to azure data lake storage using databricks free edition
Hello guys, i'm using databricks free edition (serverless) and i am trying to connect to a azure data lake storage.The problem I'm having is that in the free edition we can't configure the cluster so I tried to make the connection via notebook using ...
- 1226 Views
- 10 replies
- 8 kudos
Latest Reply
- 8 kudos
@TalessRocha thanks for getting back to us! Glad to hear you got it working, that's awesome. Best of luck with your projects.All the best,BS
- 8 kudos
- 351 Views
- 8 replies
- 1 kudos
getting 500 on embedding model invocation call
I'm getting the following error message "{"error_code": "INTERNAL_ERROR", "message": "The server received an invalid response from an upstream server."}" when making a call to bge-large-en embedding model.
- 351 Views
- 8 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @tefrati - Can you please go to the model serving endpoint and click on the use drop down (as shown in the picture below) and try the simple python or sql and see if that works for you. from openai import OpenAI import os # How to get your Databr...
- 1 kudos
- 18 Views
- 0 replies
- 0 kudos
How to make a song for peace is a love you so jao ab
How to make sure you will be sala Uddin Ahmed to deewane season of the same hai bhai send your number and the first to make it to you too baby
- 18 Views
- 0 replies
- 0 kudos
- 3087 Views
- 2 replies
- 0 kudos
New default notebook format (IPYNB) causes unintended changes on release
Dear Databricks,We have noticed the following issue since the new default notebook format has been set to IPYNB. When we release our code from (for example) DEV to TST using a release pipeline built in Azure DevOps, we see unintended changes popping ...
- 3087 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Hi @Rvwijk, please take a look at this. This should solve your issue. I suspect the mismatch is happening due to the previous ones, including output for the notebook cells. You may need to perform a rebase of your repository and allow the output to b...
- 0 kudos
- 21 Views
- 0 replies
- 1 kudos
Migrate External Tables to Managed
With managed tables, you can reduce your storage and compute costs thanks to predictive optimization or file list caching. Now it is really time to migrate external tables to managed ones, thanks to ALTER SET MANAGED functionality. Read more: - h...
- 21 Views
- 0 replies
- 1 kudos
- 26 Views
- 1 replies
- 0 kudos
Databricks Certification: Booking Slot While Taking Partner Academy Course?
Hi team,I'm currently diving into the Partnered Academy learning plan (e.g., Data Engineer Associate, ML Practitioner, etc.) and I'm looking to schedule my certification exam.I'm wondering about the official policy: Is it mandatory to complete the en...
- 26 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
From my experience everything is independent from each other
- 0 kudos
- 35 Views
- 0 replies
- 0 kudos
Cannot export embedded dashboard widget as CSV or other formats except PNG
I’ve integrated a Databricks dashboard into my web application for all my users, following the guidelines in this article:Embedding Databricks Dashboards.This integration worked perfectly initially. However, I’m now encountering an issue with exporti...
- 35 Views
- 0 replies
- 0 kudos
- 34824 Views
- 47 replies
- 30 kudos
community edition : "User is not a member of this workspace"
Something very strange has happened. When trying to login to my databricks community edition account. I'm getting the email with my verification code. but after entring that code, I'm getting the error message : "User is not a member of this workspac...
- 34824 Views
- 47 replies
- 30 kudos
Latest Reply
- 30 kudos
I'm also facing the same error on registering to community edition. After providing correct verification code, I'm getting error "User is not a member of this workspace". The url -> https://community.cloud.databricks.com/verify-code?tuuid=2cd7f036-38...
- 30 kudos
- 1842 Views
- 3 replies
- 1 kudos
Best practices for optimizing Spark jobs
What are some best practices for optimizing Spark jobs in Databricks, especially when dealing large datasets? Any tips or resources would be greatly appreciated! I’m trying to analyze data on restaurant menu prices so that insights would be especiall...
- 1842 Views
- 3 replies
- 1 kudos
Latest Reply
- 1 kudos
Their menu is full of creative options — it’s hard to pick just one favorite because everything tastes amazing!”
- 1 kudos
- 2647 Views
- 2 replies
- 0 kudos
Resolved! Whether Databricks has any plans to Integrate Databricks Genie with Azure OpenAI?
Hi,I am checking to see if Databricks has any plans to Integrate Databricks Genie with Azure OpenAI? as I know Databricks Genie is designed to work with Databricks' data ecosystem itself. But is there any plans to enhance this capability where we can...
- 2647 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
Thank you Louis, yes I believe when I posted in May it wasn’t available. Now I can see the option. Thanks for the reply.
- 0 kudos
- 181 Views
- 1 replies
- 0 kudos
AIM with Entra ID Groups – Users and Service Principals not visible in Workspace
Hello Community, I am testing Automatic Identity Management (AIM) in Databricks with Unity Catalog enabled. Steps I did: • AIM is activated • In Microsoft Entra ID I created a group g1 and added user u1 and service principal sp1 ...
- 181 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
In Azure Databricks, when AIM is enabled, Entra users, service principals, and groups are available in Azure Databricks as soon as they’re granted permissions. Group memberships, including nested groups, flow directly from Entra ID, so permissions al...
- 0 kudos
- 11426 Views
- 16 replies
- 3 kudos
Is it possible to view Databricks cluster metrics using REST API
I am looking for some help on getting databricks cluster metrics such as memory utilization, CPU utilization, memory swap utilization, free file system using REST API.I am trying it in postman using databricks token and with my Service Principal bear...
- 11426 Views
- 16 replies
- 3 kudos
Latest Reply
- 3 kudos
Is there any solution found to get cpu, memory metrics for Hive meta store backed workloads ? We are not using UC. So can't use system tables
- 3 kudos
- 71 Views
- 2 replies
- 0 kudos
DLT Pipeline Stopped working
Hello, Suddenly our DLT pipelines we're getting failures saying thatLookupError: Traceback (most recent call last): result_df = result_df.withColumn("input_file_path", col("_metadata.file_path")).withColumn( ...
- 71 Views
- 2 replies
- 0 kudos
Latest Reply
- 0 kudos
May be there is internally some updates from databricks Can Check and Switch Your Pipeline Channel, In the DLT pipeline settings (under Advanced > Channel), confirm if it's set to "Preview". Switch to "Current" for a more stable engine version, then...
- 0 kudos
| User | Count |
|---|---|
| 1841 | |
| 920 | |
| 806 | |
| 476 | |
| 317 |