Accessibility Statement Skip Navigation
  • Resources
  • Blog
  • Journalists
  • +44 (0)20 7454 5110
  • Client Login
  • Send a Release
Return to PR Newswire homepage
  • News
  • Products
  • Contact
When typing in this field, a list of search results will appear and be automatically updated as you type.

Searching for your content...

No results found. Please change your search terms and try again.
  • News in Focus
      • Browse News Releases

      • All Public Company News
      • All Multimedia News
      • View All News Releases

      • Regulatory News

      • D/A/CH Regulatory News
      • UK Regulatory News
      • View All Regulatory News

  • Business & Money
      • Auto & Transportation

      • Aerospace & Defense
      • Air Freight
      • Airlines & Aviation
      • Automotive
      • Maritime & Shipbuilding
      • Railroads & Intermodal Transportation
      • Supply Chain/Logistics
      • Transportation, Trucking & Railroad
      • Travel
      • Trucking & Road Transportation
      • View All Auto & Transportation

      • Business Technology

      • Blockchain
      • Broadcast Tech
      • Computer & Electronics
      • Computer Hardware
      • Computer Software
      • Data Analytics
      • Electronic Commerce
      • Electronic Components
      • Electronic Design Automation
      • Financial Technology
      • High Tech Security
      • Internet Technology
      • Nanotechnology
      • Networks
      • Peripherals
      • Semiconductors
      • View All Business Technology

      • Entertain­ment & Media

      • Advertising
      • Art
      • Books
      • Entertainment
      • Film & Motion Picture
      • Magazines
      • Music
      • Publishing & Information Services
      • Radio & Podcast
      • Television
      • View All Entertain­ment & Media

      • Financial Services & Investing

      • Accounting News & Issues
      • Acquisitions, Mergers & Takeovers
      • Banking & Financial Services
      • Bankruptcy
      • Bond & Stock Ratings
      • Conference Call Announcements
      • Contracts
      • Cryptocurrency
      • Dividends
      • Earnings
      • Earnings Forecasts & Projections
      • Financing Agreements
      • Insurance
      • Investments Opinions
      • Joint Ventures
      • Mutual Funds
      • Private Placement
      • Real Estate
      • Restructuring & Recapitalisation
      • Sales Reports
      • Shareholder Activism
      • Shareholder Meetings
      • Stock Offering
      • Stock Split
      • Venture Capital
      • View All Financial Services & Investing

      • General Business

      • Awards
      • Commercial Real Estate
      • Corporate Expansion
      • Earnings
      • Environmental, Social and Governance (ESG)
      • Human Resource & Workforce Management
      • Licensing
      • New Products & Services
      • Obituaries
      • Outsourcing Businesses
      • Overseas Real Estate (non-US)
      • Personnel Announcements
      • Real Estate Transactions
      • Residential Real Estate
      • Small Business Services
      • Socially Responsible Investing
      • Surveys, Polls & Research
      • Trade Show News
      • View All General Business

  • Science & Tech
      • Consumer Technology

      • Artificial Intelligence
      • Blockchain
      • Cloud Computing/Internet of Things
      • Computer Electronics
      • Computer Hardware
      • Computer Software
      • Consumer Electronics
      • Cryptocurrency
      • Data Analytics
      • Electronic Commerce
      • Electronic Gaming
      • Financial Technology
      • Mobile Entertainment
      • Multimedia & Internet
      • Peripherals
      • Social Media
      • STEM (Science, Tech, Engineering, Math)
      • Supply Chain/Logistics
      • Wireless Communications
      • View All Consumer Technology

      • Energy & Natural Resources

      • Alternative Energies
      • Chemical
      • Electrical Utilities
      • Gas
      • General Manufacturing
      • Mining
      • Mining & Metals
      • Oil & Energy
      • Oil & Gas Discoveries
      • Utilities
      • Water Utilities
      • View All Energy & Natural Resources

      • Environ­ment

      • Conservation & Recycling
      • Environmental Issues
      • Environmental Policy
      • Environmental Products & Services
      • Green Technology
      • Natural Disasters
      • View All Environ­ment

      • Heavy Industry & Manufacturing

      • Aerospace & Defence
      • Agriculture
      • Chemical
      • Construction & Building
      • General Manufacturing
      • HVAC (Heating, Ventilation & Air-Conditioning)
      • Machinery
      • Machine Tools, Metalworking & Metallurgy
      • Mining
      • Mining & Metals
      • Paper, Forest Products & Containers
      • Precious Metals
      • Textiles
      • Tobacco
      • View All Heavy Industry & Manufacturing

      • Telecomm­unications

      • Carriers & Services
      • Mobile Entertainment
      • Networks
      • Peripherals
      • Telecommunications Equipment
      • Telecommunications Industry
      • VoIP (Voice over Internet Protocol)
      • Wireless Communications
      • View All Telecomm­unications

  • Lifestyle & Health
      • Consumer Products & Retail

      • Animals & Pets
      • Beers, Wines & Spirits
      • Beverages
      • Bridal Services
      • Cannabis
      • Cosmetics & Personal Care
      • Fashion
      • Food & Beverages
      • Furniture & Furnishings
      • Home Improvement
      • Household, Consumer & Cosmetics
      • Household Products
      • Jewellery
      • Non-Alcoholic Beverages
      • Office Products
      • Organic Food
      • Product Recalls
      • Restaurants
      • Retail
      • Supermarkets
      • Toys
      • View All Consumer Products & Retail

      • Entertain­ment & Media

      • Advertising
      • Art
      • Books
      • Entertainment
      • Film & Motion Picture
      • Magazines
      • Music
      • Publishing & Information Services
      • Radio & Podcast
      • Television
      • View All Entertain­ment & Media

      • Health

      • Biometrics
      • Biotechnology
      • Clinical Trials & Medical Discoveries
      • Dentistry
      • FDA Approval
      • Fitness/Wellness
      • Health Care & Hospitals
      • Health Insurance
      • Infection Control
      • International Medical Approval
      • Medical Equipment
      • Medical Pharmaceuticals
      • Mental Health
      • Pharmaceuticals
      • Supplementary Medicine
      • View All Health

      • Sports

      • General Sports
      • Outdoors, Camping & Hiking
      • Sporting Events
      • Sports Equipment & Accessories
      • View All Sports

      • Travel

      • Amusement Parks & Tourist Attractions
      • Gambling & Casinos
      • Hotels & Resorts
      • Leisure & Tourism
      • Outdoors, Camping & Hiking
      • Passenger Aviation
      • Travel Industry
      • View All Travel

  • Policy & Public Interest
      • Policy & Public Interest

      • Animal Welfare
      • Corporate Social Responsibility
      • Economic News, Trends & Analysis
      • Education
      • Environmental
      • European Government
      • Labour & Union
      • Natural Disasters
      • Not For Profit
      • Public Safety
      • View All Policy & Public Interest

  • People & Culture
      • People & Culture

      • Aboriginal, First Nations & Native American
      • African American
      • Asian American
      • Children
      • Diversity, Equity & Inclusion
      • Hispanic
      • Lesbian, Gay & Bisexual
      • Men's Interest
      • People with Disabilities
      • Religion
      • Senior Citizens
      • Veterans
      • Women
      • View All People & Culture

  • Overview
  • Distribution
  • Paid Placement
  • Multichannel Amplification
  • Disclosure Services
  • SocialBoost
  • Rooms
    • MediaRoom
    • ESG Rooms
  • AI Tools
  • General Enquiries
  • Media Enquiries
  • Partnerships
  • Hamburger menu
  • Cision PR Newswire UK provides press release distribution, targeting, monitoring, and marketing services
  • Send a Release
    • Phone

    • +44 (0)20 7454 5110 from 8 AM - 5:30 PM GMT

    • ALL CONTACT INFO
    • Contact Us

      +44 (0)20 7454 5110
      from 8 AM - 5:30 PM GMT

  • Client Login
  • Send a Release
  • Resources
  • Blog
  • Journalists
  • News in Focus
    • Browse News Releases
    • Regulatory News
  • Business & Money
    • Auto & Transportation
    • Business Technology
    • Entertain­ment & Media
    • Financial Services & Investing
    • General Business
  • Science & Tech
    • Consumer Technology
    • Energy & Natural Resources
    • Environ­ment
    • Heavy Industry & Manufacturing
    • Telecomm­unications
  • Lifestyle & Health
    • Consumer Products & Retail
    • Entertain­ment & Media
    • Health
    • Sports
    • Travel
  • Policy & Public Interest
    • Policy & Public Interest
  • People & Culture
    • People & Culture
  • Client Login
  • Send a Release
  • Resources
  • Blog
  • Journalists
  • Overview
  • Distribution
  • Paid Placement
  • Multichannel Amplification
  • Disclosure Services
  • Cision Communications Cloud®
  • AI Tools
  • Client Login
  • Send a Release
  • Resources
  • Blog
  • Journalists
  • General Enquiries
  • Media Enquiries
  • Partnerships
  • Client Login
  • Send a Release
  • Resources
  • Blog
  • Journalists

Databricks Donates Declarative Pipelines to Apache Spark™ Open Source Project

This image opens in the lightbox

News provided by

Databricks

11 Jun, 2025, 13:00 GMT

Share this article

Share toX

Share this article

Share toX

SAN FRANCISCO, June 11, 2025 /PRNewswire/ -- Data + AI Summit -- Databricks, the Data and AI company, today announced it is open-sourcing the company's core declarative ETL framework as Apache Spark™ Declarative Pipelines. This initiative comes on the heels of Apache Spark reaching two billion downloads and the recent launch of Apache Spark 4.0. These releases build on Databricks' long-standing commitment to open ecosystems, ensuring users have the flexibility and control they need without vendor lock-in. Spark Declarative Pipelines tackles one of the biggest challenges in data engineering, making it easy to build and operate reliable, scalable data pipelines end-to-end.

Spark Declarative Pipelines provides an easier way to define and execute data pipelines for both batch and streaming ETL workloads across any Apache Spark-supported data source, including cloud storage, message buses, change data feeds and external systems. This battle-tested declarative framework for building data pipelines helps engineers address common pain points like complex pipeline authoring, manual operations overhead and siloed batch/streaming.

Spark Declarative Pipelines is based on Databricks' core declarative ETL framework, which is used by thousands of customers. With the proven ability to handle complex data engineering workloads and low-latency streaming, Spark Declarative Pipelines lays the foundation for the next generation of data processing and governance. With Spark Declarative Pipelines, more community members can begin to cut engineering time and costs and reliably support new AI agent systems and other workloads in production.

"Our commitment to open source is unwavering. With origins in academia and the open source community, Databricks was founded in 2013 by the original creators of the lakehouse architecture and open source projects including Apache Spark, Delta Lake, MLflow and Unity Catalog," said Matei Zaharia, Co-founder and CTO of Databricks. "We worked closely with the community to help remove friction around data formats that kept information siloed. Spark Declarative Pipelines now gives enterprises an open way to build high-quality pipelines."

Key benefits of Spark Declarative Pipelines include:

  • Simplifying pipeline authoring: Data engineers and analysts can quickly declare robust pipelines with minimal coding, focusing on delivering business-critical insights.
  • Improved operability by design: Spark Declarative Pipelines help catch issues earlier in development through clear pipeline definitions that are validated in full prior to execution, reducing the risk of failures downstream and making pipelines easier to troubleshoot and maintain.
  • Unified batch and streaming: Data teams can flexibly meet both real-time and periodic processing needs through a single API for defining and managing batch and streaming data pipelines, simplifying development and maintenance.

"Declarative pipelines hide the complexity of modern data engineering under a simple, intuitive programming model. As an engineering manager, I love the fact that my engineers can focus on what matters most to the business. It's exciting to see this level of innovation now being open-sourced, making it accessible to even more teams." — Jian (Miracle) Zhou, Senior Engineering Manager, Navy Federal Credit Union

"At 84.51˚ we're always looking for ways to make our data pipelines easier to build and maintain, especially as we move toward more open and flexible tools. The declarative approach has been a big help in reducing the amount of code we have to manage, and it's made it easier to support both batch and streaming without stitching together separate systems. Open-sourcing this framework as Spark Declarative Pipelines is a great step for the Spark community." — Brad Turnbaugh, Sr. Data Engineer, 84.51°

About Databricks
Databricks is the Data and AI company. More than 15,000 organizations worldwide — including Block, Comcast, Condé Nast, Rivian, Shell and over 60% of the Fortune 500 — rely on the Databricks Data Intelligence Platform to take control of their data and put it to work with AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache Spark™, Delta Lake, MLflow, and Unity Catalog. To learn more, follow Databricks on X, LinkedIn and Facebook.

Contact: Press@databricks.com

Logo - https://mma.prnewswire.com/media/1160675/Databricks_Logo.jpg

Modal title

Also from this source

Databricks and Microsoft Extend Strategic Partnership for Azure Databricks

Databricks and Microsoft Extend Strategic Partnership for Azure Databricks

Data + AI Summit -- Databricks, the Data and AI company, and Microsoft today announced a multi-year early extension of their strategic partnership,...

Databricks Announces Strategic AI Partnership with Google Cloud to Bring Gemini Models Natively to the Data Intelligence Platform

Databricks Announces Strategic AI Partnership with Google Cloud to Bring Gemini Models Natively to the Data Intelligence Platform

Databricks, the Data and AI company, today announced a new strategic product partnership with Google Cloud to make the latest Gemini models available ...

More Releases From This Source

Explore

Computer & Electronics

Computer & Electronics

Data Analytics

Data Analytics

Data Analytics

Data Analytics

Computer Software

Computer Software

News Releases in Similar Topics

Contact PR Newswire

  • +44 (0)20 7454 5110
    from 8 AM - 5:30 PM GMT
  • General Enquiries
  • Media Enquiries
  • Partnerships

Products

  • Content Distribution
  • Multimedia Services
  • Disclosure Services
  • Cision Communications Cloud®

About

  • About PR Newswire
  • About Cision
  • Partnering Opportunities
  • Careers
  • Accessibility Statement
  • APAC
  • APAC - Simplified Chinese
  • APAC - Traditional Chinese
  • Brazil
  • Canada
  • Czech
  • Denmark
  • Finland
  • France
  • Germany
  • India
  • Indonesia
  • Israel
  • Japan
  • Korea
  • Mexico
  • Middle East
  • Middle East - Arabic
  • Netherlands
  • Norway
  • Poland
  • Portugal
  • Russia
  • Slovakia
  • Spain
  • Sweden
  • United States
  • Vietnam

My Services

  • All News Releases
  • Customer Portal
  • Resources
  • Blog
  • Journalists
  • Data Privacy

Do not sell or share my personal information:

  • Submit via Privacy@cision.com 
  • Call Privacy toll-free: 877-297-8921

Contact PR Newswire

Products

About

My Services
  • All News Releases
  • Customer Portal
  • Resources
  • Blog
  • Journalists
+44 (0)20 7454 5110
from 8 AM - 5:30 PM GMT
  • Terms of Use
  • Privacy Policy
  • Information Security Policy
  • Site Map
  • RSS
  • Cookie Settings
Copyright © 2025 PR Newswire Europe Limited. All Rights Reserved. A Cision company.