get-data-into-celonis

Connect to Systems

28 páginasver na Celonis Academy

Intro

Learning Objectives

Welcome! After completing this course, you will be able to:

Choose and set up the appropriate connection approach Understand how to prepare different system types for a connection Structure your Data Pipeline into Data Pools

This course's focus on connecting to systems:

Personal Training Environment Required

This course contains hands-on exercises that require a personal Celonis Training Environment, also called a "Celonis Team".

Not sure if you have one? Click below and we'll either create one for you or show you how to access your existing Team if you have one. Please make sure to disable adblockers on this page if the button is not working for you.

Check / Create my training environment

This content is available in multiple languages: English, German, French, Japanese, Spanish, and Portuguese. To switch your language, use the selector in the top navigation bar.

In Courses: Switching languages will not affect your progress. If you are multilingual, comparing languages can even help deepen your understanding. In Exams: Do not switch the language once you start the exam. If you switch the language when you're already reviewing questions and answers, the page will refresh, causing your exam attempt to end prematurely. (Note: Once inside the exam course, you can switch the language anytime before clicking the Start Exam button.)

If you encounter any technical or access issues during this course, check out our FAQs in the Support area or reach out to us via our Academy contact form.

Accessibility Support

Celonis Academy is committed to providing an accessible learning experience for everyone. Our content is primarily text and video-based; and is also designed to be consumed via audio using screen readers or text-to-speech software.

Tips for Navigating With a Keyboard or Screen Reader

You can navigate the different interactions within this course using standard screen reader functionalities. Generally, you can use the Tab key to move between interactive elements on each screen. Celonis Academy does not implement specific custom keyboard shortcuts for navigation within our courses, allowing you to use your screen reader's default navigation. For a better viewing experience, you have the option to hide the side menu. To move between the pages of the course, you will find "Previous" and "Next" arrow buttons located at the beginning of each page (just below the page title and process line) and at the end of the content.

Using Screen Readers and Text-to-Speech

For screen reader users, we recommend using VoiceOver on macOS; and Narrator or NVDA on Windows computers. VoiceOver and Narrator are typically pre-installed on your respective operating systems. For a text-to-speech functionality, you can explore browser plugins such as Read Aloud for Google Chrome; or your computer's default accessibility features.

Adjusting Text Size: You can adjust the text size by using your computer or browser's native zoom shortcuts.

Video Captions

Where videos are included, you can enable captions by clicking the "CC" button located in the video timeline. If the course offers content in multiple languages, you will usually have the option to select your preferred language for the captions. If a video does not contain audio, this will be indicated before the video begins.

Further Resources: Discover more accessibility features on Windows or get started with accessibility on macOS.

Note: For an optimal experience, we generally recommend using the Google Chrome browser.

---

Choose and set up the right connection method

Overview

The Main Connection Methods Overview How Celonis Imports Data

Connecting to source systems is your very first step to pull process data into the Celonis platform. Celonis utilizes a broad set of technologies like message queues, Restful APIs, Soap APIs, direct database access, or system-specific solutions to connect.

Data Integration in Celonis

Data Integration is where you set up connections and your data pipeline. The main ways you can bring data into Celonis are:

Process Connectors Extractors (Data Connections) Extractor Builder File Uploads Data Ingestion API Celoxtractor

You can get a first impression of connection methods by simply jumping to the Data Integration UI.

Process Connectors: When creating a Data Pool you can download a Process Connector template:

Other methods: If you create a Data Pool from scratch then the other connection methods are at your disposal:

Note: The methods on this page focus on connecting to transactional systems for Process Mining. If you're looking for information on Task Mining, have a look at the Task Mining Basics course.

Ready to learn? Let's dive deeper into each of these methods.

Process Connectors Overview

Process Connectors Process Connectors Overview

To build your Data Pipeline, in most cases, you’ll use the Process Connectors.

Process Connectors are the fastest and most common way for you to connect and cover the most established process use cases. You can download them from the Celonis Marketplace or when you create a Data Pool.

Out-of-the-box, Connectors give you everything you need to create a standard data model based on the system and process you choose. Once you successfully connect, the connectors support you with templates across the ETL (Extract, Transform, Load) process:

For example, you may want an SAP connector for the Purchase-to-Pay process or an Oracle connector for the Order-to-Cash process, and so on. In this case the connector gives you the right connection possibilities, out-of-the-box templates for extractions and transformations, and a pre-defined data model with table relationships and a load schedule.

When you create a Data Pool in Data Integration, the option to download a Process Connector is immediately available:

After download, you can immediately visualize the assets the Connector provides:

From here, it’s up to you to adapt the Connector to any customizations that may exist in the source system such as new or unique activities in the process flow.

Note : Each connector requires a source system user with permissions to specific tables. You can find the list of required tables for a connector in the Celonis Help or in the extraction of the connector itself.

Try it out - Install a Connector

Process Connectors Try it out - Install a Connector

Follow these 3 simple steps below to install a basic Process Connector in your Personal Training Environment. Click "expand" start with the steps.

Unsure where you personal training environment is or how to access it? Use this button to check:

Check / Create my training environment

Steps

In Data → Data Integration → Click on "+ New Data Pool" in the top right corner.

Click on "Start with Process Connector template"

Search for "Get Data" , select the "Get Data into Celonis - P2P" Connector and install it with "Install Process Connector"

The Result

You should now have a new Data Pool called "Get Data into Celonis - P2P" with a Blank Connection Placeholder, a Data Job, and Task Templates.

Clicking on the Data Job in the graphic, you should see your task templates:

Keep in mind this is a stripped-down Process Connector. Normally you would have a more complex structure of Data Jobs, Schedules, and Data Model(s).

If this is overwhelming don't worry! You will learn more about these objects as you progress through the Get Data into Celonis Track.

Extractors Overview

Extractors Extractors Overview

Extractors are blank data connections with no reference to a process. With Extractors, you connect to source systems and then have to build your data pipeline from scratch—i.e. your extractions, transformations, and Data Model. You can think of Extractors as the “connection part” of Process Connectors from the Marketplace.

If you click on "Add a Data Connection" in a Data Pool and then "Connect to Data Source":

You'll see some Extractor examples based on your environment's access rights (different from screenshot below):

Accessing Databases

Any software system needs to store its data in a centralized location. In most cases the data is stored in an underlying database not developed by the solution provider—e.g. PostgreSQL or MSSQL. This a perfect use for the Database Extractor, also known as the JDBC Extractor. This Extractor supports more than 20 database types and you can extend it with custom database drivers.

Try it out - Create a New Connection

Extractors Try it out - Create a New Connection

Follow the steps below to create a new connection to a PostgreSQL database in your personal environment.

Steps

Navigate to your Data Pool "Get Data into Celonis - P2P" and click on "Data Connections".

Delete the existing Data Connection. You'll have to type in the connection name to be able to delete it, feel free to copy and paste from within the popup.

On the main graphic, you'll see that the extractions and transformations related to that connection are erased. Not to worry, you still have them as templates as a backup:

The templates are in the top right corner for later use if needed:

Now create a new Database connection by clicking on "Connect to Data Source" (2 times) and then filter for "Cloud" and click on "Database"

Make sure to pick the second Database Extractor. This is the Cloud Extractor as shown in its bottom right corner.

Insert the following Connection details:

Name: P2P Connection

Type: MySQL

Host: academymysql.cluster-ro-ckynnbglhixw.eu-central-1.rds.amazonaws.com

Port: 3306

Database Name: p2p

Username: p2p

PW: Celonis1234!

Maximum Parallel Table Extractions (Advanced Settings): 8

Test your connection. If there are any issues, double-check your entries until the connection works.

Note: Do not miss to enter the correct schema for your connection. If left to the default "public" schema, the connection will work but you will have issues accessing the right tables in your extraction.

Click on “Save”.

The Result

You should now have a functioning connection to a database in your Data Pool and have retained Task templates from the Process Connector. The templates will come in handy once you start working on Data Job Extractions.

Extractor Builder Overview

Extractor Builder Extractor Builder Overview

On some occasions, you’ll encounter systems or processes with no out-of-the-box Process Connectors or Extractors. In those cases, you can use the Extractor Builder. You'll find it under "Create custom REST API extractor" when connecting new data sources:

This tool helps you quickly build an Extractor using REST API and supports the OData V2 and V4 standards as well. It allows you to quickly connect to source systems and extract the data you need. This way, you don’t need to put a stop to your work or wait for Celonis to build a Connector or Extractor to bring data into Celonis.

In many cases, you can also customize existing Celonis Extractors that were built using the Extractor Builder.

Just as with other Extractors, once you connect to source systems, you still need to define your extractions, transformations, and data model.

Interested in learning more about the Extractor Builder?

Extractor Builder Interested in learning more about the Extractor Builder?

If so, check out our mini-course on the topic:

Extractor Builder Basics

File Uploads Overview

File Uploads File Uploads Overview

You can use the File Uploads when connecting to a new data source to upload data directly to Celonis. The files should be in one of these formats:

When would you use File Uploads?

Here are some sample use cases:

For additional static data For a list of translation terms For a list of permissions outside of your source systems For historical data that you know will never change For one-time uploads

If you can, avoid using file uploads for dynamic data that needs to be refreshed regularly. For this, you should consider out-of-the-box extractors or the Data Push API instead.

For the generation of SAP ABAP reports, you can also use SAP ABAP Generator and later upload the ABAP files under file uploads as well. For more info, have a look at this Microlearning and the Help documentation.

Try it out (optional) - Perform a File Upload

File Uploads Try it out (optional) - Perform a File Upload

A file upload really should not be rocket science. That said, it's good to experience it first hand so you know what options you have at your disposal.

Steps Download the "Perform a File Upload" file from the Resources section of this course.

Go to your Data Connections, click "Add a Data Connection" and then "Upload Files".

Select the downloaded file (or drag and drop it into the area) . It should appear below:

Click on the Menu button, visually the three dots, next to your file and then "Configure Upload" . This is where you decide into which connection (database schema) the data is upload. It goes to "Global" by default. You can also rename your target table name here if needed:

Now again click the three dots and select the second option "Configure Data Schema:

This allows you to adjust the headers, data types, and more if needed. In case you get recurring failures, it's sometimes advised to import everything as a string.

Click on the play button action and wait for the upload to complete.

If incomplete, have a look at the execution history under the three dots or the status icon to see the error. Results

Easy right? Your sheet is now uploaded as a table in your Data Pool and you will be able to perform transformations and include it in your Data Model. Note that the data is only available in Data Integration, not yet available in the Studio (This requires a Data Model load).

Note: Your files load into the 'Global Schema' by default unless specificied otherwise. To see your data, you'll need a data job based on this scope.

Data Ingestion API

In some rare cases, you cannot or should not “pull'' the data from source systems but source systems can push the data into Celonis. For this, Celonis provides the Data Ingestion API.

The Data Ingestion API relies on an ingestion architecture separate from the native extractors.

Specifications

Here are a few things you should know about the API itself:

One API call per table: Every table you create or update requires one API call. Built on S3 API: It's built on top of the S3 API with the same methods and error codes. It uses primarily this S3 PUT object endpoint. Continuous load: The load is continuous. So as soon as you ping the API, your data is loaded into Celonis. It operates on a First In First Out (FIFO) principle. Auto-delta load: All pushed records are loaded as delta records automatically and compared to existing records. Nested data possible: The API can handle nested (json) data out-of-the-box and unnest it into tables and columns. UI schema configuration: It supports a UI based configuration for table schema, i.e. table names, keys, age columns. Parquet files only: As of today, the API only supports parquet files and requires you to convert to parquet before pushes.

Creating a Data Ingestion API Connection

Data Ingestion API Creating a Data Ingestion API Connection

By now, you should understand that each connection in a Data Pool acts as a separate container to organize your data. If you are familiar with the term, it can be compared to a database schema.

For the Data Ingestion API, you can create a separate connection into which you would like to push data:

From there you can define your table schemas and get ready to push your data via the API call.

Working with Data Ingestion API

Bear in mind that in most cases you will need basic programming skills (Python or Java) to set a Data Pipeline using the Data Ingestion API. That said, you can run tests using tools such as Postman. Alternatively, you could also test and set up a permanent flow using Celonis Action Flows / Machine Learning Workbench.

For more details on the API, visit our Data Ingestion API course course.

Notice the Data Push API?

The Data Push API is an older API that was also used to push data. It is no longer recommended as a push option as the Data Ingestion API is faster, and easier to setup.

Celoxtractor for Full Control

Celoxtractor Celoxtractor for Full Control

If you have some Python programming experience and none of the other connection methods fit a certain use case, you can consider using the Celoxtractor.

The Celoxtractor is a Python package designed to let you develop your own Celonis Extractor easily. It gives you:

complete control over your data, feature parity to native Celonis extractors, and full flexibility in adjusting all aspects of your extractions.

For more information on this tool, have a look at the Help and the GitHub documentation.

What systems do we connect to?

The Main Types of Systems What systems do we connect to?

So we've looked at some ways to connect, but what systems does the Celonis typically connect to?

Below is a list of the most common systems and data sources along with a short description of how you would connect to them using Celonis.

Bear in mind that with Celonis, you can bring data from multiple sources using the available connection methods and integrate this data into one large Data Model.

If you already feel comfortable with these systems, feel free to skip this page.

Software-as-a-Service

To access data from Software-as-a-Service (SaaS) products like the Salesforce Lightning Platform, Workday, ServiceNow, Zendesk, SAP Ariba etc… you will normally use the Process Connectors from the Marketplace or the Extractors (Data Connections). With both of these methods, you establish a direct cloud-to-cloud connection between Celonis and the source system.

Typically the extraction uses one of the three API architectures provided by the source system: a REST-API (e.g. Zendesk), a SOAP web service (e.g. Workday), or a Message Queue (e.g. Salesforce Platform Events). In each scenario, the system API end-points provide the data in a JSON file, .csv file format, or similar, and the Celonis Extractor converts the file format to .parquet and inserts the data to the Celonis data store.

ERP Systems

Enterprise Resource Planning Systems (ERP) systems are software suites that integrate and automate business back office functions, services, and human resources. Most ERPs run in “on-premise” or closed hosting environments within the customer infrastructure.

As with SaaS products, you’ll establish connections to ERP systems with either native connectors or extractors. The main difference here is that closed environments require the installation of extra components before you can establish a connection. For example, you need to install an RFC module and Celonis Extractor to connect to SAP systems.

Celonis does establish a connection to on-premise ERP systems via an on-premise extractor.

Data Warehouses

A data warehouse is a system used for reporting and data analysis. Most organizations have a data warehouse as part of their business intelligence setup. Data warehouses are still mainly traditional on-premise systems like MSSQL or SAP HANA but you’ll see modern cloud applications such as Google BigQuery, Snowflake or AWS Redshift.

In most cases, you’ll access data warehouses via a Java Database Connectivity (JDBC) interface. This is the database native extractor. With it, you can connect to more than 20 different database types. If needed, you can extend the Celonis JDBC extractor with any JDBC driver and set up connectivity to any system with a JDBC interface.

Productivity Tools

Another valuable source of information are productivity tools. Systems like Microsoft Teams, Google Drive, MS365 are the systems power the day to day in any organisation. Most modern productivity tools are SaaS applications and offer APIs for data extractions. With native extractors you can establish a cloud-to-cloud connection to these systems and add supporting documents like prioritisation sheets, working documents and so on.

Alternatively, you can use file uploads or even establish an ongoing integration of flat-files with a native extractor that continuously syncs a google sheet for example.

Task Mining

For some use cases, you’ll need to capture user interaction data, also known as Desktop Data. This data capture and its analysis are called “Task Mining”. The data is anything your users do on their computer that cannot be captured by IT source systems. For the capture, Celonis provides the Task Mining Desktop Client. The Client can capture system events, screen data, and software events from browsers or source systems like SAP GUI. The data is then sent to Celonis for structuring and analysis.

Task Mining allows you to enrich existing IT-based business processes with desktop data and also to capture processes where no IT system is involved or where the underlying IT system provides no access to its data. You can learn more on this topic in our Task Mining Basics course.

Paper Documents / OCR

Many processes do still run on paper documents. Celonis can process paper invoices via Optical Character Recognition (OCR) within the Touchless Invoice Ingestion App. The paper invoice’s information can be written to any underlying IT system like SAP or Oracle. If required the information can also be routed through the App where the end-users can validate the extracted information.

Custom Data Integrations

In many cases, organizations use custom solutions. Thankfully, Celonis is built for easy extensibility with custom data sources. You can build custom integrations with the following methods:

Database access with the Database Extractor The Extractor Builder File Uploads (non-continuous) The Data Push API The Python extractor (Celoxtractor) That is the gist!

And those are the basics of the most common types of systems and the top ways you’ll connect.

Recap on Connection Methods

Recap Recap on Connection Methods

In short you've learned to connect. Here are a few takeaways on the connection methods covered.

Process Connectors - Look for Process Connectors in the Marketplace whenever possible. You can import Process Connectors directly when creating new Data Pools. Native Extractors - Next, you can connect to systems neutrally and extract data with the Extractors (neutral Connections). Extractor Builder - If no Connectors / Extractors are available, you can of course put in a request for Celonis to create one but do not wait. Build your own connection and extractions with the Extractor Builder. File Uploads - For one-time or static cases use the Upload Files option. Data Ingestion API - Use the Data Ingestion API if the use case requires you to push data such as with existing ETL tools or scheduled reports. Celoxtractor - Finally, if you are apt with Python and want to control every aspect of your extractor, you can consider the Celoxtractor.

Depending on the source systems you connect to, one or more of these methods will fit and it is up to you to decide which one to use. If you forget anything, simply navigate to the UI and have a look!

---

Prepare systems for a connection

Every Connection is Different

With so many types of systems and hosting approaches on the market, you’ll often need to go through extra steps to establish a data connection. Not to worry - for each connection, the Celonis Help documentation guides you with checklists on what to do.

For now, let’s look at three different basic connection scenarios and their associated setup and extraction flows:

Closed hosting environments (On-prem or Cloud) Open Cloud environments Real-time connections

The Setup of a Closed System Connection

Closed Hosting Environments The Setup of a Closed System Connection

By "closed" we most often refer to an on-premise system but it also points to systems hosted in the cloud not accessible through direct connections. A good example of closed hosting environments are SAP ECC and SAP S/4HANA. For both of these systems, you need to go through the following steps before you can establish a connection.

WHAT WHERE WHO

  1. Install the Celonis SAP RFC Module SAP ECC or S/4HANA system SAP Basis Team
  2. Install the on-premise Extractor Customer Network Server and infrastructure administrator
  3. Create a Data Connection Celonis Platform Data Engineer

In short, Celonis cannot connect directly to SAP and you need an RFC module (SAP specific) installed on the hosting server along with an extra Extractor Server (for all closed systems). Each step requires collaboration with different internal team members and includes minute substeps on both the source system (e.g safelist IPs and ports) and Celonis side. You can find a detailed overview of the steps on this Connect to SAP help page.

Why is the on-premise Extractor needed?

For two simple reasons:

On-premise systems typically can’t communicate with external applications. Network internal connections are a lot more stable and performant, resulting in a minimum load on the source system.

There is no need to remember all the steps here, simply be sure to check the respective documentation for each connector you install and know that you are responsible for moving these steps forward in a Celonis project.

Note: When the Extractor is on the source system side for closed systems, you set up the the connection type as "Uplinked" in Celonis. This is selected automatically when you select on an on-prem connection type in Data integration.

Connection & Extraction Flow with Closed Systems

Closed Hosting Environments Connection & Extraction Flow with Closed Systems

Here is the generic connection and data extraction flow with a closed source system. Have a read through the steps on the right to better understand. No need to learn this flow by heart, this is simply to give you an understanding of what happens in the background.

Extractor establishes connection to Data Integration.

Data Engineer defines which tables to extract in Data Integration.

Extractor informs source system of what data to extract.

Data extraction is performed.

The extracted data is sent to the Extractor Server.

The Extractor Server transforms the files into parquet files.

The Extractor Server sends the transformed to the Celonis data Storage.

As you can see, the Extractor Server is on the source system side and acts as the intermediary between the closed system and Celonis, with all cross system connections using https encrypted over TLS 1.2.

Connection to Open Systems

Open Hosting Environments Connection to Open Systems

You could say life is a little easier when connecting to open hosting environments. In most cases, these are cloud systems. Few extra steps are typically required beyond potentially safelisting IPs and having a user with proper access rights. The Extractor in this case is on the Celonis side and a direct connection (not uplinked) is established with the Cloud systems:

Connection and Extraction Flow with Open Systems

Open Hosting Environments Connection and Extraction Flow with Open Systems

Here is an overview of the connection and extraction flow between Celonis and open cloud systems:

Extractor establishes connection to the Cloud system and requests data.

Data Engineer defines which tables to extract in Data Integration.

Cloud system extracts data and sends it to the Cloud Extractor.

Cloud Extractor transforms the files into parquet files.

Cloud Extractor sends the transformed files to the Celonis Data Storage.

Real-Time Connections

Preparation for Real-time Connections Real-Time Connections

Now let’s have a closer look at real-time connection preparations. These come on top regardless of whether you work with a closed or open hosting environment.

Again we can use the SAP example. Real-time means we can track changes in source system tables fast and quickly push the data into Celonis. So how can you dynamically track changes on the source system side? There are three general steps.

Create Change Log tables to store changes Install Triggers to monitor and capture changes Activate a cleanup background job to clean up the Log tables (SAP-specific)

In a closed system use case such as SAP, your job is to advise SAP teams and forward information when the time comes. Make sure to browse and bookmark this Help page to have all the information on the setup.

What are Triggers and Change Log tables for?

Here is a simple graphic to understand these two important components:

The keywords here are "trigger" and "change". Every time records are added, updated, or deleted in source tables, the triggers go off and the changed records' primary keys, the type of update, and timestamps are stored in Change Log tables.

At frequent intervals, Celonis requests these records and the Extractor Server acts as the intermediary to prepare the records and push them to Celonis. Here is a sample change log table for a normal SAP EKKO table:

Note that this real-time setup applies to the SAP and JDBC real-time scenario. For other systems such as Azure Event Hub, the setup will differ.

Real-time connections are a larger topic covered further in the Extract Data and Transform Data courses.

Troubleshooting your Connections

Troubleshooting Troubleshooting your Connections

Here are basic steps you should follow when troubleshooting connection issues with source systems:

Help Doc First - Always make sure to follow the Help prerequisites for your particular connector or connection method. Most connectors will have a list of FAQs and potential errors. You can search in the help or bookmark the data connection help page as a good starting point. Logs - Check the logs for your connection if available. In case it is an uplink connection ( for on-prem or closed hosting cases), you should check the local extractor logs and capture the error message in case you need help from Celonis Support.

If neither the Help nor logs resolve your issue, then contact Support through the support portal. Make sure to include your connection page URL as it includes your Data Pool and Connection IDs.

Recap on Preparing Systems

Recap Recap on Preparing Systems

In preparing systems for a connection, you need to consider whether:

the hosting environment is closed (most cases on-premise)—then you install the Celonis Extractor on premise and use an uplink connection to Celonis the hosting environment is open (most cases cloud)—then you use the default Celonis Cloud Extractor with a direct connection to Celonis your connection is real-time—you then need to set up Log tables and triggers in the source system environment.

Here is a graphic clearly capturing the difference between connections in closed versus open source system environments:

Lastly, to troubleshoot connections use the available logs, refer to the Help documentation and reach out to support if needed.

Want to see how connections to specific systems are set up? Check out our Set Up System Connections playlist.

---

Structure your Data pools

Data Pools Introduction

Structure Data to your needs Data Pools Introduction

When you connect to a source system in Data Integration, the new connection lands in a new or existing Data Pool.

What’s... a Data Pool and why does it matter?

Data pools are containers for all other elements in Data Integration - Data Connections, Data Jobs, and Data Models. They pool your data. If you plan to bring together multiple systems and processes in Celonis, aligning your plans with Data Integration’s structural elements is very important and will save you re-structuring work later down the line.

Structure of Data Integration

Structure Data to your needs Structure of Data Integration

Now let’s dig into the basic data structure of Data Integration and what it means for you. You've indirectly touched on these points but here are some hard facts:

If you download a Process Connector from the Marketplace, Celonis creates a Data Pool for you. Data Pool - One Data Pool can host several Data Connections. Data Connection - A Data Connection is simply a connection to a source system and also serves a data organizer within a Data Pool. You can create one or more Data Connections, and for each, you work on one or more Data Jobs. Data Jobs combine extraction, transformation, and Data Model load “tasks” and allow you to execute them in sequence. The advantage of having multiple Data Jobs is that you can assign them different execution schedules. This becomes relevant when certain tables need to be extracted and transformed and loaded more often than others. Data Models - Lastly, with your different tasks ready, you bring the data together into one or more Data Models. Logically speaking, you can only create Data Model loading tasks once you've created Data Models. Data Models serve as the basis for all downstream work in Celonis. A Data Model can pull tables from multiple connections and data jobs within one Data Pool. (Note: you can share data across Data Pools if needed, more on this later)

Here is a visual that better captures the hierarchy:

And hopefully the structure and its effect on data flow becomes clearer the more you work with the Data Integration UI.

And lastly, here is a small table to recap what each element contains:

ELEMENT DEFINITION DATA POOLS Host one or more Data Connections DATA CONNECTIONS Contain one or more Data Jobs DATA JOBS Contain "tasks" for extraction, transformation, and Data Model load DATA MODELS Pull data from all connections in one Data Pool

Structuring your Data in Data Integration

Structure Data to your needs Structuring your Data in Data Integration

Looking at this structure, you should do your best to keep all related processes and systems within one Data Pool. Likewise, you should keep unrelated processes in separate Data Pools. For example, you can create one Data Pool for all your SAP ERP processes and another Data Pool for your IT Service Management processes.

In some cases, it also makes sense to create separate Data Pools for different regional or legal entities to restrict access to the data for certain users or user groups.

---

Recap

Course Recap

Knowledge Check & Recap Course Recap

This course was all about Connecting.

Here is a recap of what we covered:

Connection methods—Celonis offers a range of connection methods for different use cases: Process Connectors Extractors (Data Connections) Extractor Builder File Uploads Data Ingestion API Celoxtractor

Preparing Connections—How you prepare a system for a connection depends on its type: For closed environment you'll typically need the on-prem Extractor Server along with other preparatory steps. For real-time connections, you'll need change logs, triggers and other steps depending on the system. Structure of Data Integration—Data Pools are the top-level structural elements in Data Integration and a basic best practice is to keep related processes in the same Data Pool. Within Data Pools, you work with: Data Connections, Data Jobs, Tasks—extraction, transformation, Data Model load, and Data Models.

Remember to visit the Feedback page. Your feedback is optional but greatly appreciated.

Want to dig deeper into the topic of this course? Join the Celonis Community to ask your product questions, hear about the latest product releases, and remain up-to-date by subscribing.

We invite you to join our Celonis Academy Group to engage with your peers, get exclusive updates and answers directly from Academy experts, and stay connected!

---

Knowledge Check — 6 questões
1. You need to set up a Data Model for a standard process based on SAP, how should you go about it?
2. You need to get a one time report with sample data into Celonis, what is the fastest way to do this?
3. A source system has REST APIs but no Celonis Process Connectors or Extractors are available. How should you extract data from the system?
4. What is the main difference between connecting Celonis to a closed vs open hosting environment?
5. Three of your processes are related to one another. How should you go about structuring your data in Data Integration?
6. A source system connection is not possible due to security reasons but an ETL tool is available and could push data into Celonis. Which connection method should you use?