The devil is in the details

Data ingestion pipelines are brittle. Things can go wrong in a million places. We take care of all the tiny details so you don't have to.

Your Data Safety is our number one commitment

• Files are encrypted and backed up before we process them. If anything breaks, the original file will be pulled from the secure backup.
• You own the backup storage. We don't hold on to your files. You can use AWS S3, or Google Cloud Storage as the backup medium.
• We avoid locking your destination database at all costs. We use a combination of triggers and advanced codes to run database schema migrations on live databases.
• You can even host the settings database that powers Qluster. The pipelines will still run within Qluster's infrastructure. That way there is nothing from your data that will be leaked to Qluster except logs. We can even have the logs stream to your logging infrastructure.
• Data governance: You can assign users to specific data sources, i.e. assigning 3rd party vendors to only access their respective data.
• Incoming data is never lost. The bad data makes it way in the quarantine tables for you to review and resolve.

Data Firewall

• Qluster is the first layer of defense against bad data.
• The mentality of "get all the data in, deal with the problems later" will hurt you later down the line. We detect and handle data issues upstream before it even makes its way to your infrastructure.
• If some rows of data can't be imported to the database, we quarantine the bad rows for you so you can take a look at them. We provide tooling to be able to resolve the issue.
• Once you have enough data ingested through Qluster, we can enable anomaly detection.

Observablity

• We give observability to both you and 3rd party vendor about what is happening to the data and guide the user to make the right choices when things break.
• Just like you, your vendors can see and get notified about what has gone wrong. They will get notified only about the issues specific to their own data.
• Qluster Slack bot can be installed in your slack channels to provide immediate notification for data issues.

Data Lineage

• Qluster tracks how data is modified as it gets cleaned. Every value modification is recorded, along with why and who changed it.
• With a click of a button on each value, you can get the history of that value.

File Formats

• Automatically detecting file encoding. Is this a big-endian file? little-endian UTF-16 vs. UTF-32? There are so many flavors of encoding. We got you covered. No more strange characters showing up in your output.
• Field name transliteration. We automatically transliterate the field names in your files into English to make sure your destination database can handle them.
• Header line not in the first line? No problem. Some text above the header? No problem. We automatically detect all that.
• Auto handling compressed files such as ZIP, TAR, GZIP, SNAPPY
• Handling Excel files including modern XLSX files all the way to the original XLS-XML files from 1995

AI Driven Data Cleaning

• When we see a new file with fields we did not expect, we use AI (i.e. Named Entity Recognition), fuzzy matching and statistics to recommend column mapping for you.
• We use semi-supervised anomaly detection algorithms to detect anomalies in your incoming data. The model is auto-trained based on your most recent data.
• We automatically take care of the majority of common data issues such as datetime formats, fat finger errors, and field renames for you. If Qluster doesn't automatically resolve it, it will recommend solutions to you. Both the admin and the vendors can confirm the recommended solution so the issue can be resolved.

Cloud Independent

• All Qluster needs to run is Kubernetes and Postgres. It does not have a hard dependency on any cloud provider.
• Qluster can be deployed on your own VPC (Virtual Private Cloud). That way no data ever gets out of your infrastructure.

Highly Scalable

• Qluster is distributed. It can scale as big as you need it to be.
• Qluster uses Kubernetes primitives and is cloud native.

Extendable

• Qluster's SDK can be used to build custom validation and data transformation logic.
  
• Qluster's REST API can be used to by the engineers to interface with Qluster.
  
• Since each action in Qluster corresponds to a kubernetes job, if you need custom validation or ingestion code to be run as a part of your pipeline, as long as it can run as a Docker image, we have got you covered. Custom code can only be deployed in the enterprise self-hosted deployments.