Nuxia

NUXIA DATA

Nuxia Data Intelligence
Data platforms

An open-format lakehouse integrating storage, streaming, processing, SQL, search and data governance within one architecture.

Let's discuss your platformExplore the architecture

NUXIA DATA INTELLIGENCE

Explore the architecture

Select a component to discover its role or follow the journey of a payment event.

Illustrative example, with no real payment data. Valkey enriches processing and OpenMetadata documents lineage across the flow.

Storage

Serving

Ingestion

Processing

Governance

Platform

Data flow view

  1. The example starts with a payment event published to Kafka. The log retains the event so downstream processes can consume it and replay it when required.

  2. Flink consumes the stream and maintains activity features used to evaluate the payment. Processing combines the event with operational information, such as time-window aggregates and the customer profile.

  3. The customer profile and operational counters are read from Valkey to enrich Flink’s processing. This stage represents an in-memory lookup, without sending every request to the warehouse.

  4. The journey then represents writing the event and its results into Iceberg tables. A snapshot identifies the data state available to analytical processes and later reviews.

  5. Object storage holds the Parquet files, metadata and snapshots underlying the Iceberg tables. This stage shows the lakehouse’s storage foundation, shared by its different engines.

  6. During batch processing, Spark validates and transforms lakehouse data into datasets prepared for analysis. Results are written back to Iceberg and feed subsequent warehouse loads.

  7. Cloudberry or Greenplum makes the prepared data available for distributed SQL queries and aggregation. In this example, it feeds business reports and analysis with an identifiable data source.

  8. OpenMetadata links tables and reports to their sources, transformations and owners. The journey ends by showing that lineage, although the catalogue and quality controls operate across the platform.

Platform use cases

Traceable business reporting

A governed pipeline connects transactions to versioned tables and reports that can be traced back to their sources.

  1. Capture the event

    Kafka receives events and changes while preserving the source timestamp.

    Kafka

  2. Store raw data

    Iceberg records unmodified events and retains queryable snapshots.

    Iceberg

  3. Validate and conform

    Spark applies quality rules and isolates records requiring review.

    Spark

  4. Load the warehouse

    Cloudberry receives conformed data with partitions and distribution suited to queries.

    Cloudberry

  5. Compute the report

    SQL models express reporting rules and versioned reference data.

    SQL

  6. Record lineage

    OpenMetadata links report fields to source tables and storage locations.

    OpenMetadata

  7. Archive evidence

    The report, snapshot identifiers and query log are retained.

    Object storage
Real-time fraud and AML

Event processing combines profiles, rules and models to support decisions and investigations from the same data foundation.

  1. Receive authorisations

    Kafka orders related events using card or account partitions.

    Kafka

  2. Add behavioural context

    Flink enriches the event with time-window aggregates and patterns.

    Flink

  3. Read the profile

    Valkey or GemFire provide profiles, limits and risk ratings.

    Valkey / GemFire

  4. Run the model

    Flink computes a score and records the model version and features used.

    Flink

  5. Return the decision

    The decision returns to the authorisation system and is published as an event.

    GemFire

  6. Index the investigation

    OpenSearch brings together alerts, scores and features for investigation.

    OpenSearch

  7. Feed training outcomes

    Confirmed outcomes are stored in Iceberg for future Spark training.

    Iceberg
Customer 360

The platform combines channel, CRM and business-system data for analytics, applications and service teams.

  1. Ingest channel data

    Kafka and CDC collect source changes and land them in Iceberg.

    Kafka / CDC

  2. Resolve identity

    Spark matches records using identification and matching rules.

    Spark

  3. Build the customer table

    Iceberg retains attributes, products and permissions with snapshot history.

    Iceberg

  4. Serve SQL analytics

    Cloudberry exposes customer and transaction models to BI tools.

    Cloudberry

  5. Publish profiles in memory

    Valkey serves current profiles to applications and service teams.

    Valkey

  6. Search profiles

    Solr searches names, accounts and documents without an exact identifier.

    Solr

  7. Govern personal data

    OpenMetadata records classification, ownership and consent context.

    OpenMetadata
Cloud warehouse repatriation

Migration moves data, schemas, pipelines and consumers with parallel running and result validation before cutover.

  1. Inventory the source

    Schemas, queries, BI connections, service accounts and costs are reviewed.

    Assessment

  2. Export to open formats

    Data is exported to Parquet and registered as Iceberg tables.

    Iceberg

  3. Rebuild schemas

    Cloudberry adapts DDL, partitioning and distribution to query patterns.

    Cloudberry

  4. Move pipelines and queries

    Airflow, Spark and dbt incorporate workflows and validate dialect differences.

    Airflow / dbt

  5. Run in parallel

    Results are compared and differences between platforms are documented.

    Validation

  6. Cut over by domain

    Each domain switches its consumers with defined acceptance and rollback.

    Cutover

  7. Close and document

    Source retirement respects agreed retention and preserves evidence.

    Evidence
On-premise logging and audit

Application and platform logs are collected, normalised and indexed, with archives and documented retention policies.

  1. Collect logs

    Kafka receives application, database, network and platform logs.

    Kafka

  2. Normalise and enrich

    Flink applies a common schema and system, environment and classification tags.

    Flink

  3. Index recent data

    OpenSearch provides search, dashboards and alerts for operations teams.

    OpenSearch

  4. Archive historical data

    Older data moves to snapshots and Iceberg tables on object storage.

    Iceberg

  5. Query the evidence

    Search reconstructs access and activity within the retained period.

    OpenSearch

  6. Check retention and integrity

    Object locking, checksums and the catalogue document archive protection.

    Object storage
Data foundation for private AI

Governed datasets, features and document indexes feed models and assistants with traceability of the sources used.

  1. Prepare governed data

    Iceberg identifies the datasets used by each model through snapshots.

    Iceberg

  2. Compute features

    Spark shares feature definitions between Iceberg for training and Valkey for inference.

    Spark / Valkey

  3. Process documents

    Spark prepares chunks and embeddings while retaining source document and page.

    Spark

  4. Index text and vectors

    OpenSearch combines keyword search and vector similarity.

    OpenSearch

  5. Retrieve authorised context

    RAG filters passages by user permissions before sending them to the model.

    RAG

  6. Log questions and answers

    Queries, retrieved sources and responses are retained for review.

    OpenSearch

Deployment patterns

Starter: one site and one Kubernetes cluster

Kafka, RabbitMQ and CDC feed RustFS objects and Iceberg tables; Spark processes the data, while Cloudberry, Valkey and OpenSearch serve it. OpenMetadata adds catalogue and lineage on Kubernetes.

Architecture example from the reference. Figures are illustrative; sizing and availability are validated per project.

Ingestion

Kafka3 brokers
RabbitMQ3-node quorum
CDC connectorsCore, cards and channels

Lakehouse storage

RustFS object storage4 nodes, erasure coded
Iceberg tablesACID, snapshots

Processing and serving

Spark on KubernetesBatch and quality
Cloudberry1 coordinator, 4 segments
ValkeyIn-memory serving
OpenSearchLogs and search

Consumption and platform

BI and business reportsSQL consumers
OpenMetadataCatalogue and lineage
Kubernetes, one site12–16 nodes, example
Connections between components
  • CDC connectorsKafkaChange events
  • KafkaIceberg tablesSink
  • RabbitMQIceberg tablesMessages
  • RustFS object storageIceberg tablesFiles
  • Iceberg tablesSpark on KubernetesRead / write
  • Spark on KubernetesCloudberryConformed load
  • CloudberryBI and business reportsSQL
  • Spark on KubernetesValkeyPublish profiles
  • KafkaOpenSearchLogs
  • OpenMetadataIceberg tablesLineage
  • Kubernetes, one siteSpark on KubernetesRuns
Enterprise: three zones in one data centre

Distributes Kafka, storage and Kubernetes across three zones. Spark and Flink cover batch and streaming; Cloudberry, Valkey or GemFire and Solr or OpenSearch provide SQL, memory and search, with OpenMetadata governance.

Architecture example from the reference. Figures are illustrative; sizing and availability are validated per project.

Ingestion

Kafka9 brokers, 3 zones
RabbitMQQuorum queues
CDC and file ingestionCore, cards and treasury

Lakehouse storage

RustFS / CephZone-aware erasure coding
Iceberg tables600 TB–1.5 PB usable, example
HDFSDeep history, optional

Processing and serving

Apache SparkBatch and features
Apache FlinkStreaming and scoring
Cloudberry MPP8–16 mirrored segments
Valkey / GemFireIn-memory grid

Search, governance and platform

Solr / OpenSearchSearch, logs and vectors
Kubernetes across 3 zonesOpenMetadata, monitoring and backup
Connections between components
  • CDC and file ingestionKafkaEvents
  • KafkaApache FlinkStreams
  • KafkaIceberg tablesSink
  • RabbitMQApache FlinkMessages
  • RustFS / CephIceberg tablesFiles
  • HDFSApache SparkHistory
  • Iceberg tablesApache SparkRead / write
  • Apache FlinkIceberg tablesStreaming upsert
  • Apache FlinkValkey / GemFireState and decisions
  • Apache SparkCloudberry MPPConformed load
  • Iceberg tablesCloudberry MPPExternal tables
  • Cloudberry MPPSolr / OpenSearchExtracts
  • Kubernetes across 3 zonesCloudberry MPPOperations
DR: two sites with active replication

Replicates events, objects and services between primary and secondary sites using MirrorMaker 2, bucket replication and standby services. Includes shared operations, active-site selection and recovery drills; objectives are agreed per project.

Architecture example from the reference. Figures are illustrative; sizing and availability are validated per project.

Primary site

Kafka and RabbitMQPrimary ingestion
Object storage + IcebergPrimary lakehouse
Warehouse, in-memory and searchPrimary serving

Replication

MirrorMaker 2Topics and offsets
Bucket replicationContinuous, encrypted
Standby and WAN gridWarehouse and GemFire

Secondary site

Kafka and RabbitMQSecondary ingestion
Object storage + IcebergSecondary lakehouse
Warehouse, in-memory and searchReporting and recovery

Operations

Unified operationsMonitoring, backup and runbooks
Traffic steeringActive site selection
Periodic failover drillRecovery evidence
Connections between components
  • Kafka and RabbitMQObject storage + IcebergIngestion
  • Object storage + IcebergWarehouse, in-memory and searchServe
  • Kafka and RabbitMQMirrorMaker 2Mirror
  • Object storage + IcebergBucket replicationReplication
  • Warehouse, in-memory and searchStandby and WAN gridShip and replicate
  • MirrorMaker 2Kafka and RabbitMQApply
  • Bucket replicationObject storage + IcebergApply
  • Standby and WAN gridWarehouse, in-memory and searchApply
  • Unified operationsWarehouse, in-memory and searchMonitors
  • Unified operationsWarehouse, in-memory and searchMonitors
  • Traffic steeringWarehouse, in-memory and searchActive
  • Periodic failover drillWarehouse, in-memory and searchValidates

An integrated lakehouse platform

Nuxia Data Intelligence brings together storage, ingestion, processing, serving and governance. Open formats let different engines work on the same datasets. The architecture combines established services, with versions selected and validated per project.

Layered architecture

  • Foundation: Kubernetes on virtualisation or bare metal, with shared identity, networking and observability.
  • Ingestion: Kafka for events, RabbitMQ for messaging and CDC connectors for database changes.
  • Storage: S3 objects with RustFS or Ceph, Parquet files and Iceberg tables; HDFS for existing history.
  • Processing: Spark for batch and machine learning, Flink for streams and Airflow for orchestration.
  • Serving: Cloudberry or Greenplum for analytical SQL; Valkey and GemFire/Geode for memory; Solr and OpenSearch for search.
  • Cross-layer governance: OpenMetadata connects catalogue, classification, ownership, quality and lineage across layers.

From source to consumption

Business systems publish events and changes. Ingestion preserves raw data and positions for replay. Spark or Flink validate and transform; Iceberg records snapshots; SQL, BI, applications and AI consume the results. The catalogue links transformations to sources and owners.

Use cases

  • Traceable reporting: link each figure to sources, quality rules and input snapshots.
  • Risk and business analytics: use complete history with SQL and distributed processing.
  • Data for AI: combine document retrieval, operational features and model evaluation.

Planning the migration

We start with one data domain and a measurable outcome. We inventory sources and processes, build the foundation, ingest the first flow and compare results with the existing system. We then expand domains and consumers, rehearse recovery and transfer operations. TCO includes hardware, energy, licensing and operations and is compared with current costs.

Versions, licences, compatibility, support coverage and service targets are agreed after reviewing the environment. Sizing and results are validated with representative workloads and tests.

Frequently asked questions

Can we adopt only part of the catalogue?

Yes. Assessment determines what to keep, integrate and change. The architecture can start with a database, an event flow or a lakehouse domain and expand in stages.

Can it run in a disconnected environment?

This is assessed per component and version: dependencies, images, licences, updates and observability. Isolation requires internal repositories and a controlled import and verification process.

CONNECTED CAPABILITIES

Analytics and data warehouse

Analytical SQL, distributed processing and open formats for historical data. We connect warehouse and lakehouse with governance and traceability.

API gateways and 3scale

Maintain and evolve API management. We assess 3scale, its components and continuity or migration options while preserving contracts, policies and identity.

Databases and in-memory data

Relational databases, caches and distributed in-memory data for transactional applications. Architecture, performance, recovery and lifecycle for each engine.

Data governance and storage

Know where data lives, who uses it and how it changes. Catalogue, lineage, quality and S3-compatible storage for a traceable platform.

PaaS and developer platforms

Self-service application development and deployment on private infrastructure, with a shared catalogue, automation and operational controls.

Data platforms

From hypervisor to data catalogue: connected technology families and Nuxia Data Intelligence as an integrated lakehouse architecture.

Search and text analytics

Document and application search, observability and log analytics on your infrastructure. Indexes, relevance, retention and access within one design.

Messaging and streaming

Connect applications and process events with Apache Kafka, RabbitMQ and Apache Flink. We design recoverable, observable flows sized for your workload.

LET'S START WITH YOUR CONTEXT

Let's discuss your platform

Tell us what you want to solve and where you are now.

Write to Nuxia