Release notes for the Stackable Data Platform
The Stackable platform consists of multiple operators that work together. Periodically a platform release is made, including all components of the platform at a specific version.
Release 24.3
New / extended platform features
- Support for many new product versions
-
Almost all product images have been updated to their latest versions. Some notable examples are Apache Airflow 2.8.1, Trino 442 and Apache Spark 3.5.1. In addition, we started building some product binaries from source instead of packaging them from the official releases. This enables greater control over the features and the security aspects of the products. Currently, Apache Hadoop and Apache HBase are built from source and others will follow in coming releases.
- Security
-
We put a special emphasis on security in this release. For this purpose we addressed the following topics:
-
Authorization
-
Authentication with Kerberos
-
OpenID Connect integration
-
Vulnerability management
-
Starting with this release, user authorization is consistent across all products.
The Open Policy Agent (OPA) has been a core component of the platform since the beginning and in this release we enhanced its capability with the inclusion of a new component called user-info-fetcher.
It allows for authorization policies to be built on many attributes like organizational group membership, resource assignment and much more.
The first major identity provider supported by the user-info-fetcher is Keycloak but others will follow.
For more information, refer to the User Info Fetcher Usage Guide.
A lot of effort was spent on enabling policy based authorization (with OPA) within HDFS. This was in important milestone in the platform evolution and is a unique feature that has long been missed by HDFS users and administrators. It requires a Kerberos-enabled cluster as well as the HDFS extension which includes an OPA authorizer and group mapper. This is already bundled in Stackable image of HDFS and can be used by following the documentation. More details are available in the pull request.
Kerberos is the most widely used authentication protocol in the enterprise world. We added support for it in Apache Hive and Apache HBase. We also added examples for running Apache Spark applications in a Kerberos enabled environment.
OpenID Connect is the de-facto authorization standard on the Web, and it’s making its way into enterprise environments. Our platform now supports it for Apache Superset (documentation) and Trino (documentation), others will follow.
A core component of SDP is the Secret Operator. In this release, the secret operator will automatically rotate certificates it generates. The certificate lifetime is also jittered to avoid all pods restarting around the same time.
This is also the first release we’re publishing SBOMs (Software Bill of Materials) in the CycloneDX format for. We publish these for both our operators and our product images. The work on this was partially funded by the Sovereign Tech Fund. These SBOMS are published to our OCI registry as signed in-toto attestations and we have written a tutorial on how you can use it. To make it easier to use we have written a SBOM browser which allows you to download the raw CycloneDX JSON files as well.
| In an upcoming release we will enable authentication and encryption by default where possible. To ensure a smooth transition to future releases, we strongly encourage you to enable security features wherever possible in your deployments. |
- Storage
-
HDFS deployments now support rack awareness. This is another unique feature that brings the SDP platform closer to feature parity with bare metal HDFS deployments. Of course, the exact meaning of rack is different in Kubernetes environments, but the effect is the same: DataNodes are brought closer to the data they are reading and writing thus improving performance and reliability. A new topology provider is bundled with the HDFS image that maps Kubernetes labels to a cluster topology.
- Documentation
-
We are constantly working on improving the platform documentation and custom resource definitions are a significant part of that. The CRD documentation is now generated automatically and can be found at https://crds.stackable.tech.
- Command line tools
-
The
stackablectlcommand line tool has been overhauled and can now list endpoints provided by the listener operator. Also operator installation is parallelized, which considerably speeds up the process of setting up SDP on fresh Kubernetes clusters. - Custom labels for Helm charts
-
In the past, Helm users could not assign custom labels to stacklets. This is now possible and it enables better component management with third party tools.
- Bugfixes
-
-
Apache Airflow Operator: Using git-sync with the KubernetesExecutor is now possible.
-
Apache Hadoop Operator:
-
Kerberos principals are now included in the discovery ConfigMap.
-
Environment variables can now be overridden with the role group’s
envOverridesproperty.
-
-
Apache Spark Operator:
-
Applications can now be provisioned dynamically without having to fiddle with classpath settings.
-
RBAC permissions have been updated to allow the deletion of ConfigMaps when cleaning up applications.
-
-
Trino Operator: Add HDFS configuration files to the
hive.config.resourcesproperty when connecting to a HDFS cluster.
-
- Product features
-
The following are selected product features provided by new versions available in this release:
-
Apache Airflow: Introducing Airflow Object Storage and Listener hooks for Datasets plus various bug fixes.
-
Apache Druid: SQL compliance & engine enhancements, ingestion improvements, concurrent data handling.
-
Apache Kafka: Bug fixes.
-
Apache NiFi:
-
Improvements and bugfixes.
-
Over 270 issues fixed since version 1.23.2.
-
Adds new components for Slack and Zendesk integration among others.
-
-
Open Policy Agent:
-
Performance improvements, bugfixes and security fixes for third-party libraries.
-
Tooling to help prepare existing policies for the upcoming OPA 1.0 release, which will include a new version of the Rego language.
-
-
Apache Spark:
-
3.5.0 (SDP ships with 3.5.1) added first class support for distributed training and inference support, and enhancement of compatibility for Structured streaming, Spark SQL improvements.
-
Spark Connect is not yet supported in this release.
-
-
Apache Superset:
-
Latest patch release for the Superset 2.x lineage.
-
Apache Superset 3.1 includes various smaller new features/optimizations e.g. waterfall chart visualization, ECharts bubble chart, improved data set selectors, automatically format SQL queries, and country map visualization improvements.
-
-
Trino:
-
Lots of improvements and optimization since release 428.
-
Most notably we would like to highlight support for access control with the Open Policy Agent that we ourselves contributed in release 438 (#19532).
-
Also, starting from release 440, there is now row filtering and column masking in Open Policy Agent.
-
-
Apache ZooKeeper: Security and bug fixes.
-
- Support for the ARM architecture
-
During the development of this release, we started introducing support for the arm64 architecture. Currently support is experimental, and we only provide arm64 images for the previous release (23.11). For more information on how to use the ARM images, refer to the documentation.
Product versions
Deprecated versions
The following product versions are deprecated and will be removed in a later release:
-
Apache Airflow: 2.7.2, 2.7.3
-
Apache Druid: 27.0.0
-
Apache Kafka: 3.5.1, 3.5.2
-
Apache NiFi: 1.23.2
-
Apache Spark: 3.4.1, 3.5.0
-
Apache Superset: 2.1.1, 3.0.1. 3.0.3
-
Trino: 428
-
Apache ZooKeeper: 3.8.3
-
Open Policy Agent: 0.57.0
N.B. in some cases a newly supported version is also immediately marked as deprecated. This is done to allow an update path from the latest patch of a minor version (e.g. Kafka 2.8.2 -→ 3.4.1).
Removed versions
The following product versions are no longer supported (although images for released product versions remain available here):
-
Apache Airflow: 2.6.1, 2.6.3
-
Apache Druid: 27.0.0
-
Apache Hadoop: 3.2.2, 3.2.4
-
Apache HBase: 2.4.12
-
Apache Kafka: 2.8.1, 2.8.2, 3.4.1
-
Open Policy Agent: 0.51.0
-
Apache Spark: 3.4.0
-
Apache Superset: 2.1.0
-
Apache ZooKeeper: 3.8.1
Cockpit and stackablectl
A new project called Stackable Cockpit has been started.
It is a web-based management tool that allows users to interact with the Stackable data platform.
The repository also contains the stackablectl command line tool, which has been refactored for performance and stability.
Supported Kubernetes versions
This release supports the following Kubernetes versions:
-
1.29 -
1.28 -
1.27
These Kubernetes versions are no longer supported:
-
1.26 -
1.25
Breaking changes
You will need to adapt your existing CRDs due to the following breaking changes detailed below.
Stackable Operator for Apache Airflow
Breaking changes details
The following fields used to be optional but are now mandatory:
* spec.clusterConfig.credentialsSecret: Name of the secret containing the credentials for the database.
* spec.clusterConfig.exposeConfig: Set to true to export the AIRFLOWWEBSERVEREXPOSE__CONFIG environment variable.
* spec.clusterConfig.loadExamples: Set to true to load example DAGs into the Airflow cluster.
Breaking changes details
Assigning role groups to node selector was deprecated in 23.11 and has been removed in this release.
To assign roles and role groups to nodes, use the config.affinity.nodeAffinity property as described in the documentation.
Apache HBase
Breaking changes details
Assigning role groups to node selector was deprecated in SDP 23.11 and has been removed in this release.
To assign roles and role groups to nodes, use the config.affinity.nodeAffinity property as described in the documentation.
Stackable Operator for Apache Hadoop HDFS
-
Support for exposing HDFS clusters to clients outside of Kubernetes
.spec.clusterConfig.listenerClasshas been split to.spec.nameNodes.config.listenerClassand.spec.dataNodes.config.listenerClass, migration will be required when usingexternal-unstable.
Breaking changes details
This requires a change from e.g.
apiVersion: hdfs.stackable.tech/v1alpha1
kind: HdfsCluster
metadata:
name: hdfs
spec:
clusterConfig:
listenerClass: external-unstable (1)
...to:
apiVersion: hdfs.stackable.tech/v1alpha1
kind: HdfsCluster
metadata:
name: hdfs
spec:
clusterConfig:
...
nameNodes:
config:
listenerClass: external-unstable (2)
...
...
dataNodes:
config:
listenerClass: external-unstable (3)
...
journalNodes:
config:
...| 1 | Remove the cluster-wide listenerClass |
| 2 | Add the external-unstable listenerClass to the nameNodes role. You can set these at the role-group level too. |
| 3 | Add the external-unstable listenerClass to the dataNodes role. You can set these at the role-group level too. |
It should be noted that this change is not necessary if you are using the default spec.clusterConfig.listenerClass: cluster-internal.
| Unfortunately, it is not possible to patch existing HDFS stacklets in place. It will be necessary to delete and recreate the HDFS stacklet. No data will be lost during this process. |
Open Policy Agent Operator
Breaking changes details
Assigning role groups to node selector was deprecated in SDP 23.11 and has been removed in this release.
To assign roles and role groups to nodes, use the config.affinity.nodeAffinity property as described in the documentation.
Secret operator
This changes the format of the CA secrets. Old secrets will be migrated automatically, but manual intervention will be required to downgrade back to 23.11.x.
Stackable Operator for Apache Spark
Breaking changes details
The spec.version field has been removed.
The spec.mode field is now required and must be set to cluster.
The spec.mainClass field is now required and must point to a location on ths file system or S3 where the main class is located.
Breaking changes details
This is an experimental feature that was introduced to support logging in XML format.
The side effect of this removal is that the vector agent cannot aggregate output from the spark-submit containers.
On the other hand, it enables dynamic provisioning of java packages (such as Delta Lake) with Stackable stock images, which we consider more important.
Upgrade from 23.11
Using stackablectl
Uninstall the 23.11 release
$ stackablectl release uninstall 23.11
[INFO ] Uninstalling release 23.11
[INFO ] Uninstalling airflow operator
[INFO ] Uninstalling commons operator
# ...Afterwards you will need to upgrade the CustomResourceDefinitions (CRDs) installed by the Stackable Platform.
The reason for this is that helm will uninstall the operators but not the CRDs. This can be done using kubectl replace:
kubectl replace -f https://raw.githubusercontent.com/stackabletech/airflow-operator/24.3.0/deploy/helm/airflow-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/24.3.0/deploy/helm/commons-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/druid-operator/24.3.0/deploy/helm/druid-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hbase-operator/24.3.0/deploy/helm/hbase-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/24.3.0/deploy/helm/hdfs-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hello-world-operator/24.3.0/deploy/helm/hello-world-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/24.3.0/deploy/helm/hive-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/24.3.0/deploy/helm/kafka-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/24.3.0/deploy/helm/listener-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/24.3.0/deploy/helm/nifi-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/24.3.0/deploy/helm/opa-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/24.3.0/deploy/helm/secret-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/24.3.0/deploy/helm/spark-k8s-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/24.3.0/deploy/helm/superset-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/24.3.0/deploy/helm/trino-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/24.3.0/deploy/helm/zookeeper-operator/crds/crds.yamlcustomresourcedefinition.apiextensions.k8s.io "airflowclusters.airflow.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "airflowdbs.airflow.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "authenticationclasses.authentication.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "s3connections.s3.stackable.tech" replaced
...Install the 24.3 release
$ stackablectl release install 24.3
[INFO ] Installing release 23.11
[INFO ] Installing airflow operator in version 24.3.0
[INFO ] Installing commons operator in version 24.3.0
[INFO ] Installing druid operator in version 24.3.0
[INFO ] Installing hbase operator in version 24.3.0
[INFO ] Installing hdfs operator in version 24.3.0
[INFO ] Installing hive operator in version 24.3.0
[INFO ] Installing kafka operator in version 24.3.0
[INFO ] Installing listener operator in version 24.3.0
[INFO ] Installing hello-world operator in version 24.3.0
[INFO ] Installing nifi operator in version 24.3.0
[INFO ] Installing opa operator in version 24.3.0
[INFO ] Installing secret operator in version 24.3.0
[INFO ] Installing spark-k8s operator in version 24.3.0
[INFO ] Installing superset operator in version 24.3.0
[INFO ] Installing trino operator in version 24.3.0
[INFO ] Installing zookeeper operator in version 24.3.0Using Helm
Use helm list to list the currently installed operators.
You can use the following command to uninstall all operators that are part of the 23.11 release:
$ helm uninstall airflow-operator commons-operator druid-operator hbase-operator hdfs-operator hive-operator kafka-operator listener-operator nifi-operator opa-operator secret-operator spark-k8s-operator superset-operator trino-operator zookeeper-operator
release "airflow-operator" uninstalled
release "commons-operator" uninstalled
# ...Afterward you will need to upgrade the CustomResourceDefinitions (CRDs) installed by the Stackable Platform.
The reason for this is that helm will uninstall the operators but not the CRDs. This can be done using kubectl replace:
kubectl replace -f https://raw.githubusercontent.com/stackabletech/airflow-operator/24.3.0/deploy/helm/airflow-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/24.3.0/deploy/helm/commons-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/druid-operator/24.3.0/deploy/helm/druid-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hbase-operator/24.3.0/deploy/helm/hbase-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/24.3.0/deploy/helm/hdfs-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hello-world-operator/24.3.0/deploy/helm/hello-world-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/24.3.0/deploy/helm/hive-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/24.3.0/deploy/helm/kafka-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/24.3.0/deploy/helm/listener-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/24.3.0/deploy/helm/nifi-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/24.3.0/deploy/helm/opa-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/24.3.0/deploy/helm/secret-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/24.3.0/deploy/helm/spark-k8s-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/24.3.0/deploy/helm/superset-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/24.3.0/deploy/helm/trino-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/24.3.0/deploy/helm/zookeeper-operator/crds/crds.yamlInstall the 24.3 release
helm repo add stackable-stable https://repo.stackable.tech/repository/helm-stable/
helm repo update stackable-stable
helm install --wait airflow-operator stackable-stable/airflow-operator --version 24.3.0
helm install --wait commons-operator stackable-stable/commons-operator --version 24.3.0
helm install --wait druid-operator stackable-stable/druid-operator --version 24.3.0
helm install --wait hbase-operator stackable-stable/hbase-operator --version 24.3.0
helm install --wait hdfs-operator stackable-stable/hdfs-operator --version 24.3.0
helm install --wait hive-operator stackable-stable/hive-operator --version 24.3.0
helm install --wait kafka-operator stackable-stable/kafka-operator --version 24.3.0
helm install --wait listener-operator stackable-stable/listener-operator --version 24.3.0
helm install --wait hello-world-operator stackable-stable/hello-world-operator --version 24.3.0
helm install --wait nifi-operator stackable-stable/nifi-operator --version 24.3.0
helm install --wait opa-operator stackable-stable/opa-operator --version 24.3.0
helm install --wait secret-operator stackable-stable/secret-operator --version 24.3.0
helm install --wait spark-k8s-operator stackable-stable/spark-k8s-operator --version 24.3.0
helm install --wait superset-operator stackable-stable/superset-operator --version 24.3.0
helm install --wait trino-operator stackable-stable/trino-operator --version 24.3.0
helm install --wait zookeeper-operator stackable-stable/zookeeper-operator --version 24.3.0Known upgrade issues
In the case of the breaking changes detailed above it will be necessary to update the custom resources and re-apply them.
Additionally, please note the following:
All operators
If you used node selectors to assign Pods to nodes, this will not work anymore.
Use the config.affinity.nodeAffinity property instead and follow the instructions in the documentation.
Release 23.11
This release introduces two further elements of platform stability - pod disruption budgets and graceful shutdown specifications - as well as a new management tool and updated product version support. Any known issues related to this release are listed in the Known issues section below.
New / extended platform features
The following new major platform features were added:
- PodDisruptionBudgets
-
Kubernetes has mechanisms to ensure minimal planned downtime. Our product operators deploy so-called PodDisruptionBudget (PDB) resources alongside the products. For every role that you specify (e.g. HDFS namenodes or Trino workers) a PDB is created. This will determine the extent to which roles for a given application may be inactive at any given time. See the documentation for more details.
- Graceful shutdown
-
Graceful shutdown refers to the managed, controlled shutdown of service instances in the manner intended by the software authors. Typically, an instance will receive a signal indicating the intent for the server to shut down, and it will initiate a controlled shutdown. Our operators configure a sensible amount of time Pods are granted to properly shut down without disrupting the availability of the product. See the documentation for more details.
- Signed SDP product images
-
As of this release all Stackable product images are signed (the signing of operator images was delivered in SDP 23.7). Please see this tutorial for more information.
- Airflow KubernetesExecutor
-
Airflow clusters can now be configured to use Kubernetes executors, whereby pods are spun up for job tasks and terminated when complete, thus offering an alternative way to use resources without the need for job queuing.
- Overridable Java security settings
-
For JVM-based products (i.e. Druid, HBase, HDFS, Hive, Kafka, NiFi, Spark, Trino and ZooKeeper) it is now possible to provide custom security settings that override the default values. This allows the user to control things such as DNS lookup caches.
- Stackable Cockpit
-
This release includes a very early preview version of Stackable Cockpit, a browser-based management tool which interacts with the Stackable data platform to display e.g. deployed stacklets and their status.
- stackablectl
-
Our command line tool has been re-worked to use the same backbone as Stackable Cockpit: you can find out about the recent enhancements by visiting the online documentation.
- Listener operator
-
The listener-operator was introduced in release 23.1 and the associated ServiceType field in 23.4. In this release we introduce configurable ListenerClass presets that map to the service types appropriate for different environments. This is discussed in more detail in the documentation.
- Openshift certification
-
All Stackable operators in the 23.11 release have been certified for Openshift versions 4.11-4.13 and can be installed directly from the OperatorHub UI:
- Product features
-
Additionally, there are some other individual product features that are noteworthy:
Product Versions
Deprecated Versions
The following product versions are deprecated and will be removed in a later release:
-
Airflow: 2.6.1
-
HBase: 2.4.12
-
HDFS: 3.2.4, 3.3.4
-
Kafka: 2.8.2, 3.4.0
-
OpenPolicyAgent: 0.51.0
-
Spark: 3.4.0
-
Superset: 2.1.0
-
ZooKeeper: 3.8.1
N.B. in some cases a newly supported version is also immediately marked as deprecated. This is done to allow an update path from the latest patch of a minor version (e.g. Kafka 2.8.2 -→ 3.4.1).
Removed Versions
The following product versions are no longer supported (although images for released product versions remain available here):
-
Airflow: 2.2.3, 2.2.4, 2.2.5, 2.4.1
-
Druid: 0.23.0, 24.0.0
-
HBase: 2.4.6, 2.4.8, 2.4.9, 2.4.11
-
HDFS: 3.3.1, 3.3.3
-
Hive: 2.3.9
-
Kafka: 2.7.1, 3.1.0, 3.2.0, 3.3.1
-
NiFi: 1.15.x, 1.16.x, 1.18.0, 1.20.0
-
Opa: 0.27.1, 0.28.0, 0.37.2, 0.41.0, 0.45.0
-
Spark: 3.2.1, 3.3.0
-
Superset: 1.3.2, 1.4.1, 1.4.2, 1.5.1, 1.5.3, 2.0.1
-
Trino: 377, 387, 395, 396, 403
-
ZooKeeper: 3.5.8, 3.6.3, 3.7.0, 3.8.0
stackablectl
The following demo has been added to stackablectl:
Signal processing demo
This demonstrates the ingestion of streamed data into a Timescale time-series database, augmented by a moving window of anomaly detection measurements that are plotted alongside the raw data in Grafana.
This link lists the available demos.
Supported Kubernetes versions
This release supports the following Kubernetes versions:
-
1.28 -
1.27 -
1.26
These Kubernetes versions are no longer supported:
-
1.25 -
1.24
Breaking changes
You will need to adapt your existing CRDs due to the following breaking changes detailed below.
For all operators: the field spec.image.stackableVersion is no longer needed when the operator and product stackable versions match e.g. both are 23.11. If this field has been previously set this will not happen: in such cases spec.image.stackableVersion should be set explicitly, either to a release version or to null.
|
Stackable Operator for Apache Airflow
-
Removed AirflowDB
.spec.clusterConfig.databaseInitializationwas removed from the CRD. This allowed configuring logging for the database initialization job, which does not exist anymore. Instead, database initialization is done by the scheduler pod, which has its own logging configuration. This was necessary to remove theAirflowDBCustom Resource, which caused problems when upgrading or reinstalling Airflow clusters. Additionally, theAirflowDBcustom resource is not used anymore. Those CRs should be deleted. -
Implement KubernetesExecutor As an alternative to the CeleryExecutor we now also support the KubernetesExecutor, whereby pods are spun up for job tasks and are terminated afterwards. This removes the need for a Redis job queue and offers an alternative approach to resource management.
-
Rename service port name The service port name has been renamed from airflow to http for consistency reasons. This change requires that the statefulset be removed before upgrading. There might also be some e.g. Ingresses that rely on the port name that will need to be updated.
-
AuthenticationClass references
spec.clusterConfig.authenticationConfighas been consolidated tospec.clusterConfig.authenticationwhich takes a vector of AuthenticationClass references.
These consolidated changes would require a change from e.g.
Breaking changes details
apiVersion: airflow.stackable.tech/v1alpha1
kind: AirflowCluster
metadata:
name: airflow
spec:
image:
productVersion: "2.6.1"
stackableVersion: "23.7" (1)
clusterConfig:
executor: CeleryExecutor (2)
...
databaseInitialization: (3)
logging:
enableVectorAgent: false
containers:
...
authenticationConfig: (4)
authenticationClass: server-tls
userRegistrationRole: Admin
webservers:
...
workers: (5)
roleGroups:
default:
replicas: 1
...to:
apiVersion: airflow.stackable.tech/v1alpha1
kind: AirflowCluster
metadata:
name: airflow
spec:
image:
productVersion: "2.6.1"
clusterConfig:
authentication: (4)
- authenticationClass: server-tls
userRegistrationRole: Admin
webservers:
roleGroups:
default:
replicas: 1
celeryExecutors: (5)
roleGroups:
default:
replicas: 1
...| 1 | this field is no longer needed if the product and operator image versions match |
| 2 | field removed |
| 3 | section removed |
| 4 | authenticationConfig replaced with a list of authentication classes |
| 5 | workers replaced with either celeryExecutors or kubernetesExecutors |
| it will be necessary to remove the stateful sets before updating the Airflow resource due to the change to the name of container port. Any existing AirflowDB jobs should be deleted as well. This will allow a database update to be followed through where necessary. |
Stackable Operator for Apache HDFS
-
Removed field autoFormatFs This field was not used.
Breaking changes details
This requires a change from e.g.
apiVersion: hdfs.stackable.tech/v1alpha1
kind: HdfsCluster
metadata:
name: hdfs
spec:
image:
productVersion: "3.3.4"
stackableVersion: "23.7" (1)
clusterConfig:
zookeeperConfigMapName: hdfs-zk
autoFormatFs: False (2)
...to:
apiVersion: hdfs.stackable.tech/v1alpha1
kind: HdfsCluster
metadata:
name: hdfs
spec:
image:
productVersion: "3.3.4"
clusterConfig:
zookeeperConfigMapName: hdfs-zk
...| 1 | this field is no longer needed if the product and operator image versions match. |
| 2 | field removed. |
Stackable Operator for Apache Kafka
-
Certificate conversion The secret-operator now handles certificate conversion. This allows for the removal of the prepare init container but means that you can’t configure the log level for this container anymore. You will need to remove the field
spec.brokers.config.logging.container.preparein case it is specified.
Breaking changes details
This requires a change from e.g.
apiVersion: kafka.stackable.tech/v1alpha1
kind: KafkaCluster
metadata:
name: kafka
spec:
image:
productVersion: "3.4.0"
stackableVersion: "23.7" (1)
clusterConfig:
...
brokers:
config:
logging:
containers:
prepare: (2)
console:
level: INFO
...to:
apiVersion: kafka.stackable.tech/v1alpha1
kind: KafkaCluster
metadata:
name: kafka
spec:
image:
productVersion: "3.4.0"
clusterConfig:
...
brokers:
config:
logging:
...| 1 | this field is no longer needed if the product and operator image versions match. |
| 2 | section removed for prepare container. |
| for details about how Kafka uses a PVC to persist a reference to its ZNode, and how this may be relevant to upgrade scenarios, please read the documentation here. |
Stackable Operator for Apache NiFi
-
AuthenticationClass references Consolidated authentication config to a list of AuthenticationClasses.
-
Remove credential generation Removed crd support for the auto generation of admin credentials.
-
Remove redundant authentication field Removed crd support for the
nifi.security.allow.anonymous.authenticationproperty that was never used.
Breaking changes details
This requires a change from e.g.
---
apiVersion: nifi.stackable.tech/v1alpha1
kind: NifiCluster
metadata:
name: test-nifi
spec:
image:
productVersion: "1.21.0"
stackableVersion: "23.7" (1)
clusterConfig:
zookeeperConfigMapName: test-zk
authentication: (2)
allowAnonymousAccess: False (3)
method:
singleUser:
adminCredentialsSecret: nifi-admin-credentials-simple
autoGenerate: False (4)
sensitiveProperties:
keySecret: nifi-sensitive-property-key
nodes:
roleGroups:
default:
replicas: 1to:
---
apiVersion: nifi.stackable.tech/v1alpha1
kind: NifiCluster
metadata:
name: test-nifi
spec:
image:
productVersion: "1.23.2"
clusterConfig:
zookeeperConfigMapName: test-zk
authentication: (2)
- authenticationClass: nifi-users (5)
sensitiveProperties:
keySecret: nifi-sensitive-property-key
nodes:
roleGroups:
default:
replicas: 1
---
apiVersion: authentication.stackable.tech/v1alpha1
kind: AuthenticationClass
metadata:
name: nifi-users (5)
spec:
provider:
static:
userCredentialsSecret:
name: nifi-admin-credentials
---
apiVersion: v1
kind: Secret
metadata:
name: nifi-admin-credentials
stringData:
admin: supersecretpassword| 1 | this field is no longer needed if the product and operator image versions match. |
| 2 | this section has been changed to take a list of authentication classes. |
| 3 | functionality has been removed. |
| 4 | functionality has been removed. |
| 5 | the authentication class referenced by the NiFi cluster. |
Stackable Operator for Apache Spark
-
Image specification Use product image selection instead of version.
-
Configuration structure Refactored application roles to use CommonConfiguration structures from the operator framework.
Breaking changes details
This requires a change from e.g.
apiVersion: spark.stackable.tech/v1alpha1
kind: SparkApplication
metadata:
name: spark-examples
spec:
version: "1.0"
sparkImage: "docker.stackable.tech/stackable/spark-k8s:3.4.0-stackable23.7" (1)
mode: cluster
mainClass: org.apache.spark.examples.SparkALS
mainApplicationFile: "local:///stackable/spark/examples/jars/spark-examples.jar"
job: (2)
logging:
enableVectorAgent: False
driver: (2)
logging:
enableVectorAgent: False
executor: (2)
instances: 1
logging:
enableVectorAgent: Falseto:
apiVersion: spark.stackable.tech/v1alpha1
kind: SparkApplication
metadata:
name: spark-examples-2
spec:
version: "1.0"
sparkImage: (1)
productVersion: "3.5.0"
mode: cluster
mainClass: org.apache.spark.examples.SparkALS
mainApplicationFile: "local:///stackable/spark/examples/jars/spark-examples.jar"
job:
config: (2)
logging:
enableVectorAgent: False
driver:
config: (2)
logging:
enableVectorAgent: False
executor:
replicas: 1
config: (2)
logging:
enableVectorAgent: False| 1 | this field has been changed to be consistent with product image selection, documented here. |
| 2 | this section has been changed to be consistent with common configuration definitions used for other operators. |
Stackable Operator for Apache Superset
-
Remove SupersetDB
.spec.clusterConfig.loadExamplesOnInitwas removed from the CRD. Already loaded examples in Superset will not be removed by this change. Additionally, theSupersetDBcustom resource is not used anymore. Those CRs should be deleted. Loading examples is still supported, the process is now described in the documentation. -
Rename service port name The service port name has been renamed from superset to http for consistency reasons. This change requires that the statefulset be removed before upgrading. There might also be some e.g. Ingresses that rely on the port name that will need to be updated.
Breaking changes details
This requires a change from e.g.
apiVersion: superset.stackable.tech/v1alpha1
kind: SupersetCluster
metadata:
name: superset
spec:
image:
productVersion: "2.1.0"
stackableVersion: "23.7" (1)
clusterConfig:
credentialsSecret: superset-credentials
loadExamplesOnInit: false (2)
nodes:
roleGroups:
default:
replicas: 1
...to:
apiVersion: superset.stackable.tech/v1alpha1
kind: SupersetCluster
metadata:
name: superset
spec:
image:
productVersion: "2.1.0"
clusterConfig:
credentialsSecret: superset-credentials
nodes:
roleGroups:
default:
replicas: 1
...| 1 | this field is no longer needed if the product and operator image versions match. |
| 2 | this field has been removed. |
| it will be necessary to remove the stateful sets before updating the Superset resource due to the change to the name of container port. |
Stackable Operator for Trino
-
New OPA Authorizer Version 428 uses the new OPA authorizer which requires changes to existing rego rules.
Breaking changes details
This requires a change from e.g.
apiVersion: trino.stackable.tech/v1alpha1
kind: TrinoCluster
metadata:
name: trino
spec:
image:
productVersion: "414"
clusterConfig:
...to:
---
apiVersion: trino.stackable.tech/v1alpha1
kind: TrinoCluster
metadata:
name: trino
spec:
image:
productVersion: "428"
clusterConfig:
...
---To adapt the rego rules to work with the new authorizer visit the documentation here. For example, you can upgrade to the last version of Open Policy Agent and enter the relevant syntax in a ConfigMap like this:
---
apiVersion: opa.stackable.tech/v1alpha1
kind: OpaCluster
metadata:
name: opa
spec:
image:
productVersion: "0.57.0"
stackableVersion: 23.11
servers:
roleGroups:
default: {}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: simple-trino-opa-bundle
labels:
opa.stackable.tech/bundle: "trino"
data:
trino.rego: |
package trino
import future.keywords.in
default allow = false
allow {
is_admin
}
extended[i] {
some i
input.action.filterResources[i]
is_admin
}
allow {
input.action.operation in ["ExecuteQuery", "AccessCatalog"]
is_bob
}
extended[i] {
input.action.operation in ["FilterCatalogs"]
some i
input.action.filterResources[i]
is_bob
}
is_admin() {
input.context.identity.user == "admin"
}
is_bob() {
input.context.identity.user == "bob"
}Upgrade from 23.7
Using stackablectl
To uninstall the 23.7 release run
$ stackablectl release uninstall 23.7
[INFO ] Uninstalling release 23.7
[INFO ] Uninstalling airflow operator
[INFO ] Uninstalling commons operator
# ...Afterwards you will need to upgrade the CustomResourceDefinitions (CRDs) installed by the Stackable Platform.
The reason for this is that helm will uninstall the operators but not the CRDs. This can be done using kubectl replace:
kubectl replace -f https://raw.githubusercontent.com/stackabletech/airflow-operator/23.11.0/deploy/helm/airflow-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/23.11.0/deploy/helm/commons-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/druid-operator/23.11.0/deploy/helm/druid-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hbase-operator/23.11.0/deploy/helm/hbase-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/23.11.0/deploy/helm/hdfs-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hello-world-operator/23.11.0/deploy/helm/hello-world-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/23.11.0/deploy/helm/hive-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/23.11.0/deploy/helm/kafka-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/23.11.0/deploy/helm/listener-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/23.11.0/deploy/helm/nifi-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/23.11.0/deploy/helm/opa-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/23.11.0/deploy/helm/secret-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/23.11.0/deploy/helm/spark-k8s-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/23.11.0/deploy/helm/superset-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/23.11.0/deploy/helm/trino-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/23.11.0/deploy/helm/zookeeper-operator/crds/crds.yamlcustomresourcedefinition.apiextensions.k8s.io "airflowclusters.airflow.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "airflowdbs.airflow.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "authenticationclasses.authentication.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "s3connections.s3.stackable.tech" replaced
...To install the 23.11 release run
$ stackablectl release install 23.11
[INFO ] Installing release 23.11
[INFO ] Installing airflow operator in version 23.11.0
[INFO ] Installing commons operator in version 23.11.0
[INFO ] Installing druid operator in version 23.11.0
[INFO ] Installing hbase operator in version 23.11.0
[INFO ] Installing hdfs operator in version 23.11.0
[INFO ] Installing hive operator in version 23.11.0
[INFO ] Installing kafka operator in version 23.11.0
[INFO ] Installing listener operator in version 23.11.0
[INFO ] Installing hello-world operator in version 23.11.0
[INFO ] Installing nifi operator in version 23.11.0
[INFO ] Installing opa operator in version 23.11.0
[INFO ] Installing secret operator in version 23.11.0
[INFO ] Installing spark-k8s operator in version 23.11.0
[INFO ] Installing superset operator in version 23.11.0
[INFO ] Installing trino operator in version 23.11.0
[INFO ] Installing zookeeper operator in version 23.11.0Using helm
Use helm list to list the currently installed operators.
You can use the following command to uninstall all operators that are part of the 23.7 release:
$ helm uninstall airflow-operator commons-operator druid-operator hbase-operator hdfs-operator hive-operator kafka-operator listener-operator nifi-operator opa-operator secret-operator spark-k8s-operator superset-operator trino-operator zookeeper-operator
release "airflow-operator" uninstalled
release "commons-operator" uninstalled
# ...Afterwards you will need to upgrade the CustomResourceDefinitions (CRDs) installed by the Stackable Platform.
The reason for this is that helm will uninstall the operators but not the CRDs. This can be done using kubectl replace:
kubectl replace -f https://raw.githubusercontent.com/stackabletech/airflow-operator/23.11.0/deploy/helm/airflow-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/23.11.0/deploy/helm/commons-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/druid-operator/23.11.0/deploy/helm/druid-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hbase-operator/23.11.0/deploy/helm/hbase-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/23.11.0/deploy/helm/hdfs-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hello-world-operator/23.11.0/deploy/helm/hello-world-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/23.11.0/deploy/helm/hive-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/23.11.0/deploy/helm/kafka-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/23.11.0/deploy/helm/listener-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/23.11.0/deploy/helm/nifi-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/23.11.0/deploy/helm/opa-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/23.11.0/deploy/helm/secret-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/23.11.0/deploy/helm/spark-k8s-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/23.11.0/deploy/helm/superset-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/23.11.0/deploy/helm/trino-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/23.11.0/deploy/helm/zookeeper-operator/crds/crds.yamlTo install the 23.11 release run
helm repo add stackable-stable https://repo.stackable.tech/repository/helm-stable/
helm repo update stackable-stable
helm install --wait airflow-operator stackable-stable/airflow-operator --version 23.11.0
helm install --wait commons-operator stackable-stable/commons-operator --version 23.11.0
helm install --wait druid-operator stackable-stable/druid-operator --version 23.11.0
helm install --wait hbase-operator stackable-stable/hbase-operator --version 23.11.0
helm install --wait hdfs-operator stackable-stable/hdfs-operator --version 23.11.0
helm install --wait hive-operator stackable-stable/hive-operator --version 23.11.0
helm install --wait kafka-operator stackable-stable/kafka-operator --version 23.11.0
helm install --wait listener-operator stackable-stable/listener-operator --version 23.11.0
helm install --wait hello-world-operator stackable-stable/hello-world-operator --version 23.11.0
helm install --wait nifi-operator stackable-stable/nifi-operator --version 23.11.0
helm install --wait opa-operator stackable-stable/opa-operator --version 23.11.0
helm install --wait secret-operator stackable-stable/secret-operator --version 23.11.0
helm install --wait spark-k8s-operator stackable-stable/spark-k8s-operator --version 23.11.0
helm install --wait superset-operator stackable-stable/superset-operator --version 23.11.0
helm install --wait trino-operator stackable-stable/trino-operator --version 23.11.0
helm install --wait zookeeper-operator stackable-stable/zookeeper-operator --version 23.11.0Known issues
-
Spark-k8s: missing RBAC permission. This has been fixed in the nightly build of the operator.
Release 23.7
This release introduces the specification of resource quotas and pod overrides and updates the product versions supported by SDP.
New / extended platform features
The following new major platform features were added:
- Resource Quotas
-
Explicit resources are now applied to all containers, for both operators and products. This allows running the Stackable Data Platform on Kubernetes clusters with a ResourceQuota or LimitRange set. Where these are not specified directly, defaults will be used. See this issue for more information.
- Pod Overrides
-
It is now possible to add custom settings which specify elements of a pod template (Service, StatefulSet etc.) on roles or rolegroups, which the operator then merges with the objects it writes before actually applying them. This provides the user with a possibility for specifying any property that can be set on a regular Kubernetes Pod, but which is not directly exposed via the Stackable custom resource definition. Have a look at the documentation for more details.
For example, with HDFS:
roleGroups:
default:
replicas: 1
podOverrides:
spec:
containers:
- name: journalnode
resources:
requests:
cpu: 110m
limits:
cpu: 410m- Openshift certification
-
OLM bundles - a pre-requisite for the Openshift certification process - have been created for each operator. All 15 SDP operators in release 23.4.1 are now Openshift-certified and deployable directly from within an Openshift cluster.
- Signed SDP operator images
-
As of this release all Stackable operator images are signed (this feature will be added to product images in a subsequent release). More information about this, including how to verify the image signatures, can be found in this tutorial.
- New Versions
-
The following new product versions are now supported:
- Deprecated Versions
-
The following product versions are deprecated and will be removed in a later release:
-
Airflow: 2.2.3, 2.2.4, 2.2.5, 2.4.1
-
Druid: 0.23.0, 24.0.0
-
HBase: 2.4.6, 2.4.8, 2.4.9, 2.4.11
-
HDFS: 3.2.2, 3.3.1, 3.3.3
-
Hive: 2.3.9
-
Kafka: 2.7.1, 2.8.1, 3.1.0, 3.2.0, 3.3.1
-
Nifi: 1.15.0, 1.15.1, 1.15.2, 1.15.3, 1.16.0, 1.16.1, 1.16.2, 1.16.3, 1.18.0
-
Opa: 0.27.1, 0.28.0, 0.37.2, 0.41.0, 0.45.0
-
Spark: 3.2.1, 3.3.0
-
Superset: 1.3.2, 1.4.1, 1.5.1
-
Trino: 377, 387, 395, 396, 403
-
Zookeeper: 3.5.8, 3.6.3, 3.7.0, 3.8.0
-
- Removed Versions
-
No product versions have been removed.
- Product features
-
Additionally, there are some individual product features that are noteworthy:
Supported Kubernetes versions
This release supports the following Kubernetes versions:
-
1.26 -
1.25 -
1.24
This Kubernetes version is no longer supported:
-
1.23
Breaking changes
The re-structuring of configuration definitions in certain operators will require you to adapt your existing CRDs as shown below.
Stackable Operator for Apache Airflow
CRDs should be changed from e.g.
spec:
...
executor: CeleryExecutor
loadExamples: true
exposeConfig: false
credentialsSecret: test-airflow-credentials
...to:
spec:
...
clusterConfig:
executor: CeleryExecutor
loadExamples: true
exposeConfig: false
credentialsSecret: test-airflow-credentials
...Stackable Operator for Apache Superset
CRDs should be changed from e.g.
spec:
...
credentialsSecret: superset-credentials
loadExamplesOnInit: false
vectorAggregatorConfigMapName: vector-aggregator-discovery
...to:
spec:
...
clusterConfig:
credentialsSecret: superset-credentials
loadExamplesOnInit: false
vectorAggregatorConfigMapName: vector-aggregator-discovery
...Stackable Operator for Trino
CRDs should be changed from e.g.
spec:
...
clusterConfig:
authentication:
method:
multiUser:
userCredentialsSecret:
name: trino-users
...referencing a Secret with bcrypt-ed data:
---
apiVersion: v1
kind: Secret
metadata:
name: trino-users
type: kubernetes.io/opaque
stringData:
# admin:admin
admin: $2y$10$89xReovvDLacVzRGpjOyAOONnayOgDAyIS2nW9bs5DJT98q17Dy5i
# alice:alice
alice: $2y$10$HcCa4k9v2DRrD/g7e5vEz.Bk.1xg00YTEHOZjPX7oK3KqMSt2xT8W
# bob:bob
bob: $2y$10$xVRXtYZnYuQu66SmruijPO8WHFM/UK5QPHTr.Nzf4JMcZSqt3W.2.to:
spec:
...
clusterConfig:
authentication:
- authenticationClass: trino-users-auth
...referencing an AuthenticationClass (which references a Secret with plain data):
---
apiVersion: authentication.stackable.tech/v1alpha1
kind: AuthenticationClass
metadata:
name: trino-users-auth
spec:
provider:
static:
userCredentialsSecret:
name: trino-users
---
apiVersion: v1
kind: Secret
metadata:
name: trino-users
type: kubernetes.io/opaque
stringData:
admin: admin
alice: alice
bob: bobUpgrade from 23.4
Using stackablectl
To uninstall the 23.4 release run
$ stackablectl release uninstall 23.4
[INFO ] Uninstalling release 23.4
[INFO ] Uninstalling airflow operator
[INFO ] Uninstalling commons operator
# ...Afterwards you will need to upgrade the CustomResourceDefinitions (CRDs) installed by the Stackable Platform.
The reason for this is that helm will uninstall the operators but not the CRDs. This can be done using kubectl replace:
kubectl replace -f https://raw.githubusercontent.com/stackabletech/airflow-operator/23.7.0/deploy/helm/airflow-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/23.7.0/deploy/helm/commons-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/druid-operator/23.7.0/deploy/helm/druid-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hbase-operator/23.7.0/deploy/helm/hbase-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/23.7.0/deploy/helm/hdfs-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/23.7.0/deploy/helm/hive-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/23.7.0/deploy/helm/kafka-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/23.7.0/deploy/helm/listener-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/23.7.0/deploy/helm/nifi-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/23.7.0/deploy/helm/opa-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/23.7.0/deploy/helm/secret-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/23.7.0/deploy/helm/spark-k8s-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/23.7.0/deploy/helm/superset-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/23.7.0/deploy/helm/trino-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/23.7.0/deploy/helm/zookeeper-operator/crds/crds.yamlcustomresourcedefinition.apiextensions.k8s.io "airflowclusters.airflow.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "airflowdbs.airflow.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "authenticationclasses.authentication.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "s3connections.s3.stackable.tech" replaced
...To install the 23.7 release run
$ stackablectl release install 23.7
[INFO ] Installing release 23.7
[INFO ] Installing airflow operator in version 23.7.0
[INFO ] Installing commons operator in version 23.7.0
[INFO ] Installing druid operator in version 23.7.0
[INFO ] Installing hbase operator in version 23.7.0
[INFO ] Installing hdfs operator in version 23.7.0
[INFO ] Installing hive operator in version 23.7.0
[INFO ] Installing kafka operator in version 23.7.0
[INFO ] Installing listener operator in version 23.7.0
[INFO ] Installing nifi operator in version 23.7.0
[INFO ] Installing opa operator in version 23.7.0
[INFO ] Installing secret operator in version 23.7.0
[INFO ] Installing spark-k8s operator in version 23.7.0
[INFO ] Installing superset operator in version 23.7.0
[INFO ] Installing trino operator in version 23.7.0
[INFO ] Installing zookeeper operator in version 23.7.0Using helm
Use helm list to list the currently installed operators.
You can use the following command to uninstall all operators that are part of the 23.4 release:
$ helm uninstall airflow-operator commons-operator druid-operator hbase-operator hdfs-operator hive-operator kafka-operator listener-operator nifi-operator opa-operator secret-operator spark-k8s-operator superset-operator trino-operator zookeeper-operator
release "airflow-operator" uninstalled
release "commons-operator" uninstalled
# ...Afterwards you will need to upgrade the CustomResourceDefinitions (CRDs) installed by the Stackable Platform.
The reason for this is that helm will uninstall the operators but not the CRDs. This can be done using kubectl replace:
kubectl replace -f https://raw.githubusercontent.com/stackabletech/airflow-operator/23.7.0/deploy/helm/airflow-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/23.7.0/deploy/helm/commons-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/druid-operator/23.7.0/deploy/helm/druid-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hbase-operator/23.7.0/deploy/helm/hbase-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/23.7.0/deploy/helm/hdfs-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/23.7.0/deploy/helm/hive-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/23.7.0/deploy/helm/kafka-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/23.7.0/deploy/helm/listener-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/23.7.0/deploy/helm/nifi-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/23.7.0/deploy/helm/opa-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/23.7.0/deploy/helm/secret-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/23.7.0/deploy/helm/spark-k8s-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/23.7.0/deploy/helm/superset-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/23.7.0/deploy/helm/trino-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/23.7.0/deploy/helm/zookeeper-operator/crds/crds.yamlTo install the 23.7 release run
helm repo add stackable-stable https://repo.stackable.tech/repository/helm-stable/
helm repo update stackable-stable
helm install --wait airflow-operator stackable-stable/airflow-operator --version 23.7.0
helm install --wait commons-operator stackable-stable/commons-operator --version 23.7.0
helm install --wait druid-operator stackable-stable/druid-operator --version 23.7.0
helm install --wait hbase-operator stackable-stable/hbase-operator --version 23.7.0
helm install --wait hdfs-operator stackable-stable/hdfs-operator --version 23.7.0
helm install --wait hive-operator stackable-stable/hive-operator --version 23.7.0
helm install --wait kafka-operator stackable-stable/kafka-operator --version 23.7.0
helm install --wait listener-operator stackable-stable/listener-operator --version 23.7.0
helm install --wait nifi-operator stackable-stable/nifi-operator --version 23.7.0
helm install --wait opa-operator stackable-stable/opa-operator --version 23.7.0
helm install --wait secret-operator stackable-stable/secret-operator --version 23.7.0
helm install --wait spark-k8s-operator stackable-stable/spark-k8s-operator --version 23.7.0
helm install --wait superset-operator stackable-stable/superset-operator --version 23.7.0
helm install --wait trino-operator stackable-stable/trino-operator --version 23.7.0
helm install --wait zookeeper-operator stackable-stable/zookeeper-operator --version 23.7.0Known upgrade issues
In the case of the breaking changes detailed above it will be necessary to update the custom resources for Airflow, Superset and Trino clusters and re-apply them.
Additionally, please note the following:
All operators
-
If the default PVC size has been changed, then the StatefulSet must be deleted: it is not possible to change the PVC in the StatefulSet specification.
-
The error message is similar to:
StatefulSet.apps "trino-worker-default" is invalid: spec: Forbidden: updates to `StatefulSetspec for fields other than 'replicas', 'template', 'updateStrategy', […]`
-
Release 23.4
The focus in this platform release is on the support of default/custom affinities and the status field, as well as the rollout of log aggregation across the remaining operators. Additionally, all operators have been updated and tested for compatibility with OpenShift clusters (versions 4.10 and 4.11). Several operators from the 23.1 platform release were already certified against OpenShift.
Release 23.4.0
This was the first release in the 23.4 release line. It is recommended to install Release 23.4.1 instead, as it contains relevant bugfixes.
Release 23.4.1
This is a bugfix/patch-level release that fixes the following issues:
-
Fix missing custom resource defaults that are required for a release update. See here.
-
Specify the security context to run as a member of the root group (this has been implemented for the Stackable operators where it had not previously been implemented i.e. Apache HBase, Apache HDFS, Apache ZooKeeper and Apache Spark on Kubernetes). This is required by Openshift clusters so that the product can be run with a random UID. This is a requirement for at least Airflow, but is Openshift policy as described here and here.
-
Automatically migrate the name used for the bundle-builder container for OPA daemonsets. See here.
-
Automatically shorten the registration socket path used in listener-operator for Microk8s compatibility, migrated during upgrade. See here.
New / extended platform features
The following new major platform features were added:
- Cluster Operation
-
The first part of Cluster operations was rolled out in every applicable Stackable Operator. This supports pausing the cluster reconciliation and stopping the cluster completely. Pausing reconciliation will not apply any changes to the Kubernetes resources (e.g. when changing the custom resource). Stopping the cluster will set all replicas of StatefulSets, Deployments or DaemonSets to zero and therefore result in the deletion of all Pods belonging to that cluster (not the PVCs)
- Status Field
-
Operators of the Stackable Data Platform create, manage and delete Kubernetes resources: in order to easily query the health state of the products - and react accordingly - Stackable Operators use several predefined condition types to capture different aspects of a product’s availability. See this ADR for more information.
- Default / Custom Affinities
-
In Kubernetes there are different ways to influence how Pods are assigned to Nodes. In some cases it makes sense to co-locate certain services that communicate a lot with each other, such as HBase regionservers with HDFS datanodes. In other cases it makes sense to distribute the Pods among as many Nodes as possible. There may also be additional requirements e.g. placing important services - such as HDFS namenodes - in different racks, datacenter rooms or even datacenters. This release implements default affinities that should suffice for many scenarios out-of-the box, while also allowing for custom affinity rules at a role and/or role-group level. See this ADR for more information.
- Log Aggregation
-
The logging framework (added to the platform in Release 23.1) offers a consistent custom resource configuration and a separate, persisted sink (defaulting to OpenSearch). This has now been rolled out across all products. See this ADR and this concepts page for more information.
- Service Type
-
The Service type can now be specified in all products. This currently differentiates between the internal ClusterIP and the external NodePort and is forward compatible with the ListenerClass for the automatic exposure of Services via the Listener Operator. This change is not backwards compatible with older platform releases. For security reasons, the default is set to the cluster-internal (ClusterIP) ListenerClass. A cluster can be exposed outside of Kubernetes by setting clusterConfig.listenerClass to external-unstable (NodePort) or external-stable (LoadBalancer).
- New Versions
-
No new product versions are supported in this platform release.
- Deprecated Versions
-
No product versions have been deprecated in this platform release.
- Product features
-
Additionally, there are some individual product features that are noteworthy:
stackablectl
The following have been added to stackablectl:
Trino-iceberg demo
This is a condensed form of the data-lakehouse-iceberg-trino-spark demo focusing on using the lakehouse to store and modify data. It demonstrates how to integrate Trino and Iceberg and should run on a local workstation.
Jupyterhub/Spark demo
This demo showcases the integration between Jupyter and Apache Hadoop deployed on the Stackable Data Platform (SDP) Kubernetes cluster. This demo can be installed on most cloud managed Kubernetes clusters as well as on premise or on a reasonably provisioned laptop.
The quickstart guide shows how to get started with stackablectl. This
link lists the available demos.
Supported Kubernetes versions
This release supports the following Kubernetes versions:
-
1.26 -
1.25 -
1.24 -
1.23(it is planned to discontinue support for this version in the next release)
Breaking changes
You will need to adapt your existing CRDs due to the following breaking changes detailed below.
All Stackable Operators
As mentioned above, specifying the service type is a breaking change for all operators. The default value is set to the
cluster-internal ListenerClass: if the cluster requires external access outside of Kubernetes then set
clusterConfig.listenerClass to external-unstable or external-stable:
spec:
image:
productVersion: "396"
stackableVersion: "23.4.1"
clusterConfig:
listenerClass: external-unstableThis is an example for Trino, but the pattern is the same across all operators.
Stackable Operator for Apache Airflow
Existing Airflow clusters need to be deleted and recreated. Airflow metadata held in the database and DAGs saved on disk are not affected.
This is required because the UID of the Airflow user has changed to be in line with the rest of the platform.
Stackable Operator for Apache HBase
CRDs should be changed from e.g.
spec:
...
hdfsConfigMapName: simple-hdfs
zookeeperConfigMapName: simple-znodeto:
spec:
...
clusterConfig:
hdfsConfigMapName: simple-hdfs
zookeeperConfigMapName: simple-znodeStackable Operator for Apache Hadoop
CRDs should be changed from e.g.
spec:
...
zookeeperConfigMapName: simple-hdfs-znode
dfsReplication: 3
vectorAggregatorConfigMapName: vector-aggregator-discoveryto:
spec:
...
clusterConfig:
zookeeperConfigMapName: simple-hdfs-znode
dfsReplication: 1
vectorAggregatorConfigMapName: vector-aggregator-discoveryStackable Operator for Apache Nifi
CRDs should be changed from e.g.
spec:
...
zookeeperConfigMapName: simple-nifi-znodeto:
spec:
...
clusterConfig:
zookeeperConfigMapName: simple-nifi-znodeStackable Operator for Apache Spark-k8s
-
Support has been dropped for the use of the
spec.{driver,executor}.nodeSelectorfield. Usespec.{driver,executor}.affinity.nodeSelectorinstead - this is part of Deploy default and support custom affinities in our operators
CRDs should be changed from e.g.
spec:
...
driver:
nodeSelector:to:
spec:
...
driver:
affinity:Stackable Operator for Apache Trino
CRDs should be changed from e.g.
spec:
...
opa:
configMapName: simple-opa
package: trino
authentication:
method:
multiUser:
userCredentialsSecret:
name: simple-trino-users-secret
catalogLabelSelector:
matchLabels:
trino: simple-trino
vectorAggregatorConfigMapName: vector-aggregator-discoveryto:
spec:
...
clusterConfig:
authentication:
method:
multiUser:
userCredentialsSecret:
name: simple-trino-users-secret
authorization:
opa:
configMapName: simple-opa
package: trino
catalogLabelSelector:
matchLabels:
trino: simple-trino
vectorAggregatorConfigMapName: vector-aggregator-discoveryUpgrade from 23.1
Using stackablectl
You can list the available releases as follows
$ stackablectl release list
RELEASE RELEASE DATE DESCRIPTION
23.4 2023-04-25 Fifth release focusing on affinities and product status
23.1 2023-01-27 Fourth release focusing on image selection and logging
22.11 2022-11-08 Third release focusing on resource management
22.09 2022-09-09 Second release focusing on security and OpenShift support
22.06 2022-06-30 First official release of the Stackable Data PlatformTo uninstall the 23.1 release run
$ stackablectl release uninstall 23.1
[INFO ] Uninstalling release 23.1
[INFO ] Uninstalling airflow operator
[INFO ] Uninstalling commons operator
# ...Afterwards you will need to upgrade the CustomResourceDefinitions (CRDs) installed by the Stackable Platform.
The reason for this is that helm will uninstall the operators but not the CRDs. This can be done using kubectl replace:
kubectl replace -f https://raw.githubusercontent.com/stackabletech/airflow-operator/23.4.1/deploy/helm/airflow-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/23.4.1/deploy/helm/commons-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/druid-operator/23.4.1/deploy/helm/druid-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hbase-operator/23.4.1/deploy/helm/hbase-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/23.4.1/deploy/helm/hdfs-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/23.4.1/deploy/helm/hive-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/23.4.1/deploy/helm/kafka-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/23.4.1/deploy/helm/listener-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/23.4.1/deploy/helm/nifi-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/23.4.1/deploy/helm/opa-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/23.4.1/deploy/helm/secret-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/23.4.1/deploy/helm/spark-k8s-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/23.4.1/deploy/helm/superset-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/23.4.1/deploy/helm/trino-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/23.4.1/deploy/helm/zookeeper-operator/crds/crds.yamlcustomresourcedefinition.apiextensions.k8s.io "airflowclusters.airflow.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "airflowdbs.airflow.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "authenticationclasses.authentication.stackable.tech" replaced
customresourcedefinition.apiextensions.k8s.io "s3connections.s3.stackable.tech" replaced
...To install the 23.4 release run
$ stackablectl release install 23.4
[INFO ] Installing release 23.4
[INFO ] Installing airflow operator in version 23.4.1
[INFO ] Installing commons operator in version 23.4.1
[INFO ] Installing druid operator in version 23.4.1
[INFO ] Installing hbase operator in version 23.4.1
[INFO ] Installing hdfs operator in version 23.4.1
[INFO ] Installing hive operator in version 23.4.1
[INFO ] Installing kafka operator in version 23.4.1
[INFO ] Installing listener operator in version 23.4.1
[INFO ] Installing nifi operator in version 23.4.1
[INFO ] Installing opa operator in version 23.4.1
[INFO ] Installing secret operator in version 23.4.1
[INFO ] Installing spark-k8s operator in version 23.4.1
[INFO ] Installing superset operator in version 23.4.1
[INFO ] Installing trino operator in version 23.4.1
[INFO ] Installing zookeeper operator in version 23.4.1Using helm
Use helm list to list the currently installed operators.
You can use the following command to uninstall all operators that are part of the 23.1 release:
$ helm uninstall airflow-operator commons-operator druid-operator hbase-operator hdfs-operator hive-operator kafka-operator listener-operator nifi-operator opa-operator secret-operator spark-k8s-operator superset-operator trino-operator zookeeper-operator
release "airflow-operator" uninstalled
release "commons-operator" uninstalled
# ...Afterwards you will need to upgrade the CustomResourceDefinitions (CRDs) installed by the Stackable Platform.
The reason for this is that helm will uninstall the operators but not the CRDs. This can be done using kubectl replace:
kubectl replace -f https://raw.githubusercontent.com/stackabletech/airflow-operator/23.4.1/deploy/helm/airflow-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/commons-operator/23.4.1/deploy/helm/commons-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/druid-operator/23.4.1/deploy/helm/druid-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hbase-operator/23.4.1/deploy/helm/hbase-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/23.4.1/deploy/helm/hdfs-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/hive-operator/23.4.1/deploy/helm/hive-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/kafka-operator/23.4.1/deploy/helm/kafka-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/listener-operator/23.4.1/deploy/helm/listener-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/nifi-operator/23.4.1/deploy/helm/nifi-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/opa-operator/23.4.1/deploy/helm/opa-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/secret-operator/23.4.1/deploy/helm/secret-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/23.4.1/deploy/helm/spark-k8s-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/superset-operator/23.4.1/deploy/helm/superset-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/trino-operator/23.4.1/deploy/helm/trino-operator/crds/crds.yaml
kubectl replace -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/23.4.1/deploy/helm/zookeeper-operator/crds/crds.yamlTo install the 23.4 release run
helm repo add stackable-stable https://repo.stackable.tech/repository/helm-stable/
helm repo update stackable-stable
helm install --wait airflow-operator stackable-stable/airflow-operator --version 23.4.1
helm install --wait commons-operator stackable-stable/commons-operator --version 23.4.1
helm install --wait druid-operator stackable-stable/druid-operator --version 23.4.1
helm install --wait hbase-operator stackable-stable/hbase-operator --version 23.4.1
helm install --wait hdfs-operator stackable-stable/hdfs-operator --version 23.4.1
helm install --wait hive-operator stackable-stable/hive-operator --version 23.4.1
helm install --wait kafka-operator stackable-stable/kafka-operator --version 23.4.1
helm install --wait listener-operator stackable-stable/listener-operator --version 23.4.1
helm install --wait nifi-operator stackable-stable/nifi-operator --version 23.4.1
helm install --wait opa-operator stackable-stable/opa-operator --version 23.4.1
helm install --wait secret-operator stackable-stable/secret-operator --version 23.4.1
helm install --wait spark-k8s-operator stackable-stable/spark-k8s-operator --version 23.4.1
helm install --wait superset-operator stackable-stable/superset-operator --version 23.4.1
helm install --wait trino-operator stackable-stable/trino-operator --version 23.4.1
helm install --wait zookeeper-operator stackable-stable/zookeeper-operator --version 23.4.1Release 23.1
This release marks a major change in the way operator and product images are versioned. Up until now, operators were versioned independently of each other and a platform release was a loosely coupled set of operator versions. This had major disadvantages both technical and organisational.
On the technical side, a multi-dimensional matrix of versions had to be tested, documentation cross-references had to be maintained and coordinating a platform release was extremely difficult.
Organizationally the biggest challenge was communication and coordination within the teams as well as to and with users.
As a result, starting with this release, all operator and product images are versioned in lock-step. This platform release is marked 23.1 and all included components are tagged with 23.1.0. Eventual patch versions of the components that might follow and will be tagged with 23.1.1, 23.1.2 and so on.
The focus in this platform release is on the support of offline (or on-premise) product images and the partial rollout of logging support.
New platform features
The following new major platform features were added:
- Product image selection
-
Product image selection has been expanded to cover different scenarios:
-
Stackable-provided product images, defined with the repository, the product version and the stackable tag
-
As above, but without the stackable tag (whereby the most recent tagged image will be taken)
-
The product version and a full repository path (this allows fully-customized product images)
-
These options are described in more detail in this ADR and on this concepts page.
N.B. this is a breaking change across all operators as spec.version has been replaced by spec.image.
- Logging Aggregation
-
Component activity within the platform is logged in a way that makes it difficult to find, persist and consolidate this information. Log configuration is also a challenge. To address these two issues a logging framework has been added to the platform, offering a consistent custom resource configuration and a separate, persisted sink (the current implementation support OpenSearch). This is discussed in more detail in this ADR and on this concepts page.
In this release this has been added to the following components:
Support for other products will be added in future releases.
- New Versions
-
The following new product version is now supported:
- Deprecated Versions
-
The following product version is no longer supported:
- Product features
-
Additionally, there are some individual product features that are noteworthy
stackablectl
The following have been added to stackablectl:
Logging demo
This illustrates how to set up logging for Zookeeper and browse the results in an Open Search dashboard. This has been implemented for HBase, Hadoop and Zookeeper and will eventually be available for all Stackable operators.
LDAP stack and tutorial
LDAP support has now been added to multiple products. An explanation of the overall approach is given here but in order to make the configuration steps a little clearer a tutorial has been added that uses a dedicated Stackable stack for OpenLDAP and shows its usage.
The quickstart guide shows how to get started with stackablectl. This
link lists the available demos.
Supported Kubernetes versions
This release supports the following Kubernetes versions:
-
1.25 -
1.24 -
1.23 -
1.22
Breaking changes
This release brings with it several breaking changes needed to future-proof the platform. You will need to adapt your existing CRDs due to the following breaking changes:
All Stackable Operators
As mentioned above, product image selection is a breaking for all operators. Previously the product image was declared
using spec.version:
spec:
version: 396-stackable23.1(example for Trino)
This must now be replaced with spec.image:
spec:
image:
productVersion: 396
stackableVersion: 23.1This is the same pattern across operators. so for Hive the change would look like this. From:
spec:
version: 3.1.3-stackable23.1to
spec:
image:
productVersion: 3.1.3
stackableVersion: 23.1Stackable Operator for Apache Druid
This means a stackable version >= 23.1 has to be used for the product image.
Deep storage, Ingestion spec, discovery config maps, authentication etc. are now subfields of spec.clusterConfig instead of being top level under spec. Change the resource from e.g.:
zookeeperConfigMapName: simple-druid-znode
metadataStorageDatabase:
dbType: derby
connString: jdbc:derby://localhost:1527/var/druid/metadata.db;create=true
host: localhost
port: 1527
deepStorage:
hdfs:
configMapName: simple-hdfs
directory: /datato
clusterConfig:
deepStorage:
hdfs:
configMapName: simple-hdfs
directory: /data
metadataStorageDatabase:
dbType: derby
connString: jdbc:derby://localhost:1527/var/druid/metadata.db;create=true
host: localhost
port: 1527
tls: null
zookeeperConfigMapName: simple-druid-znodeStackable Operator for Apache Hadoop
As part of the change mentioned above we also did some code cleanup that allowed us to remove arbitrary hard-coded
values from the operator. This change affects the directory structure the operator creates inside of the
PersistentVolumes used for permanent storage.
The old folder naming was:
-
DataNode →
data -
JournalNode →
journal -
NameNode →
name
which has now been adopted to match the actual rolename:
-
DataNode →
datanode -
JournalNode →
journalnode -
NameNode →
namenode
Unfortunately, this means that for cluster that where initially rolled out with an older operator version, a one-time migration step becomes necessary to rename these directories.
You can either do this manually by attaching the PVs to a pod and performing the rename (cluster needs to be stopped for this) or use the script provided below.
| Please be aware that if this script runs after the cluster was already restarted with the newer operator version it will delete any data that was written to the empty post-upgrade HDFS that was stood up by the new operator. |
#!/usr/bin/env bash
if [ $# -ne 1 ] ; then
echo "Usage: $0 CLUSTER_NAME"
exit 1
else
HDFS_CLUSTER_NAME=$1
fi
kubectl get pvc -l app.kubernetes.io/name=hdfs -l app.kubernetes.io/instance="$HDFS_CLUSTER_NAME"
for pvc in $(kubectl get pvc -l app.kubernetes.io/name=hdfs -l app.kubernetes.io/instance="$HDFS_CLUSTER_NAME" -l app.kubernetes.io/component=journalnode -o name | sed -e 's#persistentvolumeclaim/##'); do
kubectl apply -f - << EOF
apiVersion: batch/v1
kind: Job
metadata:
name: migrate-journalnode-${pvc}
spec:
template:
spec:
containers:
- name: migrate
image: docker.stackable.tech/stackable/hadoop:3.3.4-stackable23.1.0
command: ["bash", "-c", "ls -la /stackable/data && if [ -d /stackable/data/journal ]; then echo Removing might existing target dir && rm -rf /stackable/data/journalnode && echo Renaming folder && mv /stackable/data/journal /stackable/data/journalnode; else echo Nothing to do; fi"]
volumeMounts:
- name: data
mountPath: /stackable/data
volumes:
- name: data
persistentVolumeClaim:
claimName: ${pvc}
restartPolicy: Never
backoffLimit: 1
EOF
done
for pvc in $(kubectl get pvc -l app.kubernetes.io/name=hdfs -l app.kubernetes.io/instance="$HDFS_CLUSTER_NAME" -l app.kubernetes.io/component=namenode -o name | sed -e 's#persistentvolumeclaim/##'); do
kubectl apply -f - << EOF
apiVersion: batch/v1
kind: Job
metadata:
name: migrate-namenode-${pvc}
spec:
template:
spec:
containers:
- name: migrate
image: docker.stackable.tech/stackable/hadoop:3.3.4-stackable23.1.0
command: ["bash", "-c", "ls -la /stackable/data && if [ -d /stackable/data/name ]; then echo Removing might existing target dir && rm -rf /stackable/data/namenode && echo Renaming folder && mv /stackable/data/name /stackable/data/namenode; else echo Nothing to do; fi"]
volumeMounts:
- name: data
mountPath: /stackable/data
volumes:
- name: data
persistentVolumeClaim:
claimName: ${pvc}
restartPolicy: Never
backoffLimit: 1
EOF
done
for pvc in $(kubectl get pvc -l app.kubernetes.io/name=hdfs -l app.kubernetes.io/instance="$HDFS_CLUSTER_NAME" -l app.kubernetes.io/component=datanode -o name | sed -e 's#persistentvolumeclaim/##'); do
kubectl apply -f - << EOF
apiVersion: batch/v1
kind: Job
metadata:
name: migrate-datanode-${pvc}
spec:
template:
spec:
containers:
- name: migrate
image: docker.stackable.tech/stackable/hadoop:3.3.4-stackable23.1.0
command: ["bash", "-c", "ls -la /stackable/data/data && if [ -d /stackable/data/data/data ]; then echo Removing might existing target dir && rm -rf /stackable/data/data/datanode && echo Renaming folder && mv /stackable/data/data/data /stackable/data/data/datanode; else echo Nothing to do; fi"]
volumeMounts:
- name: data
mountPath: /stackable/data/data
volumes:
- name: data
persistentVolumeClaim:
claimName: ${pvc}
restartPolicy: Never
backoffLimit: 1
EOF
doneThe migration process for this now becomes:
-
Stop HDFS cluster by either removing the HdfsCluster definition object or scaling all roles to 0 replicas
-
Uninstall Stackable Operator for Apache Hadoop
-
Run migration script
-
Install newer version of Stackable Operator for Apache Hadoop
Stackable Operator for Apache Hive
These two changes mean that resources previously defined like this:
s3:
reference: minio
metastore:
roleGroups:
default:
replicas: 1
config:
database:
connString: jdbc:postgresql://hive-postgresql:5432/hive
user: hive
password: hive
dbType: postgreswill now be defined like this:
clusterConfig:
database:
connString: jdbc:postgresql://hive-postgresql:5432/hive
user: hive
password: hive
dbType: postgres
s3:
reference: minio
metastore:
roleGroups:
default:
replicas: 1Stackable Operator for Apache Kafka
This means a stackable version >= 23.1 has to be used for the product image.
spec:
...
zookeeperConfigMapName: simple-kafka-znode
config:
authentication:
- authenticationClass: kafka-client-auth-tls
tls:
secretClass: tls
clientAuthentication:
authenticationClass: kafka-client-auth-tls
internalTls:
secretClass: kafka-internal-tlsChanges to:
spec:
...
clusterConfig:
authentication:
- authenticationClass: kafka-client-auth-tls
tls:
internalSecretClass: kafka-internal-tls
serverSecretClass: tls
zookeeperConfigMapName: simple-kafka-znodeStackable Operator for Apache Nifi
This means a stackable version >= 23.1 has to be used for the product image.
Stackable Operator for Trino
This means a stackable version >= 23.1 has to be used for the product image.
This changes the secret definition from:
stringData:
LDAP_USER: cn=admin,dc=example,dc=org
LDAP_PASSWORD: adminto:
stringData:
user: cn=admin,dc=example,dc=org
password: adminStackable Operator for Apache Zookeeper
Similar to the Kafka example above, the configuration settings are consolidated under .spec i.e. from:
config:
tls:
secretClass: tls
clientAuthentication:
authenticationClass: zk-client-tls
quorumTlsSecretClass: tlsto:
clusterConfig:
authentication:
- authenticationClass: zk-client-tls
tls:
serverSecretClass: tls
quorumSecretClass: tlsUpgrade from 22.11
Using stackablectl
You can list the available releases as follows
$ stackablectl release list
RELEASE RELEASE DATE DESCRIPTION
23.1 2023-01-27 Fourth release focusing on image selection and logging
22.11 2022-11-08 Third release focusing on resource management
22.09 2022-09-09 Second release focusing on security and OpenShift support
22.06 2022-06-30 First official release of the Stackable Data PlatformTo uninstall the 22.11 release run
$ stackablectl release uninstall 22.11
[INFO ] Uninstalling release 22.11
[INFO ] Uninstalling airflow operator
[INFO ] Uninstalling commons operator
# ...Afterwards you will need to update the CustomResourceDefinitions (CRDs) installed by the Stackable Platform. The reason for this is that helm will uninstall the operators but not the CRDs.
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/airflow-operator/23.1.0/deploy/helm/airflow-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/commons-operator/23.1.0/deploy/helm/commons-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/druid-operator/23.1.0/deploy/helm/druid-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hbase-operator/23.1.0/deploy/helm/hbase-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/23.1.0/deploy/helm/hdfs-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hive-operator/23.1.0/deploy/helm/hive-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/kafka-operator/23.1.0/deploy/helm/kafka-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/nifi-operator/23.1.0/deploy/helm/nifi-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/opa-operator/23.1.0/deploy/helm/opa-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/secret-operator/23.1.0/deploy/helm/secret-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/23.1.0/deploy/helm/spark-k8s-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/superset-operator/23.1.0/deploy/helm/superset-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/trino-operator/23.1.0/deploy/helm/trino-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/23.1.0/deploy/helm/zookeeper-operator/crds/crds.yamlTo install the 23.1 release run
$ stackablectl release install 23.1
[INFO ] Installing release 23.1
[INFO ] Installing airflow operator in version 23.1.0
[INFO ] Installing commons operator in version 23.1.0
[INFO ] Installing druid operator in version 23.1.0
[INFO ] Installing hbase operator in version 23.1.0
[INFO ] Installing hdfs operator in version 23.1.0
[INFO ] Installing hive operator in version 23.1.0
[INFO ] Installing kafka operator in version 23.1.0
[INFO ] Installing listener operator in version 23.1.0
[INFO ] Installing nifi operator in version 23.1.0
[INFO ] Installing opa operator in version 23.1.0
[INFO ] Installing secret operator in version 23.1.0
[INFO ] Installing spark-k8s operator in version 23.1.0
[INFO ] Installing superset operator in version 23.1.0
[INFO ] Installing trino operator in version 23.1.0
[INFO ] Installing zookeeper operator in version 23.1.0
# ...Using helm
Use helm list to list the currently installed operators.
You can use the following command to uninstall all operators that are part of the release 22.11:
$ helm uninstall airflow-operator commons-operator druid-operator hbase-operator hdfs-operator hive-operator kafka-operator nifi-operator opa-operator secret-operator spark-k8s-operator superset-operator trino-operator zookeeper-operator
release "airflow-operator" uninstalled
release "commons-operator" uninstalled
# ...Afterwards you will need to update the CustomResourceDefinitions (CRDs) installed by the Stackable Platform. This is because helm will uninstall the operators but not the CRDs.
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/airflow-operator/23.1.0/deploy/helm/airflow-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/commons-operator/23.1.0/deploy/helm/commons-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/druid-operator/23.1.0/deploy/helm/druid-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hbase-operator/23.1.0/deploy/helm/hbase-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/23.1.0/deploy/helm/hdfs-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hive-operator/23.1.0/deploy/helm/hive-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/kafka-operator/23.1.0/deploy/helm/kafka-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/nifi-operator/23.1.0/deploy/helm/nifi-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/opa-operator/23.1.0/deploy/helm/opa-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/secret-operator/23.1.0/deploy/helm/secret-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/23.1.0/deploy/helm/spark-k8s-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/superset-operator/23.1.0/deploy/helm/superset-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/trino-operator/23.1.0/deploy/helm/trino-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/23.1.0/deploy/helm/zookeeper-operator/crds/crds.yamlTo install the release 23.1 run
$ helm repo add stackable-stable https://repo.stackable.tech/repository/helm-stable/
$ helm repo update stackable-stable
$ helm install --wait airflow-operator stackable-stable/airflow-operator --version 23.1.0
$ helm install --wait commons-operator stackable-stable/commons-operator --version 23.1.0
$ helm install --wait druid-operator stackable-stable/druid-operator --version 23.1.0
$ helm install --wait hbase-operator stackable-stable/hbase-operator --version 23.1.0
$ helm install --wait hdfs-operator stackable-stable/hdfs-operator --version 23.1.0
$ helm install --wait hive-operator stackable-stable/hive-operator --version 23.1.0
$ helm install --wait kafka-operator stackable-stable/kafka-operator --version 23.1.0
$ helm install --wait listener-operator stackable-stable/listener-operator --version 23.1.0
$ helm install --wait nifi-operator stackable-stable/nifi-operator --version 23.1.0
$ helm install --wait opa-operator stackable-stable/opa-operator --version 23.1.0
$ helm install --wait secret-operator stackable-stable/secret-operator --version 23.1.0
$ helm install --wait spark-k8s-operator stackable-stable/spark-k8s-operator --version 23.1.0
$ helm install --wait superset-operator stackable-stable/superset-operator --version 23.1.0
$ helm install --wait trino-operator stackable-stable/trino-operator --version 23.1.0
$ helm install --wait zookeeper-operator stackable-stable/zookeeper-operator --version 23.1.0Release 22.11
This is the third release of the Stackable Data Platform, which this time focuses on resource management.
New platform features
The following new major platform features were added:
- CPU and memory limits configurable
-
The operators now request resources from Kubernetes for the products and required CPU and memory can now also be configured for all products. If your product instances are less performant after the update, the new defaults might be set too low and we recommend to set custom requests for your cluster.
- Orphaned Resources
-
The operators now properly clean up after scaling down products. This means for example deleting StatefulSets that were left over after scaling down.
- New Versions
-
New product versions are supported.
- Product features
-
Additionally there are some individual product features that are noteworthy
-
NiFi: repository sizes are now adjusted based on declared PVC sizes
-
The github repositories contain new and improved READMEs.
Supported Kubernetes versions
This release supports the following Kubernetes versions:
-
1.25(new) -
1.24 -
1.23 -
1.22
Upgrade from 22.09
Using stackablectl
You can list the available releases as follows
$ stackablectl release list
RELEASE RELEASE DATE DESCRIPTION
22.11 2022-11-08 Third release focusing on resource management
22.09 2022-09-09 Second release focusing on security and OpenShift support
22.06 2022-06-30 First official release of the Stackable Data PlatformTo uninstall the 22.09 release run
$ stackablectl release uninstall 22.09
[INFO ] Uninstalling release 22.09
[INFO ] Uninstalling airflow operator
[INFO ] Uninstalling commons operator
# ...Afterwards you will need to update the CustomResourceDefinitions (CRDs) installed by the Stackable Platform. The reason for this is that helm will uninstall the operators but not the CRDs.
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/airflow-operator/0.6.0/deploy/helm/airflow-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/commons-operator/0.4.0/deploy/helm/commons-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/druid-operator/0.8.0/deploy/helm/druid-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hbase-operator/0.5.0/deploy/helm/hbase-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/0.6.0/deploy/helm/hdfs-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hive-operator/0.8.0/deploy/helm/hive-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/kafka-operator/0.8.0/deploy/helm/kafka-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/nifi-operator/0.8.0/deploy/helm/nifi-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/opa-operator/0.11.0/deploy/helm/opa-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/secret-operator/0.6.0/deploy/helm/secret-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/0.6.0/deploy/helm/spark-k8s-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/superset-operator/0.7.0/deploy/helm/superset-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/trino-operator/0.8.0/deploy/helm/trino-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/0.12.0/deploy/helm/zookeeper-operator/crds/crds.yamlTo install the 22.11 release run
$ stackablectl release install 22.11
[INFO ] Installing release 22.11
[INFO ] Installing airflow operator in version 0.6.0
[INFO ] Installing commons operator in version 0.4.0
[INFO ] Installing druid operator in version 0.8.0
[INFO ] Installing hbase operator in version 0.5.0
[INFO ] Installing hdfs operator in version 0.6.0
[INFO ] Installing hive operator in version 0.8.0
[INFO ] Installing kafka operator in version 0.8.0
[INFO ] Installing nifi operator in version 0.8.0
[INFO ] Installing opa operator in version 0.11.0
[INFO ] Installing secret operator in version 0.6.0
[INFO ] Installing spark-k8s operator in version 0.6.0
[INFO ] Installing superset operator in version 0.7.0
[INFO ] Installing trino operator in version 0.7.0
[INFO ] Installing zookeeper operator in version 0.12.0
# ...Using helm
Use helm list to list the currently installed operators.
You can use the following command to uninstall all of the operators that are part of the release 22.09:
$ helm uninstall airflow-operator commons-operator druid-operator hbase-operator hdfs-operator hive-operator kafka-operator nifi-operator opa-operator secret-operator spark-k8s-operator superset-operator trino-operator zookeeper-operator
release "airflow-operator" uninstalled
release "commons-operator" uninstalled
# ...Afterwards you will need to update the CustomResourceDefinitions (CRDs) installed by the Stackable Platform. This is because helm will uninstall the operators but not the CRDs.
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/airflow-operator/0.6.0/deploy/helm/airflow-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/commons-operator/0.4.0/deploy/helm/commons-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/druid-operator/0.8.0/deploy/helm/druid-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hbase-operator/0.5.0/deploy/helm/hbase-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/0.6.0/deploy/helm/hdfs-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hive-operator/0.8.0/deploy/helm/hive-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/kafka-operator/0.8.0/deploy/helm/kafka-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/nifi-operator/0.8.0/deploy/helm/nifi-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/opa-operator/0.11.0/deploy/helm/opa-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/secret-operator/0.6.0/deploy/helm/secret-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/0.6.0/deploy/helm/spark-k8s-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/superset-operator/0.7.0/deploy/helm/superset-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/trino-operator/0.8.0/deploy/helm/trino-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/0.12.0/deploy/helm/zookeeper-operator/crds/crds.yamlTo install the release 22.11 run
$ helm repo add stackable-stable https://repo.stackable.tech/repository/helm-stable/
$ helm repo update stackable-stable
$ helm install --wait airflow-operator stackable-stable/airflow-operator --version 0.6.0
$ helm install --wait commons-operator stackable-stable/commons-operator --version 0.4.0
$ helm install --wait druid-operator stackable-stable/druid-operator --version 0.8.0
$ helm install --wait hbase-operator stackable-stable/hbase-operator --version 0.5.0
$ helm install --wait hdfs-operator stackable-stable/hdfs-operator --version 0.6.0
$ helm install --wait hive-operator stackable-stable/hive-operator --version 0.8.0
$ helm install --wait kafka-operator stackable-stable/kafka-operator --version 0.8.0
$ helm install --wait nifi-operator stackable-stable/nifi-operator --version 0.8.0
$ helm install --wait opa-operator stackable-stable/opa-operator --version 0.11.0
$ helm install --wait secret-operator stackable-stable/secret-operator --version 0.6.0
$ helm install --wait spark-k8s-operator stackable-stable/spark-k8s-operator --version 0.6.0
$ helm install --wait superset-operator stackable-stable/superset-operator --version 0.7.0
$ helm install --wait trino-operator stackable-stable/trino-operator --version 0.7.0
$ helm install --wait zookeeper-operator stackable-stable/zookeeper-operator --version 0.12.0Breaking changes
You will need to adapt your existing CRDs due to the following breaking changes:
Stackable Operator for Apache Spark
The configuration of pod resource requests has been changed to be consistent with other operators that are part of the Stackable Data Platform (#174).
In the previous version, these were configured like this:
driver:
cores: 1
coreLimit: "1200m"
memory: "512m"`From now on, Pod resources can be configured in two different ways. The first and recommended way is to add a resources section for each role as the following examples shows:
driver:
resources:
cpu:
min: "1"
max: "1500m"
memory:
limit: "1Gi"The second method is to use the sparkConf section and and set them individually as spark properties:
sparkConf:
spark.kubernetes.submission.waitAppCompletion: "false"
spark.kubernetes.driver.pod.name: "resources-sparkconf-driver"
spark.kubernetes.executor.podNamePrefix: "resources-sparkconf"
spark.kubernetes.driver.request.cores: "2"
spark.kubernetes.driver.limit.cores: "3"When both methods are used, the settings in the sparkConf section override the resources configuration.
Note that none of the settings above have any influence over the parallelism used by Spark itself. The only supported way to affect this is as follows:
sparkConf:
spark.driver.cores: "3"
spark.executor.cores: "3"Release 22.09
This is the second release of the Stackable Data Platform. It contains lots of new features and bugfixes. The main features focus on OpenShift support and security.
New platform features
The following new major platform features were added:
- OpenShift compatibility
-
We have made continued progress towards OpenShift compability, and the following operators can now be previewed on OpenShift. Further improvements are expected in future releases, but no stability or compatibility guarantees are currently made for OpenShift clusters.
- Support for internal and external TLS
-
The following operators support operating the products at a maximal level of transport security by using TLS certificates to secure internal and external communication:
- LDAP authentication
-
Use a central LDAP server to manage all of your user identities in a single place. The following operators added support for LDAP authentication:
stackablectl
stackablectl now supports deploying ready-to-use demos, which give an end-to-end demonstration of the usage of the
Stackable Data Platform. The quickstart guide shows how to get started
with stackablectl. Here you can see the available demos.
Supported Kubernetes versions
This release supports the following Kubernetes versions:
-
1.24 -
1.23 -
1.22
Support for 1.21 was dropped.
Upgrade from 22.06
Using stackablectl
You can list the available releases as follows
$ stackablectl release list
RELEASE RELEASE DATE DESCRIPTION
22.11 2022-11-08 Third release candidate of 22.11
22.09 2022-09-09 Second release focusing on security and OpenShift support
22.06 2022-06-30 First official release of the Stackable Data PlatformTo uninstall the 22.06 release run
$ stackablectl release uninstall 22.06
[INFO ] Uninstalling release 22.06
[INFO ] Uninstalling airflow operator
[INFO ] Uninstalling commons operator
# ...Afterwards you will need to update the CustomResourceDefinitions (CRDs) installed by the Stackable Platform. The reason is, that helm will uninstall the operators but not the CRDs.
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/airflow-operator/0.5.0/deploy/helm/airflow-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/commons-operator/0.3.0/deploy/helm/commons-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/druid-operator/0.7.0/deploy/helm/druid-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hbase-operator/0.4.0/deploy/helm/hbase-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hdfs-operator/0.5.0/deploy/helm/hdfs-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/hive-operator/0.7.0/deploy/helm/hive-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/kafka-operator/0.7.0/deploy/helm/kafka-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/nifi-operator/0.7.0/deploy/helm/nifi-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/opa-operator/0.10.0/deploy/helm/opa-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/secret-operator/0.5.0/deploy/helm/secret-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/0.5.0/deploy/helm/spark-k8s-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/superset-operator/0.6.0/deploy/helm/superset-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/trino-operator/0.6.0/deploy/helm/trino-operator/crds/crds.yaml
$ kubectl apply -f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/0.11.0/deploy/helm/zookeeper-operator/crds/crds.yamlTo install the 22.09 release run
$ stackablectl release install 22.09
[INFO ] Installing release 22.09
[INFO ] Installing airflow operator in version 0.5.0
[INFO ] Installing commons operator in version 0.3.0
[INFO ] Installing druid operator in version 0.7.0
[INFO ] Installing hbase operator in version 0.4.0
[INFO ] Installing hdfs operator in version 0.5.0
[INFO ] Installing hive operator in version 0.7.0
[INFO ] Installing kafka operator in version 0.7.0
[INFO ] Installing nifi operator in version 0.7.0
[INFO ] Installing opa operator in version 0.10.0
[INFO ] Installing secret operator in version 0.5.0
[INFO ] Installing spark-k8s operator in version 0.5.0
[INFO ] Installing superset operator in version 0.6.0
[INFO ] Installing trino operator in version 0.6.0
[INFO ] Installing zookeeper operator in version 0.11.0
# ...Using helm
Use helm list to list the currently installed operators.
You can use the following command to uninstall all of the operators that are part of the release 22.06:
$ helm uninstall airflow-operator commons-operator druid-operator hbase-operator hdfs-operator hive-operator kafka-operator nifi-operator opa-operator secret-operator spark-k8s-operator superset-operator trino-operator zookeeper-operator
release "airflow-operator" uninstalled
release "commons-operator" uninstalled
# ...Afterwards you will need to update the CustomResourceDefinitions (CRDs) installed by the Stackable Platform. The reason is, that helm will uninstall the operators but not the CRDs.
$ kubectl apply \
-f https://raw.githubusercontent.com/stackabletech/airflow-operator/0.5.0/deploy/helm/airflow-operator/crds/crds.yaml \
-f https://raw.githubusercontent.com/stackabletech/commons-operator/0.3.0/deploy/helm/commons-operator/crds/crds.yaml \
-f https://raw.githubusercontent.com/stackabletech/druid-operator/0.7.0/deploy/helm/druid-operator/crds/crds.yaml \
-f https://raw.githubusercontent.com/stackabletech/hbase-operator/0.4.0/deploy/helm/hbase-operator/crds/crds.yaml \
-f https://raw.githubusercontent.com/stackabletech/hdfs-operator/0.5.0/deploy/helm/hdfs-operator/crds/crds.yaml \
-f https://raw.githubusercontent.com/stackabletech/hive-operator/0.7.0/deploy/helm/hive-operator/crds/crds.yaml \
-f https://raw.githubusercontent.com/stackabletech/kafka-operator/0.7.0/deploy/helm/kafka-operator/crds/crds.yaml \
-f https://raw.githubusercontent.com/stackabletech/nifi-operator/0.7.0/deploy/helm/nifi-operator/crds/crds.yaml \
-f https://raw.githubusercontent.com/stackabletech/opa-operator/0.10.0/deploy/helm/opa-operator/crds/crds.yaml \
-f https://raw.githubusercontent.com/stackabletech/secret-operator/0.5.0/deploy/helm/secret-operator/crds/crds.yaml \
-f https://raw.githubusercontent.com/stackabletech/spark-k8s-operator/0.5.0/deploy/helm/spark-k8s-operator/crds/crds.yaml \
-f https://raw.githubusercontent.com/stackabletech/superset-operator/0.6.0/deploy/helm/superset-operator/crds/crds.yaml \
-f https://raw.githubusercontent.com/stackabletech/trino-operator/0.6.0/deploy/helm/trino-operator/crds/crds.yaml \
-f https://raw.githubusercontent.com/stackabletech/zookeeper-operator/0.11.0/deploy/helm/zookeeper-operator/crds/crds.yamlTo install the release 22.09 run
$ helm repo add stackable-stable https://repo.stackable.tech/repository/helm-stable/
$ helm repo update stackable-stable
$ helm install --wait airflow-operator stackable-stable/airflow-operator --version 0.5.0
$ helm install --wait commons-operator stackable-stable/commons-operator --version 0.3.0
$ helm install --wait druid-operator stackable-stable/druid-operator --version 0.7.0
$ helm install --wait hbase-operator stackable-stable/hbase-operator --version 0.4.0
$ helm install --wait hdfs-operator stackable-stable/hdfs-operator --version 0.5.0
$ helm install --wait hive-operator stackable-stable/hive-operator --version 0.7.0
$ helm install --wait kafka-operator stackable-stable/kafka-operator --version 0.7.0
$ helm install --wait nifi-operator stackable-stable/nifi-operator --version 0.7.0
$ helm install --wait opa-operator stackable-stable/opa-operator --version 0.10.0
$ helm install --wait secret-operator stackable-stable/secret-operator --version 0.5.0
$ helm install --wait spark-k8s-operator stackable-stable/spark-k8s-operator --version 0.5.0
$ helm install --wait superset-operator stackable-stable/superset-operator --version 0.6.0
$ helm install --wait trino-operator stackable-stable/trino-operator --version 0.6.0
$ helm install --wait zookeeper-operator stackable-stable/zookeeper-operator --version 0.11.0Breaking changes
You will need to adapt your existing CRDs to the following breaking changes:
druid-operator
-
HDFS deep storage is now configurable via the HDFS discovery config map instead of a url to a HDFS name node (#262). Instead of
deepStorage:
hdfs:
storageDirectory: hdfs://druid-hdfs-namenode-default-0:8020/datause
deepStorage:
hdfs:
configMapName: druid-hdfs
directory: /druidkafka-operator
-
Add TLS encryption and authentication support for internal and client communications. This is breaking for clients because the cluster is secured per default, which results in a client port change (#442). If you don’t want to use TLS to secure your Kafka cluster you can restore the old behavior by using the
tlsattribute as follows:
apiVersion: kafka.stackable.tech/v1alpha1
kind: KafkaCluster
# ...
spec:
config:
tls: null
# ...trino-operator
-
TrinoCatalogs now have their own CRD object and get referenced by the TrinoCluster (#263). Instead of
apiVersion: trino.stackable.tech/v1alpha1
kind: TrinoCluster
# ...
spec:
hiveConfigMapName: hive
s3:
inline:
host: minio
port: 9000
accessStyle: Path
credentials:
secretClass: s3-credentials
# ...use
apiVersion: trino.stackable.tech/v1alpha1
kind: TrinoCluster
# ...
spec:
catalogLabelSelector:
trino: trino
# ...
---
apiVersion: trino.stackable.tech/v1alpha1
kind: TrinoCatalog
metadata:
name: hive
labels:
trino: trino
spec:
connector:
hive:
metastore:
configMap: hive
s3:
inline:
host: minio
port: 9000
accessStyle: Path
credentials:
secretClass: s3-credentialsRelease 22.06
This is our first release of the Stackable Data Platform, bringing Kubernetes operators for 12 products as well as stackablectl, the commandline tool to easily install data products in Kubernetes. Operators spin up production ready product applications. Also, there are some common features across all operators, such as monitoring, service discovery and configuration overrides. Find the Platform features, stackablectl features and Operators below.
Please report any issues you find in the specific operator repositories or in our dedicated github.com/stackabletech/issues/[issues] repository. You may also join us in our Slack community or contact us via our homepage.
While we are very proud of this release it is our first one and we’ll add new features and fix bugs all the time and will have regular releases from now on.
Platform features
- Easily install production ready data applications
-
Using a familiar declarative approach, users can easily install data applications such as Apache Kafka or Trino across multiple cloud Kubernetes providers or on their own data centers. The installation process is fully automated while also providing the flexibility for the user to tune relevant aspects of each application.
- Monitoring
-
All products have monitoring with prometheus enabled. Learn more
- Service discovery
-
Products on the Stackable platform use service discovery to easily interconnect with each other. Learn more
- Configuration overrides
-
All operators support configuration overrides, these are documented in the specific operator documentation pages.
- Common S3 configuration
-
Many products support connecting to S3 to load and/or store data. There is a common resource for S3 connections and buckets across all operators that can be reused. Learn more
- Roles and role groups
-
To support hybrid hardware clusters, the Stackable platform uses the concept of role groups. Services and applications can be configured to maximize hardware efficiency.
- Standardized
-
Learn once reuse everywhere. We use the same conventions in all our operators. Configure your LDAP or S3 connections once and reuse them everywhere. All our operators reuse the same CRD structure as well.
stackablectl
stackablectl is used to install and interact with the operators, either individually or with multiple at once.
Operators
This is the list of all operators in this current release, with their versions for this release.
-
Stackable Operator for Apache Airflow (0.4.0)
-
Load DAGs from ConfigMaps or PersistentVolumeClaims
-
-
Stackable Operator for Apache Druid (0.6.0)
-
S3 and HDFS as deep storage options
-
ingestion from S3 buckets
-
authorization using OPA
-
-
Stackable Operator for Apache Hive (0.6.0)
-
Hive Metastore can index S3
-
-
Stackable Operator for Apache Kafka (0.6.0)
-
Seamless integration with NiFi and Druid
-
Supports OPA authorization
-
-
Stackable Operator for Apache Superset (0.5.0)
-
connects to Druid as a backend
-
Supports LDAP authentication
-
-
Stackable Operator for Trino (0.4.0)
-
Supports OPA and file-based authorization
-
Connects to the Hive Metastore
-
Query data from S3
-
TLS support
-
-
Stackable Operator for Apache ZooKeeper (0.10.0)
-
Supports creating ZNodes with CRDs
-
Read up on the supported versions for each of these products.
-
Stackable Operator for OPA (OpenPolicyAgent) (0.9.0)
-
Create RegoRules in ConfigMaps
-
-
Stackable Commons Operator (0.2.0)
-
Stackable Secret Operator (0.5.0)