Kubernetes Environments

The added users and groups display in the User / Group table.

• Read - View the details of the environment. Create and edit a deployment for the environment.
• Write - Edit, activate, deactivate, and delete the environment.

For more information, see Environment Permissions.

kubectl cluster-info

kubectl cluster-info

kubectl apply -f ./<file_name>.yaml

kubectl apply -f ./streamsets-kubernetes-agent.yaml

- name: STREAMSETS_KUBERNETES_AGENT_JAVA_OPTS
  value: -Dhttps.proxyHost=<proxy host> -Dhttps.proxyPort=<port> -Dhttp.proxyHost=<proxy host> -Dhttp.proxyPort=<port> -Dhttp.nonProxyHosts=<pipe-separated no proxy hosts> -Dhttp.proxyUser=<proxy user> -Dhttp.proxyPassword=<password> -Dhttps.proxyUser=<proxy user> -Dhttps.proxyPassword=<password> -Djdk.http.auth.tunneling.disabledSchemes=

For the -Dhttp.nonProxyHosts option, include the ClusterIP of the Kubernetes API server and any other hosts that the agent can connect to without going through the proxy server. Separate each entry with the pipe character (|).

kubectl get svc/kubernetes --namespace=default

The properties are the same proxy properties that you configure for StreamSets engines. For a description of each property, see the Data Collector documentation or the Transformer documentation.

kubectl apply -f ./<file_name>.yaml

For a description of each proxy property, see the Data Collector documentation or the Transformer documentation.

For detailed instructions, see Starting Deployments.

For example, to set all messages to the DEBUG log level, add the following ConfigMap:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: custom-agent-log4j
data:
  custom_log4j_file_name: "custom-agent-log4j.properties"
  custom-agent-log4j.properties: |
    name = streamsets-dpm-kubernetes-agent
    appender.stdout.type = Console
    appender.stdout.name = stdout
    appender.stdout.layout.type = PatternLayout
    appender.stdout.layout.pattern = %d{ISO8601} [thread:%t] %-5p %c{1} - %m%n

    appender.streamsets.type = RollingFile
    appender.streamsets.name = streamsets
    appender.streamsets.fileName = ${sys:streamsets.kubernetes.agent.log.dir}/agent.log
    appender.streamsets.filePattern = ${sys:streamsets.kubernetes.agent.log.dir}/agent.%i.log
    appender.streamsets.layout.type = PatternLayout
    appender.streamsets.layout.pattern = %d{ISO8601} [thread:%t] %-5p %c{1} - %m%n
    appender.streamsets.append = true
    appender.streamsets.policies.type = Policies
    appender.streamsets.policies.size.type = SizeBasedTriggeringPolicy
    appender.streamsets.policies.size.size = 256MB
    appender.streamsets.strategy.type = DefaultRolloverStrategy
    appender.streamsets.strategy.max = 10

    rootLogger.level = DEBUG
    rootLogger.appenderRef.stdout.ref = stdout

    logger.l1.name = com.streamsets
    logger.l1.level = DEBUG
    logger.l1.appenderRef.streamsets.ref = streamsets

apiVersion: apps/v1
kind: Deployment
...
    spec:
      containers:
        - name: streamsets-agent
          image: streamsets/kubernetes-agent:1.1.2
          ...
          resources:
            requests:
              memory: 1Gi
              cpu: 0.5
          volumeMounts:
          - name: config
            mountPath: "/opt/streamsets-kubernetes-agent/etc/"
            readOnly: true
      dnsPolicy: Default
      serviceAccountName: streamsets-agent-sa-f00bb553-59d8-48c5-b681-5bec6eb4f430
volumes:
   - name: config
        configMap:
          name: custom-agent-log4j
          items:
          - key: "custom-agent-log4j.properties"
            path: "custom-agent-log4j.properties"

- name: STREAMSETS_KUBERNETES_AGENT_JAVA_OPTS
  value: -Dlog4j2.configurationFile=/opt/streamsets-kubernetes-agent/etc/custom-agent-log4j.properties

kubectl apply -f ./<file_name>.yaml