@mdf.js/kafka-provider

Mytra Development Framework - @mdf.js/kafka-provider

Typescript tools for development

Table of contents

• @mdf.js/kafka-provider
  • Table of contents
  • Introduction
  • Installation
  • Information
  • Use
  • Environment variables
  • License

Introduction

Kafka provider for @mdf.js based on kafkajs.

Installation

Using npm:

npm install @mdf.js/kafka-provider
Copy

Using yarn:

yarn add @mdf.js/kafka-provider
Copy

Information

Check information about @mdf.js providers in the documentation of the core module @mdf.js/core.

Use

Checks included in the provider:

• status: Checks the status of the kafka nodes using the admin client of the KafkaJS library, performing several requests about the status of the nodes and groups.
  • observedValue: actual state of the consumer/producer provider instance [error, running, stopped] based on the response, or not, to admin client requests. error if there is errors during the requests, running if the requests are successful, and stopped if the instance has been stopped or not initialized.
  • status: pass if the status is running, warn if the status is stopped, fail if the status is error.
  • output: Shows the error message in case of error state (status fail).
• topics: Checks the topics available in the Kafka connection
  • observedValue: List of topics available in the Kafka connection.
  • observedUnit: topics.
  • status: pass if the topics are available, fail in other cases.
  • output: No topics available if the topics are not available.

Environment variables

• CONFIG_KAFKA_PRODUCER__METADATA_MAX_AGE (default: 300000): Maximum time in ms that the producer will wait for metadata
• CONFIG_KAFKA_PRODUCER__ALLOW_AUTO_TOPIC_CREATION (default: true): Allow auto topic creation
• CONFIG_KAFKA_PRODUCER__TRANSACTION_TIMEOUT (default: 60000): Transaction timeout in ms
• CONFIG_KAFKA_PRODUCER__IDEMPOTENT (default: false): Idempotent producer
• CONFIG_KAFKA_PRODUCER__TRANSACTIONAL_ID (default: undefined): Transactional id
• CONFIG_KAFKA_PRODUCER__MAX_IN_FLIGHT_REQUEST (default: undefined): Maximum number of in-flight requests
• CONFIG_KAFKA_PRODUCER__RETRY__MAX_RETRY_TIME (default: 300000): Maximum time in ms that the producer will wait for metadata
• CONFIG_KAFKA_PRODUCER__RETRY__INITIAL_RETRY_TIME (default: 300): Initial value used to calculate the retry in milliseconds (This is still randomized following the randomization factor)
• CONFIG_KAFKA_PRODUCER__RETRY__FACTOR (default: 0.2): A multiplier to apply to the retry time
• CONFIG_KAFKA_PRODUCER__RETRY__MULTIPLIER (default: 2): A multiplier to apply to the retry time
• CONFIG_KAFKA_PRODUCER__RETRY__RETRIES (default: 5): Maximum number of retries per call
• CONFIG_KAFKA_CONSUMER__GROUP_ID (default: 'hostname()'): Consumer group id
• CONFIG_KAFKA_CONSUMER__SESSION_TIMEOUT (default: 30000): The timeout used to detect consumer failures when using Kafka's group management facility. The consumer sends periodic heartbeats to indicate its liveness to the broker. If no heartbeats are received by the broker before the expiration of this session timeout, then the broker will remove this consumer from the group and initiate a rebalance.
• CONFIG_KAFKA_CONSUMER__REBALANCE_TIMEOUT (default: 60000): The maximum time that the coordinator will wait for each member to rejoin when rebalancing the group.
• CONFIG_KAFKA_CONSUMER__HEARTBEAT_INTERVAL (default: 3000): The expected time between heartbeats to the consumer coordinator when using Kafka's group management facility. Heartbeats are used to ensure that the consumer's session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than sessionTimeout, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances.
• CONFIG_KAFKA_CONSUMER__METADATA_MAX_AGE (default: 300000): The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions.
• CONFIG_KAFKA_CONSUMER__ALLOW_AUTO_TOPIC_CREATION (default: true): Allow automatic topic creation on the broker when subscribing to or assigning non-existing topics.
• CONFIG_KAFKA_CONSUMER__MAX_BYTES_PER_PARTITION (default: 1048576): The maximum amount of data per-partition the server will return.
• CONFIG_KAFKA_CONSUMER__MIN_BYTES (default: 1): Minimum amount of data the server should return for a fetch request. If insufficient data is available the request will wait until some is available.
• CONFIG_KAFKA_CONSUMER__MAX_BYTES (default: 10485760): The maximum amount of data the server should return for a fetch request.
• CONFIG_KAFKA_CONSUMER_MAX_WAIT_TIME_IN_MS (default: 5000): The maximum amount of time the server will block before answering the fetch request if there isn't sufficient data to immediately satisfy minBytes.
• CONFIG_KAFKA_CONSUMER__RETRY__MAX_RETRY_TIME (default: 30000): Maximum time in milliseconds to wait for a successful retry
• CONFIG_KAFKA_CONSUMER__RETRY__INITIAL_RETRY_TIME (default: 300): Initial value used to calculate the retry in milliseconds (This is still randomized following the randomization factor)
• CONFIG_KAFKA_CONSUMER__RETRY__FACTOR (default: 0.2): A multiplier to apply to the retry time
• CONFIG_KAFKA_CONSUMER__RETRY__MULTIPLIER (default: 2): A multiplier to apply to the retry time
• CONFIG_KAFKA_CONSUMER__RETRY__RETRIES (default: 5): Maximum number of retries per call
• CONFIG_KAFKA_CONSUMER__READ_UNCOMMITTED (default: false): Whether to read uncommitted messages
• CONFIG_KAFKA_CONSUMER__MAX_IN_FLIGHT_REQUEST (default: undefined): Maximum number of in-flight requests
• CONFIG_KAFKA_CONSUMER__RACK_ID (default: undefined): The consumer will only be assigned partitions from the leader of the partition to which it is assigned.
• CONFIG_KAFKA_LOG_LEVEL (default: `error`): Define the log level for the kafka provider, possible values are: - error - warn - info - debug - trace
• CONFIG_KAFKA_CLIENT__CLIENT_ID (default: hostname): Client identifier
• CONFIG_KAFKA_CLIENT__BROKERS (default: '127.0.0.1:9092'): Kafka brokers
• CONFIG_KAFKA_CLIENT__CONNECTION_TIMEOUT (default: 1000): Time in milliseconds to wait for a successful connection
• CONFIG_KAFKA_CLIENT__AUTHENTICATION_TIMEOUT (default: 1000): Timeout in ms for authentication requests
• CONFIG_KAFKA_CLIENT__REAUTHENTICATION_THRESHOLD (default: 1000): When periodic reauthentication (connections.max.reauth.ms) is configured on the broker side, reauthenticate when reauthenticationThreshold milliseconds remain of session lifetime.
• CONFIG_KAFKA_CLIENT__REQUEST_TIMEOUT (default: 30000): Time in milliseconds to wait for a successful request
• CONFIG_KAFKA_CLIENT__ENFORCE_REQUEST_TIMEOUT (default: true): The request timeout can be disabled by setting this value to false.
• CONFIG_KAFKA_MAX_RETRY_TIME (default: 30000): Maximum time in milliseconds to wait for a successful retry
• CONFIG_KAFKA_INITIAL_RETRY_TIME (default: 300): Initial value used to calculate the retry in milliseconds (This is still randomized following the randomization factor)
• CONFIG_KAFKA_RETRY_FACTOR (default: 0.2): Randomization factor
• CONFIG_KAFKA_RETRY_MULTIPLIER (default: 2): Exponential factor
• CONFIG_KAFKA_RETRIES (default: 5): Maximum number of retries per call
• CONFIG_KAFKA_CLIENT_SSL_ENABLED (default: false): Whether to use SSL
• CONFIG_KAFKA_CLIENT__SSL__REJECT_UNAUTHORIZED (default: true): Whether to verify the SSL certificate.
• CONFIG_KAFKA_CLIENT__SSL__SERVER_NAME (default: undefined): Server name for the TLS certificate.
• CONFIG_KAFKA_CLIENT_SSL_CA_PATH (default: undefined): Path to the CA certificate.
• CONFIG_KAFKA_CLIENT_SSL_CERT_PATH (default: undefined): Path to the client certificate.
• CONFIG_KAFKA_CLIENT_SSL_KEY_PATH (default: undefined): Path to the client key.
• CONFIG_KAFKA_CLIENT__SASL_USERNAME (default: undefined): SASL username
• CONFIG_KAFKA_CLIENT__SASL_PASSWORD (default: undefined): SASL password
• NODE_APP_INSTANCE (default: undefined): Used as default container id, receiver name, sender name, etc. in cluster configurations.

License

Copyright 2024 Mytra Control S.L. All rights reserved.

Use of this source code is governed by an MIT-style license that can be found in the LICENSE file or at https://opensource.org/licenses/MIT.