The execution of VaToolBxSendToAvroKafka and VaToolBxSendToAvroKafkaWithKey functions is based on a configuration file in YML format defining the configuration of each Adelia logical broker manipulated in the 4GL source by the two VaToolBxSendToAvroKafka and VaToolBxSendToAvroKafkaWithKey functions.
Example of 4GL:
CALL_DLL 'VaToolBx.dll' 'VaToolBxSendToAvroKafka' 'myKafkaLogicalBroker' myTopic myInstance myReturnCode myReturnMessage
CALL_DLL 'VaToolBx.dll' 'VaToolBxSendToAvroKafkaWithKey' 'myKafkaLogicalBroker' myTopic myInstance myKey myReturnCode myReturnMessage
In this example, myKafkaLogicalBroker corresponds to a logical broker whose characteristics are defined in this configuration file.
This file is defined when starting up the JVM with the environment variable com.hardis.adelia.kafka.configuration
.. -Dcom.hardis.adelia.kafka.configuration=C:\directory\AdeliaKafkaConfig.yml
Example of configuration to start up Tomcat (by adapting catalina.bat):
Before:
%_EXECJAVA% %LOGGING_CONFIG% %LOGGING_MANAGER% %JAVA_OPTS% %CATALINA_OPTS% %DEBUG_OPTS% -D%ENDORSED_PROP%="%JAVA_ENDORSED_DIRS%" -classpath "%CLASSPATH%" -Dcatalina.base="%CATALINA_BASE%" -Dcatalina.home="%CATALINA_HOME%" -Djava.io.tmpdir="%CATALINA_TMPDIR%" %MAINCLASS% %CMD_LINE_ARGS% %ACTION%
After:
%_EXECJAVA% %LOGGING_CONFIG% %LOGGING_MANAGER% %JAVA_OPTS% %CATALINA_OPTS% %DEBUG_OPTS% -D%ENDORSED_PROP%="%JAVA_ENDORSED_DIRS%" -classpath "%CLASSPATH%" -Dcatalina.base="%CATALINA_BASE%" -Dcatalina.home="%CATALINA_HOME%" -Djava.io.tmpdir="%CATALINA_TMPDIR%" -Dcom.hardis.adelia.kafka.configuration="D:\MyConfig\AdeliaKafkaConfig.yml" %MAINCLASS% %CMD_LINE_ARGS% %ACTION%
This file is read during the first execution of:
LOAD_DLL 'VaToolBx'
or CALL_DLL 'VaToolBxSendToAvroKafka'
or CALL_DLL 'VaToolBxSendToAvroKafkaWithKey'
Example of configuration file:
#-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
#
# Adelia Kafka settings (logical brokers from Adelia VaToolBxSendToAvroKafka & VaToolBxSendToAvroKafkaWithKey)
# Example: CALL_DLL 'VaToolBx.dll' 'VaToolBxSendToAvroKafka' 'myKafkaLogicalBroker' Topic Instance ReturnCode ReturnMessage
#
# This file is defined when running jvm with -Dcom.hardis.adelia.kafka=C:\...\KafkaAdeliaConfig.yml
# Example: update catalina.bat to launch Tomcat with this property
#
# Properties names:
# Kafka properties : bootstrap.servers, etc. please refer https://kafka.apache.org/documentation/#producerconfigs
# Adelia properties : name starting with 'adelia.' for example 'adelia.sendmode'
# Adelia Kafka logical broker 'myKafkaLogicalBroker'
myKafkaLogicalBroker:
# bootstrap.servers => a list of Kafka brokers
bootstrap.servers: ps972.hardis.fr:9092
# schema.registry.url => schema registry url (for example http://localhost:8081
schema.registry.url:
# acks => number of acknowledgments the producer requires the leader to have received before considering a request complete (string value so "0" instead of 0).
acks: "0"
# retries => 0 so if the request fails then the producer does not retry
retries: 0
# max.block.ms => how long KafkaProducer.send() and KafkaProducer.partitionsFor() will block
max.block.ms: 123
# key.serializer => serializer class for key that implements the org.apache.kafka.common.serialization.Serializer interface.
key.serializer: org.apache.kafka.common.serialization.StringSerializer
# value.serializer => serializer class for value that implements the org.apache.kafka.common.serialization.Serializer interface.
value.serializer: org.apache.kafka.common.serialization.ByteArraySerializer
# producer.type => async
producer.type: async
# linger.ms => delay (default value 0) to define how the producer groups together any records that arrive in between request transmissions into a single batched request.
linger.ms: 0
# adelia.client.id => client.id: id to pass to the server when making requests so the server can track the source of requests beyond just IP/port by passing a producer name for things like server-side request logging ('auto' value for host name for IP address sender).
adelia.client.id: auto
# adelia.partition => partition (if a valid partition number is specified that partition will be used when sending the record. If no partition is specified but a key is present a partition will be chosen using a hash of the key. If neither key nor partition is present a partition will be assigned in a round-robin fashion).
# adelia.partition:
# adelia.sendmode => sending mode ('fireandforget' is default value)
# adelia.sendmode: fireandforget
# Adelia Kafka logical broker 'myKafkaLogicalBroker2'
myKafkaLogicalBroker2:
bootstrap.servers: ps123.hardis.fr:9092
..
# Adelia Kafka logical broker 'myKafkaLogicalBroker3'
myKafkaLogicalBroker3:
bootstrap.servers: ps456.hardis.fr:9092
#-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
In this example, this file defines the configuration of 3 logical brokers: myKafkaLogicalBroker, myKafkaLogicalBroker2 and myKafkaLogicalBroker3.
YML document structure:
nameAdeliaLogicalBroker1:
# this is a comment line in YML as it begins with a hash
propertyname1 : value1
propertyname2 : value2
propertyname3 : value3
..nameAdeliaLogicalBroker2:
this is a comment line in YML as it begins with a hash
propertyname1 : value1
propertyname8 : value8
propertyname11 : value11
....
Each property is either an:
- Apache Kafka Producer property:
all the properties are documented here.
For example: bootstrap.servers, acks, retries, etc.
- Special Adelia property (its name starts with adelia.)
For example:
adelia.partition
If a valid partition number (in the Kafka sense) is specified, this partition will be used when sending the record.
If no partition is specified but a key is present, a partition will be chosen using a hash of the key.
If neither key nor partition is present, a partition will be assigned is a round-robin fashion.
adelia.client.id
Mapped on the client.id property
Identifier to transmit to the server during requests so this can track the source of requests beyond the IP/port address by transmitting a producer name for tasks such as server-side request logging.
'auto' = sender's IP address