[Программирование, Java, Apache, Промышленное программирование] Spring Boot app with Apache Kafka in Docker container (перевод)
Автор
Сообщение
news_bot ®
Стаж: 6 лет 9 месяцев
Сообщений: 27286
Privet, comrads!In this article i’ll show how easy it is to setup Spring Java app with Kafka message brocker. We will use docker containers for kafka zookeeper/brocker apps and configure plaintext authorization for access from both local and external net.Final project link to github can be picked up at the end of the article
by Esperanza GatesApache Kafka + Zookeeper docker image selectionFirst, you have to decide on the vendor of the Apache Kafka image for container. The requirements of each specific project differ in the level of security and reliability of the solution, in some cases, of course, you will have to build your own image, but for most projects it will be reasonable to choose one from docker hub. For 2020, there are three main popular image vendors kafka + zookeper:
- images from confluent [subjectively, a big plus — they are supplied by the developers of Apache Kafka itself and are the most reliable in terms of information security]
- images from bitnami [in comparison with confluent, the images need minimal configuration and start out of the box]
- images from wurstmeister [in this example I will use them because the project has the most active community on github and hub.docker, as well as excellent documentation]
Installing Docker-composeDocker-compose is a tool to run and configure multi-container applications . It is a greate choice for Kafka setup because the minimum kafka configuration consist of zookeeper and at least one broker.
Docker-compose comes bundled with Docker app for mac and windows, so you only need to to download and install the Docker app from https://docs.docker.com/engine/install/.
Linux users required to install it compose and docker as separate tools. For a guide to installing docker + compose, it’s best to refer to the official documentation https://docs.docker.com/compose/install/.Writing a yaml file for composeThe general scheme of our cluster with clients and containers will be as follows:
One broker will connect to one coordinator with open ports 9093 for localhost connections and 9092 for external connections.For access from external networks, authorization is enabled via login password without ssl. To work through ssl, you need to issue your own keys and sign certificates, the whole process is well described in the official documentation.As a result, the file will look like this (the <your public IP here> address needs to be replaced with your public IP to be able to access from other networks):Извините, данный ресурс не поддреживается. :( Client authorizationIn order to enable authorization, you need to transfer the kafkaserverjaas.conf file to kafka. In the folder with the compose yaml file, create kafkaserverjaas.conf and add one admin user with the admin-secret password:Извините, данный ресурс не поддреживается. :( This file will be transferred to the image with the kafka broker via:
- specifying the path to the file in the kafka config directlyKAFKA_OPTS: "-Djava.security.auth.login.config=/etc/kafka/kafka_server_jaas.conf"
- mapping local directory containig file to /etc/kafka image directory volumes: - ./:/etc/kafka
Launching containersAs a result we should get 2 files, which are located in the same directory:
- docker-compose.yml
- kafka_server_jaas.conf
In that directory call:$ docker-compose up -dThe -d flag allows you to start in detached mode and close the console if necessary without turning off the containers.Spring Boot Java clientLet’s write a minimalistic client to publish/consume messages from kafka. To do this, we will create a new project through Spring Initializr or any other tool you are used to. It will be enough to connect one dependency:
I prefer to configure Spring Boot applications via yaml config files with minimal Java configurations. Due to the built-in structuring in Yaml-format, these files are much more human-readable than .properties files and allow you to group settings by modules. The only drawback of the .yml configuration is that it is not supported in tests, but it is easily bypassed by the built-in Spring Boot tools.
I highly recommend to try and check how your perception of the old .properties file changes (especially for large projects) if you rewrite it in yaml. This can be done in a couple of clicks at https://www.toyaml.com/index.html.
In project resources create an application.yml file with the following content:Извините, данный ресурс не поддреживается. :( This setup works for sequentially reading one message at a time (max-poll-records: 1) by one listener (concurrency: 1) and requires manual confirmation of message processing in listener (ack-mode: manual_immediate).Add producer and consumer classesИзвините, данный ресурс не поддреживается. :( The listener receives the message and logs all available information about it to the console.Извините, данный ресурс не поддреживается. :( Let’s start adding messages to the queue every 3 seconds
to enable the scheduler, we need to add the @EnableScheduling annotation to our spring application
Извините, данный ресурс не поддреживается. :( Voilà! the application is ready to run. You can run the dev-option with a local client and local kafka, and you can also access kafka from other networks if you open port 9092 and specify the host address in application.yml.The finished project available in the repository - https://github.com/layonez/kafka-example
===========
Источник:
habr.com
===========
===========
Автор оригинала: @Layonez
===========Похожие новости:
- [Java] Собеседование Backend-Java-разработчика: вопросы и где искать ответы. Часть 2
- [Java] Собеседование Backend-Java-разработчика: вопросы и где искать ответы. Часть 1
- [CMS, PHP, Программирование, Разработка под e-commerce, Интернет-маркетинг] Генератор ocmod-файла для интернет-магазина на Opencart
- [Программирование, Apache, Hadoop] Экономичная конфигурация исполнителей Apache Spark (перевод)
- [Java] Ласточка в мире микросервисов
- [Python, Программирование] Functools – Инструменты для работы с функциями (перевод)
- [Open source, Программирование, Системное программирование, Компиляторы, Rust] Rust 1.48.0: упрощение создания ссылок и псевдонимы поиска (перевод)
- [Разработка веб-сайтов, JavaScript, Angular, ReactJS, TypeScript] Schedulers в RxJS
- [Разработка веб-сайтов, Программирование, Разработка под e-commerce, DevOps] «Kubernetes для разработчиков»: трехдневный интенсив
- [JavaScript] Делаем симулятор двухпозиционного регулятора на JavaScript
Теги для поиска: #_programmirovanie (Программирование), #_java, #_apache, #_promyshlennoe_programmirovanie (Промышленное программирование), #_java, #_kafka, #_docker, #_springboot, #_dockercompose, #_programmirovanie (
Программирование
), #_java, #_apache, #_promyshlennoe_programmirovanie (
Промышленное программирование
)
Вы не можете начинать темы
Вы не можете отвечать на сообщения
Вы не можете редактировать свои сообщения
Вы не можете удалять свои сообщения
Вы не можете голосовать в опросах
Вы не можете прикреплять файлы к сообщениям
Вы не можете скачивать файлы
Текущее время: 22-Ноя 08:31
Часовой пояс: UTC + 5
| Автор | Сообщение |
|---|---|
|
news_bot ®
Стаж: 6 лет 9 месяцев |
|
|
Privet, comrads!In this article i’ll show how easy it is to setup Spring Java app with Kafka message brocker. We will use docker containers for kafka zookeeper/brocker apps and configure plaintext authorization for access from both local and external net.Final project link to github can be picked up at the end of the article  by Esperanza GatesApache Kafka + Zookeeper docker image selectionFirst, you have to decide on the vendor of the Apache Kafka image for container. The requirements of each specific project differ in the level of security and reliability of the solution, in some cases, of course, you will have to build your own image, but for most projects it will be reasonable to choose one from docker hub. For 2020, there are three main popular image vendors kafka + zookeper:
Docker-compose comes bundled with Docker app for mac and windows, so you only need to to download and install the Docker app from https://docs.docker.com/engine/install/. Linux users required to install it compose and docker as separate tools. For a guide to installing docker + compose, it’s best to refer to the official documentation https://docs.docker.com/compose/install/.Writing a yaml file for composeThe general scheme of our cluster with clients and containers will be as follows:  One broker will connect to one coordinator with open ports 9093 for localhost connections and 9092 for external connections.For access from external networks, authorization is enabled via login password without ssl. To work through ssl, you need to issue your own keys and sign certificates, the whole process is well described in the official documentation.As a result, the file will look like this (the <your public IP here> address needs to be replaced with your public IP to be able to access from other networks):Извините, данный ресурс не поддреживается. :( Client authorizationIn order to enable authorization, you need to transfer the kafkaserverjaas.conf file to kafka. In the folder with the compose yaml file, create kafkaserverjaas.conf and add one admin user with the admin-secret password:Извините, данный ресурс не поддреживается. :( This file will be transferred to the image with the kafka broker via:
 I prefer to configure Spring Boot applications via yaml config files with minimal Java configurations. Due to the built-in structuring in Yaml-format, these files are much more human-readable than .properties files and allow you to group settings by modules. The only drawback of the .yml configuration is that it is not supported in tests, but it is easily bypassed by the built-in Spring Boot tools. I highly recommend to try and check how your perception of the old .properties file changes (especially for large projects) if you rewrite it in yaml. This can be done in a couple of clicks at https://www.toyaml.com/index.html.
to enable the scheduler, we need to add the @EnableScheduling annotation to our spring application
=========== Источник: habr.com =========== =========== Автор оригинала: @Layonez ===========Похожие новости:
Программирование ), #_java, #_apache, #_promyshlennoe_programmirovanie ( Промышленное программирование ) |
|
Вы не можете начинать темы
Вы не можете отвечать на сообщения
Вы не можете редактировать свои сообщения
Вы не можете удалять свои сообщения
Вы не можете голосовать в опросах
Вы не можете прикреплять файлы к сообщениям
Вы не можете скачивать файлы
Вы не можете отвечать на сообщения
Вы не можете редактировать свои сообщения
Вы не можете удалять свои сообщения
Вы не можете голосовать в опросах
Вы не можете прикреплять файлы к сообщениям
Вы не можете скачивать файлы
Текущее время: 22-Ноя 08:31
Часовой пояс: UTC + 5