RabbitMQ is a popular message broker that facilitates the exchange of data between applications. However, as with any system, it’s important to have visibility into the logs generated by RabbitMQ to identify issues and ensure smooth operation. In this blog post, we’ll walk you through the process of shipping RabbitMQ logs to Elasticsearch, a distributed search and analytics engine. By centralising and analysing RabbitMQ logs with Elasticsearch, you can gain valuable insights into your system and easily troubleshoot any issues that arise.

Logs processing system architecture

To build that architecture, we’re going to set up 4 components in our system. Each one of them has got its own set of features. Here there are:

• A logs Publisher
• A RabbitMQ Server With a Queue To Publish data to and receive data from.
• A Logstash Pipeline To Process Data From The RabbitMQ Queue.
• An Elasticsearch Index To Store The Processed Logs.

Installation

1. Logs Publisher

Logs can come from any software. It can be from a web server (Apache, Nginx), a monitoring system, an operating system, a web or mobile application, and so on. The logs give information about the working history of any software.

If don’t have any choices yet, you can use my simple stuff here: https://github.com/baoanh194/rabbitmq-simple-publisher-consumer

2. RabbitMQ

The logs publisher will be publishing the logs to a RabbitMQ queue.

Instead of going through a very long RabbitMQ installation, we’re going to go with a RabbitMQ Docker instance to make things simple. You can find your preferred operating system here: https://docs.docker.com/engine/install/

To start a RabbitMQ container. You can do this by running the following command:

This command starts a RabbitMQ container with the management plugin enabled. After enabling the plugin, you can access the RabbitMQ management console by going to http://localhost:15672/ in your web browser. Normally the username/password is guest/guest .

3. Elasticsearch

Go and check this link to install and configure Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html

To store RabbitMQ data for visualisation in Kibana, you need to start an Elasticsearch container. You can do this by running the following command (I’m using Docker to set up Elasticsearch):

When you start Elasticsearch for the first time, there are some security configuration required.

4. Logstash

If you haven’t installed or worked with Logstash before, don’t worry. Have a look at the Elastic docs: https://www.elastic.co/guide/en/logstash/current/installing-logstash.html

It’s very detailed and easy to read.

For me, I installed Logstash on MacOS by Homebrew:

Once Logstash is installed on your machine, let’s create the Pipeline to process data.

Paste the code below to your pipelines.conf file:

(Put new config file under: /opt/homebrew/etc/logstash)

Run your pipeline with Logstash:

Here is a screenshot of what you should get if your RabbitMQ Docker Instance is running well and everything works pretty well on your Logstash pipeline side:

Let’s ship some logs

Now everything is ready. Go to the logs publisher root folder and run the send.js script

You can check  the data is sent to Elastic:

curl -k -u elastic https://localhost:9200/_search?pretty

If everything goes well, you will get the result as below screenshot:

Configure Kibana to Visualize RabbitMQ Data

Additionally, you can configure Kibana to visualize the RabbitMQ data on Elastic. By configuring Kibana, you can create visualisations such as charts, graphs, and tables that make it easy to understand the data and identify trends or anomalies. For example, you could create a chart that shows the number of messages processed by RabbitMQ over time, or a table that shows the top senders and receivers of messages.

Kibana also allows you to build dashboards, which are collections of visualisations and other user interface elements arranged on a single screen. Dashboards can be shared with others in your organization, making it easier for team members to collaborate and troubleshoot issues. You can refer to this link for how to set up Kibana: https://www.elastic.co/pdf/introduction-to-logging-with-the-elk-stack

Conclusion

In summary, shipping RabbitMQ logs to Elasticsearch offers benefits such as centralized log storage, quick search and analysis, and improved system troubleshooting. By following the steps outlined in this blog post, you can set up a system to handle large volumes of logs and gain real-time insights into your messaging system. Whether you’re running a small or large RabbitMQ instance, shipping logs to Elasticsearch can help you optimise and scale your system.

The post Ship RabbitMQ logs to Elasticsearch appeared first on Erlang Solutions .

We are very proud to announce the release of PGPainless-WOT , an implementation of the OpenPGP Web of Trust specification using PGPainless.

The release is available on the Maven Central repository .

The work on this project begun a bit over a year ago as an NLnet project which received funding through the European Commission’s NGI Assure program . Unfortunately, somewhere along the way I lost motivation to work on the project, as I failed to see any concrete users. Other projects seemed more exciting at the time.

Fast forward to end of May when Wiktor reached out and connected me with Heiko, who was interested in the project. We two decided to work together on the project and I quickly rebased my – at this point ancient and outdated – feature branch onto the latest PGPainless release. At the end of June, we started the joint work and roughly a month later today, we can release a first version

Big thanks to Heiko for his valuable contributions and the great boost in motivation working together gave me
Also big thanks to NLnet for sponsoring this project in such a flexible way.
Lastly, thanks to Wiktor for his talent to connect people

The Implementation

We decided to write the implementation in Kotlin. I had attempted to learn Kotlin multiple times before, but had quickly given up each time without an actual project to work on. This time I stayed persistent and now I’m a convinced Kotlin fan Rewriting the existing codebase was a breeze and the line count drastically reduced while the amount of syntactic sugar which was suddenly available blow me away! Now I’m considering to steadily port PGPainless to Kotlin. But back to the Web-of-Trust.

Our implementation is split into 4 modules:

• pgpainless-wot parses OpenPGP certificates into a generalized form and builds a flow network by verifying third-party signatures. It also provides a plugin for pgpainless-core .
• wot-dijkstra implements a query algorithm that finds paths on a network. This module has no OpenPGP dependencies whatsoever, so it could also be used for other protocols with similar requirements.
• pgpainless-wot-cli provides a CLI frontend for pgpainless-wot
• wot-test-suite contains test vectors from Sequoia PGP’s WoT implementation

The code in pgpainless-wot can either be used standalone via a neat little API, or it can be used as a plugin for pgpainless-core to enhance the encryption / verification API:

/* Standalone */
Network network = PGPNetworkParser(store).buildNetwork();
WebOfTrustAPI api = new WebOfTrustAPI(network, trustRoots, false, false, 120, refTime);

// Authenticate a binding
assertTrue(
    api.authenticate(fingerprint, userId, isEmail).isAcceptable());

// Identify users of a certificate via the fingerprint
assertEquals(
    "Alice <alice@example.org>",
    api.identify(fingerprint).get(0).getUserId());

// Lookup certificates of users via userId
LookupAPI.Result result = api.lookup(
    "Alice <alice@example.org>", isEmail);

// Identify all authentic bindings (all trustworthy certificates)
ListAPI.Result result = api.list();


/* Or enhancing the PGPainless API */
CertificateAuthorityImpl wot = CertificateAuthorityImpl
    .webOfTrustFromCertificateStore(store, trustRoots, refTime)

// Encryption
EncryptionStream encStream = PGPainless.encryptAndOrSign()
    [...]
    // Add only recipients we can authenticate
    .addAuthenticatableRecipients(userId, isEmail, wot)
    [...]

// Verification
DecryptionStream decStream = [...]
[...]  // finish decryption
MessageMetadata metadata = decStream.getMetadata();
assertTrue(metadata.isAuthenticatablySignedBy(userId, isEmail, wot));

The CLI application pgpainless-wot-cli mimics Sequoia PGP’s neat sq-wot tool, both in argument signature and output format. This has been done in an attempt to enable testing of both applications using the same test suite.

pgpainless-wot-cli can read GnuPGs keyring, can fetch certificates from the Shared OpenPGP Certificate Directory (using pgpainless-cert-d of course :P) and ingest arbitrary .pgp keyring files.

$ ./pgpainless-wot-cli help     
Usage: pgpainless-wot [--certification-network] [--gossip] [--gpg-ownertrust]
                      [--time=TIMESTAMP] [--known-notation=NOTATION NAME]...
                      [-r=FINGERPRINT]... [-a=AMOUNT | --partial | --full |
                      --double] (-k=FILE [-k=FILE]... | --cert-d[=PATH] |
                      --gpg) [COMMAND]
  -a, --trust-amount=AMOUNT
                         The required amount of trust.
      --cert-d[=PATH]    Specify a pgp-cert-d base directory. Leave empty to
                           fallback to the default pgp-cert-d location.
      --certification-network
                         Treat the web of trust as a certification network
                           instead of an authentication network.
      --double           Equivalent to -a 240.
      --full             Equivalent to -a 120.
      --gossip           Find arbitrary paths by treating all certificates as
                           trust-roots with zero trust.
      --gpg              Read trust roots and keyring from GnuPG.
      --gpg-ownertrust   Read trust-roots from GnuPGs ownertrust.
  -k, --keyring=FILE     Specify a keyring file.
      --known-notation=NOTATION NAME
                         Add a notation to the list of known notations.
      --partial          Equivalent to -a 40.
  -r, --trust-root=FINGERPRINT
                         One or more certificates to use as trust-roots.
      --time=TIMESTAMP   Reference time.
Commands:
  authenticate  Authenticate the binding between a certificate and user ID.
  identify      Identify a certificate via its fingerprint by determining the
                  authenticity of its user IDs.
  list          Find all bindings that can be authenticated for all
                  certificates.
  lookup        Lookup authentic certificates by finding bindings for a given
                  user ID.
  path          Verify and lint a path.
  help          Displays help information about the specified command

The README file of the pgpainless-wot-cli module contains instructions on how to build the executable.

Future Improvements

The current implementation still has potential for improvements and optimizations. For one, the Network object containing the result of many costly signature verifications is currently ephemeral and cannot be cached. In the future it would be desirable to change the network parsing code to be agnostic of reference time, including any verifiable signatures as edges of the network, even if those signatures are not yet – or no longer valid. This would allow us to implement some caching logic that could write out the network to disk, ready for future web of trust operations.

That way, the network would only need to be re-created whenever the underlying certificate store is updated with new or changed certificates (which could also be optimized to only update relevant parts of the network). The query algorithm would need to filter out any inactive edges with each query, depending on the queries reference time. This would be far more efficient than re-creating the network with each application start.

But why the Web of Trust?

End-to-end encryption suffers from one major challenge: When sending a message to another user, how do you know that you are using the correct key? How can you prevent an active attacker from handing you fake recipient keys, impersonating your peer? Such a scenario is called Machine-in-the-Middle (MitM) attack .

On the web, the most common countermeasure against MitM attacks are certificate authorities, which certify the TLS certificates of website owners, requiring them to first prove their identity to some extent. Let’s Encrypt for example first verifies, that you control the machine that serves a domain before issuing a certificate for it. Browsers trust Let’s Encrypt, so users can now authenticate your website by validating the certificate chain from the Let’s Encrypt CA key down to your website’s certificate.

The Web-of-Trust follows a similar model, with the difference, that you are your own trust-root and decide, which CA’s you want to trust (which in some sense makes you your own “meta-CA”). The Web-of-Trust is therefore far more decentralized than the fixed set of TLS trust-roots baked into web browsers. You can use your own key to issue trust signatures on keys of contacts that you know are authentic. For example, you might have met Bob in person and he handed you a business card containing his key’s fingerprint. Or you helped a friend set up their encrypted communications and in the process you two exchanged fingerprints manually.

In all these cases, in order to initiate a secure communication channel, you needed to exchange the fingerprint via an out-of-band channel. The real magic only happens, once you take into consideration that your close contacts could also do the same for their close contacts, which makes them CAs too. This way, you could authenticate Charlie via your friend Bob, of whom you know that he is trustworthy, because – come on, it’s Bob! Everybody loves Bob!

An example for an OpenPGP Web-of-Trust. Simply by delegating trust to the Neutron Mail CA and to Vincenzo, Aaron is able to authenticate a number of certificates.

The Web-of-Trust becomes really useful if you work with people that share the same goal. Your workplace might be one of them, your favorite Linux distribution’s maintainer team, or that non-Profit organization/activist collective that is fighting for a better tomorrow. At work for example, your employer’s IT department might use a local CA (such as an instance of the OpenPGP CA ) to help employees to communicate safely. You trust your workplace’s CA, which then introduces you safely to your colleagues’ authentic key material. It even works across business’ boundaries, e.g. if your workplace has a cooperation with ACME and you need to establish a safe communication channel to an ACME employee. In this scenario, your company’s CA might delegate to the ACME CA, allowing you to authenticate ACME employees.

As you can see, the Web-of-Trust becomes more useful the more people are using it. Providing accessible tooling is therefore essential to improve the overall ecosystem. In the future, I hope that OpenPGP clients such as MUAs (e.g. Thunderbird) will embrace the Web-of-Trust.

The Ignite Realtime community is happy to announce a new release of the Jabber Browsing plugin for Openfire.

This is a plugin for the Openfire Real-time Communications server. It provides an implementation for service discovery using the jabber:iq:browse namespace, as specified in XEP-0011: Jabber Browsing . Note that this feature is considered obsolete! The plugin should only be used by people that seek backwards compatibility with very old and very specific IM clients.

This release is a maintenance release. It adds translations and fixes one bug. More details are available in the changelog .

Your instance of Openfire should automatically display the availability of the update in the next few hours. Alternatively, you can download the new release of the plugin at the Jabber Browsing plugin archive page .

If you have any questions, please stop by our community forum or our live groupchat .

For other release announcements and news follow us on Twitter and Mastodon .

1 post - 1 participant

Read full topic

The Ignite Realtime community is happy to announce a new release of the Agent Information plugin for Openfire.

This plugin implements the XEP-0094 ‘Agent Information’ specification for service discovery using the jabber:iq:agents namespace. Note that this feature is considered obsolete! The plugin should only be used by people that seek backwards compatibility with very old and very specific IM clients.

This release is a maintenance release. It adds translations and fixes one bug. More details are available in the changelog .

Your instance of Openfire should automatically display the availability of the update in the next few hours. Alternatively, you can download the new release of the plugin at the JmxWeb plugin archive page .

If you have any questions, please stop by our community forum or our live groupchat .

For other release announcements and news follow us on Twitter and Mastodon .

1 post - 1 participant

Read full topic

The Ignite Realtime community is happy to announce a new release of the Certificate Manager plugin for Openfire.

This plugin allows you to automate TLS certificate management tasks. This is particularly helpful when your certificates are short-lived, like the ones issued by Let’s Encrypt.

This release is a maintenance release. It adds translations. More details are available in the changelog .

Your instance of Openfire should automatically display the availability of the update in the next few hours. Alternatively, you can download the new release of the plugin at the Certificate Manager plugin archive page .

If you have any questions, please stop by our community forum or our live groupchat .

For other release announcements and news follow us on Twitter and Mastodon .

1 post - 1 participant

Read full topic

The Ignite Realtime community is happy to announce a new release of the JmxWeb plugin for Openfire.

This plugin provides a web based platform for managing and monitoring Openfire via JMX

This release is a maintenance release. It adds translations and fixes one bug. More details are available in the changelog .

Your instance of Openfire should automatically display the availability of the update in the next few hours. Alternatively, you can download the new release of the plugin at the JmxWeb plugin archive page .

If you have any questions, please stop by our community forum or our live groupchat .

For other release announcements and news follow us on Twitter and Mastodon .

1 post - 1 participant

Read full topic

The Ignite Realtime community is happy to announce a new release of the Push Notification plugin for Openfire.

This plugin enables clients to register for push notifications.

This release is a maintenance release. It adds translations and a configuration page. More details are available in the changelog

Your instance of Openfire should automatically display the availability of the update in the next few hours. Alternatively, you can download the new release of the plugin at the Push Notification plugin archive page .

If you have any questions, please stop by our community forum or our live groupchat .

For other release announcements and news follow us on Twitter and Mastodon .

1 post - 1 participant

Read full topic

The Ignite Realtime community is happy to announce a new release of the Search plugin for Openfire.

This plugin adds features to Openfire that makes it easier for users to find each-other.

This release is a maintenance release. It adds translations. More details are available in the changelog

Your instance of Openfire should automatically display the availability of the update in the next few hours. Alternatively, you can download the new release of the plugin at the Search plugin archive page .

If you have any questions, please stop by our community forum or our live groupchat .

For other release announcements and news follow us on Twitter and Mastodon .

1 post - 1 participant

Read full topic

The Ignite Realtime community is happy to announce a new release of the Openfire plugin for Candy.

Candy is a third-party chat client . The Openfire plugin makes deploying it a one-click affair!

This release is a maintenance release. It adds translations and updates dependencies on third-party libraries. More details are available in the changelog .

Your instance of Openfire should automatically display the availability of the update in the next few hours. Alternatively, you can download the new release of the plugin at the candy plugin archive page .

If you have any questions, please stop by our community forum or our live groupchat .

For other release announcements and news follow us on Twitter and Mastodon .

1 post - 1 participant

Read full topic