A new approach to the viewing of logs – InformTFB

A new approach to the viewing of logs

A new approach to the viewing of logs

At one time, I had to work a lot with logs. They could be large and located on different servers. It was not necessary to find something specific, but to understand why the system does not behave as it should. For some reason, there was no log aggregator.

I wanted to have a log viewer that allows you to open any file at any time, without downloading it to the local machine, like the less command in the linux console. But at the same time, there should be convenient text highlighting, as in the IDE, and filtering entries by various parameters. Filtering and searching should work by events in the log, and not by lines, like grep. this is important when there are multi-line entries, such as errors with stacktraces. It should also be possible to view entries from multiple files on the same page at once, by checking them by timestamp, even if the files are located on different nodes.

And I figured out how to make such a utility!

Log Viewer is a small web application that runs on nodes where logs are stored, and shows these logs through the Web interface. This is a viewer, not an aggregator. there is no indexing or reading the entire log into memory, only the part of the log that the user is currently viewing is read. This approach allows you to take up almost no resources.

I anticipate performance questions like ” Is it possible to quickly filter records without indexing? In bad cases, you will have to scan the entire log to find at least one entry that fits the filter.” First, scanning the log works quite quickly, 1 GB is read in about 3.5 seconds, this is tolerable. Secondly, we usually know the time interval in which we are looking for a problem. if a filter is set by date, then only the part of the file that contains records related to that time will be scanned. You can find the boundary of a time interval in a file very quickly by binary search.

Log display

To make it easier to distinguish between the borders of one record, the entry under the cursor is highlighted with a rectangle; the severity field is highlighted in different colors depending on the value, and paired brackets are highlighted when you hover the cursor over one of them.

Pay attention to the stack trace of the experiment, only the most interesting lines are shown, the rest are taken under ” + “and”…”. interesting lines are considered to be classes from packages belonging to the main application , their neighbors, and the first line. Packages of the main application are set in the configuration. In this form, the stacktrace takes up much less space on the screen and is more convenient to watch. Perhaps this idea will appeal to developers of the Java IDE.

The logger name is also abbreviated:”~. SecurityManager”. Only the class name is shown, and the package is collapsed to”~”.

Folding only affects the display, and the search works based on the original text. If a match is found in a shortened part of the text, this part of the text will automatically appear. Also, if the user selects text and presses Ctrl+C, the original text is copied to the buffer, without any abbreviations.

The architecture makes it easy to hang highlighting or tooltips on the text. thanks to this, various nice little things are made such as showing the date in human format, if it is printed as a number:

Filtering

The set of filters depends on the log format. Some filters are always available, such as a substring filter, and some appear if a certain type of field is present in the log. This allows you to create specialized filters for certain types of fields. For example, if the log contains the severity field, then the following UI component will appear in the top panel::Severity filter

It is very convenient to add filters from the context menu. You can select the text, right – click and select “Do not show entries with this text”. A filter by text is automatically added to the filters panel, which hides entries with this text. It helps when the log is filled up with monotonous entries that are not interesting at the moment.Adding filters from the context menu

You can click on an entry and choose ” Hide subsequent entries “or” Hide previous entries ” to work only with a specific part of the log. Hiding is done by adding a filter by date.

For complex cases, you can set a filter with a condition written in JavaScript. This filter is a function that accepts one record and returns true or false.JavaScript filter example

When changing filters, the viewer tries to preserve the position in the log as much as possible. If there is a selected entry, changing the filters will not change its position on the screen, and the entries around it will disappear or appear. The user can set a filter so that only errors are visible, find a suspicious error, then remove the filter and see what happened around this error.

The status of the filter panel is displayed in the URL parameters so that the current configuration can be bookmarked in the browser.

Small but useful features

When you find something interesting — you want to share it with the team. to do this, you can create a special link to the current position in the log, and anyone who opens it will see exactly the same page that was when the link was created, including the state of filters, text in the search field, the selected entry, etc.

If the server is located in a different timezone, a tooltip with the date in the user’s timezone will appear above the text with the date.

Configuration

I tried to make the configuration as simple as possible so that everything works out of the box. If you ask the user to set the log format, most people will just close the app and go to watch it the old-fashioned way. Therefore, the log format is recognized automatically. Of course, this doesn’t always work and often isn’t accurate. For such cases, you can set the log format manually in the configuration file. You can use the log4j, logback, or just regexp patterns. If your log is not recognized, but you think that it should be-create an issue on GitHub, this will help the project.

The most important setting is the list of visible files. By default, all files with the extension “.log” are available and the entire directory structure is visible, but this is not very good from the security point of view. In the configuration file, you can limit the visibility of files by using a list of patterns like this:

logs = [
  {
    path: "/opt/my-app/logs/*.log"
  },
  {
    path: ${HOME}"/work/**"
  }
]

Only .log files in the /opt/my-app/logs directory and any files in the ~/work directory and its subdirectories will be available to the user.

For more information, see the documentation on GitHub.

Working with multiple nodes

Merging files located on different nodes is a killer feature, for which the project was started. As I said before, the file is never fully downloaded from one node to another and is not indexed. Therefore, Log Viewer should be running on each node. The user opens the web UI on one of the nodes, specifies the location of logs, and Log Viewer connects to other LogViewer instances to load log content through them. Records from all open files are stored in a timestamp and are shown as if they are just one file.

I’ll briefly describe how it works under the hood. When the user opens the page, need to show the logs on each node send the request to “give the last N posts”, where N is the number of rows fit on the screen. The received records are sorted by timestamp, the last N records are taken and shown to the user. When the user scrolls up the page, a request is sent to all nodes “give the last N records with a timestamp less than T”, where T is the timestamp of the topmost record on the screen. The resulting records are sorted and added to the page. When scrolling down, the same thing happens, only in the other direction. Finding the position in the file where entries are older/younger than T is very fast, since the entries are sorted by timestamp and binary search can be used. There are many nuances, but the General scheme is as follows. Merge works only if the system was able to determine the log date and each entry has a full timestamp.

At the moment, there is no UI for selecting files on different nodes, you have to register files in the URL parameters in this form:
http://localhost:8111/log?path=/opt/my-app/logs/a.log@hostname1&path=/opt/my-app/logs/b.log@hostname1&path=/opt/my-app/logs/c.log@hostname2
here, each “path” parameter specifies a single file. after”@”, the host on which the file is located and the log viewer instance is running is indicated. You can specify multiple hosts separated by commas. If ” @ ” is missing, the file is located on the current node. To avoid dealing with huge URLS, you can set short links in the configuration, in the section log-paths = { … }.

Embedding the viewer in your app

Log Viewer can be connected to your Java Web application as a library so that it can show the user its logs. Sometimes this is more convenient than running a separate application. Just add a library dependency to the library via Maven / Gradle and connect one configuration class to the spring context. Everything else is configured automatically. log viewer itself recognizes which logging system is used and takes the location and format of logs from its configuration. By default, the UI is mapped to /logs, but everything can be customized. So far, the automatic configuration only works with Log4j and Logback.

This was tested on a small number of apps. if you have any problems, feel free to write to discussions on GitHub.

What is planned to be done in the future

It would be convenient if you could leave comments on your posts. For example, attach the ticket number to an error message. The comment should be visible to all users and when the same error occurs next time, it will be clear what to do with it.

There are many ideas for small UI improvements. For example, if a piece of JSON is found in the text, then you want the viewer to be able to show it in formatted form, and not in one line. I want to be able to set a filter by severity for a single class, and not just for all of them.

Sometimes it is not possible to open a port on the server for viewing logs, there is only SSH access. You can make support for working via SSH. The Web UI will be raised on the local machine, connect via SSH to the server, and run a special agent there. The agent will accept commands via the input stream and return the necessary parts of the log via the output stream.

Valery Radokhleb
Valery Radokhleb
Web developer, designer

Leave a Reply

Your email address will not be published. Required fields are marked *