Table of Contents
Setup Syslog-ng in the server
Configure Syslog-ng to accept logs from sources
Configure remote nodes to send logs to the Ubuntu server
Configure destinations to store logs
Develop web tool/ install database client tool/ data analysis tool to visualize logs
After installing Syslog in the server, we can configure it to receive log from UDP or TCP port and perform some filtering before sending it to our destination (pgsql database).
Versions used in this article:
Syslog-ng: 3.30
PostgreSQL: 9.5
Server: ubuntu 16.04
Default config directory of syslog-ng in ubuntu server is:
/etc/syslog-ng/syslog-ng.conf
@version: 3.30
@include “scl.conf”
The first line specifies the version of Syslog-ng. The following line is Source Configuration Library (SCL) that generates a reusable configuration file, which can be used to implement existing templates. [More about SCL]
Now, lets define some global parameters.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
options { chain_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no); dns_cache(no); owner("root"); group("adm"); perm(0640); stats_freq(0); bad_hostname("^gconfd$"); use-uniqid(yes); }; |
We have defined use-uniqid to get a unique id for each log message. The id is a non-zero 48-bit integer number, and the value starts again from 1 when it exceeds the max value limit.
Sources driver from which syslog-ng will collect log.
|
1 2 3 4 5 |
source src_name { system(); internal(); udp(); }; |
Destination location where log messages will be stored/processed:
|
1 2 3 4 5 |
destination des_name1{ file("/data /syslog-ng3_30/$HOST/$YEAR/$MONTH/$DAY/$HOST-$YEAR-$MONTH-$DAY.log" owner(root) group(root) perm (0600) create_dirs ( yes ) dir_perm( 0700 )); }; |
|
1 2 3 4 5 6 7 8 9 |
destination des_name2 { sql(type(pgsql) host("host ip") username("username") password("password") database("db_name") table("table_name") columns("RCPTID int","datetime TIMESTAMP", "host inet", "message") values("${RCPTID}","${ISODATE}", "${HOST}","${MSGONLY}") indexes("RCPTID","datetime", "host", "message")); }; |
Destination 1 stores logs in .text format in respective directories. Moreover, destination two stores log to the PostgreSQL database. We need to install pgsql and create a database there before sending logs.
Finally, the log statement combines multiple sources and destinations.
|
1 2 3 4 5 6 7 8 9 |
log { source(src_name); destination(des_name1); }; # to store text format log log { source(src_name); destination(des_name2); }; # to store in database |
Visualizing pgsql database stored log in DBeaver:
Web-tool to visualize log from PostgreSQL database with search feature:
A web app that has been developed to visualize syslog logs stored in a PostgreSQL database. This app provides users with IP, log message, and time range-based search features that make it easy to identify trends and patterns in log data.
The web app, called “PostgreSQL Syslog Viewer,” is a powerful tool for anyone who needs to analyze syslog logs stored in a PostgreSQL database. With its intuitive interface, users can quickly and easily navigate through log files and search for specific data points.
One of the key features of the PostgreSQL Syslog Viewer is its IP search functionality. This allows users to search for specific IP addresses within log files. This is especially useful when looking for specific events or errors that may have occurred on a particular system or network. The search function is lightning-fast and can quickly locate specific entries in even the largest log files.
Another powerful feature of the PostgreSQL Syslog Viewer is its log message search functionality. Users can search for specific text within log files. This is especially useful when looking for specific events or errors that may have occurred within the syslog logs. The search function is intuitive and user-friendly, making it easy for even novice users to find the information they need.
Finally, the PostgreSQL Syslog Viewer includes a time range-based search feature. This allows users to search for specific log entries within a specific time range. This is especially useful when looking for events or errors that may have occurred during a particular time period.
In addition to its search features, the PostgreSQL Syslog Viewer also includes several visualization options. Users can view logs in a variety of different formats, including tables, graphs, and charts. This allows users to quickly identify trends and patterns within the data.
0 Comments