# logs_agent

**Repository Path**: carrywl/logs_agent

## Basic Information

- **Project Name**: logs_agent
- **Description**: 日志解析客户端+进程检测
- **Primary Language**: Unknown
- **License**: MIT
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2024-02-26
- **Last Updated**: 2025-05-20

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# 日志解析客户端

## 功能介绍

1. 根据解析规则和定时任务策略指定目录下的实时日志文件（日志一直在输出），把解析结果保存下来
2. 监控某个进程，并定时上报情况

## 项目环境

* python 3.10.11

快速启动：`pip install -r requirements.txt`

运行main.py



打包命令：` pyinstaller -F .\main.py  `

## 初始化SQL
```sql


CREATE TABLE activity_log (
	id bigserial NOT NULL,
	log_time timestamp NOT NULL,
	activity_id int4 NULL,
	event_name varchar(16) NULL,
	event_detail varchar(64) NULL,
	state varchar(16) NULL,
	remark1 varchar(64) NULL,
	remark2 varchar(64) NULL,
	CONSTRAINT de_ipa_activity_log_pk PRIMARY KEY (id)
);



-- DROP TABLE client_log;

CREATE TABLE client_log (
	id bigserial NOT NULL,
	log_time timestamp NOT NULL,
	log_level varchar(8) NOT NULL,
	node_id varchar(16) NULL,
	ip_address inet NULL,
	request_id varchar(16) NULL,
	log_msg varchar(256) NULL,
	CONSTRAINT de_ipa_client_log_pk PRIMARY KEY (id)
);


-- DROP TABLE run_log;

CREATE TABLE run_log (
	id bigserial NOT NULL,
	pid int4 NOT NULL,
	process_name varchar(64) NULL,
	state varchar(16) NULL,
	start_time timestamp NULL,
	fqdn varchar(64) NULL,
	CONSTRAINT de_ipa_run_log_pk PRIMARY KEY (id)
);
```

## 项目配置

* db_config: 数据库配置
  * dbname: 数据库名称
  * user: 数据库用户名
  * password: 数据库密码
  * host: 主机地址
  * port: 端口号
  * min_conn: 最小连接数
  * max_conn: 最大连接数
* log_tasks: 日志任务（按照日志路径/日志解析规则去解析日志并入库，任务执行周期使用cron控制）
  * name: 任务名称
  * log_path: 扫描的日志路径数组（支持配置多个，如果日志路径不为空 则使用日志路径，否则使用日志规则解析）
  * log_path_rule: 日志路径规则（支持的规则类型: regex_line_rule[正则解析行, 对每行进行正则解析]）
  * cron_rule: 任务执行周期，数组分别代表【秒，分，时，天，月，年】，详细规则参考地址：[apscheduler.triggers.cron — APScheduler 3.10.4.post1 documentation](https://apscheduler.readthedocs.io/en/3.x/modules/triggers/cron.html#module-apscheduler.triggers.cron)
  * db_rule: 数据库规则
    * table_name: 数据库名称
    * columns: 列名
  * pid_tasks: 进程任务（查询进程状态并入库，任务执行周期使用cron控制）
    * name: 任务名称
    * process_name: 进程名称
    * cron_rule: 任务执行周期，数组分别代表【秒，分，时，天，月，年】，详细规则参考地址：[apscheduler.triggers.cron — APScheduler 3.10.4.post1 documentation](https://apscheduler.readthedocs.io/en/3.x/modules/triggers/cron.html#module-apscheduler.triggers.cron)
    * db_rule: 数据库规则
      * table_name: 数据库名称
      * columns: 列名



```json
{
  "db_config": {
    "dbname": "test",
    "user": "root",
    "password": "123456",
    "host": "127.0.0.1",
    "port": "5432",
    "min_conn": 1,
    "max_conn": 10
  },
  "log_tasks": [
    {
      "name": "客户端日志任务",
      "log_path": [
        "C:\\wanglei\\workSpaces\\pycharm\\logs_agent\\example\\client.log"
      ],
      "log_path_rule": "",
      "parse_rule": {
        "regex_line_rule": "(?P<log_time>\\d{4}-\\d{2}-\\d{2}\\s{1}\\d{2}:\\d{2}:\\d{2},\\d{3})\\s{1}(?P<log_level>\\w+)\\s{1}(?P<node_id>\\w+)\\|(?P<ip_address>((2(5[0-5]|[0-4]\\d))|[0-1]?\\d{1,2})(\\.((2(5[0-5]|[0-4]\\d))|[0-1]?\\d{1,2})){3})\\s{1}(?P<request_id>\\w+)(\\S+)\\s{1}(?P<log_msg>.*)"
      },
      "cron_rule": [
        "*/30",
        "*",
        "*",
        "*",
        "*",
        "*"
      ],
      "db_rule": {
        "table_name": "client_log",
        "columns": [
          "log_time",
          "log_level",
          "node_id",
          "ip_address",
          "request_id",
          "log_msg"
        ]
      }
    },
    {
      "name": "机器人活动日志任务",
      "log_path": "",
      "log_path_rule": {
        "regex_rule": [
          "C:\\wanglei\\workSpaces\\pycharm\\logs_agent\\example\\",
          "(\\d+)_activity.log"
        ]
      },
      "parse_rule": {
        "regex_line_rule": "{\\\"logV2\":\"\\[(?P<log_time>\\d{4}-\\d{2}-\\d{2}\\s\\d{2}:\\d{2}:\\d{2},\\d{3})]\\s(?P<activity_id>\\d+)\\s(?P<event_name>.*)\\s(?P<event_detail>.*)\\s(?P<state>.*)\\s(?P<remark1>\\d+)\\s+(?P<remark2>\\d+)\"}"
      },
      "cron_rule": [
        "0",
        "*/1",
        "*",
        "*",
        "*",
        "*"
      ],
      "db_rule": {
        "table_name": "activity_log",
        "columns": [
          "log_time",
          "activity_id",
          "event_name",
          "event_detail",
          "state",
          "remark1",
          "remark2"
        ]
      }
    }
  ],
  "pid_tasks": [
    {
      "name": "Typora",
      "process_name": "Typora.exe",
      "cron_rule": [
        "*/30",
        "*",
        "*",
        "*",
        "*",
        "*"
      ],
      "db_rule": {
        "table_name": "run_log",
        "columns": [
          "pid",
          "process_name",
          "state",
          "start_time",
          "fqdn"
        ]
      }
    }
  ]
}
```