[Logging] Implement Player Event Logging system (#2833)

* Plumbing

* Batch processing in world

* Cleanup

* Cleanup

* Update player_event_logs.cpp

* Add player zoning event

* Use generics

* Comments

* Add events

* Add more events

* AA_GAIN, AA_PURCHASE, FORAGE_SUCCESS, FORAGE_FAILURE

* FISH_SUCCESS, FISH_FAILURE, ITEM_DESTROY

* Add charges to ITEM_DESTROY

* WENT_ONLINE, WENT_OFFLINE

* LEVEL_GAIN, LEVEL_LOSS

* LOOT_ITEM

* MERCHANT_PURCHASE

* MERCHANT_SELL

* SKILL_UP

* Add events

* Add more events

* TASK_ACCEPT, TASK_COMPLETE, and TASK_UPDATE

* GROUNDSPAWN_PICKUP

* SAY

* REZ_ACCEPTED

* COMBINE_FAILURE and COMBINE_SUCCESS

* DROPPED_ITEM

* DEATH

* SPLIT_MONEY

* TRADER_PURCHASE and TRADER_SELL

* DISCOVER_ITEM

* Convert GM_COMMAND to use new macro

* Convert ZONING event to use macro

* Revert some code changes

* Revert "Revert some code changes"

This reverts commit d53682f.

* Add cereal generation support to repositories

* TRADE

* Formatting

* Cleanup

* Relocate discord_manager to discord folder

* Discord sending plumbing

* Rename UCS's Database class to UCSDatabase to be more specific and not collide with base Database class for repository usage

* More discord sending plumbing

* More discord message formatting work

* More discord formatting work

* Discord formatting of events

* Format WENT_ONLINE, WENT_OFFLINE

* Add merchant purchase event

* Handle Discord MERCHANT_SELL formatter

* Update player_event_discord_formatter.cpp

* Tweaks

* Implement retention truncation

* Put mutex locking on batch queue, put processor on its own thread

* Process on initial bootup

* Implement optional QS processing, implement keepalive from world to QS

* Reload player event settings when logs are reloaded in game

* Set settings defaults

* Update player_event_logs.cpp

* Update player_event_logs.cpp

* Set retention days on boot

* Update player_event_logs.cpp

* Player Handin Event Testing.

Testing player handin stuff.

* Cleanup.

* Finish NPC Handin.

* set a reference to the client inside of the trade object as well for plugins to process

* Fix for windows _inline

* Bump to cpp20 default, ignore excessive warnings on windows

* Bump FMT to 6.1.2 for cpp20 compat and swap fmt::join for Strings::Join

* Windows compile fixes

* Update CMakeLists.txt

* Update CMakeLists.txt

* Update CMakeLists.txt

* Create 2022_12_19_player_events_tables.sql

* [Formatters] Work on Discord Formatters

* Handin money.

* Format header

* [Formatters] Work on Discord Formatters

* Format

* Format

* [Formatters] More Formatter work, need to test further.

* [Formatters] More Work on Formatters.

* Add missing #endif

* [Formatters] Work on Formatters, fix Bot formatting in ^create help

* NPC Handin Discord Formatter

* Update player_event_logs.cpp

* Discover Item Discord Formatter

* Dropped Item Discord Formatter

* Split Money Discord Formatter

* Trader Discord Formatters

* Cleanup.

* Trade Event Discord Formatter Groundwork

* SAY don't record GM commands

* GM_Command don't record #help

* Update player_event_logs.cpp

* Fill in more event data

* Post rebase fixes

* Post rebase fix

* Discord formatting adjustments

* Add event deprecation or unimplemented tag support

* Trade events

* Add return money and sanity checks.

* Update schema

* Update ucs.cpp

* Update client.cpp

* Update 2022_12_19_player_events_tables.sql

* Implement archive single line

* Replace hackers table and functions with PossibleHack player event

* Replace very old eventlog table since the same events are covered by player event logs

* Update bot_command.cpp

* Record NPC kill events ALL / Named / Raid

* Add BatchEventProcessIntervalSeconds rule

* Naming

* Update CMakeLists.txt

* Update database_schema.h

* Remove logging function and methods

* DB version

* Cleanup SendPlayerHandinEvent

---------

Co-authored-by: Kinglykrab <kinglykrab@gmail.com>
Co-authored-by: Aeadoin <109764533+Aeadoin@users.noreply.github.com>

common/CMakeLists.txt

@@ -33,9 +33,11 @@ SET(common_sources
     eq_stream_proxy.cpp
     eqtime.cpp
     event_sub.cpp
+    events/player_event_logs.cpp
+    events/player_event_discord_formatter.cpp
     expedition_lockout_timer.cpp
     extprofile.cpp
-    discord_manager.cpp
+    discord/discord_manager.cpp
     faction.cpp
     file.cpp
     guild_base.cpp
@@ -198,7 +200,6 @@ SET(repositories
     repositories/base/base_dynamic_zones_repository.h
     repositories/base/base_dynamic_zone_members_repository.h
     repositories/base/base_dynamic_zone_templates_repository.h
-    repositories/base/base_eventlog_repository.h
     repositories/base/base_expeditions_repository.h
     repositories/base/base_expedition_lockouts_repository.h
     repositories/base/base_faction_association_repository.h
@@ -218,7 +219,6 @@ SET(repositories
     repositories/base/base_guilds_repository.h
     repositories/base/base_guild_ranks_repository.h
     repositories/base/base_guild_relations_repository.h
-    repositories/base/base_hackers_repository.h
     repositories/base/base_horses_repository.h
     repositories/base/base_instance_list_repository.h
     repositories/base/base_instance_list_player_repository.h
@@ -264,6 +264,8 @@ SET(repositories
     repositories/base/base_pets_equipmentset_repository.h
     repositories/base/base_pets_equipmentset_entries_repository.h
     repositories/base/base_player_titlesets_repository.h
+    repositories/base/base_player_event_log_settings_repository.h
+    repositories/base/base_player_event_logs_repository.h
     repositories/base/base_quest_globals_repository.h
     repositories/base/base_raid_details_repository.h
     repositories/base/base_raid_members_repository.h
@@ -376,7 +378,6 @@ SET(repositories
     repositories/dynamic_zones_repository.h
     repositories/dynamic_zone_members_repository.h
     repositories/dynamic_zone_templates_repository.h
-    repositories/eventlog_repository.h
     repositories/expeditions_repository.h
     repositories/expedition_lockouts_repository.h
     repositories/faction_association_repository.h
@@ -396,7 +397,6 @@ SET(repositories
     repositories/guilds_repository.h
     repositories/guild_ranks_repository.h
     repositories/guild_relations_repository.h
-    repositories/hackers_repository.h
     repositories/horses_repository.h
     repositories/instance_list_repository.h
     repositories/instance_list_player_repository.h
@@ -442,6 +442,8 @@ SET(repositories
     repositories/pets_equipmentset_repository.h
     repositories/pets_equipmentset_entries_repository.h
     repositories/player_titlesets_repository.h
+    repositories/player_event_log_settings_repository.h
+    repositories/player_event_logs_repository.h
     repositories/quest_globals_repository.h
     repositories/raid_details_repository.h
     repositories/raid_members_repository.h
@@ -507,7 +509,7 @@ SET(common_headers
     dbcore.h
     deity.h
     discord/discord.h
-    discord_manager.h
+    discord/discord_manager.h
     dynamic_zone_base.h
     emu_constants.h
     emu_limits.h
@@ -530,6 +532,9 @@ SET(common_headers
     eq_stream_locator.h
     eq_stream_proxy.h
     eqtime.h
+    events/player_event_logs.h
+    events/player_event_discord_formatter.h
+    events/player_events.h
     errmsg.h
     event_sub.h
     expedition_lockout_timer.h
@@ -608,6 +613,7 @@ SET(common_headers
     event/event_loop.h
     event/task.h
     event/timer.h
+    json/json_archive_single_line.h
     json/json.h
     json/json-forwards.h
     net/console_server.h
@@ -661,8 +667,7 @@ SET(common_headers
     StackWalker/StackWalker.h
     util/memory_stream.h
     util/directory.h
-    util/uuid.h
-    )
+    util/uuid.h)
 
 SOURCE_GROUP(Event FILES
     event/event_loop.h

common/database.cpp

@@ -1281,44 +1281,6 @@ bool Database::MoveCharacterToZone(const char *charname, uint32 zone_id)
 	return results.RowsAffected() != 0;
 }
 
-bool Database::SetHackerFlag(const char* accountname, const char* charactername, const char* hacked) {
-	std::string query = StringFormat("INSERT INTO `hackers` (account, name, hacked) values('%s','%s','%s')", accountname, charactername, hacked);
-	auto results = QueryDatabase(query);
-
-	if (!results.Success()) {
-		return false;
-	}
-
-	return results.RowsAffected() != 0;
-}
-
-bool Database::SetMQDetectionFlag(const char* accountname, const char* charactername, const char* hacked, const char* zone) {
-	//Utilize the "hacker" table, but also give zone information.
-	std::string query = StringFormat("INSERT INTO hackers(account,name,hacked,zone) values('%s','%s','%s','%s')", accountname, charactername, hacked, zone);
-	auto results = QueryDatabase(query);
-
-	if (!results.Success())
-	{
-		return false;
-	}
-
-	return results.RowsAffected() != 0;
-}
-
-bool Database::SetMQDetectionFlag(const char* accountname, const char* charactername, const std::string &hacked, const char* zone) {
-	//Utilize the "hacker" table, but also give zone information.
-	auto query = fmt::format("INSERT INTO hackers(account, name, hacked, zone) values('{}', '{}', '{}', '{}')",
-				 accountname, charactername, hacked, zone);
-	auto results = QueryDatabase(query);
-
-	if (!results.Success())
-	{
-		return false;
-	}
-
-	return results.RowsAffected() != 0;
-}
-
 uint8 Database::GetRaceSkill(uint8 skillid, uint8 in_race)
 {
 	uint16 race_cap = 0;

common/database.h

@@ -108,9 +108,6 @@ class Database : public DBcore {
 	bool MoveCharacterToZone(uint32 character_id, uint32 zone_id);
 	bool ReserveName(uint32 account_id, char *name);
 	bool SaveCharacterCreate(uint32 character_id, uint32 account_id, PlayerProfile_Struct *pp);
-	bool SetHackerFlag(const char *accountname, const char *charactername, const char *hacked);
-	bool SetMQDetectionFlag(const char *accountname, const char *charactername, const char *hacked, const char *zone);
-	bool SetMQDetectionFlag(const char *accountname, const char *charactername, const std::string &hacked, const char *zone);
 	bool UpdateName(const char *oldname, const char *newname);
 	bool CopyCharacter(
 		const std::string& source_character_name,

common/database_schema.h

@@ -321,13 +321,11 @@ namespace DatabaseSchema {
 			"discord_webhooks",
 			"dynamic_zone_members",
 			"dynamic_zones",
-			"eventlog",
 			"expedition_lockouts",
 			"expeditions",
 			"gm_ips",
 			"group_id",
 			"group_leaders",
-			"hackers",
 			"instance_list",
 			"ip_exemptions",
 			"item_tick",
@@ -343,6 +341,8 @@ namespace DatabaseSchema {
 			"respawn_times",
 			"saylink",
 			"server_scheduled_events",
+			"player_event_log_settings",
+			"player_event_logs"
 			"shared_task_activity_state",
 			"shared_task_dynamic_zones",
 			"shared_task_members",

common/discord/discord.cpp

@@ -1,22 +1,17 @@
+#include <cereal/archives/json.hpp>
+#include <cereal/archives/binary.hpp>
 #include "discord.h"
 #include "../http/httplib.h"
 #include "../json/json.h"
 #include "../strings.h"
 #include "../eqemu_logsys.h"
+#include "../events/player_event_logs.h"
 
 constexpr int MAX_RETRIES = 10;
 
 void Discord::SendWebhookMessage(const std::string &message, const std::string &webhook_url)
 {
-	// validate
-	if (webhook_url.empty()) {
-		LogDiscord("[webhook_url] is empty");
-		return;
-	}
-
-	// validate
-	if (webhook_url.find("http://") == std::string::npos && webhook_url.find("https://") == std::string::npos) {
-		LogDiscord("[webhook_url] [{}] does not contain a valid http/s prefix.", webhook_url);
+	if (!ValidateWebhookUrl(webhook_url)) {
 		return;
 	}
 
@@ -28,7 +23,7 @@ void Discord::SendWebhookMessage(const std::string &message, const std::string &
 	std::string endpoint = Strings::Replace(webhook_url, base_url, "");
 
 	// client
-	httplib::Client cli(base_url.c_str());
+	httplib::Client cli(base_url);
 	cli.set_connection_timeout(0, 15000000); // 15 sec
 	cli.set_read_timeout(15, 0); // 15 seconds
 	cli.set_write_timeout(15, 0); // 15 seconds
@@ -46,9 +41,9 @@ void Discord::SendWebhookMessage(const std::string &message, const std::string &
 	int  retries     = 0;
 	int  retry_timer = 1000;
 	while (retry) {
-		if (auto res = cli.Post(endpoint.c_str(), payload.str(), "application/json")) {
+		if (auto res = cli.Post(endpoint, payload.str(), "application/json")) {
 			if (res->status != 200 && res->status != 204) {
-				LogError("Code [{}] Error [{}]", res->status, res->body);
+				LogError("[Discord Client] Code [{}] Error [{}]", res->status, res->body);
 			}
 			if (res->status == 429) {
 				if (!res->body.empty()) {
@@ -81,6 +76,74 @@ void Discord::SendWebhookMessage(const std::string &message, const std::string &
 	}
 }
 
+void Discord::SendPlayerEventMessage(
+	const PlayerEvent::PlayerEventContainer &e,
+	const std::string &webhook_url
+)
+{
+	if (!ValidateWebhookUrl(webhook_url)) {
+		return;
+	}
+
+	auto s = Strings::Split(webhook_url, '/');
+
+	// url
+	std::string base_url = fmt::format("{}//{}", s[0], s[2]);
+	std::string endpoint = Strings::Replace(webhook_url, base_url, "");
+
+	// client
+	httplib::Client cli(base_url);
+	cli.set_connection_timeout(0, 15000000); // 15 sec
+	cli.set_read_timeout(15, 0); // 15 seconds
+	cli.set_write_timeout(15, 0); // 15 seconds
+	httplib::Headers headers = {
+		{"Content-Type", "application/json"}
+	};
+
+	std::string payload = PlayerEventLogs::GetDiscordPayloadFromEvent(e);
+	if (payload.empty()) {
+		return;
+	}
+
+	bool retry       = true;
+	int  retries     = 0;
+	int  retry_timer = 1000;
+	while (retry) {
+		if (auto res = cli.Post(endpoint, payload, "application/json")) {
+			if (res->status != 200 && res->status != 204) {
+				LogError("Code [{}] Error [{}]", res->status, res->body);
+			}
+			if (res->status == 429) {
+				if (!res->body.empty()) {
+					std::stringstream ss(res->body);
+					Json::Value       response;
+
+					try {
+						ss >> response;
+					}
+					catch (std::exception const &ex) {
+						LogDiscord("JSON serialization failure [{}] via [{}]", ex.what(), res->body);
+					}
+
+					retry_timer = std::stoi(response["retry_after"].asString()) + 500;
+				}
+
+				LogDiscord("Rate limited... retrying message in [{}ms]", retry_timer);
+				std::this_thread::sleep_for(std::chrono::milliseconds(retry_timer + 500));
+			}
+			if (res->status == 204) {
+				retry = false;
+			}
+			if (retries > MAX_RETRIES) {
+				LogDiscord("Retries exceeded for player event message");
+				retry = false;
+			}
+
+			retries++;
+		}
+	}
+}
+
 std::string Discord::FormatDiscordMessage(uint16 category_id, const std::string &message)
 {
 	if (category_id == Logs::LogCategory::MySQLQuery) {
@@ -89,3 +152,20 @@ std::string Discord::FormatDiscordMessage(uint16 category_id, const std::string
 
 	return message + "\n";
 }
+
+bool Discord::ValidateWebhookUrl(const std::string &webhook_url)
+{
+	// validate
+	if (webhook_url.empty()) {
+		LogDiscord("[webhook_url] is empty");
+		return false;
+	}
+
+	// validate
+	if (!Strings::Contains(webhook_url, "http://") && !Strings::Contains(webhook_url, "https://")) {
+		LogDiscord("[webhook_url] [{}] does not contain a valid http/s prefix.", webhook_url);
+		return false;
+	}
+
+	return true;
+}

common/discord/discord.h

@@ -4,11 +4,16 @@
 
 #include <string>
 #include "../types.h"
+#include "../http/httplib.h"
+#include "../repositories/player_event_logs_repository.h"
+#include "../events/player_events.h"
 
 class Discord {
 public:
 	static void SendWebhookMessage(const std::string& message, const std::string& webhook_url);
 	static std::string FormatDiscordMessage(uint16 category_id, const std::string& message);
+	static void SendPlayerEventMessage(const PlayerEvent::PlayerEventContainer& e, const std::string &webhook_url);
+	static bool ValidateWebhookUrl(const std::string &webhook_url);
 };
 
 

common/discord_manager.cpp → common/discord/discord_manager.cpp

@@ -1,7 +1,6 @@
 #include "discord_manager.h"
-#include "../common/discord/discord.h"
-#include "../common/eqemu_logsys.h"
-#include "../common/strings.h"
+#include "../../common/discord/discord.h"
+#include "../events/player_event_logs.h"
 
 void DiscordManager::QueueWebhookMessage(uint32 webhook_id, const std::string &message)
 {
@@ -55,7 +54,6 @@ void DiscordManager::ProcessMessageQueue()
 				message = "";
 			}
 		}
-
 		// final flush
 		if (!message.empty()) {
 			Discord::SendWebhookMessage(
@@ -67,3 +65,11 @@ void DiscordManager::ProcessMessageQueue()
 	webhook_message_queue.clear();
 	webhook_queue_lock.unlock();
 }
+
+void DiscordManager::QueuePlayerEventMessage(const PlayerEvent::PlayerEventContainer& e)
+{
+	auto w = player_event_logs.GetDiscordWebhookUrlFromEventType(e.player_event_log.event_type_id);
+	if (!w.empty()) {
+		Discord::SendPlayerEventMessage(e, w);
+	}
+}

common/discord_manager.h → common/discord/discord_manager.h

@@ -4,12 +4,15 @@
 #include <mutex>
 #include <map>
 #include <vector>
-#include "../common/types.h"
+#include "../../common/types.h"
+#include "../repositories/player_event_logs_repository.h"
+#include "../events/player_events.h"
 
 class DiscordManager {
 public:
 	void QueueWebhookMessage(uint32 webhook_id, const std::string& message);
 	void ProcessMessageQueue();
+	void QueuePlayerEventMessage(const PlayerEvent::PlayerEventContainer& e);
 private:
 	std::mutex webhook_queue_lock{};
 	std::map<uint32, std::vector<std::string>> webhook_message_queue{};