v2.5.3

Skipping v2.5.2 because I accidentally released v2.5.1 as internal version "v2.5.2-dev".

• Fixed Twitter Scraper - previously used the now archived https://github.com/n0madic/twitter-scraper
  • Updated to fork maintained by imperatrona - https://github.com/imperatrona/twitter-scraper
• sendFile data keys were previously very limited, now it supports the same data keys as discord messages / downloads.
• Data key {{serverName}} is more reliable, previously often failed to parse. Discord-related data keys are now more reliable in general.
• Added subfoldersFallback setting alongside subfolders. If this setting is present when subfolders encounters an unparsed data key ({{example}}), it will use subfoldersFallback in place of subfolders.
• Fixed line breaks not parsing correctly for sendFileCaption
• Added data key {{usernameNoLeadPeriod}} - username of message author but with leading periods (".") stripped.
• 403 will once again cause a download to immediately bail. A recent update undid this and made it retry X times.

v2.5.1

v2.5.1 Patch:

NOTE: Had to re-upload the binaries and they show as v2.5.2-dev in-program. Treat this as v2.5.1.

• Fixes numerous longtime issues with user accounts, particularly regarding thread functionality and history processing.

v2.5.0 (released earlier today):

• Fixed Discord CDN 403 errors.
• Restructured how the bot handles sources (servers/categories/channels/threads).
  • Seems to work much better for forums and threads. Note that user accounts cannot access active threads, only archived.
  • Issues may occur, report anything related to this.
• Fixed issues with history date range filtering.
• Added commandPrefix to source configs.
  • Override the command prefix per-group, including leading space, (default "ddg ").
• blockedReactions & allowedReactions in filters now accepts standard Discord emoji names.
• Added support for Instagram 2FA. Use instagramTOTP in the credentials section with your TOTP seed, or enter your OTP code upon login. TOTP seed should always work, OTP code will apply to your login cache but if the cache expires, you'll need to re-enter the code upon new login.
• Added filepathNormalizeText & filepathStripSymbols to global config, source configs, and source message/download logging.
  • Both false by default.
  • Normalize text will convert letter-like symbols/fonts to their standard equivalent.
  • Improved filepath clearing padded spaces. Generally improved filename/filepath scrubbing.
• Improved reliability of fetching server and channel data.
• Added outputHistoryStatus & outputHistoryErrors globally and per-source to optionally disable history outputting.
• Added content setting to download & message logging. Currently accepts {{message}} for message logging and {{msgContent}} and {{link}} and {{embedDesc}} for download logging.
• Added {{attachmentID}} key to filenameFormat data keys.
• Changed subfolder data key {{userName}} to {{username}} for consistency.
• Fixed nil error crash when encountering command-message error while handling history.
• Restructured various areas, issues could occur, open an issue if you encounter anything.
• Upgraded Dependencies

v2.5.0

• Fixed Discord CDN 403 errors.
• Restructured how the bot handles sources (servers/categories/channels/threads).
  • Seems to work much better for forums and threads. Note that user accounts cannot access active threads, only archived.
  • Issues may occur, report anything related to this.
• Fixed issues with history date range filtering.
• Added commandPrefix to global config and source configs.
  • Override the command prefix per-group, including leading space, (default "ddg ").
• blockedReactions & allowedReactions in filters now accepts standard Discord emoji names.
• Added support for Instagram 2FA. Use instagramTOTP in the credentials section with your TOTP seed, or enter your OTP code upon login. TOTP seed should always work, OTP code will apply to your login cache but if the cache expires, you'll need to re-enter the code upon new login.
• Added filepathNormalizeText & filepathStripSymbols to global config, source configs, and source message/download logging.
  • Both false by default.
  • Normalize text will convert letter-like symbols/fonts to their standard equivalent.
  • Improved filepath clearing padded spaces. Generally improved filename/filepath scrubbing.
• Improved reliability of fetching server and channel data.
• Added outputHistoryStatus & outputHistoryErrors globally and per-source to optionally disable history outputting.
• Added content setting to download & message logging. Currently accepts {{message}} for message logging and {{msgContent}} and {{link}} and {{embedDesc}} for download logging.
• Added {{attachmentID}} key to filenameFormat data keys.
• Changed subfolder data key {{userName}} to {{username}} for consistency.
• Fixed nil error crash when encountering command-message error while handling history.
• Restructured various areas, issues could occur, open an issue if you encounter anything.
• Upgraded Dependencies

v2.4.0

!! v2.3.0 was released last week, see changelog for settings breaking changes !!

• Fixed critical issues with the new "subfolders" setting.
• Removed "#" from {{channelName}} for "subfolders".
• Added "alias" (string) and "aliases" (list of strings) to sources. These keywords can be used to call history commands. If a server/channel/etc config has something like "alias": "abc", you can use ddg history abc. Multiple configs can use the same aliases to call them all with 1 command. Aliases will not work if they contain spaces or commas.
• logLinks and logMessages reworked - "divideBy____" settings removed and replaced with "subfolders" and "filenameFormat" like the download sources.
• logLinks and logMessages global default options added (unused by default).
• Improved Discord source validation at startup, added source Discord permission checking to source validation.
• Improved a lot of logging appearance and info.
• Some log outputs formerly sent under "debug" are now under "verbose".
• Upgraded Dependencies

logLinks & logMessages Data Keys

Applies to subfolders, filenameFormat, linePrefix, lineSuffix WITHIN logLinks and logMessages, globally and per-source.

Key	Value
{{year}}	Year message was sent (2016+)
{{monthNum}}	Month of year message was sent (1-12)
{{dayOfMonth}}	Day of month message was sent (1-31)
{{hour}}	Hour of day message was sent (0-23)
{{minute}}	Minute of hour message was sent (0-59)
{{second}}	Second of minute message was sent (0-59)
{{timestamp}}	Message timestamp, "2006-01-02 15-04-05" format
{{timestampYYYYMMDD}}	Message timestamp, "2006-01-02" format
{{timestampHHMMSS}}	Message timestamp, "15-04-05" format
{{messageID}}	Discord Source Message ID
{{channelID}}	Discord Source Message Channel ID
{{serverID}}	Discord Source Message Server ID
{{userID}}	ID of Discord user who sent the message
{{username}}	Username of Discord user who sent the message
{{userDisc}}	User discriminator number of Discord user who sent the message
{{channelName}}	Discord Source Message Channel Name
{{channelTopic}}	Discord Source Message Channel Topic
{{categoryID}}	Discord Source Message Category ID
{{categoryName}}	Discord Source Message Category Name
{{threadID}}	Discord Source Message Thread ID (if thread)
{{threadName}}	Discord Source Message Thread Name (if thread)
{{threadTopic}}	Discord Source Message Thread Topic (if thread)
{{forumID}}	Discord Source Message Forum ID (if forum)
{{forumName}}	Discord Source Message Forum Name (if forum)
{{downloadStatus}}	[logLinks ONLY] Short Status of Download
{{downloadStatusLong}}	[logLinks ONLY] Long Status of Download
{{downloadFilename}}	[logLinks ONLY] Filename of Download
{{downloadExt}}	[logLinks ONLY] File Extension of Download
{{downloadPath}}	[logLinks ONLY] Save Path for Download

v2.3.0

!! IMPORTANT !!
All divideBy\_\_\_\_\_ settings have been removed and been replaced with subfolders, see below.

• Added "subfolders" and removed all divideBy settings, a list of strings for each subfolder title.

  • Replaceable keys are: {{serverID}}, {{serverName}}, {{categoryID}}, {{categoryName}}, {{channelID}}, {{channelName}}, {{userID}}, {{userName}}, {{fileType}}, {{year}}, {{monthNum}}, {{dayOfMonth}}, {{hour}}, {{message}}, {{messageID}}.
  • Default is "subfolders": [ "{{fileType}}" ],.
  • Present both globally in the main settings and individually in source settings. If these are missing from source (i.e. no override), it will use the main settings as default.
  • Something like "subfolders": [ "{{serverName}}", "{{categoryName}}", "{{channelName}}", "{{year}}", "{{fileType}}" ], will equate to "BASE_DESTINATION/My Server/Category/Channel/2023/images/".

• Reworked Emoji & Sticker Support

  • Removed ddg emojis command.
  • Runs at launch after all other initial startup tasks finish.
  • Added settings to main settings group:
    • "emojisServers", "stickersServers" are lists of server IDs to download from. ex: "emojisServers": [ "123", "456" ],
    • "emojisFilenameFormat", "stickersFilenameFormat" to override the filename format. Default is "{{ID}} {{name}}, only accepts {{ID}} and {{name}} as keys.
    • "emojisDestination", "stickersDestination" to override the destination folders. If these settings are missing, it will use "emojis" and "stickers" within the bot folder. The bot will automatically create subfolders by channel name within whatever destination.

• Hopefully fixed Discord timestamp snowflake issues regarding history date range filters.

• Fixed error upon exit if Twitter / X is disabled.

• Added "verbose" and "debugV" but they currently do nothing, just future variants of "debug".

v2.2.2

• Fixed history command not finding channels when going by category.
• Fixed history not working on news / announcement channels.
• Fixed history command including categories themselves in history queues when going by server or by all.
• Added warning to history command if the bot detects issues with Discord cache.
• Fixed filters.allowedFilenames not working.
• Fixed some minor crashing issues.
• Optimized and cleaned up some things.

v2.2.1

• Fixed folder division sometimes causing fatal crash.

See v2.2.0 changelog for changes from 2 days prior.

v2.2.0

• Settings now supported in .YAML format. Settings are identical to the JSON structuring, just an alternative format option. Upon launch, if using a .JSON/.JSONC config, the bot will translate it to "cache/settings.yaml" and alternatively when using a .YAML config it will translate it to "cache/settings.json".
• Reduced which settings are hidden in settingsOutput and program-generated settings.
• Twitter authentication now caches to reduce login requests.
• Fixed "x.com" links not being recognized as Twitter.
• If not authenticated with Twitter/X, those links are skipped entirely to avoid hangups.
• Download speed metrics are far more accurate, factoring the elapsed file download duration rather than overall duration.
• Default for downloadRetryMax changed from 3 to 2.
• Added setting to main section: overwriteDefaultReaction (text), "✅" by default.
• Added setting to main section: overwriteCachePath (text), "cache" by default.
• Added setting to main section: overwriteHistoryPath (text), "cache/history" by default.
• Added setting to main section: overwriteDuploPath (text), "cache/.duplo" by default.
• Added setting to main section: overwriteTwitterPath (text), "cache/twitter.json" by default.
• Added setting to main section: overwriteInstagramPath (text), "cache/instagram.json" by default.
• Added setting to main section: overwriteConstantsPath (text), "cache/constants.json" by default.
• Added setting to main section: overwriteDatabasePath (text), "database" by default.
• Added setting to main section: overwriteDatabaseBackupsPath (text), "backups" by default.
• Added setting to main section: historyManagerRate (number), 5 by default, seconds between rechecking the queued history jobs.
• Added setting to credentials section: twitterEnabled (bool), true by default, optionally completely disable the Twitter Scraper section.
• Added setting to credentials section: instagramEnabled (bool), true by default, optionally completely disable the Instagram Scraper section.
• Added setting to credentials section: twitterProxy (text), if this setting isn't blank, the Twitter scraper will use it as a proxy address.
• Added setting to credentials section: instagramProxy (text), if this setting isn't blank, the Instagram scraper will use it as a proxy address.
• Added setting to credentials section: instagramProxyInsecure (bool), false by default.
• Added setting to credentials section: instagramProxyForceHTTP2 (bool), false by default.
• Added setting to main section: logOutput (text), if this setting isn't blank the program will log all console output as a text file, the setting should be a complete filepath including .txt or the desired extension.
• Added setting to main section: logIndent (bool), enabled by default, if disabled the program will not indent between timestamps and log content.
• Fixed logMessages and logLinks timezones.
• Improved Twitter & Instagram authentication logging.
• Improved general output.
• Improved History status output.
• Optimized some things.
• Upgraded dependencies.
• Fixed some typos.

v2.1.1

Over the next week (since posting this) I'll be improving the readme.

• Fixed Twitter Scraper (I didn't actually do anything, the developer updated it).
• Added commandTagging to main settings. True by default. Controls whether command senders should be tagged in replies from the bot.
• Added historyRequestDelay to delay between requests of history messages.
• Added historyRequestCount to limit messages per-request in history. Default and max value is 100.
• Added delayHandling and delayHandlingHistory (in milliseconds) to delay handling/downloading.
• Added divideByDay and divideByHour to main settings and source settings.
• Fixed categoryBlacklist not functioning properly.
• Fixed blockedPhrases and allowedPhrases not ignoring empty entries.
• Download errors no longer tag sender
• savePossibleDuplicates now false by default
• Updated Websocket Dependency for security patch
• Updated Dependencies

v2.1.0

• Twitter - They ditched their Free API tiers and banned access to all of those existing applications. I've replaced it with a Twitter Scraper library. It's more crude than the official API was but at least it works. I've noticed it will only download the first video/gif if there's multiple in 1 tweet. It seems to be fine for images. I'll see if I can fix this soon. Every other aspect of it seems to be fine so far. You can optionally login via email+password ("twitterEmail" and "twitterPassword" in the credentials section) to download from private accounts your user follows.
• Thread Support - proper thread support, even for users (mostly). Treats threads as subchannels of a channel, you can use direct thread IDs or the forum channel ID. It will fetch any archived or active threads it can find. USER ACCOUNTS CANNOT FETCH ACTIVE THREADS, ONLY ARCHIVED. Message logging works properly for threads.
• Improved general reliability of how the bot checks channel and server data.
• Fixed unwarranted Discord permission errors, particularly regarding user accounts and DM channels.
• Fixed unwarranted Discord source invalidation warnings, particularly regarding DM channels.
• Fixed categories not being handled properly by history.
• Added "watchSettings" (true or false) to main settings. True by default. Controls whether settings are updated live when the file is modified.
• Added "backupDatabaseOnStart" (true or false) to main settings. False by default. Backs up database folder into a dated zip file under a local "backups" folder. This feature can be expanded upon request.