Skip to content

Commit

Permalink
RUMM-513 Validating JSON payload before sending to intake
Browse files Browse the repository at this point in the history
We observe non-parseable logs sent to intake
In a certain org which has high volumes of logs,
%0.3-0.5 of all logs are not parsed properly in dashboard

Hypothesis #1
Data being read is malformed, because every malformed log has their malformed part at the very end
So if we can read the file later, we can have valid payload
This commit implements this hypothesis

Hypothesis #2
Write op corrupts the file and we need to watch write ops closely so that we can catch and recover errors
This requires major refactoring in File.swift and related files
  • Loading branch information
buranmert committed Jul 27, 2020
1 parent 969c2d4 commit 15ee54c
Showing 1 changed file with 6 additions and 0 deletions.
6 changes: 6 additions & 0 deletions Sources/Datadog/Core/Persistence/FileReader.swift
Original file line number Diff line number Diff line change
Expand Up @@ -38,11 +38,17 @@ internal final class FileReader {
}
}

private static let utf8NewlineData = "\n".data(using: .utf8)!.first! // swiftlint:disable:this force_unwrapping
private func synchronizedReadNextBatch() -> Batch? {
if let file = orchestrator.getReadableFile(excludingFilesNamed: Set(filesRead.map { $0.name })) {
do {
let fileData = try file.read()
let batchData = dataFormat.prefixData + fileData + dataFormat.suffixData
// Validate data here, return nil if corrupt
// NOTE: dataFormat.validate(batchData) would make more sense
try batchData.split(separator: Self.utf8NewlineData).forEach {
try JSONSerialization.jsonObject(with: $0, options: [])
}
return Batch(data: batchData, file: file)
} catch {
developerLogger?.error("🔥 Failed to read file: \(error)")
Expand Down

0 comments on commit 15ee54c

Please sign in to comment.